Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

5.7 Collection and use of information

Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

Page last updated: 17 April 2012

5.7.1 Continuous quality improvement
5.7.2 NT Aboriginal Health Key Performance Indicators
5.7.3 Robustness of the NT Aboriginal Health Key Performance Indicator data
5.7.4 Use of NT Aboriginal Health Key Performance Indicator data
5.7.5 The tension between accountability and improvement
5.7.6 The need for context-specific information
5.7.7 Summary and conclusions

This section addresses the evaluation objectives of how the EHSDI has had an impact on the availability and use of information to support improved health service delivery. The EHSDI reform agenda includes a focus on CQI of health service delivery. Improved information systems and data availability are critical for monitoring health system performance and supporting quality improvement. Information is needed on the state of NT Aboriginal population health, and the functioning of PHC services.

5.7.1 Continuous quality improvement

CQI uses data to drive organisational change in a continuous cycle of improvement (Kritchevsky and Simmons 1991). In a health setting, focus is placed on ensuring that health systems are operating efficiently and effectively. CQI is characterised by the regular and cyclic collection of data to direct resources and activities to improve health service delivery. CQI endeavours to build a culture of learning by improving organisational capacity to analyse and improve systems to enhance the services provided.

CQI is a key pillar of PHC system reform under the EHSDI. A total of $3.001 million of EHSDI funding has been spent on CQI to date, including $0.212 million in 2008–09 and $2.789 million in 2009–10. Systematic investment in CQI is seen by the NT AHF as a critical platform for NT remote PHC sector reform.

There was no coordinated and coherent CQI model across the NT remote PHC sector before the EHSDI. Bailie et al (2008) state that in recent years there has been a high level of interest isn CQI activities in the Aboriginal health sector, as demonstrated by the voluntary uptake of CQI systems by health services. Before the EHSDI, the DHF instigated a system of quality assurance in its clinics and engaged the Audit and Best Practice in Chronic Disease (ABCD) program developed by the Menzies School of Health Research. Several ACCHOs also used the ABCD program or used other CQI programs including the Australian Primary Care Collaboratives, DoHA’s Health Quality Improvement Initiatives including the Continuous Improvement Projects (CIP), and the Healthy for Life Program funded by DoHA.

These programs were introduced with improved PIRS in clinics; however, the uptake of CQI activities was variable across the NT Aboriginal health sector. While some services were participating in CQI at a high level using two or three formal CQI systems, others lacked the capacity to implement or effectively utilise CQI systems.

PHRG internal documentation notes that health service involvement in CQI has been demonstrated to improve the quality and appropriateness of care, leading to improved health outcomes (PHRG briefing paper, February 2009). There is acknowledgment by the NT AHF partners that significant disparities existed in the quality of care offered between different health services across the NT and the need for an effective CQI process in the NT PHC system had been recognised for some time (PHRG EHSDI CQI Proposal, November 2008). Embedding quality improvement into PHC is one of the key priorities of the PHRG. Australian Government investment in the NT health system under the EHSDI was seen as an opportunity to build and implement a sector-wide CQI model.

In December 2008 the NT AHF agreed that there was a need for ‘a strong and effective CQI process in PHC in the NT’ (NT AHF Proposed EHSDI investment in continual quality improvement, February 2009). An EHSDI CQI strategy was endorsed in March 2009, with the core aim of building on existing quality improvement activities to support long-term, coordinated and sustainable service improvement across the NT remote PHC sector. The strategy has five major components:
  1. Recruitment of two CQI Coordinators (Top End and Central) to be based at AMSANT to provide expert leadership in the developing a sector-wide CQI model and training and support to CQI positions in the HSDAs.
  2. Recruitment of a CQI Facilitator for each HSDA to assist health services with implementing CQI and to support and mentor practitioners in the use of CQI.
  3. Development of a CQI model which will provide consistency and sustainability across the NT with the flexibility to allow for local circumstances.
  4. Establishment of a CQI program planning committee to provide advice, develop the NT CQI model and monitor implementation.
  5. Evaluation of the CQI program to inform future development and CQI investment (AMSANT 2009c).
There has been some progress towards implementing CQI activities as part of the EHSDI. The two lead CQI Coordinator positions have been filled and a CQI planning committee has been set up and has met regularly since mid-2009. The recruitment of CQI Facilitators in each HSDA has been slow with most positions not yet filled at the time of the evaluation case studies. This was due to the slower than expected roll out of funding and negotiations regarding the role of the Facilitators in HSDAs with multiple providers or where existing positions included some CQI responsibilities. These issues have largely been resolved and, as at February 2011, 10 of the positions had been successfully filled and three were vacant (CQI briefing paper to PHRG, February 2011).
Top of page
A CQI model for NT Aboriginal PHC was developed by the PHRG and endorsed by the NT AHF. This model sets out a framework for CQI to be shared by both DHF and ACCHO service providers. The framework outlines 10 key elements that each service needs to address as part of their CQI approach. These include:
  • support from leadership and management, including incorporating CQI into health service planning and devoting management and staff time to CQI
  • involving and orienting the entire health service team to CQI protocols and practices
  • providing orientation, training and rewarding achievements to gain staff support for CQI
  • effective communication such as newsletters, teleconferences and an email network
  • encouraging Aboriginal engagement to provide input into CQI planning and policies at an NT-wide, regional and service level
  • consumer input based on strategies to seek the views of patients and communities
  • on-site and regional level training for PHC staff and shared learning through the promotion of success stories
  • high quality data, including NT AHKPIs and other clinical health data
  • identifying and using appropriate CQI tools
  • timely and relevant feedback at all levels including health boards, staff, consumers and communities (NT AHF Briefing paper for NT AHF meeting 47, December 2009).

Discussion of the EHSDI continuous quality improvement program

The EHSDI CQI program is relatively new and yet to be fully implemented. There has been limited progress in developing the key elements of the CQI framework outlined above. Several of the main elements emphasise the engagement and input of personnel at all levels of the PHC system, including management, health staff, consumer and Aboriginal community input. The majority of informants we spoke to, including both health service staff and health centre managers, indicated that their knowledge of, and experience in, CQI was limited. The initial delay in recruitment of CQI Facilitators meant that many health service staff received little information and assistance in implementing CQI activities. This has led to mixed feelings among informants, with some articulating cautious enthusiasm, while others felt anxious about their ability to put CQI systems into practice.

Gaining the support of community-level PHC staff is essential to increase the uptake and effectiveness of CQI activities and to gain community support and involvement. This can best be achieved through training and dissemination of information about CQI to increase the confidence of health service providers. The Facilitator positions will play an important role in ‘selling’ the benefits of CQI and providing guidance for quality improvement activities. Further investment is needed in training at both a regional and local level, and for health service boards, managers and staff.
As discussed earlier, there are a range of CQI systems and tools available including the ABCD program used by DHF clinics and ACCHOs and the Australian Primary Care Collaboratives used by ACCHOS. There has been some discussion among the NT AHF regarding a standardised approach to CQI to allow comparability between services and as a basis for quality standards. A clear message that we heard through interviews of both health service staff and management was that health services were reluctant to discard current systems that were considered effective and useful and that there was a need for flexibility in choosing and adapting tools to local circumstances.

The NT AHF CQI framework notes that staff will be supported to identify the most appropriate tools and trained to use them. We believe that this is an appropriate approach, at least for the shorter-term. This would be assisted by developing a resource (such as a manual or a website) outlining the features of the various systems, costs, guidance as to what kind of health setting the tool may be most appropriate for, mechanisms to celebrate and publicise CQI success stories and a peer-support system.

The cost of the CQI tools can be reduced if they are collectively purchased by a number of health services. As the EHSDI regionalisation processes progress and as CQI becomes embedded as common practice, the standardisation of CQI approaches could be pursued as a longer-term goal. Whether a standardised approach is adopted or not, an important consideration is to ensure CQI is ‘owned’ at the service level to generate enthusiasm for its use and so that its results are more likely to influence service changes (Phillips et al 2010). In the meantime the NT AHKPIs, as discussed below, provide a standardised means of collecting data across the NT remote health sector.

5.7.2 NT Aboriginal Health Key Performance Indicators

The NT AHKPIs were developed in response to an identified need for a common set of indicators to establish a baseline against which Aboriginal health outcomes could be measured. While they were developed before the EHSDI, the NT AHKPIs are seen as a critical system-wide tool to support CQI activities under the EHSDI. The indicators are intended to be used at all community-based Aboriginal health services in the NT including both DHF and ACCHO services. The goal of the system is to contribute to improved PHC services by providing data on various aspects of health service functioning and on measures of population health. The information gained through NT AHKPIs aims to present a NT-wide view of processes and activities in Aboriginal PHC, and to highlight trends at both a local and NT-wide level and inform Aboriginal health planning and policy development.

In 2003 the NT AHF agreed on 44 NT AHKPIs of which 19 have been developed. The 19 KPIs provide information on four domains (Table 61) including 12 quantitative indicators of health services and seven qualitative indicators. These qualitative indicators cover management and support services (four indicators), linkages, policy and advocacy (one indicator) and community involvement (two indicators). Data has been systematically collected against the 12 quantitative KPIs, but the seven qualitative KPIs have yet to be finalised.
Top of page

Table 61: NT AHKPIs by domain
Health servicesManagement and support services
1.1 Episodes of health care and client contacts
1.2 First antenatal visit
1.3 Birth weight
1.4 Fully immunised children
1.5 Underweight children
1.6 Anaemic children
1.7 Chronic disease management plan
1.8 HbA1c tests(a)
1.9 ACE inhibitor and/or ARB(b)
1.10 Adult aged 15–54 health check
1.11 Adult aged 55 and over health check
1.12 PAP smear tests
2.13 Unplanned staff turnover
2.14 Recruits completing orientation training
2.15 Overtime workload
2.16 Quality improvement
Links, policy and advocacy
3.17 Report on service activities
Community involvement
4.18 Community involvement in determining health
4.19 Evidence of appropriate reporting to community

(a) The HbA1c test is used to determine whether diabetes is under control.
(b) The Angiotensin converting enzyme (ACE) inhibitors and Angiotensin Receptor Blockers (ARB) are oral medications used primarily to lower blood pressure.

The system is under the governance of a steering committee, under the NT AHF. The main focus of the committee to date has been on establishing effective systems and processes for the collection and use of 
KPI data.

NT AHKPI system logic

The NT AHKPI system is intended to act as a monitoring and information tool. It involves the systematic documentation of health service activities and outputs, which are tracked at regular intervals with reports for each community and HSDA produced twice per year. The monitoring of NT AHKPIs provides baseline information on processes of care at the PHC level and on the state of NT Aboriginal population health. The establishment of a baseline means goals can be set and interventions planned and monitored for improvements in health outcomes (Pope 2003).

KPI data has two key purposes—providing information to guide decision making to strengthen health service performance and offering accountability to stakeholders. Indicators can be useful in establishing a benchmark from which to track health service progress towards goals. KPI data can also be used as a means of comparison between health services, acting as a point of reference to provide context to the performance measures. It enables the health service to see how well it is doing compared to other areas. The information can be used to direct development decisions and modify the activities in which service providers are engaged, forming part of an ongoing innovation cycle.

At a higher level, KPIs can be used by policy makers and funders to obtain useful information for evidence-based policy decisions and can assist in planning new or targeted interventions. The establishment of baselines is also intended to allow for examining the impact of interventions on population health outcomes.

Data collection

In DHF health centres, NT AHKPI data is predominantly collected by PCIS which is currently being implemented across all DHF health centres. During the implementation phase of PCIS an interim data collection tool was used to capture NT AHKPI data using paper-based medical records. This will be used until it is replaced by PCIS which provides for automated data. A small amount of data is also captured from other sources such as hospital birth records. ACCHOs capture NT AHKPI data through their own patient electronic records systems (such as Communicare and Ferret).

Informants from health clinics that use electronic records systems generally felt positive towards NT AHKPI data collection, reporting that the data was easy to extract and did not present a burden to staff. The Communicare system, in particular, received positive reports for ease of use. In health services without electronic records, NT AHKPI data collection was seen as time consuming and onerous, taking time away from ‘on the ground’ work. The need for electronic data capture systems across all health centres was highlighted as a key priority to make data collection part of daily operation and not an additional task. Health service managers also believed that education of staff was essential to ensure that they understand the benefits of the NT AHKPIs and the reasons for collecting data.

Health centres that submit NT AHKPI data receive two reports—a community NT AHKPI report (containing information specific to that health centre) and an HSDA-level report (containing combined NT AHKPI data for the region). The HSDA report provides information on the performance of the HSDA against the indicators as well as in comparison with overall NT results. The reports contain quantitative data relating to indicators 1.1–1.12 and include a ‘key message’ section in which dialogue is supplied to assist with interpretation of the data in the reports. Reports are produced biannually—the calendar year report released in February and the financial year report released in August.
Top of page

5.7.3 Robustness of the NT Aboriginal Health Key Performance Indicator data

At the time of writing three NT AHKPI reporting cycles had been completed—the 2008–09 financial year, 2009 calendar year and 2009–10 financial year. All cycles achieved good coverage (over 95 per cent), but as one would expect in a new data system there were some problems with data integrity and some technical issues. Issues with the datasets include:
  • incomplete data collection at health centres which do not have an electronic data collection system
  • under reporting of some data (such as immunisation rates)
  • irregularities of interpretation of certain indicators (such as counting of pap smear requests) rather than results received
  • inconsistent resident/visitor status causing an over count of resident population
  • incomplete data for some sites due to data entry backlog.
The NT AHKPI HSDA reports draw attention to these data issues, containing a disclaimer that data quality will vary across health services ‘depending on their experience in using data and indicators, and the capacity of their data systems’ (p. 7). The reports further note that the quality of the information collected is being reviewed and that ‘data quality and collection methods will continue to be analysed and improved’ (p. 7).

In December 2009 a system review and assessment structure was formed to perform a CQI role. This structure consists of three working groups that focus on policy, clinical and technical issues and aims to ensure that the biannual reports will be continuously improved in quality and further developed for content scope. It is believed that the next round of reports will be more useful and accurate.

5.7.4 Use of NT Aboriginal Health Key Performance Indicator data

There is evidence that a minority of health services are currently seeking out and using NT AHKPI data for quality improvement purposes. In these services the community NT AHKPI reports are generally used to provide information to health service boards, although utility has at present been restricted due to the quality of data. Informants praised the reports as easy to interpret with the use of the graphs and tables providing ‘at a glance’ information, without the need to read large tracts of text. These services characterised the process as a useful means to reflect on service delivery and stated that the NT AHKPI reports provided a means for communicating improvements to the community.

Some health services indicated that a lack of organisational capacity was a barrier to successful use of the NT AHKPI data to facilitate quality improvement. The majority of informants that we spoke to during the evaluation case studies (February–March 2010) indicated that their knowledge of CQI was limited and reported a lack of understanding about the NT AHKPI system. Evaluation participants stated that they were unsure how to use the information contained in the reports and were often uncertain as to how the indicators would drive service provision and improvement. Barriers to success included use of paper-based records and concerns about the extra workload associated with data collection.

Training and dissemination of information about CQI was identified as critical to increase the confidence of health service providers. The CQI Facilitator positions funded under the EHSDI will play an important role in helping organisations understand CQI and interpret and use their NT AHKPI results. Workshops for PHC staff on using NT AHKPI data for quality improvement purposes are anticipated to be run within the next few months and teleconferences with CQI Facilitators have commenced. There are indications that with increased knowledge of, and familiarity with, the NT AHKPI system, use of data by health services will increase. Several health centre managers expressed a desire for information on the performance of their service and believed that the NT AHKPI system had the potential to be a useful tool.

Interviews with government officials suggested that the NT AHKPIs are generally viewed positively as a tool with the potential to inform higher-level Aboriginal health planning and decision making. Although there was acknowledgment that it would take some time to gain benefits from the use of KPI data, informants stated that they valued the ability to gain a broad understanding of trends in population health issues.
Key benefits articulated included the establishment of a benchmark from which to measure progress towards health system goals and an evidence base on which to build policy decisions.

Evaluation participants also raised several suggestions regarding how the NT AHKPI data could be more effectively disseminated. Informants stated that there had been a lot of negative media coverage of Aboriginal health and that getting news of achievements out and celebrating those achievements remained a challenge. Gains revealed through the indicators could be used to highlight successes at both a local level, leading to increased community pride, and at a national level to encourage further investment in Aboriginal health.

Informants also emphasised the need for greater communication with the Australian Government concerning health issues in remote NT communities. It was felt that quantitative data was considered the ‘gold standard’ by many in the Australian Government. Once issues with data quality are resolved, NT AHKPI data has the potential to provide clear and concise evidence to highlight health needs in Aboriginal communities.

Any use of NT AHKPI data needs to conform to the data management strategy (NT AHF 2008b) which outlines when and how data can be released. This strategy is currently under review and will offer protocols for receiving, release and privacy of data, as well as an overarching governance structure to ensure data is protected from unauthorised use and is provided under consistent protocols.

5.7.5 The tension between accountability and improvement

While the primary use of the NT AHKPIs is to facilitate continuous improvement, international trends show an increasing demand for accountability data on the performance of publicly-funded initiatives, as part of a quest for an evidence base for the investment of public money (Newcomer et al 2004). Tensions may arise in the KPI system between the conflicting demands of data use for accountability and data use for improvement and learning. Under the improvement paradigm the purpose of KPI data is to enhance the quality of services provided. For the accountability paradigm, the primary purpose is to demonstrate to funders, oversight agencies and the public that the service is achieving results and is therefore worth continuing support.

Several informants raised concerns that the NT AHKPI data could be used as leverage for funding cuts if health services are underperforming against the indicators. While the intended purpose of the NT AHKPI data collection is not to inform funding decisions, international experience shows an increasing interest among policy makers in ‘pay-for-performance’ funding approaches (Mullen et al 2010). Such approaches reward healthcare providers for meeting certain measures of quality and efficiency. Basing funding decisions on performance against KPIs has the potential to encourage ‘gaming’ and manipulation of the data to maximise income which could compromise the quality of the data for internal CQI purposes. This indicates a need for agreement between the health sector and the government that NT AHKPI data will not be used for performance-related funding decisions.

KPI data can offer value in informing funding decisions provided it is used appropriately. For example, poor performance against indicators may be due to under-resourcing and a strong case could be made for increased funding. Conversely, if there is poor performance in a situation of relatively high resources there may be a case for closer scrutiny or review of management processes.
Top of page

5.7.6 The need for context-specific information

KPI data can be useful for monitoring health service activities for progress in the desired direction. The NT AHKPIs monitor activities and quantifiable outputs, such as the number and proportion of underweight children, but do not provide data on the impacts or health outcomes of such activities. As such, they offer a tangible means of reporting on health service outputs via numbers and trends, but cannot provide sufficient data to evaluate the performance related to the impact of health services on Aboriginal health.

A number of complex factors and situations can affect health in any given area and in remote Aboriginal communities in particular. The critical issue is in the appropriate interpretation and use of KPI data taking account of the unique contexts of each health service. The community NT AHKPI reports offer organisations an opportunity to provide comments on their data. Health centres receive an initial copy of the report and are invited to comment on any known issues that may affect their NT AHKPI data. This feedback is then inserted into the final report. Informants indicated that at present many felt unsure as to how to provide relevant comments on the data. To address this, a prompt sheet has been developed to facilitate the provision of systematic feedback.

While this offers health organisations a chance to outline potential reasons for performance, the NT AHKPI data could be further enhanced by a greater emphasis on understanding the background drivers behind the data. Data generated through KPI systems is only moderately useful until it is given meaning by an analysis of context. This might include considering the characteristics of the communities and each HSDA and other determinants (such as social factors) which may be affecting the data. To facilitate deeper understanding of the results, the NT AHKPI data could be linked with qualitative information such as health impact assessments, using mechanisms such as surveys or story collection. This view was supported by several informants who indicated that supplementing the information contained in the reports with qualitative stories would facilitate a deeper understanding of the experiences behind the data.

The NT AHKPIs are currently in the development and implementation phase and the emphasis has been on ensuring systems and processes are in place and functioning effectively. The seven qualitative indicators are still in development and data has not been systematically collected against these. Such data (for example on unplanned staff turnover) would help to put the clinical data into context. As the system matures, consideration could be given to expanding the scope of the indicator set to include outcome and impact measures. These measures would need to be carefully chosen so that they are sensitive to change in PHC performance. Adding these to the existing process or ‘quality of care’ indicators would enable a more comprehensive picture of Aboriginal health in the NT and would enable the tracking of progress towards desired health outcomes.
Top of page

5.7.7 Summary and conclusions

CQI is a core element of the EHSDI and is seen by the NT AHF as critical to support service improvement in the remote NT health sector. Just over $3 million has been spent on CQI over a two-year period. An EHSDI CQI strategy has been endorsed by the NT AHF and a model has been developed which outlines 10 areas as key elements which each service needs to address as part of their CQI approach. The CQI program is in the initial phases and there has been varied progress in implementing the components of the strategy.

The CQI process has documented support for its ability to improve health service delivery and contribute to improved health outcomes (Bailie et al 2008; Si et al 2008) and we conclude that investment in CQI should be continued. Investment priority should be given to providing initial and ongoing training for all levels of health service staff, as well as developing resources and tools for additional support.

Evaluation participants highlighted flexibility to select and adapt context-appropriate CQI tools as important. Further consideration should be given to developing a resource to assist health services to select suitable CQI tools and enable users to provide feedback and peer support for CQI. As a longer-term goal, the standardisation of CQI approaches across the NT remote health sector could be pursued. Initially effort should be directed towards building capacity at the local level.

The NT AHKPIs are a system-wide tool to provide data to support continuous quality improvement activities under the EHSDI. Although there is cautious enthusiasm regarding the NT AHKPIs among health service providers, a lack of understanding on how to interpret the results is currently hindering their full use. The current NT AHKPI system lacks clear guidance for staff on how data could be used to inform health service planning. Emphasis needs to be placed on developing staff capacity to use data effectively.

The current indicators gather quantitative data on health service processes and outputs. The system has enabled, for the first time, the construction of an NT-wide view of health service activities and offers a tangible means of highlighting trends and changes. The NT AHKPIs are therefore useful for performance reporting and accountability purposes.

The NT AHKPIs are a useful tool for monitoring health service outputs but do not attempt to evaluate health service impacts. Future development of the NT AHKPI system could consider supplementing the data with specific qualitative information to provide context to the numerical results and enable a broader understanding of the background drivers for the trends and results revealed by the indicators.
Top of page