5.7.1 Continuous quality improvement
5.7.2 NT Aboriginal Health Key Performance Indicators
5.7.3 Robustness of the NT Aboriginal Health Key Performance Indicator data
5.7.4 Use of NT Aboriginal Health Key Performance Indicator data
5.7.5 The tension between accountability and improvement
5.7.6 The need for context-specific information
5.7.7 Summary and conclusions
This section addresses the evaluation objectives of how the EHSDI has had an impact on the availability and use of information to support improved health service delivery. The EHSDI reform agenda includes a focus on CQI of health service delivery. Improved information systems and data availability are critical for monitoring health system performance and supporting quality improvement. Information is needed on the state of NT Aboriginal population health, and the functioning of PHC services.
CQI is a key pillar of PHC system reform under the EHSDI. A total of $3.001 million of EHSDI funding has been spent on CQI to date, including $0.212 million in 2008–09 and $2.789 million in 2009–10. Systematic investment in CQI is seen by the NT AHF as a critical platform for NT remote PHC sector reform.
There was no coordinated and coherent CQI model across the NT remote PHC sector before the EHSDI. Bailie et al (2008) state that in recent years there has been a high level of interest isn CQI activities in the Aboriginal health sector, as demonstrated by the voluntary uptake of CQI systems by health services. Before the EHSDI, the DHF instigated a system of quality assurance in its clinics and engaged the Audit and Best Practice in Chronic Disease (ABCD) program developed by the Menzies School of Health Research. Several ACCHOs also used the ABCD program or used other CQI programs including the Australian Primary Care Collaboratives, DoHA’s Health Quality Improvement Initiatives including the Continuous Improvement Projects (CIP), and the Healthy for Life Program funded by DoHA.
These programs were introduced with improved PIRS in clinics; however, the uptake of CQI activities was variable across the NT Aboriginal health sector. While some services were participating in CQI at a high level using two or three formal CQI systems, others lacked the capacity to implement or effectively utilise CQI systems.
PHRG internal documentation notes that health service involvement in CQI has been demonstrated to improve the quality and appropriateness of care, leading to improved health outcomes (PHRG briefing paper, February 2009). There is acknowledgment by the NT AHF partners that significant disparities existed in the quality of care offered between different health services across the NT and the need for an effective CQI process in the NT PHC system had been recognised for some time (PHRG EHSDI CQI Proposal, November 2008). Embedding quality improvement into PHC is one of the key priorities of the PHRG. Australian Government investment in the NT health system under the EHSDI was seen as an opportunity to build and implement a sector-wide CQI model.
In December 2008 the NT AHF agreed that there was a need for ‘a strong and effective CQI process in PHC in the NT’ (NT AHF Proposed EHSDI investment in continual quality improvement, February 2009). An EHSDI CQI strategy was endorsed in March 2009, with the core aim of building on existing quality improvement activities to support long-term, coordinated and sustainable service improvement across the NT remote PHC sector. The strategy has five major components:
- Recruitment of two CQI Coordinators (Top End and Central) to be based at AMSANT to provide expert leadership in the developing a sector-wide CQI model and training and support to CQI positions in the HSDAs.
- Recruitment of a CQI Facilitator for each HSDA to assist health services with implementing CQI and to support and mentor practitioners in the use of CQI.
- Development of a CQI model which will provide consistency and sustainability across the NT with the flexibility to allow for local circumstances.
- Establishment of a CQI program planning committee to provide advice, develop the NT CQI model and monitor implementation.
- Evaluation of the CQI program to inform future development and CQI investment (AMSANT 2009c).
Top of page
A CQI model for NT Aboriginal PHC was developed by the PHRG and endorsed by the NT AHF. This model sets out a framework for CQI to be shared by both DHF and ACCHO service providers. The framework outlines 10 key elements that each service needs to address as part of their CQI approach. These include:
- support from leadership and management, including incorporating CQI into health service planning and devoting management and staff time to CQI
- involving and orienting the entire health service team to CQI protocols and practices
- providing orientation, training and rewarding achievements to gain staff support for CQI
- effective communication such as newsletters, teleconferences and an email network
- encouraging Aboriginal engagement to provide input into CQI planning and policies at an NT-wide, regional and service level
- consumer input based on strategies to seek the views of patients and communities
- on-site and regional level training for PHC staff and shared learning through the promotion of success stories
- high quality data, including NT AHKPIs and other clinical health data
- identifying and using appropriate CQI tools
- timely and relevant feedback at all levels including health boards, staff, consumers and communities (NT AHF Briefing paper for NT AHF meeting 47, December 2009).
Discussion of the EHSDI continuous quality improvement programThe EHSDI CQI program is relatively new and yet to be fully implemented. There has been limited progress in developing the key elements of the CQI framework outlined above. Several of the main elements emphasise the engagement and input of personnel at all levels of the PHC system, including management, health staff, consumer and Aboriginal community input. The majority of informants we spoke to, including both health service staff and health centre managers, indicated that their knowledge of, and experience in, CQI was limited. The initial delay in recruitment of CQI Facilitators meant that many health service staff received little information and assistance in implementing CQI activities. This has led to mixed feelings among informants, with some articulating cautious enthusiasm, while others felt anxious about their ability to put CQI systems into practice.
Gaining the support of community-level PHC staff is essential to increase the uptake and effectiveness of CQI activities and to gain community support and involvement. This can best be achieved through training and dissemination of information about CQI to increase the confidence of health service providers. The Facilitator positions will play an important role in ‘selling’ the benefits of CQI and providing guidance for quality improvement activities. Further investment is needed in training at both a regional and local level, and for health service boards, managers and staff.
As discussed earlier, there are a range of CQI systems and tools available including the ABCD program used by DHF clinics and ACCHOs and the Australian Primary Care Collaboratives used by ACCHOS. There has been some discussion among the NT AHF regarding a standardised approach to CQI to allow comparability between services and as a basis for quality standards. A clear message that we heard through interviews of both health service staff and management was that health services were reluctant to discard current systems that were considered effective and useful and that there was a need for flexibility in choosing and adapting tools to local circumstances.
The NT AHF CQI framework notes that staff will be supported to identify the most appropriate tools and trained to use them. We believe that this is an appropriate approach, at least for the shorter-term. This would be assisted by developing a resource (such as a manual or a website) outlining the features of the various systems, costs, guidance as to what kind of health setting the tool may be most appropriate for, mechanisms to celebrate and publicise CQI success stories and a peer-support system.
The cost of the CQI tools can be reduced if they are collectively purchased by a number of health services. As the EHSDI regionalisation processes progress and as CQI becomes embedded as common practice, the standardisation of CQI approaches could be pursued as a longer-term goal. Whether a standardised approach is adopted or not, an important consideration is to ensure CQI is ‘owned’ at the service level to generate enthusiasm for its use and so that its results are more likely to influence service changes (Phillips et al 2010). In the meantime the NT AHKPIs, as discussed below, provide a standardised means of collecting data across the NT remote health sector.
In 2003 the NT AHF agreed on 44 NT AHKPIs of which 19 have been developed. The 19 KPIs provide information on four domains (Table 61) including 12 quantitative indicators of health services and seven qualitative indicators. These qualitative indicators cover management and support services (four indicators), linkages, policy and advocacy (one indicator) and community involvement (two indicators). Data has been systematically collected against the 12 quantitative KPIs, but the seven qualitative KPIs have yet to be finalised.
Top of page
|Health services||Management and support services|
|1.1 Episodes of health care and client contacts|
1.2 First antenatal visit
1.3 Birth weight
1.4 Fully immunised children
1.5 Underweight children
1.6 Anaemic children
1.7 Chronic disease management plan
1.8 HbA1c tests(a)
1.9 ACE inhibitor and/or ARB(b)
1.10 Adult aged 15–54 health check
1.11 Adult aged 55 and over health check
1.12 PAP smear tests
|2.13 Unplanned staff turnover|
2.14 Recruits completing orientation training
2.15 Overtime workload
2.16 Quality improvement
|Links, policy and advocacy|
|3.17 Report on service activities|
|4.18 Community involvement in determining health |
4.19 Evidence of appropriate reporting to community
(a) The HbA1c test is used to determine whether diabetes is under control.
(b) The Angiotensin converting enzyme (ACE) inhibitors and Angiotensin Receptor Blockers (ARB) are oral medications used primarily to lower blood pressure.
The system is under the governance of a steering committee, under the NT AHF. The main focus of the committee to date has been on establishing effective systems and processes for the collection and use of KPI data.
NT AHKPI system logicThe NT AHKPI system is intended to act as a monitoring and information tool. It involves the systematic documentation of health service activities and outputs, which are tracked at regular intervals with reports for each community and HSDA produced twice per year. The monitoring of NT AHKPIs provides baseline information on processes of care at the PHC level and on the state of NT Aboriginal population health. The establishment of a baseline means goals can be set and interventions planned and monitored for improvements in health outcomes (Pope 2003).
KPI data has two key purposes—providing information to guide decision making to strengthen health service performance and offering accountability to stakeholders. Indicators can be useful in establishing a benchmark from which to track health service progress towards goals. KPI data can also be used as a means of comparison between health services, acting as a point of reference to provide context to the performance measures. It enables the health service to see how well it is doing compared to other areas. The information can be used to direct development decisions and modify the activities in which service providers are engaged, forming part of an ongoing innovation cycle.
At a higher level, KPIs can be used by policy makers and funders to obtain useful information for evidence-based policy decisions and can assist in planning new or targeted interventions. The establishment of baselines is also intended to allow for examining the impact of interventions on population health outcomes.
Data collectionIn DHF health centres, NT AHKPI data is predominantly collected by PCIS which is currently being implemented across all DHF health centres. During the implementation phase of PCIS an interim data collection tool was used to capture NT AHKPI data using paper-based medical records. This will be used until it is replaced by PCIS which provides for automated data. A small amount of data is also captured from other sources such as hospital birth records. ACCHOs capture NT AHKPI data through their own patient electronic records systems (such as Communicare and Ferret).
Informants from health clinics that use electronic records systems generally felt positive towards NT AHKPI data collection, reporting that the data was easy to extract and did not present a burden to staff. The Communicare system, in particular, received positive reports for ease of use. In health services without electronic records, NT AHKPI data collection was seen as time consuming and onerous, taking time away from ‘on the ground’ work. The need for electronic data capture systems across all health centres was highlighted as a key priority to make data collection part of daily operation and not an additional task. Health service managers also believed that education of staff was essential to ensure that they understand the benefits of the NT AHKPIs and the reasons for collecting data.
Health centres that submit NT AHKPI data receive two reports—a community NT AHKPI report (containing information specific to that health centre) and an HSDA-level report (containing combined NT AHKPI data for the region). The HSDA report provides information on the performance of the HSDA against the indicators as well as in comparison with overall NT results. The reports contain quantitative data relating to indicators 1.1–1.12 and include a ‘key message’ section in which dialogue is supplied to assist with interpretation of the data in the reports. Reports are produced biannually—the calendar year report released in February and the financial year report released in August.
Top of page
- incomplete data collection at health centres which do not have an electronic data collection system
- under reporting of some data (such as immunisation rates)
- irregularities of interpretation of certain indicators (such as counting of pap smear requests) rather than results received
- inconsistent resident/visitor status causing an over count of resident population
- incomplete data for some sites due to data entry backlog.
In December 2009 a system review and assessment structure was formed to perform a CQI role. This structure consists of three working groups that focus on policy, clinical and technical issues and aims to ensure that the biannual reports will be continuously improved in quality and further developed for content scope. It is believed that the next round of reports will be more useful and accurate.
Some health services indicated that a lack of organisational capacity was a barrier to successful use of the NT AHKPI data to facilitate quality improvement. The majority of informants that we spoke to during the evaluation case studies (February–March 2010) indicated that their knowledge of CQI was limited and reported a lack of understanding about the NT AHKPI system. Evaluation participants stated that they were unsure how to use the information contained in the reports and were often uncertain as to how the indicators would drive service provision and improvement. Barriers to success included use of paper-based records and concerns about the extra workload associated with data collection.
Training and dissemination of information about CQI was identified as critical to increase the confidence of health service providers. The CQI Facilitator positions funded under the EHSDI will play an important role in helping organisations understand CQI and interpret and use their NT AHKPI results. Workshops for PHC staff on using NT AHKPI data for quality improvement purposes are anticipated to be run within the next few months and teleconferences with CQI Facilitators have commenced. There are indications that with increased knowledge of, and familiarity with, the NT AHKPI system, use of data by health services will increase. Several health centre managers expressed a desire for information on the performance of their service and believed that the NT AHKPI system had the potential to be a useful tool.
Interviews with government officials suggested that the NT AHKPIs are generally viewed positively as a tool with the potential to inform higher-level Aboriginal health planning and decision making. Although there was acknowledgment that it would take some time to gain benefits from the use of KPI data, informants stated that they valued the ability to gain a broad understanding of trends in population health issues.
Key benefits articulated included the establishment of a benchmark from which to measure progress towards health system goals and an evidence base on which to build policy decisions.
Evaluation participants also raised several suggestions regarding how the NT AHKPI data could be more effectively disseminated. Informants stated that there had been a lot of negative media coverage of Aboriginal health and that getting news of achievements out and celebrating those achievements remained a challenge. Gains revealed through the indicators could be used to highlight successes at both a local level, leading to increased community pride, and at a national level to encourage further investment in Aboriginal health.
Informants also emphasised the need for greater communication with the Australian Government concerning health issues in remote NT communities. It was felt that quantitative data was considered the ‘gold standard’ by many in the Australian Government. Once issues with data quality are resolved, NT AHKPI data has the potential to provide clear and concise evidence to highlight health needs in Aboriginal communities.
Any use of NT AHKPI data needs to conform to the data management strategy (NT AHF 2008b) which outlines when and how data can be released. This strategy is currently under review and will offer protocols for receiving, release and privacy of data, as well as an overarching governance structure to ensure data is protected from unauthorised use and is provided under consistent protocols.
Several informants raised concerns that the NT AHKPI data could be used as leverage for funding cuts if health services are underperforming against the indicators. While the intended purpose of the NT AHKPI data collection is not to inform funding decisions, international experience shows an increasing interest among policy makers in ‘pay-for-performance’ funding approaches (Mullen et al 2010). Such approaches reward healthcare providers for meeting certain measures of quality and efficiency. Basing funding decisions on performance against KPIs has the potential to encourage ‘gaming’ and manipulation of the data to maximise income which could compromise the quality of the data for internal CQI purposes. This indicates a need for agreement between the health sector and the government that NT AHKPI data will not be used for performance-related funding decisions.
KPI data can offer value in informing funding decisions provided it is used appropriately. For example, poor performance against indicators may be due to under-resourcing and a strong case could be made for increased funding. Conversely, if there is poor performance in a situation of relatively high resources there may be a case for closer scrutiny or review of management processes.
Top of page
A number of complex factors and situations can affect health in any given area and in remote Aboriginal communities in particular. The critical issue is in the appropriate interpretation and use of KPI data taking account of the unique contexts of each health service. The community NT AHKPI reports offer organisations an opportunity to provide comments on their data. Health centres receive an initial copy of the report and are invited to comment on any known issues that may affect their NT AHKPI data. This feedback is then inserted into the final report. Informants indicated that at present many felt unsure as to how to provide relevant comments on the data. To address this, a prompt sheet has been developed to facilitate the provision of systematic feedback.
While this offers health organisations a chance to outline potential reasons for performance, the NT AHKPI data could be further enhanced by a greater emphasis on understanding the background drivers behind the data. Data generated through KPI systems is only moderately useful until it is given meaning by an analysis of context. This might include considering the characteristics of the communities and each HSDA and other determinants (such as social factors) which may be affecting the data. To facilitate deeper understanding of the results, the NT AHKPI data could be linked with qualitative information such as health impact assessments, using mechanisms such as surveys or story collection. This view was supported by several informants who indicated that supplementing the information contained in the reports with qualitative stories would facilitate a deeper understanding of the experiences behind the data.
The NT AHKPIs are currently in the development and implementation phase and the emphasis has been on ensuring systems and processes are in place and functioning effectively. The seven qualitative indicators are still in development and data has not been systematically collected against these. Such data (for example on unplanned staff turnover) would help to put the clinical data into context. As the system matures, consideration could be given to expanding the scope of the indicator set to include outcome and impact measures. These measures would need to be carefully chosen so that they are sensitive to change in PHC performance. Adding these to the existing process or ‘quality of care’ indicators would enable a more comprehensive picture of Aboriginal health in the NT and would enable the tracking of progress towards desired health outcomes.
Top of page
The CQI process has documented support for its ability to improve health service delivery and contribute to improved health outcomes (Bailie et al 2008; Si et al 2008) and we conclude that investment in CQI should be continued. Investment priority should be given to providing initial and ongoing training for all levels of health service staff, as well as developing resources and tools for additional support.
Evaluation participants highlighted flexibility to select and adapt context-appropriate CQI tools as important. Further consideration should be given to developing a resource to assist health services to select suitable CQI tools and enable users to provide feedback and peer support for CQI. As a longer-term goal, the standardisation of CQI approaches across the NT remote health sector could be pursued. Initially effort should be directed towards building capacity at the local level.
The NT AHKPIs are a system-wide tool to provide data to support continuous quality improvement activities under the EHSDI. Although there is cautious enthusiasm regarding the NT AHKPIs among health service providers, a lack of understanding on how to interpret the results is currently hindering their full use. The current NT AHKPI system lacks clear guidance for staff on how data could be used to inform health service planning. Emphasis needs to be placed on developing staff capacity to use data effectively.
The current indicators gather quantitative data on health service processes and outputs. The system has enabled, for the first time, the construction of an NT-wide view of health service activities and offers a tangible means of highlighting trends and changes. The NT AHKPIs are therefore useful for performance reporting and accountability purposes.
The NT AHKPIs are a useful tool for monitoring health service outputs but do not attempt to evaluate health service impacts. Future development of the NT AHKPI system could consider supplementing the data with specific qualitative information to provide context to the numerical results and enable a broader understanding of the background drivers for the trends and results revealed by the indicators.
Top of page