Review of the Multicultural Mental Health Australia (MMHA) project

4. Documentation review

Page last updated: November 2009

This chapter presents the findings of the documentation review against the broad domains covered by the review objectives. The results of the documentation review were later re-examined alongside the findings of the stakeholder consultations (reported in chapter 5) to inform the formulation of conclusions against the review objectives.

4.1 Documentation review purpose
4.2 Documentation review process
4.3 Documentation review findings
4.3.1 MMHA identity and consistency of terminology
4.3.2 Project governance and accountability
4.3.3 Project model
4.3.4 Project planning
4.3.5 Project financial management
4.3.6 Project reporting and performance measures

4.1 Documentation review purpose

Both DoHA and MMHA provided the review team with a range of relevant documentation pertaining to the Project. Broadly this encompassed:
  • Administrative and general/background information
  • Previous reviews of MMHA
  • Funding Agreements and variations
  • Planning processes
  • Project management and operations
  • Project achievements.
The review and analysis of available documentation served to assist in:
  • Familiarisation with the background, establishment and achievements of the Project
  • Identification of any gaps in information that needed to be accessed in order to address the review objectives
  • Development and formulation of the lines of enquiry/questions to be explored in the course of the required stakeholder consultations.
Top of page

4.2 Documentation review process

Initial review of the documentation served to ensure that the review team had a sound understanding of the history and establishment of the Project. Of particular relevance were documents covering the background to the Project, past reviews of MMHA, and the current Funding Agreement. This ensured that initial consultations with DoHA, MMHA and SWAHS were underpinned by a good understanding of their respective roles in the management and operation of the Project.

During the course of concurrent stakeholder consultations, the need to examine additional documentation related to the review objectives became evident. Following submission of requests for additional documentation to MMHA and DoHA, this was provided (where available) to the review team. An objective review of all documentation was subsequently undertaken against the review objectives.

As evident from appendix A, a considerable volume of documentation was acquired by the review team over the course of the review. It is not intended that this chapter present a detailed account or summary of all the material provided and reviewed. Some material served primarily as background/context to the project. Not all documentation provided proved to be relevant or central to answering the review objectives.

The approach taken with respect to this component of the review, and what this chapter aims to establish, is:
  • whether documentation exists in relation to procedures, processes and mechanisms related to the broad domains explored in this review; and
  • the degree to which this documentation contributes to an objective evidence base to inform conclusions against the review objectives. This judgment is based on an assessment of the quality of the documentation. Specifically, its clarity, relevance, appropriateness and completeness/comprehensiveness.
Where the review team has been provided with, or requested specific feedback/advice from stakeholders in relation to particular documentation, this is included in this chapter.

4.3 Documentation review findings

4.3.1 MMHA identity and consistency of terminology

Across the documentation reviewed in the course of this review (as in our stakeholder consultations) there was considerable variability in how MMHA was described. To illustrate, the following extracts are taken from several key documents:
  • The MMHA Consortium Governance document (2007)states that: 'The program [MMHA] represents an alliance of consumers, carers, the community, statewide specialist services in multicultural mental health and suicide prevention,'
  • Schedule A of the current Funding Agreement describes MMHA as: '...the national project under the National Mental Health Strategy by the Australian Government..'
  • The following paragraph in the current Funding Agreement however refers to MMHA as: '...the Australian Government funded peak body established to undertake the Project and to represent and promote the interests of the mental health sector in CALD communities'
  • The Schedule further states the principles on which the Funding Agreement is based, one of which is: 'recognition of the role of peak advocacy organisations on contributing to the development of responsive and well-informed policy'
This inconsistent use of terminology only serves to foster confusion among stakeholders as to MMHA's role and function (discussed further in the next chapter). It is in the opinion of the reviewers this represents one of the key if not focal issues that needs to be addressed particularly if ongoing public monies is continued to be invested in such an entity. Specifically it has implications for the project going forward, with respect to the extent to which the current governance and accountability arrangements are changed.
Top of page
The remaining sections summarise the key findings of the documentation review are presented against the following broad domains:
  • Project governance
  • Program model
  • Project planning
  • Project financial and service management
  • Project reporting and performance measures.
It is important to note that there is a high degree of interdependence between these domains and accordingly some of the observations made in the documentation review have been highlighted in more than one area.

Questions related to project financial and service management, and to project reporting and performance measures were asked of only the small number of respondents who were sufficiently informed to comment. These included representatives of DoHA, SWAHS and MMHA. Therefore for the results of responses obtained in relation to these areas are incorporated into this chapter (rather than in the following chapter).

In presenting the findings of the documentation review, we include commentary on best practice sourced from respected authorities on public sector operations (e.g. The Australian National Audit Office, Australian Public Service Commission).

4.3.2 Project governance and accountability

Governance refers to the processes, policies and procedures whereby entities/organisations are directed, controlled and held to account. The exercise of governance arrangements provide strategic direction, ensure objectives are achieved, assist in risk management and responsible use of resources. It provides the framework within which managers make decisions and take actions to optimise outcomes. Governance arrangements should include at least the roles and responsibilities of those involved, rules and procedures for decision making and integration of the project/program/initiative governance arrangements within the agency's broader governance framework.17

The principles of good governance are accountability, transparency, integrity, stewardship, leadership and efficiency. Accountability is the process whereby entities/public sector organisations and the individuals within them, are responsible for their decisions and actions. It is achieved by all parties having a clear understanding of those responsibilities and having clearly defined roles through a robust structure.18

Sound governance arrangements are critical to the successful implementation of programs, projects, initiatives and policy. They serve to strengthen community confidence in a public entity, and help ensure entities' reputations are maintained and enhanced. Good governance enables entities to perform efficiently and effectively, and to respond strategically to changing demands.19

Governance structures

Review of key documentation shows two key national governance structures: the Consortium and the Joint Officers Group (JOG).

The MMHA Consortium Governance Document describes MMHA as 'the program representing an alliance of consumers, carers, the community, state-wide specialist services in multicultural mental health and suicide prevention, population and public health and the tertiary sector operating as Consortium'. This document sets out the purpose, objectives, structure and membership of the Consortium including MMHA members (state-wide specialist service providers, national peak consumer/carer and community organisations and founder organisations); MHHA Advisors (mental health experts) and Associate Organisations. It further defines the roles and relationships between the three key parties: DoHA, SWAHS and MMHA. The financial and service management arrangements between these parties are discussed further in section 4.3.5.

As discussed in Chapter 2, the Consortium was the forum through which MMHAs policy priorities and strategic directions were set and monitored. We requested and reviewed all available records of Consortium meetings between 2003 and the 2008 to gain some insight into the effectiveness of its operation. Overall, the level of record keeping was poor, particularly prior to 2006. There was one meeting in 2003, two in 2004 and three in the remaining years. Meetings in early years have agendas but no minutes or record of attendance. Meetings in later years however (2007-2008) showed improved records with agenda, a record of minutes, actions taken and a record of attendance. In the absence of good recordkeeping the transparency of its operations is compromised. Without good records of these meetings, it is difficult to assess the effectiveness of the meetings and the basis on which decisions were made.Top of page

The MMHA Consortium has operated since 2003. The documentation provided to the review team show that up until 2007, there were three different documents regarding the Consortium's governance arrangements – MMHA Terms of Reference, MMHA Operating Guidelines and a Memorandum of Understanding. The review team has been informed that members did not sign up to the MOU. In January 2007, it was decided to combine these documents into one document (the current MMHA Consortium Governance document). This document was not endorsed by the members until six months later (June 2007).

Review of a range of additional documentation over this period (e.g. internal memos, electronic and written correspondence between Consortium members and DoHA, and between MMHA/SWAHS and DoHA) reveal less than satisfactory adherence to these governance arrangements by some Consortium members. This resulted in increasing dissatisfaction with the Consortium operation among members. This is supported by the findings of our stakeholder consultations presented in the next chapter.

In 2005, MMHA underwent a review to assess its performance and achievements against the Funding Agreement for the period 2003-05. One of the recommendations was that MMHA should establish the JOG in conjunction with DoHA as an annual forum for State and Territory representatives to meet with DoHA and the Consortium. The purpose of this forum was to review priority issues for the Program, discuss plans for mental health reform in each jurisdiction, identify opportunities for collaboration and determine priorities for Projects of National significance.

Following the 2005 evaluation of MMHA, the Joint Officer Group (JOG) was established. According to the Terms of Reference, the JOGs role is to 'facilitate nation-wide implementation of the Framework for Implementation of the National Mental Health Plan 2003-06 in Multicultural Australia'. Its membership (to meet bi-annually) comprises state and Territory Mental Health Directors. Its specific responsibilities are to:
  • Provide a formal mechanism to ensure all states and territories have an opportunity to contribute to priority setting within MMHA and to the development of MMHAs national program and work plan, particularly where MMHA has responsibility for driving or facilitating national action on the service responsiveness of the Framework

  • Act as a forum for state and territory representatives to meet with DoHA and the MMHA consortium to:
    • Review priority issue for the program
    • Discuss plans for multicultural mental health reform in each jurisdiction
    • Identify opportunities for collaboration
    • Determine priorities for projects of national significance
    • Make recommendations on program development to the Mental Health Standing Committee (formerly the National Mental Health Working Group) and facilitate practical partnerships between MMHA and each jurisdiction.
We reviewed the entire set of available minutes for the JOG meetings between 2006 and 2009. Overall, we consider they are a good record of these meeting, with records of agenda, minutes, actions arising and attendance. It is noted however that with the exception of the first and most recent (May 2009) meetings, many Directors did not attend the meetings, instead delegating to officer level in most instances. While they may have better specific knowledge, they do not have the decision making power. The recommendation to strengthen the JOG was aimed at securing commitment by the State Directors to attend and bring their jurisdictional issues to the table. It sought to encourage their consultation with local CALD stakeholders and to get exposure 'on the ground' on initiatives. If the JOG is to fulfil its intended purpose then it will be important that it is capable of attracting the high-level representation of decision makers.

In mid 2008, SWAHS put forward a proposal to DoHA to implement an alternative model to the Consortium. The rationale was that MMHA had "become out of step with recent changes in the mental health sector' and that the Consortium model was 'limited and outdated'. This was mainly attributed to the 'inequitable representation and exclusion of important stakeholders' to date and significant running costs. It was argued that a more democratic, participatory and considerate structure with better consultation and communication processes was required. It was proposed that a better model would include, among other things, 'a strengthening of the Joint Officers Group' through limiting the membership to State Mental Health Directors and MMHA, and the conduct of regular state/territory based consultations (for needs identification and priority setting) involving all key stakeholders ('not just the select few that are currently on the Consortium') and establishment of the Carer and Consumer Reference Groups. This proposal was accepted by DoHA without consultation with members of the Consortium. The last meeting of the Consortium was held in November 2008.
Top of page

4.3.3 Project model

The MMHA Consortium document describes the MMHA model as one of collaboration and partnership aimed at addressing issues of transcultural mental health and suicide prevention. This is achieved by building strategic alliances and networks which focus on national mainstream programs, state and territory mental health services, specialist transcultural, refugee and torture and trauma services, consumers, carers and community sector and the ethnic media.

The model further involves engagement with the community, State and territory specialist and mainstream services, other relevant government agencies and the tertiary sector to improve access, responsiveness and quality of services and to facilitate access to information about services, mental health and mental illness, and to promote good mental health in diverse communities.

A group of Advisors who are expert in the multicultural mental health arena are also part of the MMHA model. These individuals provide strategic advice to the project on particular issues and projects. MMHA also undertakes particular projects as well as working collaboratively with partner organisations in order to achieve specific project objectives. In addition, MMHA has strong affiliations with organisations who share a similar interest in promoting the mental health and wellbeing of Australians from a CALD background.

The model is driven by an alliance of individuals and organisations including consumers and carers, statewide specialist multicultural mental health and suicide prevention services, population and public health and the tertiary sector operating as a Consortium.

It works to:
  • Develop new and improved partnerships
  • More firmly embed transcultural mental health and suicide prevention in the broader mental health reform agenda through formal relationships with generic programs funded under the NMHS and NSPS
  • Enhance the profile of the issues nationally.
It also provides a range of services including policy development, advice and consultancy, management of special projects, resources and publications development, information and communication strategies and training.

Given the complex nature of multicultural mental health service delivery in Australia, to better enable CALD communities to access quality services MMHA had established a formal system of collaboration (i.e. the Consortium) that aims to:
  • Facilitate national implementation of activity based on identified priorities
  • Facilitate and market the imperatives of providing culturally competent services in a diverse range of settings
  • Support small states and territories to develop adequate levels of service delivery for CALD communities
  • Take into account the needs of CALD consumers, their families and communities and engage with key stakeholders to acknowledge and meet those needs
  • Be stable, flexible and able to transcend jurisdictional-specific boundaries to consider issues of national significance.

Management structure and responsibilities

MMHA is funded under the National Mental Health Strategy and National Suicide Prevention Strategy by DoHA. The 2007 MMHA Consortium Governance document sets out the service and financial management structure as follows:
  • DoHA determines the overall role and functions of MMHA via the Funding Agreement it puts into place during a given funding period.

  • SWAHS (a division of the NSW Department of Health and a body corporate) is the contract holder with responsibility to manage the MMHA program. As the contract Holder/Leading Agency, it bears all legal and financial liability and obligations to deliver the agreed program outcomes under the contract with DoHA. It also receives and reports to DoHA for all MMHA funds. SWAHS is further required to ensure that:
    • The MMHA Secretariat submits acceptable reports to DHA in line with contract obligations
    • The MMHAs finances are managed in a manner acceptable to the requirements of SWAHSs Internal Audit Department and so as not to expose SWAHS to risk
    • It provides a safe and healthy working environment for MMHA program staff
    • It undertakes recruitment and management of all MMHA staff in compliance with a range of NSW public sector regulations, policies and procedures.

  • MMHA operates as a national program (fully funded by the Commonwealth) and implemented by SWAHS. It is required to prepare and provide comprehensive program reports, and to have its finances audited annually. Continued funding is contingent upon meeting these requirements.
Top of pageThe review team considers that one aspect of these structural arrangements serves to undermine the effective operation and management of the Project. Specifically, this relates to:
  • Line of reporting – under the terms of the contract, MMHA is accountable to DOHA through SWAHS. The MMHA National Program Manager (NPM) reports to the SWAHS Multicultural Health Network Director on operational matters. For all strategic and policy matters the NPM reports to the SWAHS Executive Director (SWAHS's delegate responsible for the overall management of MMHA). We consider that having the line of reporting outside of MMHA does not promote accountability and transparency and impacts on its ability to be independent
In the course of the documentation review, we cited records of Consortium meetings in 2004 and 2005. In light of the above comments, it is of concern that these very issues had been discussed by the Consortium 4-5 years ago yet with no apparent successful resolution. Of note are the following extracts:
  • Members expressed concerns in October 2004 that MMHA needed to 'do some work in relation to governance...including cohesion of the management group, decision making communication, strategic decision making and program capacity'.

  • In April 2005 MMHA formally began the process of reviewing and re-designing its governance practices. This decision is recorded as being made in view of the upcoming evaluation of MMHA and the new funding round. Issues noted at that meeting included:
    • 'There is a lot of informality about decision-making; worked so far but won't work forever'
    • 'Contract is managed by SWAHS. This makes accountability difficult'. Options considered were an NGO community-based structure or Management Committee Structure
    • 'MMHA is a virtual organisation that sits within a structure that manages it'

  • in June 2005 the Consortium met for full day discussion on Governance issues including its membership/MMHA role, MMHA structure, decision making and sustainability. Records indicate that agreement was reached on, among other things a re-design of the management structure and an updated policies and procedure manual. At the time of preparing this report it was unclear whether these activities had been effectively undertaken or completed.
    • With respect to MMHA's role, we note that consortium members indicated they were comfortable with MMHA claiming peak body status with the roles of advocacy, representation, advice, education and information and community service.
    • With respect to decision making, it is noted that Consortium members indicated that it is 'not an ideal situation (among other things because of limited accountability) that MMHA currently employs a national coordinator who reports directly to a line manager [in the Diversity Health Institute of SWAHS] who has no connection to MMHA'.
Despite the advantages of the auspice arrangement (e.g. resource/infrastructure sharing), it is apparent that SWAHS's role as fund holder and its reporting responsibilities have contributed to a 'blurring of lines', and confusion around roles and responsibilities. Our consultations revealed that (particularly during 2006-08 funding period) SWAHS increasingly 'stepped outside its role' having to be reminded about its role as contract holder and not project management.

The current structural and functional status of MMHA is, in part, a legacy of its past. It is of some concern however that despite issues around the transparency and accountability of this auspice arrangement being raised (even internally) during the 2003-2005 Funding Agreement, MMHA was refunded in 2006-08 and again recently for a further three years with the same auspicing and management structure in place.

Administration

MMHA uses operational policies of the Sydney West Area Health Service (SWAHS) in its day-to-day operations with specific ones being developed and implemented to meet the unique needs of the MMHA program. SWAHS' operational policies and guidelines are those set by the New South Wales Department of Health, which is a body incorporated under the Health Services Act 1997 (NSW).

MMHA uses the SWAHS financial systems to record and report on its finances, and the SWAHS policies and procedures to guide and ensure accurate recording and reporting which is in line with the Australian Accounting Standards. MMHA uses an access database program that had been designed to incorporate MMHA's reporting requirements to DOHA by program area and by item type. As information is entered into this database on a daily basis, reports can be generated on income and expenditure at any given time. Ad hoc reports can also be generated as needed by MMHA.

SWAHS advised that common expenditure items (such as telephone, cleaning, security, salaries, superannuation) are programmed to be paid via the SWAHS systems, rather than through processing of invoices by the MMHA Administration Officer. There is however provision for these to be provided to MMHA on a monthly basis by SWAHS. This assists the National Program Manager in monitoring expenditure and income against funding allocations.
Top of page

4.3.4 Project planning

Strategic and business/operational plan

Sound planning processes incorporate the development of a strategic plan for a project/program/initiative. Key components include the mission statement or long term vision, objectives and expected outcomes; strategic directions that prioritise the work areas; and a work plan that outlines how the desired outcomes will be achieved. A strategic plan should be developed in consultation with key stakeholders, be supported by a detailed implementation plan, and be regularly reviewed in consultation with stakeholders.

A strategic plan enables 4 key questions to be answered:
  • Where are we now
  • Where do we want to be
  • How do we get there
  • How do we measure our progress.20
Given the increasing complexity of the multicultural mental health landscape we would expect that the operation and management of the project be underpinned by a strategic plan. This establishes a systematic process for identifying intended outcomes, how outcomes are to be achieved and how success will be measured. In fulfilling its intended role to provide national leadership in mental health and suicide prevention for Australians from CALD backgrounds the development of a strategic plan would provide a clear statement of the project aims, objectives and expected outcomes, and assist in strengthening stakeholder engagement and support. Moreover, it would serve as a framework for monitoring and evaluating the progress and effectiveness of the work program. Finally, it would promote collaboration between the various stakeholder groups so that the needs of the target groups are better met.

The MMHA Project does not have a current strategic plan, nor has a previous strategic plan been developed for the project since its implementation six years ago. The review team was advised that 'A strategic plan has not been developed for the term of the new funding period due to DOHA's request of a review of MMHA in early 2009 and finalised within the 2008-09 financial year, the aim of which was to identify priorities and scope of work for future consideration'. We consider that the development of a strategic plan is a key component of sound planning for the implementation of a program/project and its development should be independent of any external evaluation.

As part of good business practice, agencies generally develop annual business/operational plans. Business planning should take place within an integrated framework which cascades from strategic priorities to divisional priorities and activities. These goals are then distilled into individual performance and development plans. This allows every employee to see exactly how their individual work affects their team goals, their division's goals and their agency's goals. It also shows how working towards these goals helps achieve the agency's overall priorities.21

Copies of the current (2008-11) and previous (2006-08) Operational Plans for the Project were provided to the review team together with corresponding work/project plans. For each of the five priority areas in the current Funding Agreement, the plan sets out the "strategy", "activity", 'output and outcome" and "timeframe".

With the commencement of the new funding period, an operational plan was developed and submitted with the funding proposal to DoHA in July 2008. Work plans (some of which were sighted) have also been developed for each of the five priority areas. Individual work plans have also been developed by respective staff for: the Carers & Consumer's Priority Area, the Communications, Information & Promotions area, and for Policy & Community Capacity Building area. An action plan has also been developed by the national CALD Consumer Reference Group based on the DOHA-defined Carers & Consumer's Priority Area. The emphasis in each of these areas is predominantly on mental health issues with suicide prevention being less evident.

Based on our review of the 06-08 and 08-11 Operational plans, we conclude that both meet the expected good practice.

A draft copy of the MMHA Communication Consultation & Promotional Plan 2008-11 was also reviewed. This plan presents the communication consultation and promotion framework and strategies for the next three-year period. It identifies a total of 14 key strategies to be implemented over this period. We were advised that this is the strategic plan for this area of operations and that finalisation of this document has been deferred pending the outcomes of this review. The Plan includes a description of MMHA's services, products and publications, analysis of its customers and stakeholders, a description of its proposed stakeholder management processes, a description of its communication and promotional strategies and activities, and communication, information and promotion management via the development of a database. Attached to this document is an Action Plan which sets out for each of the five priority areas: the strategies, associated activities, outputs and outcomes, and timeframe.
Top of page

Needs identification

Review of relevant documentation confirms that since its establishment in 2003, the MMHA Consortium was the primary vehicle for MMHA planning processes (including needs identification and analysis). With expansion of the MMHA program in 2006 from one program area (communication and information) to five, MMHA has undertaken a needs identification and analysis process on a state-by-state basis. In addition, it has conducted needs identification at a national level through discussions with the Consortium (until late 2008 when it was disbanded), JOG, and other stakeholder groups like Federation of Ethnic Communities Councils of Australia (FECCA). To date four state-based consultation forums have been held - Tasmania (July 2007), South Australia (March 2008), Northern Territory (May 2008) and Western Australia (March 2009). All stakeholders were invited to participate in needs identification and priority setting and experts permitted to contribute and comment on multicultural mental health issues in their state. Reports on these forums document the program, what was discussed, recommendations, listing of participants and documentation of the recommendations raised by the smaller discussion groups. MMHA has then mapped out stakeholder needs into an action plan with the collaboration of the state mental health branches

With the establishment of the MMHA National CALD Consumer Reference Group, this Group serves as a forum to identify the specific needs of this target group. This reference group also advices on relevant and suitable action to be taken by MMHA to address the needs and they are then built into the Action Plan designed and determined by the reference group members with the assistance of the Consumer and Carers Project Officer.

Work prioritisation

The MMHA Consortium Governance (2007) document sets out the objectives of the Consortium, one of which is 'to set and monitor the MMHA program's Policy Priorities and Strategic directions...' The MMHA Consortium is identified as 'the forum through which Consortium members set and monitor program priorities for and during the funding period...'. Its role is defined as 'ensuring equity in priority setting, to foster collaboration between members and to ensure equitable representation of the needs of all members'. The review team was provided with documentation that confirms this process and that Consortium input was sought on draft priorities prior to submission to DoHA.

The priorities put forward to DOHA for the last funding round were generated from a number of sources. These included the Carers and Consumers Scoping Study, the MMHA CALD Consumer Reference Group, the Consortium, the Joint Officers Group, and the various consultations and forums that MMHA ran, from information received from the public and/or presented at conferences and forums and that documented in the literature.

A comparison of the priorities as set by DoHA were strongly informed by MMHAs funding proposal.

4.3.5 Project financial management

Information contained in the Funding Agreements covering the 2003-05, 2006-08 and 2009-11 respectively were reviewed. The financial allocation to the project based on these funding agreements is presented in table 4.1.

Since the inception of the MMHA project the Federal Government has committed over $6.9million dollars to the promotion of transcultural mental health agenda supporting the overarching National Mental Health Strategy. The allocation of funds is based on a 37.8% commitment to salary and wages costs and 62.2% on expenses. The expenses typically relate to:
  • Production of magazines
  • Conferences
  • Committee work (JOG, Consortium, Carers & Consumers Reference & Working Groups)
  • Workforce capacity building and development through the conduct of clinical symposiums, workshops, seminars
  • Community capacity building and development through individual projects, training, education, etc.
  • Website, clearing house, e-bulletins, media – development and maintenance activities
  • General goods and services.
During the 2006-08 funding period, there were 2 contract variations. One related to an additional $650K for MMHA to deliver two more projects (development of multilingual resources for CALD communities, for the print challenged community, and for World Mental Health Day). A second variation for $24,000 was to enable the CALD Consumer Reference Group members to attend the 2008 national Diversity in Health Conference in Sydney. Separate reports were provided by MMHA on these two projects.

The proportion of funds allocated on committee activity, general goods and services and other expenses have been aggregated and are summarised in table 4.2. Overall the general goods and services expense line represents 10.87% of the overall project budget which is considered within accepted benchmarks quoted in the literature,22 of between 7.5 and 15%. Based on the information contained in table 4.2 there has been a significant redistribution of the expense budget for the MMHA project with a decrease in the general goods and services budget and a corresponding increase in capacity building, website maintenance and other such activities.Top of page

Over the 2005/06 to 2007/08 period the allocation of funds to Committee activity – namely the support of the Consortium, Consortium sub-committees and interest groups was low in comparison to allocations made in previous and subsequent years. Given the size of the Consortium relative to the newly constituted JOG the disparity in allocations is incongruous.

The sizeable shifts in allocation of funds in these three categories of expenses raises questions as to whether consistent and valid comparisons can be made in terms of the projects expense expenditure between the Funding Agreements and more importantly whether consistent chart of accounts and methods of accounting were adhered to between the respective funding agreement periods.

Part of the Funding Agreement between the DoHA and SWAHS required the submission of fully audited financial statements at the end of each fiscal period. This was not adhered to, with the argument being made that funds were not made available within the project budget to engage an auditor. Further it was argued by the Director of MMHA in April 2004 that if the SWAHS were to comply with the conditions regarding external auditing, given the financial implications of engaging an external auditor there would be substantial impact on the capacity of the program, to meet its obligation both financially and within specified timeframes. Accordingly, the Department issued a variation to the contract in late April 2004 which stated that the Department would accept financial reports that have been audited by the Area Health Services Internal Audit Department. This practice is considered to be highly irregular and a deviation from normal practice executed by the Department on similar funded projects. The review team questions the financial impact the engagement of an external auditor would have on the project, specifically in light of the fact that it has not been able to expend its funds on a regular basis. Further the engagement of an external auditor should have no impact upon the projects capacity to deliver and perform as per the timelines stipulated in the Funding Agreement. The provision of financial reports and statements that have been audited by an external third party with no vested interest in the project, funding body or funding agency is critical in terms of maintaining public accountability of tax payers' funds. This impartiality is at risk of being compromised in the opinion of the review team, as the financial reports are prepared by a division of the fund holder.

The review team considers that this aspect of the structural arrangements relating to financial and service management contributes to the undermining of the potential perspectives held in terms of the effective operation and transparent management of the Project. Specifically, this relates to:
  • Internal auditing of finances - We consider that this arrangement does not reflect transparency and accountability of operations and management and impacts on the independence of MMHA (as a fully funded Commonwealth project).
A full set of audited financial statements were not made available to the review team. However based upon correspondence and financial statements for 2006/07 and 2007/08 the review team noted the trend to carry over funds which represent a significant proportion of the allocated budget (refer table 4.3).

The question arises as to whether the project a) requires the level of funding currently allocated given this trend to carry forward such significant funds and b) whether the project is equipped to expend the funds in an appropriate manner to meet the intended objectives of the project. The feedback from stakeholders presented in chapter 5 indicates that one of the barriers to the MMHA project's effectiveness is the limited budget that is currently available to address the needs of the transcultural sector. The figures contained in table 4.3 do not support this argument, and make it increasingly difficult to justify allocating further or increased public funds to a project and program area that currently cannot expedite its existing funds. It must be noted that the review team are not advocating the expenditure of funds for the sake of acquitting allocated resources, and note the responsible manner in which MMHA has informed DoHA of its inability to fully expend funds each year. What is of concern is whether the project is effectively achieving its objectives in a timely manner, whether with appropriate reconfiguration it could achieve more given the available resources the project has currently available to it.

Table 4.1: Financial allocations to MMHA 2003-11

Area of funding2003-052005-062006-072007-082008-092009-102010-11Total
Salaries & wages
$555,000
$161,214
$462,793
$462,793
$240,000
$360,000
$380,000
$2,621,800
% increase for salaries & wages
187.07%
0.00%
-48.14%
50.00%
5.56%
Expenses
$695,700
$117,243
$402,979
$392,979
$740,000
$974,000
$986,000
$4,308,900
% increase for expenses
243.71%
-2.48%
88.31%
31.62%
1.23%
Total allocation
$1,250,700
$278,457
$865,772
$855,772
$980,000
$1,334,000
$1,366,000
$6,930,700
% increase for total allocation
210.92%
-1.16%
14.52%
36.12%
2.40%
Top of page

Table 4.2: Expense budget distribution MMHA 2003-11

Area of funding2003-052005-062006-072007-082008-092009-10 2010-11Total
Committee activity
$60,000
$5,000
$5,000
$5,000
$60,000
$70,000
$80,000
$285,000
% of expense budget spent on committee activity
8.62%
4.26%
1.24%
1.27%
8.11%
7.19%
8.11%
6.61%
% of overall projects budget spent on committee activity
4.80%
1.80%
0.58%
0.58%
6.12%
5.25%
5.86%
4.11%
Goods and services
$150,000
$47,243
$202,979
$202,979
$45,000
$50,000
$55,000
$753,200
% of expense budget spent on goods and services
21.56%
40.29%
50.37%
51.65%
6.80%
5.13%
5.58%
17.48%
% of overall projects budget spent on goods and services
11.99%
16.97%
23.44%
23.72%
4.59%
3.75%
4.03%
10.87%
Other expense - capacity building, website, magazine etc.
$485,700
$65,000
$195,000
$185,000
$635,000
$854,000
$851,000
$3,270,700
% of expense budget spent on other expenses
69.81%
55.44%
48.39%
47.08%
85.81%
87.68%
86.31%
75.91%
% of overall projects budget spent on other expenses
38.83%
23.34%
22.52%
21.62%
64.80%
64.02%
62.30%
47.19%
Total expense budget
$695,700
$117,243
$402,979
$392,979
$740,000
$974,000
$986,000
$4,308,900
Overall budget
$1,250,700
$278,457
$865,772
$855,722
$980,000
$1,334,000
$1,366,000
$6,930,700

Table 4.3: Funds carried over MMHA 2006-07 to 2008-09

Fiscal period ending 2006-07Fiscal period ending 2007-08Fiscal period ending 2008-09
Funds requested to be carried over
$546,478.34
$683,110.81*
$644,744.92**

* Includes $281,834.50 from special grant of $650,000 made in 2007
** Includes $196,437.42 from special grants/funds

Under-expenditure

We note that a request was made to DoHA in January 2009 to carry over funds from the 2006-08 Funding Agreement totalling $644,744. This comprised approximately two-thirds of the total Project budget. MMHA has advised that this request was due to items not being paid for in the last phase of the 06-08 Funding Agreement. The primary reason for this request was related to very short delivery time frame (6 weeks) for an additional grant ($650K) from DoHA for the development of multilingual resources in 21 different languages. The demand of this project (together with some staff vacancies) impacted on its ability to meet the requirements of a number of other concurrent projects, resulting in an underexpenditure.

This inability to expend funds however is not restricted to a single fiscal period but is a re-occurring event and cannot be solely attributed to delays in transference of funds from DoHA accounts to SWAHS accounts. Based on annual reports provided to DoHA that indicate that the MMHA is meeting its obligations in terms of deliverables specified in the Funding Agreement the alternate question arises, namely given the inability to expend funds, whether MMHA can meet its specified obligations on a reduced budget. This is discussed further in chapter 5 where stakeholder feedback regarding funding implications of the project are presented.

In kind support

The Funding Agreements, variations and financial statements provide evidence of the funding allocated to the project by DoHA. In addition, MMHA/SWAHS provided the review team with details of its estimation of 'in-kind' support which has been made available to the project since 2003 through SWAHS. Broadly this support includes assistance/support with:
  • financial management, payment of accounts, invoices, assistance with audit processes, access to finance data base
  • payment to staff and contractors, payroll services, management of superannuation
  • HR support and advice for managers and staff, access to learning and development courses, orientation for staff, leadership development, OHS support and structures
  • access to motor vehicle pool
  • IT and communication support and infrastructure
  • the organisation of travel for staff, carers and consumers
  • line management to the National Program Manager (e.g. proportion of Service Director's time in providing input on management and supervision of program personnel and assistance with recruitment and selection of personnel for the program)
  • Sharing of networks and linkages within multicultural mental health fields (access to SWAHS multicultural personnel, health care interpreters and translator expertise)
  • Diversity in Health Conferences.
MMHA secretariat/SWAHS advised that the level of this support has remained similar since the establishment of MMHA (i.e. approximately 10-15% of the total grant exclusive of GST). The basis for this estimate could not be provided. Accordingly, for the period March 2003 to February 2006 (total grant of $1.25M), it is estimated that SWAHS in- kind contribution totalled between $125,070 and $187,605. For the period March 2006 to June 2008 (total grant $2.0M), the in-kind contribution is estimated to be between $200,000 and $300,000.
Top of page

Management process

Documentation review and stakeholder consultations revealed several issues in relation to the project management and communication processes between DoHA and MMHA/SWAHS. It is evident that funding has not always been allocated to SWAHS in a timely manner which has impacted on recruitment of staff, and ultimately on ability to meet funding requirements

'There are issues with the timely receipt of funds from DOHA for new funding period for the continuation of meeting outputs for the program. The agreement period has in the past commenced on July 1 of a financial year with funds not arriving to the project (ie SWAHS) until the end of the same calendar year. This impedes engagement of staff and commencement of project work'
The review team was also advised that on expiry of the 2006-08 Funding Agreement, there was a considerable delay (4 months) in confirmation of a new three-year Funding Agreement by DoHA. This impacted on its ability to recruit suitable staff in a timely manner (yet the Funding Agreement was backdated to commence 1 July 2008). At the time of our consultations (May 2009), almost one year into the funding period, only 50% of the required project staff had been employed. Such delays add to the existing challenge of finding suitably skilled staff from a limited workforce pool for whom only short term/temporary employment contracts can be offered.

Stakeholder consultations also revealed that project management and communications were not as effective as they could be at times. This was attributed to a relatively high turnover of DoHA staff across earlier funding periods. This resulted in some frustration as it created difficulties in building rapport and effective working relationships.

Sustainability of the project

We note that Schedule A of the current Funding Agreement states 'the Project will build upon the outcomes and learnings of the 2006-08 project'. It is this approach to funding allocation – a project with a limited life cycle and no certainty of continued funding, that undermines its sustainability.

The model underpinning the MMHA project requires building relationships, alliances and networks to support collaborative partnerships aimed at addressing issues of transcultural mental health. This needs adequate time if transcultural mental health and suicide prevention are to become embedded in the broader mental health reform agenda.

If MMHA is indeed 'the Australian Government funded peak body' (as stated in Schedule A of the Funding Agreement) whose purpose is to drive the multicultural mental health agenda, then a longer funding cycle is needed. This would allow MMHA to build upon its work, and assist in the recruitment and retention of suitably qualified personnel. More importantly, it would support much needed longerterm planning.

4.3.6 Project reporting and performance measures

Ongoing monitoring and reporting on performance is an important component of any program or project. It is the means by which a program is able to assess its success in meeting its stated objectives; drive performance improvement and provide accountability to the community for its activities.

A sound performance management system can be expected to involve:
  • development of performance indicators to measure the achievement of the programs objectives
  • reporting internally and externally on the programs performance against the established indicators and providing explanation for poor performance
  • reviewing performance and taking appropriate action to address poor performance.
In this section we present our findings on the adequacy of the Project's current reporting requirements and performance measures.

Reporting performance

We examined the current and previous Funding Agreements together with supporting funding proposals and external progress reports submitted to DoHA. A range of other documentation related to reporting on the Project's performance and achievements was also reviewed (e.g. presentations and reports to the Consortium and JOG, MMHA website, e-Bulletin and the Synergy Magazine).

MMHAs performance measures and its performance against these are reported to DoHA. The reporting requirements are clearly set out in the Funding Agreement between DoHA and MMHA. These include submission of interim progress reports (with unaudited financial statements) and final reports at the end of each financial year (with audited financial statements). Schedule A sets out the outputs, activities and performance measures for each of the work areas. Expenditure by Activities for each financial year are set out in Schedule B. The Reporting Schedule and Payment Schedule are provided in Schedules D and E respectively.

MMHA's performance is measured by DOHA upon the periodic delivery of its funded outputs. Payment is made on receipt and acceptance of required deliverables denoting satisfactory progress on the Project's performance. Up to the 30 June 2008, MMHA was required by DOHA to provide quarterly progress reports and it was only upon their acceptance that DOHA then released the next progress payment to SWAHS. For the current funding period (2008-2011) progress reports are required twice a year.

Documentation reviewed indicates that all progress payments were made to SWAHS for the MMHA program during the 2006-08 funding period, and two to date for the current period 2008-2011. All financial reporting requirements and obligations to DOHA were also met through the provision of regular financial reports and certified audited financial statements as required.

Aside from the formal reporting requirements to DoHA, some stakeholders believed that there would be merit in additional reporting on the Project's achievements. This would serve to enhance accountability and transparency and better inform stakeholders and the general community on the projects outcomes. One such example is the development of a communication tool or report card to stakeholders on the project's achievements. This could be published in MMHAs publication, Synergy. While the Funding Agreement sets out particular reporting requirements, we do not consider that this constrains MMHA from reporting on additional aspects of performance in order to better inform the community and stakeholders on the Project's achievement and outcomes.
Top of page

Performance measures and quality

Ongoing monitoring of performance is an important component of sound program management and operation. Performance measures should cover the effective and efficient delivery of Government policy and program objectives, as well as the internal management of the agency/program.

Key performance indicators (KPIs) can benefit an entity/organisation in a number of ways. These include making performance more transparent allowing assessment of whether program objectives have been met; helping clarify government objectives and responsibilities; informing the community about performance; encouraging ongoing performance improvement and encouraging efficient service delivery.23

Key characteristics of useful KPIs include that they are:
  • Relevant – have a logical and consistent relationship to the agency/program objectives and is linked to government desired outcomes

  • Appropriate –gives sufficient info to assess the extent to which the agency/program has achieved a target, goal outcome (e.g. trend over time, performance related to the performance of other similar agencies, performance relative to predetermined benchmarks); and

  • Fair – information must be capable of measurement, represent what it purports to indicate and be accurate and auditable.
Performance information is most useful when it provides a comprehensive and balanced coverage of a given program through a mix of quantitative and qualitative performance indicators which can be understood and are well defined.24, 25

We examined the key performance indicators as set out in the current and previous Funding Agreement order to assess their adequacy and usefulness.

As a suite of KPIs, we consider they are relevant, appropriate and fair. However, across the 5 priority work areas, they largely focus on activities or delivery of outputs. Examples include number of national programs MMHA have implemented, number of presentations to conferences, number of participants at national workshops; monthly use of MMHA website. They do not address outcomes and how MMHAs products and services contributed to better mental health among CALD communities. There are few indicators that assess the achievement of MMHA objectives (effectiveness) or efficiency of its operations.

Further the reporting and KPIs mandated through the Funding Agreement do not require MMHA to provide any evidence as to the quality of the products produced by the project. The KPIs and reporting focus on provision of evidence to support a process evaluation and little else.

It is noted that the current quantitative/process focus of the KPIs is acknowledged by SWAHS/MMHA, together with the importance of being able to measure outcomes...to establish that the project has had a positive impact on CALD communities and service providers.

'A lot of our performance measures are process measures/indicators We need to improve the knowledge base...5 years from now, where do we want to be re CALD mental health? What will be the evidence of change?'Top of page
We consider the current performance measures could be improved through the development of more qualitative KPIs to better assess program effectiveness, (in particular quality) so as to provide a more balanced mix of quantitative and qualitative measures.

Other KPIs that MMHA could consider related to effectiveness include: 'waiting times for services' such as workforce training (KPI: access); 'extent to which services meet need' (KPI: appropriateness); 'stakeholder satisfaction with products/services' (KPI: quality). With respect to Efficiency, it could consider 'average cost and time to provide products and service' (KPI: quality).

In relation to MMHA products and services, the review team was advised that MMHA monitors stakeholder requests for its products and services (e.g. reports, books, fact sheets). It does not however have a process in place to assess stakeholder satisfaction and views on the appropriateness, quality and usefulness of this material. MMHA acknowledges that this is a current gap in quality control. To address this MMHA is planning to develop an online survey as a means of monitoring stakeholder satisfaction with its products and services. It will be important that MMHA regularly reviews this feedback and uses it for continuous improvement purposes.

It is acknowledged that the paucity of baseline data on the mental health needs of individuals from CALD backgrounds impacts on its ability to develop more appropriate and relevant performance measures. Further, the difficulties in measuring outcomes is noted particularly when operating within a relatively short-term funding cycle.

During the conduct of the evaluation the review team were made aware of some concerns expressed by the public about the quality of some of the translated materials recently produced by MMHA. Whilst quality assurance processes exist within MMHA there is a query as to whether this has been effectively implemented or adhered if such instances can arise. It is the evaluation teams understanding that the Department of Health and Ageing has instigated a separate investigation to address this issue.

A general observation of the materials produced through the MMHA is that currently it has a strong bias to mental health resources and there is less emphasis currently on addressing the other element of the project's portfolio, namely suicide prevention. A more even balancing of proposed work priorities may be worthy of consideration by MMHA in its development of a new strategic plan.

Footnotes

17 Australian National Audit Office and Department of the Prime Minister and Cabinet (2006) Implementation of Programme and Policy Initiatives: Making Implementation Matter, Better Practice Guide. Commonwealth of Australia Canberra.
18 Australian National Audit Office (2003) Public Sector Governance, Volumes 1 & 2: Better Practice Guide. Commonwealth of Australia, Canberra.
19 Australian Public Service Commission (2007) Building Better Governance. Commonwealth of Australia, Canberra.
20 Hill CW and Jones GR (2004) Strategic Management. Houghton Mifflin. 6th edition. New York.
21 Australian Public Service Commission (2007) Building Better Governance. Commonwealth of Australia, Canberra.
22 Australian Research Council, Submission to the Higher Education Review, July 2002
23 Steering Committee for the Review of Government Service Provision (2008) Approach to Performance Measurement, Chapter 1 in Report on Government Services. Productivity Commission, Canberra.
24 Victorian Auditor General's Office (2002) Measuring the Effectiveness of Mental Health Service in Mental Health Services for People in Crisis, Performance Audit Report. Chapter 7. Auditor General Victoria, Melbourne.
25 Steering Committee for the Review of Government Service Provision (2008) Approach to Performance Measurement, Chapter 1 in Report on Government Services. Productivity Commission, Canberra.