Performance Indicators, Accountability and Academic Freedom
Last century, in the eighties, and in a climate of public sector accountability driven by Thatcherism the British Government imposed a series of performance indicators upon the UK university sector. The goal was improved efficiency and quality, but the clear message was that the universities had to held accountable to the government, as representatives of the public and primary funders of the universities.
Similarly, we have the Australian Government clearly conveying its position that the Australian Universities must be "accountable" for the money they recieve from the public purse.
And now, finally, we have the New Zealand Government version (emphasis added):
"The tertiary education reforms are all about improving the sector's ability to deliver on government priorities, and meet the education and training needs of stakeholders such as learners, employers and communities on a regional as well as national basis."
The second half of that statement is sound and hard to argue with, but where did the first one come from?
Its worth pausing and looking at the first substantive section of the Education Act 1989 that governs universities (emphasis added):
161 Academic freedom
- It is declared to be the intention of Parliament in enacting the provisions of this Act relating to institutions that academic freedom and the autonomy of institutions are to be preserved and enhanced.
- For the purposes of this section, academic freedom, in relation to an institution, meansÑ
- The freedom of academic staff and students, within the law, to question and test received wisdom, to put forward new ideas and to state controversial or unpopular opinions:
- The freedom of academic staff and students to engage in research:
- The freedom of the institution and its staff to regulate the subject-matter of courses taught at the institution:
- The freedom of the institution and its staff to teach and assess students in the manner they consider best promotes learning:
- The freedom of the institution through its chief executive to appoint its own staff.
- In exercising their academic freedom and autonomy, institutions shall act in a manner that is consistent withÑ
- The need for the maintenance by institutions of the highest ethical standards and the need to permit public scrutiny to ensure the maintenance of those standards; and
- The need for accountability by institutions and the proper use by institutions of resources allocated to them.
The problem appears to be section 3b. Its tucked away there at the bottom, much like the important parts of PowerPoint slides. The Government appears to think that this section overwhelms everything listed above. Now I'm not a lawyer, but I have heard laywers say many times that you have to read law in context - all of the words matter, not just the ones that sound good in election speeches.
At this point its also worth emphasising that I don't think the universities have a carte blanche to do as they please and I'm already on record as saying that we could do much better. I just don't think that the degree of control the Government is seeking over the sector is necessarily going to work any better than what we currently have.
For evidence of where I think the cracks are already showing, its worth looking at the TEC "Investment Guidance" document, recently published as part of the current Tertiary Reform process. For any institution in the sector this is a critical document. In part, it defines the means by which the Government will agree to fund institutions and to what extent. Its by no means complete, and notes that
"Further work is required to complete the changes set out in this Investment Guidance. A new quality assurance process is under development and will include the use of organisational selfassessment, and external evaluation and review tools. These tools are being developed by the New Zealand Qualifications Authority, using an Expert Advisory Group and will be trialled in 2008 with the aim of introducing this process in 2009." (para 14)
Despite its incomplete nature, the Government is requiring that the institutions funded by TEC on its behalf provide an "investment plan" as per the specifications of this document. Failing to achieve the plan's agreed outcomes will destroy the "high trust" that the sector apparently enjoys from TEC and will result in an "increased frequency" of TEC engagement - whatever that actually means.
As part of these plans, a variety of "key shifts" are identified - basically things the Government wants institutions to do. These appear to be non-negotiable. How an institution proves it is doing what its told is measured by "key performance indicators" and that's the bit that really has me twitching.
"Performance indicators differ from simple indicators in that they imply a point of reference, for example a standard, an objective, an assessment, or a comparator, and are therefore relative rather than absolute in character. Although a simple indicator is the more neutral of the two, it may become a performance indicator if a value judgement is involved." Cuenin, 1986 quoted in Cave et al. (2nd Ed., 1991) p21
Value judgements are tricky things. Its clear by their actions that the Government values accountability over and above the other elements of academic freedom listed in the Education Act, whereas most academics strongly value the others. Wisely however, TEC aren't defining most of the performance indicators they want measured, they've initiated an "experts group" from the sector to work on the definition.
The problem is that keen eyed readers will note that I said "most". Very specific indictors are identified regarding participation and attainment by some sectors (page 10):
"Increase the proportion of EFTS for students aged under 25 enrolled in qualifications at degree level and above." "Increase completion rates of students aged under 25 enrolled in qualifications at degree level and above. "
The goal is unarguable (especially if you want funding) "Increasing educational success for young New Zealanders" - everybody wants this. The question is, will the KPIs listed above really result in the achievement of this goal? There's an easy way to increase completion - pass everyone. What we actually want are people well enough educated that they have the skills and knowledge that qualifications represent - a subtle but important difference and one that completion rates do not measure.
The other KPI is even worse. Assuming that we can't keep educating an ever greater proportion of society due to capped resources (a pretty safe assumption) and that the population of New Zealand is aging overall (also well established), the only way that we can educate proportionally more younger students is by educating less older students.
Its hard to get KPIs right and starting this process with flawed measures is not going to help. The major problem is that its very hard to define and measure the quality of an individual student's educational experience. Over fifteen years ago the British started realising this when they started taking a hard look at their KPIs:
"Uncertainty about output quality can lead to conflicting interpretations of a high cost per student. By one interpretation high unit cost (say a high staff-student ratio) may be taken as an indicator of a high quality educational process. If there were a direct relationship between the amount of time allocated to students and the quality of the degree obtained, then higher staff-student ratios would be associated with higher quality output. However, we have no information on th equality of degrees and hence we are unable to investigate the relationship between teachign time and student quality. In this sense, the existence of a high staff-student ratio is not sufficient to indicate high quality." (Cave, 1991, pp92)
Another proxy measure for quality proposed in the TEC document is "engagement with students" a nicely vague concept that appears to be a synonym for "customer satisfaction"
"Many individuals and organisations have a stake in the tertiary sector. The most important stakeholder group is students. The main reason for government investment in tertiary education is to meet the needs of students, and this should be reflected throughout a TEOÕs operation." (para 22)
Exactly how universities are to demonstrate that they are meeting student needs is unclear but it is clear that TEC think we don't do it:
"A related issue is the feedback that the TEC has received from stakeholders (including the feedback received in consultation on this guidance). Several stakeholders (particularly students and industry) expressed concern about the willingness of TEOs to engage. Furthermore, some stakeholders commented that, whilst TEOs may understand their needs, the TEOs do not make changes to address those needs. Whilst it is acknowledged that a TEO cannot address the needs of all stakeholders, it is expected that TEOs will have ongoing, effective relationships with stakeholders in which needs are investigated and meaningfully discussed." (para 27)
All in all, it appears that the intent of the Government in pursuing this process is to make institutions customer friendly agents of Government policy - so much for academic freedom and institutional autonomy. I look forward to the process unfolding and I really hope that we do get meaningful KPIs but I also hope that we don't repeat history:
"Any questioning of ... performance indicators is positioned as a resistance to public accountability, a refusal to be questioned according to the logic of contemporary capitalism, which requires 'clear measures to establish university performance' (A Measure of Excellence, Macleans' magazine 15/11/1993 p48 quoted in Readings, 1996)"
While working through the TEC's TEC "Investment Guidance" document as part of the other post on my blog, I was struck by this passage:
"In addition to TEO Component policy work, funding policy decisions are yet to be taken on managing over-delivery, fund rationalisation into the SAC and TEO Component3, SAC subsidy rates for taught postgraduate, distance and e-learning..." (para 16)
It appears we may be in for yet another discussion about what form of delivery costs more - oh goody. Why do I not think that this means that the Government finally understands the risk premium associated with e-learning and is going to provide additional funding to assist institutions building capability?
I feel a Tui moment coming on:
