Many of us thought it was a joke when Jo Johnson announced his intention to create a KEF to join the REF and TEF, but the smile has long since been wiped from the face of the sector, and in today's consultation we see for the first time what the KEF might actually look like. The good news is that there is minimal additional data collection needed to make the submission, as most will be drawn from existing sources. What is most interesting for me, is the proposed overlap between the KEF and impact in REF. In this blog, I will give you a rapid overview of everything you need to know about impact in the KEF proposals, and pose some questions about what the KEF is really for.
Is the KEF a renewed attempt to instrumentalise the impact of research as return on investment?
There are two things that jump out immediately at me from the proposals: a bias towards economic impact; and overlap with what is currently being measured in the impact section of REF. The focus on economic impacts has clear political roots if you look back to Jo Johnson's original speech in which he announced the need for a KEF. In his 2017 speech, Johnson pointed out that we are falling behind other countries in generating economic returns on public investment in research:
"Comparisons of our commercialisation activity with that of the US are revealing. We require about £5m more research spending to generate each new spin-off than the US does. And US higher education institutions earn almost 40% more IP licence income as a percentage of research resources than those in the UK."
If this sounds familiar then you may be remembering some of the original rhetoric around the introduction of impact to REF more than a decade ago. As Jenn Chubb's analysis of the origins of impact in the UK show (I co-authored the paper so get in touch if you don't have institutional access and I can email you a copy), there was a strong emphasis on economic impacts throughout the early years, including the Lambert review of business and community collaboration in 2003, the Warry Report (2006) that challenged the Research Councils to do more in terms of "achieving and demonstrating a step change in the economic impact of RCUK investments", the Leitch Review of skills (2006) which focused on "entrepreneurship and innovation for the knowledge economy", and the Government's Science and Innovation Investment Framework 2004-2014 which made the economic case for investment in research on the basis of benefits to the UK economy and enhancing open innovation.
The piloting and consultation process ultimately led to an assessment of research impact in REF that was far broader in scope than its initial framing. This is because the academy did not believe it was appropriate to narrow the value of research to instrumental indicators such as monetary value to the UK economy, given the breadth, depth and international nature of the work we do. Is the KEF a renewed attempt to do what the research policy community originally wanted REF to do? If so, it is likely that the push-back will only exacerbate the second issue with the KEF: its overlap with impact in REF.
Is KEF actually about knowledge exchange and impact, or an attempt to incentivise improved economic performance?
Evidence that this may be the subtext for KEF comes not only from the way in which it was launched by Jo Johnson, but in the consultation document itself, which identifies seven perspectives of knowledge exchange to be assessed via 18 indicators. Eleven out of the 18 indicators proposed are economic indicators. It is proposed that two of the perspectives will include additional narrative or contextual information (local growth and regeneration public engagement) and that these narratives will be fact-checked but not assessed.
Two out of the seven perspectives focus on the pathways we take to impact (research partnerships and public engagement), though in reality there are many more pathways that could in theory be evaluated. The remaining five perspectives are types of impact, and four of these five focus on economic impacts. Two are solely about economic impacts (working with business and IP and commercialisation) and two more are about economics and other things (skills, enterprise and entrepreneurship and local growth and regeneration).
It is not clear what the theoretical or empirical basis might be for this highly selective approach, especially in terms of the types of impact that are considered. If the rationale of the KEF is (as the consultation states) to "provide HEIs with a useful source of information and data on their knowledge exchange (KE) activities, for the purposes of understanding, benchmarking and improving their own performance", then why focus so narrowly on KE activities that lead to economic impacts? If the purpose of KEF (as the consultation states) is to provide not just businesses but "other users (and potential users) of HEI knowledge with another source of information, which may increase visibility of potential university partners and their strengths, and contribute to their internal decision making processes" then why focus primarily on economic impacts when it is highly likely that these other users will have far more diverse interests? Is the KEF about actually about helping us do better knowledge exchange to achieve benefits for whoever needs them, whether they are well resourced businesses or the poor, marginalised and voiceless in society, or is it actually about enhancing return on investment in UK research?
If the KEF is about the full range of impacts that can arise from KE rather than just an assessment of the extent to which KE yields economic returns, then in addition to benefits for public policy and the third sector, we might seek to understand the extent to which research actually generates environmental benefits, rather than making assumptions of benefit based on the amount of consultancy income a University gets from environmental charities, as proposed in the KEF consultation. We might seek to evaluate the health and wellbeing benefits of new drugs (rather than measuring the economic value of licensing the IP to pharmaceutical companies, as proposed in the KEF) and the numbers of lives saved by new health policies (rather than measuring the economic value of Government contracts to Universities, as proposed in KEF). Policies and initiatives that went badly wrong and caused significant damage to the economy and killed people would come up smelling of roses under the proposed metrics, as long as they earned the University money. In this new framing of the value of research, how do we value initiatives that enrich or change cultures, behaviours and norms, helping us transition to a more sustainable society?
How might the KEF actually drive impact?
Supporters of KEF would argue that these more esoteric impacts are what impact in REF was designed to capture. I would argue that impact in REF was also designed to capture economic impacts, but in a more even-handed way that the academy would not allow to trump other types of impact. Given that the (primarily) economic indicators proposed in KEF are already collected and reported annually, the key value of the KEF lies in its integration of economic metrics to assess the relative performance of Universities in terms of their contribution to the UK economy. The linked purpose appears to be clear: to create an incentive system that will drive a stronger contribution to the economy from the HEI sector. If this bias and purpose were explicit and justified, then there would be no overlap with impact in REF, and we could evaluate the exercise for what it is.
Currently, the KEF is a metricised, partial assessment of one small slice of what impact in REF attempts to evaluate. For this narrow set of economic impacts it will do something different to REF by providing a comparative assessment of all data against each of the selected indicators for all Universities. As an assessment of economic performance therefore, it overcomes the biases of the REF process in which a low performing institution in economic terms (based on the indicators proposed in KEF) can select their highest performing economic case studies to showcase and create a quite different impression for prospective business stakeholders. As a business, looking for expertise, the KEF will provide a benchmarked evidence-base against which to evaluate potential research partners.
Another key difference between KEF and REF is that the impacts don't have to be linked to excellent underpinning research in KEF, as they are in REF. This means that certain types of impact are likely to be captured that have not been captured before in REF. Responding to an earlier version of this article on Twitter, Jenny Ames gave Continuing Professional Development (CPD) courses as an example. Many of these generate significant income but are based on research by other Universities and so cannot be claimed in REF. However, there are biases again in which missing impacts from REF will be included in KEF, favouring those with economic impacts (like the CPD example) unless they happen to fit into the two categories that allow narrative descriptions of impact. However, for impacts in the latter category, narratives are not assessed and so will not contribute towards any future distribution of funding linked to KEF.
For economic impacts then, given the availability of existing data, there is a clear benefit of introducing a KEF for a Government that wants to justify further investment in research, and enhance the contribution that the HEI sector makes to economic growth. But let's not pretend it is a Knowledge Exchange Framework in the holistic sense in which it has been framed. It is an innovation and investment framework designed to drive innovation, investment and economic growth. Framed as such, we can debate the politics of the need and role for such a framework, and this debate is what I hope the KEF consultation will inspire.