In a nutshell, the Stern Review is a bold attempt to broaden and deepen the role of research impact in the next Research Excellence Framework (REF). It seeks to capture a far wider range of impacts than were submitted previously, including impacts on public understanding and culture, and on University teaching. It does this by broadening the definition of impact and relaxing some of the rules on the way impacts are linked to research, Units of Assessment and the number of staff submitted.
The long-anticipated Stern Review was published today. Lord Stern was commissioned by the Minister of Universities and Science in November 2015 to review the Research Excellence Framework (REF), which evaluated the excellence and impact of UK research in 2014. His recommendations will shape the next REF, which will take place in 2021. We now know that we will have until 2020 to make our submissions (the precise date is expected to be confirmed next summer). But how will the Stern Review influence the way impact is evaluated in the next REF?
Research impact was evaluated for the first time in 2014, and despite major concerns being raised about this component of the assessment when it was first proposed, feedback from across the sector during the consultation for the Stern Review was highly supportive of retaining impact in the next REF (albeit at similar levels of recognition to last time). However, a number of issues were raised. For example, linking impacts to specific, high quality research outputs meant that many impacts on industry, public engagement and policy advice were not captured.
Learning from our experience evaluating impact in the last REF, Lord Stern is recommending a number of changes to the way impact is assessed in the next REF. These are the key changes.
1. Impact needs to be defined more broadly
Research impact should be more broadly conceived to explicitly include impacts on public understanding and cultural life, and impacts on curricula/pedagogy. In May, I published an article in Times Higher raising concerns about the narrowing of our conception of impact to increasingly instrumental impacts that are most likely to score highly in REF.
Lord Stern’s recommendations go some way towards resolving this issue. However, as he points out, it was not HEFCE who constrained the definition of impact; it was the way that the academy interpreted HEFCE’s very broad definition in strategic and instrumentalist terms. Indeed, the indicative range of impacts for Panel D in REF2014 explicitly included benefits to 'cultural life' and 'public discourse'. It remains unclear whether or not this recommendation will encourage a broader range of impacts to be sought and reported, or if Universities will continue to play safe and submit a more narrow range in 2020. However, this recommendation should empower panels to reward those who do submit a broader range of impacts.
Perhaps one of the reasons why some institutions steered clear of impacts on public understanding and cultural life is that these sorts of impacts are notoriously difficult to capture and evidence, and the sector will need to build capacity in this area. That’s one of the reasons why Fast Track Impact has been working with the National Coordinating Centre for Public Engagement and Dialogue Matters to produce a Public Engagement Evaluation Toolkit for Queen Mary University London, which I hope will become heavily used across the sector.
2. Relaxing rules on the number of impact case studies that need to be submitted
According to the Stern Review, rules on the number of impact case studies that need to be submitted per member of staff should be relaxed, so Units of Assessment across an institution may have different ratios of impacts to staff. Stern recommends that each Unit of Assessment within an institution should submit at least one case study. Overall, it is likely that the average number of case studies per member of staff at an institution will be lower than the last REF, assuming more staff are submitted and the recommendation is accepted that the total number of case studies submitted should not increase significantly.
3. Expanding the range of research that impacts can arise from
Although impacts will still need to be based on research of “demonstrable quality”, it is proposed that rather than only focusing on specific research outputs, impacts may also be based on research activity or a broader body of work, encompassing “the research expertise, facilities and networks of an individual, group or institution that underpin or lead to the eventual impact of research”
4. New institutional level, interdisciplinary impact case studies
In the next REF, it should be possible to submit joint impact case studies that are shared between multiple Units of Assessment within an institution for interdisciplinary research impacts. It should still be possible for more than one institution to submit case studies about the same impact where the research was conducted across multiple institutions. A new assessment panel will be set up to evaluate these “institutional level impact case studies”.
Finally, it is worth noting that research impact will still be worth 20%, however this will all now be based on impact case studies, as the “impact statement” from REF2014 is now to be included in the assessment of the research environment. This in theory means that each case study will be worth proportionally more than in the last REF, however in practice the value of an individual case study will be influenced by the number submitted by each institution, which may now vary.
Reaction from the sector
Reaction across the sector today has inevitably been mixed. However, most negative comment has focussed on proposals to submit all researchers to REF (which it is argued may lead to mass conversion of less productive researchers to teaching-only contracts), and the inability of researchers to take their research outputs with them when they move jobs (which some have suggested may incentivise University to end contracts with high cost staff or shut down entire departments and then claim their outputs for REF).
However, the general tone of the reaction to proposals on impact has been positive so far. Prof Phil Purnell, from University of Leeds described changes to the link between impacts and research outputs as “sensibleness”, especially for interdisciplinary work. Grace Williams, a Public Engagement Officer from University of Exeter, is “really delighted” to see the explicit focus on public engagement, and Dominic Dean, PhD researcher at University of Warwick, said it was “great to see no narrowing of eligibility for impact, but rather reinforcement in the other direction”. Jeremy Barraud, Deputy Director for Research Management and Administration at University of the Arts London, described references to cultural impacts as "re-assuring", while Clare Taylor from the Society for Applied Microbiology described the proposals as "pleasing".
In my Times Higher Education article, I warned that the “impact revolution is faltering”. The Stern Review has given new impetuous to this revolution in the way we do research.
Prof Mark Reed is a recognized international expert in impact research with >100 publications that have been cited >7000 times. He has won awards for the impact of his work as a Professor at Newcastle University and Research Manager for an international charity. His work has been funded by ESRC, NERC, AHRC and BBSRC, and he regularly collaborates and publishes with scholars ranging from the arts and humanities to physical sciences.
He has been commissioned to write reports and talk to international policy conferences by the United Nations and is a science advisor to the BBC. His interdisciplinary approach to impact has been featured by Research Councils UK as an example of good practice and he has trained their staff and been invited by them to provide internal seminars on impact.
Mark provides training and advice to Universities, research funders, NGOs and policy-makers internationally, and regularly works with business. Find out more about his work at: www.profmarkreed.com or follow him on Twitter @profmarkreed
Find out more
Read reviews and get your copy of the Research Impact Handbook by Mark Reed
Free research impact training for researchers
Get Fast Track Impact training in your institution
Sign up for our newsletter