The role and shape of impact in the next REF just got a lot clearer with the publication of HEFCE’s consultation on the second Research Excellence Framework last week.
There were no big surprises, with the focus on how not if to implement Stern’s recent recommendations. However there are a few significant revelations we can glean from the consultation:
Institution-level case studies could play a major role in the next REF, accounting for 10-20% or up to 25% of impact scores in two different proposals being consulted upon. However, as I explain below, this proposal has the potential to achieve the opposite of Stern’s intention to better capture interdisciplinary and collaborative impacts;
Larger units may only be allowed to submit 1 case study for every 20 staff they submit, based on the proposed definition of “research active” staff and HESA data that show there were approximately 130,000 eligible staff employed across the sector in 2014. It appears that HEFCE are not minded to accept Stern's recommendation to "relax the tight coupling between the number of staff submitted to a Unit of Assessment and the number of case studies required". Rather, a fixed ratio is being consulted on, based on the number of research active staff, with flexibility being granted for smaller submissions (which would only have to submit 1 case study, thereby revealing their scores). As a result, some less research intensive Universities (that were more selective in the staff they submitted to REF2014) could have to find twice the number of case studies they needed in 2014 if they want to make a submission in 2021. For example, a unit with 80 academic staff that only submitted their 10 best researchers could have done so with two impact case studies in REF2014 but may need to find four case studies to be able to make a submission to REF2021. This may incentivize the submission of low grade and in some cases “unclassifiable” case studies that are not based on credible research in order to enable submissions to be made;
Limits may be placed on the proportion of case studies that can be resubmitted from REF2014, and only cases where additional impacts have occurred may be eligible. The news that case studies from REF2014 are definitely going to be eligible for resubmission in REF2021 will be particularly welcome for those whose case studies received low grades because they were still in progress. A significant proportion of case studies fall into this category. One significant group is policy impacts that had often passed into law but without evidence of the law yet being implemented or achieving results on the ground.
Why institutional-level case studies may be backfiring on Stern
HEFCE’s consultation suggests that Stern’s recommendation to include institution-level interdisciplinary case studies is likely to be adopted. HEFCE are consulting on whether institutions should have to submit between 10-20% of their case studies at institutional level. Later in the consultation document, they suggest that institutional impact case studies could be worth as much as 25% of the overall impact score (based on a split of 5% and 15% for institutional impacts versus those based on submissions to other Units of Assessment, as part of the overall quality profile).
Given the high proportion of impact case studies that could be described as interdisciplinary, institutions will have considerable latitude in their allocation of case studies to this panel. However, few of the institutions I have spoken to are focusing on case studies that fell between panels due to their interdisciplinary nature, which may have been under-scored in REF2014, as Stern intended. Instead, most appear to be viewing the institutional-level case studies as a “show-case panel” that will produce a ranking of Universities rather than subject areas, to which they are planning to submit their most iconic impacts. Significance and (in particular) reach were interpreted differently across various Units of Assessment to reflect the different nature of impacts in their subject areas. In some subject areas, the majority of top-graded impact case studies were instrumental in nature and global in reach, while others rewarded a much wider range of impacts and hardly saw any case studies with global reach. As a result, interdisciplinary case studies that were downgraded because they fell between subject areas in REF2014, may be judged even more harshly in an institutional-level panel that is dominated by iconic, global impacts.
What else does the consultation tell us about the role of impact in REF2021?
There are a few other insights that are useful to note:
REF2021 will adopt the same definition of impact as REF2014, keeping the concepts of significance and reach at its heart. It is proposed that this definition is used both by those who fund and assess research. The differences between the REF and RCUK definitions are subtle, but the REF definition is more comprehensive, broadening “creative output” to encompass cultural impacts more widely, and including environmental impacts.
The types of impact to be included in REF2021 are to be broadened, in line with Stern’s recommendations. However, as I blogged recently, this is unlikely to be heeded by most of the sector, other than UOAs in Panel D where most public engagement impacts were reported last time. It is being proposed that more guidance is given to assessment panels in the evaluation of impacts arising from public engagement. The National Coordinating Centre on Public Engagement is about to publish a report on the role of public engagement in REF2014, which may help provide useful guidance.
Stern’s recommendation for 20% of the overall score to be based purely on impact case studies is likely to be accepted and the environment template will now explicitly ask about how the unit “enabled” impact in terms of strategy and capacity building.
It is likely that impacts will need to be based on research that was published after 1 January 2000.
The research that led to the impacts should be of “demonstrable quality” but impacts will not have to be tied to a specific research outputs. Rather, it is likely that Stern’s recommendation will be adopted, so that impacts may instead be linked to “a research activity and a body or work”, with greater weight being given to the rigour than the originality of the research output, activity or body of work upon which the impact is based.
Impacts that take place after 31 July 2020 will not be eligible for submission to REF2021 under the timetable currently being consulted upon.
The role and shape of impact in REF2021 is growing clearer. It will be just as important as it was in 2014, but it will attempt to assess impacts more accurately and fairly this time. The extent to which this actually happens depends on how we now respond to the consultation. Have your say here.