In 2014, the UK became the first country to comprehensively assess the impact of its research as part of a national assessment. Although scores were not made public for individual cases, we looked for institutions whose case studies were all given grades in the same range to identify 175 (out of 7,000) high- or low-scoring case studies from a cross-section of disciplines. We combined qualitative thematic analysis with quantitative linguistic analysis to explore what made a high-scoring submission.
In a nutshell, high-scoring case studies clearly articulated evidence of significance and far-reaching benefits that could be clearly be attributed to research conducted at submitting institutions.
However, new rules are being introduced for the next REF and best practice is changing rapidly in some areas. This could mean sticking to best practice from REF 2014 may not be enough to reach top scores in 2021, so the lessons from this research should be seen as the minimum required to write a top-scoring case study for REF2021.
Part 1: Three key lessons from high-scoring case studies in REF2014
Methods
Scores for individual case studies in REF2014 were not made public, but based on the distribution of scores received by some institutions (e.g. where all were given the same score based on publically available data from HEFCE), we identified 175 of the highest and lowest scoring case studies and analysed them using a combination of qualitative thematic analysis and quantitative linguistic analysis. To achieve balance across all Main Panels, the sample included 85% of all known 4* case studies, supplemented by additional cases from Main Panel B where at least 85% of case studies for a given institution scored 4* and the remaining cases scored 3* (n = 85). These were compared with a sample of case studies known to have scored either 1* or 2* (n = 90). Initial coding was conducted by 6 coders with intercoder reliability (based on 5% sample) assessed at >90%, with subsequent thematic analysis conducted by two of the coders.
The sample for quantitative linguistic analysis includes 124 high-scoring and 93 low-scoring ICS. This includes all identifiable high-scoring ICS in any UoA and all identifiable low-scoring ICS in those UoAs where high-scoring ICS could be identified. There is a 75% overlap with the sample for qualitative analysis. In order to explore the lexical profile of ICS, lexical bundles of 2-4 words were extracted with AntConc from each corpus section (High-Overall, Low-Overall, High-MP A / C / D, Low-MP A / C / D) and then compared for significant over- and under-use across the corresponding corpus parts (High-Overall vs Low-Overall, High-MP A vs Low-MP A etc.; not enough data for MP B: only 6 high-scoring and 2 low-scoring case studies). The Keyword statistics used were Log Likelihood for significance combined with minimum expected values and Log Ratio for effect size . Each lexical bundle that met the “keyness” (i.e. specificity) threshold was assigned a code according to its predominant function in the text based on an inspection of concordance lines showing the bundle in context as it appears in the texts. This resulted in a list of typical bundles and functions that appear significantly more often in high- or low-scoring case studies, from which the above conclusions and examples are drawn. In order to quantify the readability of the texts, they were analysed using the Coh-Metrix online tool. This tool gives 106 descriptive indices of language features, including 8 principal component scores made up of combinations of the other indices. Relevant indices have been selected and compared across corpus sections using t-tests. As with the analysis of lexical bundles, comparisons were made between high-scoring ICS and low-scoring ICS in each of Main Panels A, C and D, as well as between all high- and all low-scoring ICS.
1. Articulate how specific groups have benefited and provide evidence of significance and reach
84% of high-scoring cases articulated significant and far-reaching benefits, compared to 32% of low-scoring cases, which typically focused on pathway.
Findings from the quantitative linguistic analysis show how high-scoring impact case studies contained more phrases that specified reach (e.g. “in England and”, “in the US”), compared to low-scoring case studies that used the more generic term “international”, leaving the reader in doubt about the actual reach. They also include more phrases implicitly indicating the significance of the impact (e.g. “the government’s” or “to the House of Commons”), compared to low-scoring cases that emphasized pathways (e.g. “the event”, “has been disseminated”)
Phrases containing variations on the word “dissemination” were more common in low-scoring case studies, which supports the claim that these case studies focused more on pathways than impacts and may indicate a perception among these case study authors of impact as one-way knowledge transfer
2. Establish links between research (cause) and impact (effect) convincingly
Only 50% of low-scoring case studies clearly linked the underpinning research to claimed impacts (compared to 97% of high-scoring cases).
Findings from the quantitative linguistic analysis show that high-scoring case studies were significantly more likely to include phrases that attributed impact to research like “cited in (policy documents)”, “used to (inform)” and “resulted in”. Those phrases indicate a focus on the effect, that is, on what the activity led to. In contrast, low-scoring case studies tended to link backwards, foregrounding the research activity (e.g. “(findings) of the research”). Moreover, they were more likely to include ambiguous or uncertain phrases like “a number of” (implying that it is not known how many) and “an impact on” (implying that the nature of the impact is not known)
High-scoring case studies made causal relationships more explicit (above average when compared to general English texts, where low-scoring case studies are below average), which makes a text easier to process
3. Make your narrative easy to understand
High-scoring impact case studies scored more highly on a readability measure correlated with reading speed because they included shorter, less complex sentences. The Flesch Reading Ease score, out of 100, was 30.0 on average for 4* and 27.5 on average for 1*/2*. While this is a significant difference (p<0.01), the scores indicate that ICS are generally of “college-graduate” difficulty. As such, these technical documents do not need to be “dumbed down” to the readability of a newspaper article (as some have suggested) but maintained at interested and educated non-specialist level.
Low-scoring cases were more likely to include academic phrasing with unnecessary phrases such as: in relation, in terms of, the way(s) in which
These findings are based on a preliminary analysis of data presented at the 2019 KMb conference, held in Newcastle on 21–23 March 2019.
Part 2: Writing your summary, underpinning research and corroborating sources
The following two parts of this guide are based on experience reviewing and advising on case studies from a wide range of disciplines during the current REF period.
Make sure you are submitting the impact and not the pathway
To ensure you are submitting the actual impact, and not just the pathway to impact, keep asking “what was the benefit and why was this important?” and describe the benefits more than the process through which those benefits were derived. If you don’t know why it was important, ask the beneficiaries to tell you what was meaningful or valuable to them
Dissemination is not impact: even if you have impressive numbers of reads, downloads, views or listens, how do you know if anyone learned anything from it, benefited, or did anything different as a result? Keep asking “what happened next” until you find the benefit. To do this, design your communications so you can legitimately follow up longitudinally with audiences to re-engage, deepen interest and learning, and ask them how they benefited
Developing resources for schools and doing work in schools is a pathway to impact, not an impact in its own right. Identify specific changes you would like to see (e.g. increased attainment in a specific subject, reducing an attainment gap between boys and girls or ethnic groups for a particular subject or influencing choice of subject at University) and use your materials and work in schools as interventions designed to achieve these impacts. Follow-up to find out if your interventions worked or not
Make sure the majority of the words in your summary are about the impact, rather than the context and pathway to impact
Spend time making sure your summary resonates strongly with readers, communicating your impact straight-forwardly and persuasively
Get multiple reviews of your summary and keep polishing till you can improve it no further
Target your case study to the appropriate UoA based on the publications in the “underpinning research” section
Case studies submitted to UoAs by researchers from outside the discipline because they think assessment criteria will be easier may find a less sympathetic or understanding audience than they expected
Make sure the underpinning research reaches the 2* quality threshold (if possible based on the judgment of two independent assessors)
Being published in a peer-reviewed journal or having a book deal with a prestigious academic publisher doesn’t not guarantee 2* status if it is weak work
Outputs that are not peer-reviewed or academic books, such as final reports to funders, can reach the 2* threshold if they are strong work and publicly available
Explain why your body of research (the listed outputs underpinning your impact) is at least 2* quality, drawing on any indicators of esteem that you think will play well with others in your discipline (e.g. highlighting prestigious funding sources but avoiding citation data or journal impact factors as these aren't allowed) and creating a narrative justification explaining why the work is academically significant, original and rigorous in the context of your discipline
Describe the key findings from the underpinning research that pertain to your impact
Only include essential contextual material, and avoid unnecessary detail on methods or other findings that were not integral to the impact
Number your list of underpinning research outputs and cite each output by number in your description of the underpinning research
Ensure your corroborating evidence is robust and credible, for example:
Work with credible stakeholders to conduct independent evaluations of your impact e.g. by stimulating a policy review or making the case to a project partner that their stakeholders might be interested to know the impact of the work you did with them (where resources prevent this, consider providing your stakeholders with funding for an independent consultant to evaluate the impact on your behalf, acknowledging the source of funding in the report that is published on the organisation’s website). Offer help designing the evaluation to ensure it is robust
If you can’t find an organization to do this for or with you, do it yourself and publish the findings in a well-respected peer-reviewed journal. There are a number of ways you can publish corroborating evidence in the peer-reviewed literature. The evaluation will not be not independent, but if well designed and written, few reviewers will doubt the veracity of your claims.
If that's not possible, then publishing in the grey literature (e.g. on the website of an organisation you've worked with in your pathway to impact) is still an acceptable source of corroborating evidence, and better than simply supplying raw data to the panel, as it is easier to interrogate claims when data has been analysed and interpreted, and most reviewers will accept that researchers are unlikely to try and publish fabricated findings linked to their name
Although some impacts will require rigorous research to prove cause and effect beyond reasonable doubt (e.g. Randomised Control Trials to demonstrate the efficacy of an intervention, or social science studies of attitude or cultural change), the evidence just needs to be credible and convincing enough, which in many cases will be less resource intensive. Take advice from your Unit of Assessment lead if you are worried that the effort required to evaluate your impact is disproportionate, as they will have a more strategic overview of the number of case studies needed and their maturity, and can help decide if the effort is worth it in this context
Assess impacts longitudinally, to look for longer term benefits that may not have been apparent at the time you did the work
For policy impacts, where possible include evidence that the policy was implemented, enforced and worked on the ground
For public engagement impacts, create opportunities to re-engage with participants to deepen their interest and learning and find out how they have benefited in the long-term. Impacts from public engagement are among the most challenge to evaluate and evidence, but there are a number of methods available to evaluate different types of impact arising from public engagement
Make sure case studies that were previously submitted to REF2014 only claim additional, new impacts that took place since 2014. You will need to declare if your case study is a continuation of a REF2014 impact. See my blog on Everything you need to know about impact in the final REF2021 guidance for more details on this
Part 3: Writing well
Give your most significant, far-reaching and well-evidenced impacts room to shine
Choose your words carefully
High scoring case studies are typically written in concise, accessible language, making specific claims with clear causal links back to the underpinning research
Tell a story
Create a clear narrative that respects the timeline and realities of the pathway to impact, but that builds logically and sequentially towards the impact as the culmination of the story
Link disparate impacts together into a coherent narrative using thematic links if there were not links in practice
Use structure to your advantage
Sign-post each of your key impacts so they cannot be missed
Systematically explain why each impact is significant and far-reaching, referring to evidence
Break up the text to make it easier to read, using sub-headings and other structural devices
Use context to add shine to your pathway to impact and evidence of impact:
Put evidence in context to explain its significance e.g. rather than just citing visitor numbers, say if these was the largest number of visitors in any week that year, or explain who visited and why they are significant (perhaps they are from groups who don’t typically visit museums). A small number may be significant in context, but you will need to create the argument to justify this interpretation of your evidence
If you worked with a range of organisations, name those that are widely recognized and respected to lend further credibility to your pathway to impact
Integrate testimonials to more richly describe and explain the significance of your impact
Ask for quotable, concise summaries of key points (or ask for a draft testimonial, extract the key points as a summary and ask them to amend/approve the summary to be used as a quote). Integrate these quotes with the text of your case study, in case assessors do not have time read the full testimony
Ask direct questions to obtain specific answers about impacts you are interested in, backed up by numbers and evidence where possible
Consider obtaining testimonials via interviews so you can probe to get the most relevant information, transcribe relevant extracts from the interview, requesting any clarifications needed, and ask for a final version on letter headed paper. Consider whether you need ethics approval and informed consent for this process
Probe beneath statements that could be considered opinion to see if there is further evidence or data to support the statement, and follow these leads. If nothing else, get the person to substantiate their claim by describing the basis for their opinion
Get testimonials from high profile individuals at senior levels in organisations, but consider also giving voice to low status beneficiaries such as school children or patients
Don’t wait – request testimonials now. People move on, forget or worse..
Impacts in top-scoring case studies, significant, far-reaching, clearly articulated and convincingly evidenced. While there are examples of 2* and 3* impact case studies that might have scored 3* or 4* if they had been better written and evidenced, it is not possible to get top grades unless you actually have significant and far-reaching impact. There is time to deepen and extend the impact of your research. Check out Fast Track Impact resources to help you.
Find out more
For more information and links, see other Fast Track Impact resources, listen to our podcast or check out our free magazine and The Research Impact Handbook.
If you are interested in impact training, find out about our free online course here.
View the 2nd edition of 'The Research Impact Handbook' here
Disclaimer
Fast Track Impact provides expert guidance on impact case studies based on benchmarking case studies from the same unit of assessment in the Research Excellence Framework (REF) 2014. Fast Track Impact cannot predict how REF panels will grade case studies in REF2021, nor can we judge how the submitting institution will treat case studies. We cannot guarantee that case studies for which we provide advice will be submitted by your institution to REF 2021 or, if submitted, will receive the predicted/desired scores in REF2021, even if all advice has been implemented. It is recommended that Fast Track Impact’s advice is used to supplement the guidance and instructions provided by HEFCE and your submitting institution.