What makes a 4* research impact case study for REF2021?

There is an art to writing a strong research impact case study. Part 1 of this guide presents preliminary findings of a larger qualitative and quantitative linguistic analysis of REF2014 case studies to be published in the peer-reviewed literature later this year. Parts 2 and 3, based on experience advising case study authors from across multiple disciplines in the current REF period, give you further tips on writing your summary, underpinning research and corroborating sources, and how to use language, narrative, structure and testimonials to show off your impact in the best possible light. For more detailed guidance on getting the impact sections of your REF2021 submission right, see Everything you need to know about the final REF2021 guidance on impact in less than a minute.

 

In 2014, the UK became the first country to comprehensively assess the impact of its research as part of a national assessment. Although scores were not made public for individual cases, we looked for institutions whose case studies were all given grades in the same range to identify 175 (out of 7,000) high- or low-scoring case studies from a cross-section of disciplines. We combined qualitative thematic analysis with quantitative linguistic analysis to explore what made a high-scoring submission. 

 

In a nutshell, high-scoring case studies clearly articulated evidence of significance and far-reaching benefits that could be clearly be attributed to research conducted at submitting institutions. 
 

However, new rules are being introduced for the next REF and best practice is changing rapidly in some areas. This could mean sticking to best practice from REF 2014 may not be enough to reach top scores in 2021, so the lessons from this research should be seen as the minimum required to write a top-scoring case study for REF2021. 
 

 

Part 1: Three key lessons from high-scoring case studies in REF2014

 

Methods

 

Scores for individual case studies in REF2014 were not made public, but based on the distribution of scores received by some institutions (e.g. where all were given the same score based on publically available data from HEFCE), we identified 175 of the highest and lowest scoring case studies and analysed them using a combination of qualitative thematic analysis and quantitative linguistic analysis. To achieve balance across all Main Panels, the sample included 85% of all known 4* case studies, supplemented by additional cases from Main Panel B where at least 85% of case studies for a given institution scored 4* and the remaining cases scored 3* (n = 85). These were compared with a sample of case studies known to have scored either 1* or 2* (n = 90). Initial coding was conducted by 6 coders with intercoder reliability (based on 5% sample) assessed at >90%, with subsequent thematic analysis conducted by two of the coders. 

 

The sample for quantitative linguistic analysis includes 124 high-scoring and 93 low-scoring ICS. This includes all identifiable high-scoring ICS in any UoA and all identifiable low-scoring ICS in those UoAs where high-scoring ICS could be identified. There is a 75% overlap with the sample for qualitative analysis. In order to explore the lexical profile of ICS, lexical bundles of 2-4 words were extracted with AntConc from each corpus section (High-Overall, Low-Overall, High-MP A / C / D, Low-MP A / C / D) and then compared for significant over- and under-use across the corresponding corpus parts (High-Overall vs Low-Overall, High-MP A vs Low-MP A etc.; not enough data for MP B: only 6 high-scoring and 2 low-scoring case studies). The Keyword statistics used were Log Likelihood for significance combined with minimum expected values and Log Ratio for effect size . Each lexical bundle that met the “keyness” (i.e. specificity) threshold was assigned a code according to its predominant function in the text based on an inspection of concordance lines showing the bundle in context as it appears in the texts. This resulted in a list of typical bundles and functions that appear significantly more often in high- or low-scoring case studies, from which the above conclusions and examples are drawn. In order to quantify the readability of the texts, they were analysed using the Coh-Metrix online tool. This tool gives 106 descriptive indices of language features, including 8 principal component scores made up of combinations of the other indices. Relevant indices have been selected and compared across corpus sections using t-tests. As with the analysis of lexical bundles, comparisons were made between high-scoring ICS and low-scoring ICS in each of Main Panels A, C and D, as well as between all high- and all low-scoring ICS.

 

 

1. Articulate how specific groups have benefited and provide evidence of significance and reach

  • 84% of high-scoring cases articulated significant and far-reaching benefits, compared to 32% of low-scoring cases, which typically focused on pathway. 

  • Findings from the quantitative linguistic analysis show how high-scoring impact case studies contained more phrases that specified reach (e.g. “in England and”, “in the US”), compared to low-scoring case studies that used the more generic term “international”, leaving the reader in doubt about the actual reach. They also include more phrases implicitly indicating the significance of the impact (e.g. “the government’s” or “to the House of Commons”), compared to low-scoring cases that emphasized pathways (e.g. “the event”, “has been disseminated”)

  • Phrases containing variations on the word “dissemination” were more common in low-scoring case studies, which supports the claim that these case studies focused more on pathways than impacts and may indicate a perception among these case study authors of impact as one-way knowledge transfer 

 

2. Establish links between research (cause) and impact (effect) convincingly

  • Only 50% of low-scoring case studies clearly linked the underpinning research to claimed impacts (compared to 97% of high-scoring cases).

  • Findings from the quantitative linguistic analysis show that high-scoring case studies were significantly more likely to include phrases that attributed impact to research like “cited in (policy documents)”, “used to (inform)” and “resulted in”. Those phrases indicate a focus on the effect, that is, on what the activity led to. In contrast, low-scoring case studies tended to link backwards, foregrounding the research activity (e.g. “(findings) of the research”). Moreover, they were more likely to include ambiguous or uncertain phrases like “a number of” (implying that it is not known how many) and “an impact on” (implying that the nature of the impact is not known)

  • High-scoring case studies made causal relationships more explicit (above average when compared to general English texts, where low-scoring case studies are below average), which makes a text easier to process

 

3. Make your narrative easy to understand 

  • High-scoring impact case studies scored more highly on a readability measure correlated with reading speed because they included shorter, less complex sentences. The Flesch Reading Ease score, out of 100, was 30.0 on average for 4* and 27.5 on average for 1*/2*. While this is a significant difference (p<0.01), the scores indicate that ICS are generally of “college-graduate” difficulty. As such, these technical documents do not need to be “dumbed down” to the readability of a newspaper article (as some have suggested) but maintained at interested and educated non-specialist level.

  • Low-scoring cases were more likely to include academic phrasing with unnecessary phrases such as: in relation, in terms of, the way(s) in which

 

These findings are based on a preliminary analysis of data presented at the 2019 KMb conference, held in Newcastle on 21–23 March 2019.

 

 

Part 2: Writing your summary, underpinning research and corroborating sources

 

The following two parts of this guide are based on experience reviewing and advising on case studies from a wide range of disciplines during the current REF period. 

  • Make sure you are submitting the impact and not the pathway

    • To ensure you are submitting the actual impact, and not just the pathway to impact, keep asking “what was the benefit and why was this important?” and describe the benefits more than the process through which those benefits were derived. If you don’t know why it was important, ask the beneficiaries to tell you what was meaningful or valuable to them

    • Dissemination is not impact: even if you have impressive numbers of reads, downloads, views or listens, how do you know if anyone learned anything from it, benefited, or did anything different as a result? Keep asking “what happened next” until you find the benefit. To do this, design your communications so you can legitimately follow up longitudinally with audiences to re-engage, deepen interest and learning, and ask them how they benefited

    • Developing resources for schools and doing work in schools is a pathway to impact, not an impact in its own right. Identify specific changes you would like to see (e.g. increased attainment in a specific subject, reducing an attainment gap between boys and girls or ethnic groups for a particular subject or influencing choice of subject at University) and use your materials and work in schools as interventions designed to achieve these impacts. Follow-up to find out if your interventions worked or not

  • Make sure the majority of the words in your summary are about the impact, rather than the context and pathway to impact

    • Spend time making sure your summary resonates strongly with readers, communicating your impact straight-forwardly and persuasively

    • Get multiple reviews of your summary and keep polishing till you can improve it no further

  • Target your case study to the appropriate UoA based on the publications in the “underpinning research” section

    • Case studies submitted to UoAs by researchers from outside the discipline because they think assessment criteria will be easier may find a less sympathetic or understanding audience than they expected

  • Make sure the underpinning research reaches the 2* quality threshold (if possible based on the judgment of two independent assessors)

    • Being published in a peer-reviewed journal or having a book deal with a prestigious academic publisher doesn’t not guarantee 2* status if it is weak work

    • Outputs that are not peer-reviewed or academic books, such as final reports to funders, can reach the 2* threshold if they are strong work and publicly available

    • Explain why your body of research (the listed outputs underpinning your impact) is at least 2* quality, drawing on any indicators of esteem that you think will play well with others in your discipline (e.g. highlighting prestigious funding sources but avoiding citation data or journal impact factors as these aren't allowed) and creating a narrative justification explaining why the work is academically significant, original and rigorous in the context of your discipline

  • Describe the key findings from the underpinning research that pertain to your impact

    • Only include essential contextual material, and avoid unnecessary detail on methods or other findings that were not integral to the impact

    • Number your list of underpinning research outputs and cite each output by number in your description of the underpinning research

  • Ensure your corroborating evidence is robust and credible, for example:

    • Work with credible stakeholders to conduct independent evaluations of your impact e.g. by stimulating a policy review or making the case to a project partner that their stakeholders might be interested to know the impact of the work you did with them (where resources prevent this, consider providing your stakeholders with funding for an independent consultant to evaluate the impact on your behalf, acknowledging the source of funding in the report that is published on the organisation’s website). Offer help designing the evaluation to ensure it is robust

    • If you can’t find an organization to do this for or with you, do it yourself and publish the findings in a well-respected peer-reviewed journal. There are a number of ways you can publish corroborating evidence in the peer-reviewed literature. The evaluation will not be not independent, but if well designed and written, few reviewers will doubt the veracity of your claims.

    • If that's not possible, then publishing in the grey literature (e.g. on the website of an organisation you've worked with in your pathway to impact) is still an acceptable source of corroborating evidence, and better than simply supplying raw data to the