As anyone who has attended my trainings will know, I’m not the world’s biggest fan of the Research Excellence Framework (REF2021). I see it is part of my job to undermine the all-pervasive dominance of REF in UK academic life, by encouraging researchers to stop chasing their 4* impact case study, and remember why they originally fell in love with research. Most people didn’t fall in love with impact – they fell in love with curiosity and that’s what drives them to this day. My message: do impact if you want to, and do it for your own reasons, not because REF, your funders or your University want you to do impact. If you’re motivated by curiosity, ask if engaging with people outside the academy might enable you to ask more interesting questions, and pursue the impacts that’s fascinate you. Be proud of your identity as a researcher and your motives, and don’t let anyone tell you curiosity driven research is self-indulgent, or that their motivation to change the world is somehow “purer” than your motives.
It therefore feels very strange to be defending REF in response to a comprehensive and well-researched full-frontal attack on the assessment by Richard Watermeyer. His new book, published this month by Edward Elgar, is a root and branch critique of REF impact that will resonate strongly with many. His argument is that “the confluence of market principles with public needs” forces academics to compete with each other for high scoring impacts, “in a perpetual game of one-upmanship and struggle to prove who's best or who can claim a greater portion of "excellence””. This, he argues, “stimulates performance-based anxieties that are corruptive to academics' self-concept as active contributors to the public sphere”, resulting in “inflated or otherwise counterfeit rationalisations of the public value of academic research”. The effect is a “dehumanising… deprofessionalisation of the academic profession”.
The problem, Richard argues, lies with the capitalist system that our Universities and Government are embedded within, which we are then co-opted into without our knowledge or consent:
“Too much internalised, too far assimilated and homogenized, academics have become estranged from a duty to dissent and to wilfully contest and/or conflict with authority, which has culminated in the obfuscation and unchecked distension of unequal and unjust regimes and, moreover, the kinds of capitalistic rapacity now endemic within Universities.”
In an article last year with Jenn Chubb in British Politics, I too have argued that it is this politicisation of impact that is at the root of most negative unintended consequences arising from the assessment of impact.
Two alternative responses to a dystopian future driven by the impact agenda
What has become clear to me from reading Richard’s new book, is that there are two alternative approaches to this problem. First, as Richard cogently argues, we can further politicise the issue by understanding the neoliberal political roots of the impact agenda, raise awareness of these roots and raise up an army of empowered academics who see it as their duty to resist the impact agenda.
Alternatively however, we can understand these roots, accept that they are problematic (for some not all, depending on our values and beliefs), and find our own reasons for generating impact on our own terms. In so doing, we transplant our impact in our own soil and enable it to produce very different fruits. Now rather than accepting that impact is a process of “academics …leveraging positional goods into their universities… through a commercial sensibility”, I can leverage justice and give voice to disadvantaged communities by undermining the corporate interests that dispossessed them. I can challenge dehumanising policies and regimes, and give strength to those who challenge them through my research. I can choose the political roots of my own impact, rather than having to accept those given to me by REF or my University.
Few of the researchers I enable to achieve impact from their work would view themselves as “entrepreneurs who view the future as an investment opportunity and research as a strategic career move rather than a civic and collective effort to improve the public good”. I work across the natural and physical sciences, social sciences, arts and humanities with people whose impact is to inform policies that can protect the environment, often at the expense of big business, or whose impact is to awaken parents to the danger of sugar, protecting public health at the expense of corporate profits. While for-profit spin-out companies may be prevalent in some disciplines, it is a small minority of researchers who have the opportunity to become entrepreneurs, and an even smaller minority who want to.
This is a book that seeks to understand the political roots of the impact agenda, and the consequences of this political dimension. It achieves both of these goals very successfully, eloquently diagnosing both the problem and its cause. The impact agenda has led to unintended negative consequences that other countries embarking on the design of their own assessments of impact should pay heed to. Gemma Derrick calls this problem “grimpact”, starting with research that linked MMR vaccine to autism but pointing out how REF increases the likelihood of grimpacts arising from a wider range of incentivised behaviours (not just research misconduct). Richard extends the critique based on extensive qualitative data from interviews with academics and members of the policy community.
Collecting testimonials for REF impact case studies is undermining trust in the academy
There is growing evidence as we reach the 2021 census date and evidence is required to support impact claims, that the evidence capture process is compromising relationships and trust between researchers, publics and stakeholders. One researcher interviewed for the book described how they discovered that their University had contacted their stakeholders to ask for testimonials to be re-written to their specification: “you can imagine how that went down with the organisations we were working with” (p56). Others felt embarrassed for other reasons (p57):
“I felt ashamed that the research-users may have thought that the interactions I had with them based on my research had a hidden egoistic objective to satisfy university business.”
Other interviewees explained the pressure they were under to collect evidence, and how this led to compromising situations:
“I was pushed at times to gather evidence in premature and improperly direct ways, which affected some of my relations adversely. I learned from this and now refuse to adversely affect and instrumentalise users.” (p59)
“Last time it was mainly people who were long-term engaged with impact – now it seems like all ambitious careerist types are trying to get in on the act (given it’s now even more important). I’m hearing some very negative stories from the policy/NGO community about the behaviour of some academics. This risks the relations that people like me have built up over decades, so while I agree with the impact weighting, I think this increases risks of having perverse consequences.” (p63)
Informally, I have heard an increasing number of stories from people I train over the last year about relationships that have been inadvertently compromised through innocent emails asking for testimonials that have backfired. In addition to the issues raised by Richard’s interviewees, there is a very real issue of contested claims, given that most impacts are co-generated with those we want to help, or other organisations trying to help the same group as us. Publicly claiming that a policy is based on a single study, as a result of close engagement with civil servants, is highly unlikely to be an accurate representation of the policy-making process, and if it was then the evidence analyst who was meant to synthesise from across the evidence-base may now be in trouble. If an NGO is funded to deliver an outcome and reaches out to a researcher for help, they don’t expect the researcher to then claim credit for the ultimate impact, undermining their attempt to claim impact to their own funders. I heard of one case where just such a double counting claim only came to light after the REF2014 case studies were published.
I have written about these and other issues on the Fast Track Impact blog, arguing that REF creates a conflict of interest for researchers submitting impact case studies. Based on my recent paper in British Politics with Jenn Chubb, who co-authored some of the work that underpins Richard’s book, we showed that many researchers fear the impact agenda is changing the questions researchers ask and compromising research quality.
But it isn’t all doom and gloom. As the recent Real-Time REF Review showed, researchers generally approve of the REF2021 reforms, even if 57.4% still hold negative attitudes towards the assessment. However, 29.1% perceive REF positively, and Richard offers some explanation for these views too. Applied researchers in particular, who had previously felt undervalued by their peers and institutions are now valued more than ever before for their capacity to generate impact, correcting a bias against applied research and legitimising those whose research falls into this category. This left a bitter taste in the mouth for some, because they were only being valued now for the “monetary and positional value” of their impact, and not the research underpinning it (p50). Richard suggests that these researchers were hence being “made complicit in a phony declaration of their research excellence and concurrently find their neoliberal identity - perhaps unwittingly – consummated” (p54). However, for most applied researchers, myself included, it would appear that this rebalancing is welcome, no matter how over-due it may be.
What should we do about all this?
As the evidence grows, we should all be very concerned about the negative unintended consequences of the impact agenda, and specifically the impact component of REF. However, Richard and I differ on the solution to these very real problems. Richard’s book further politicises impact in an attempt to waken researchers from their slumber to rise up against the corporate academy and its political masters (p11):
“Too much internalised, too far assimilated and homogenized, academics have become estranged from a duty to dissent and to wilfully contest and/or conflict with authority, which has culminated in the obfuscation and unchecked distension of unequal and unjust regimes and, moreover, the kinds of capitalistic rapacity now endemic within Universities.”
He goes on to argue that our unwitting collusion with our employers extends to collusion with policymakers, implying that some of us are now willing to alter research findings in line with political values to achieve policy impacts that can be claimed in REF:
“For social scientists especially, prominence in the public sphere is achieved most often via a close relationship with or cosying up to the policy making community and machinery of government.” (p85)
“Academics exceeding the parameters of their expertise or stepping outside of their comfort zone for the sake of being politically pragmatic. For some the co-mixing of academic facts with, or their subjugation to, political values may feel like a betrayal of their scientific code of practice.” (p89)
For me however, there is an uneasy relationship between the argument that we should voice our dissent and actively push back against the neo-liberal politics of REF, whilst avoiding any kind of political engagement if that means talking to the neo-liberal elite. For me this simplifies the world of politics into something black and white, and suggests that having values is a betrayal of research ethics, unless those values are opposed to the capitalist machine. For me, any scientific code that suggests researchers are value-free, independent and objective, is of limited use in the policy world, where alternative lines of conflicting evidence have to be weighed against each other and against lines of argument, including moral argument.
What is important for me, is that researchers interrogate their motives and understand why they do what they do. In so doing, they make their many identities and values explicit to themselves, and can see how these then influence the questions they ask and the research they do. When we look deeply, we may find that our own personal values fit on the left or right of the political spectrum, and I think we need to respect these differences. I should not be judged if, as a mathematician, the world’s largest investment bank approaches me with a problem that fascinates me, and I fix it for them, enabling them to make even more money. But I should interrogate my values before embarking on any such journey if I want to ensure I don’t later regret what I got myself into. Most of the researchers I know are naturally more liberal and left-leaning in their politics, and they believe in the power of research to make the world a better place. If my data shows that current Government policy is making the poorest in society more vulnerable, then I should not be judged if I want to use that evidence to challenge Government policy or empower those fighting for a different future. As someone who works with climate scientists, the whole thrust of climate science challenges the political status quo and will require entire business sectors to cease operating, if we are to avoid catastrophic climate change. Many of the researchers I know in this space would describe themselves as “researcher activists” and exemplify an altogether more hopeful vision of academic freedom, and the value of knowledge.
But of course, the problem remains: how do you know if your research actually made a difference? For me, whether or not we use the answer to this question to create a REF impact case study, the fact that we are now asking the question is hugely important. There are many of us who would be content to continue engaging with publics and stakeholders with no evidence to tell us whether or not it is helping. But unless we know if it is helping, there is always the danger that we are wasting our time or worse, causing negative unintended consequences. For me, this should always be part of “responsible research and innovation”, and REF is normalising the monitoring and evaluation of engagement and impact in the UK research system in ways that became normal decades ago in international development.
Many of the researchers I meet when I am doing impact training, especially those from more applied disciplines, tell me that they always wanted to generate impact from their research but there was never enough time. The incentives that have been created by REF have enabled many of these researchers to justify making the time, shifting impact up their "to do" list, and are inspired by what they have achieved as a result. When I train researchers from pure sciences, arts and humanities disciplines around the world, I recommend taking a look at the impact case study database for inspiration, and I regularly see light-bulb moments as researchers realise that there are more relevant, creative and meaningful ways of generating impact from their work than they had previously thought possible.
Did REF panels ignore evidence?
These case studies are not the sum total of the impact of UK research, as some of erroneously suggested. They are a highly polished and skewed tip of an iceberg – selected and crafted by researchers who are assessing their own work for a specific purpose. While that may seem like an obvious conflict of interests or flaw in the assessment process, case studies have to be supported by evidence, and assessment panels have to be satisfied that there is sufficient evidence to support any given claim. Richard undermines the validity of the entire database of case studies by suggesting that these panels did not look at the evidence. While I think that the assessment process was far from perfect, I think it is important to challenge this assertion, given its wide-ranging ramifications, if correct.
Here’s what Richard says:
“…evidence is treated as an encumbrance on the evaluation process - the scrutiny of which would be time and labour costly - and analogously therefore, expendable, superfluous and ideally circumvented, certainly where trying to adjudicate the veracity and quality of the impact promulgated in case studies in any efficient manner.” (p79)
“Where evidence was consulted, its incorporation was also seen to prejudice evaluation decisions, where certain forms of research produced impact that was easier to evidence or produced more obvious or striking forms of evidence of impact, which was further reasoning why evidence was best left alone.” (p80)
“The apparent eschewal of corroborating evidence by panellists I, in the context of a process of scientific decision-making or rather decision-making by scientists, altogether peculiar and perplexing, and self-evidently contrary to the scientific method. It also reveals the significant effort made, and no doubt expense incurred, by universities in reconnoitring impact evidence was all for nothing.” (p80)
I think a very simple misunderstanding may explain this interpretation of the quotes from interviewees. It is true that very few sub-panels requested the full evidence held by Universities, but in well-written case studies they didn’t need to, because the evidence was laid out before them in the narrative. While can be argued that panellists should not have taken as much on trust as they did, it cannot be argued that they ignored evidence. As the next quote below makes clear, testimonials were almost always quoted in the case study, and all manner of other evidence, including numbers, statistics and graphs, where presented as part of the case study. What the panels failed to do systematically was to check that the evidence presented was robust. So it may be the case, that had they checked the original testimonials, they would have found quotes had been massaged or negative comments left out. They may have discovered that some of the statistics were made up. But in a time-poor assessment process, instead, they only requested the full, original evidence where there was something that looked suspicious or didn’t make sense. Such an insight might mean we question the robustness of the process, but does not provide evidence that the process was so deeply flawed that the resulting scores were entirely subjective, as Richard suggests.
Finally, Richard claims that testimonial based evidence is inherently untrustworthy:
“The impact of academic research is largely confined to statements of support or testimonials from parliamentarians recruited by academics (as favours) for the purpose of corroborating their ICS claims - collected "evidence", ironically it would seem, that was ultimately largely ignored by REF panellists”. (p92)
“Perhaps only the most infirm or unfit of researcher-research user relationships will produce an account that is anything other than complimentary.” (p56)
First, it is important to point out that testimonials were only one form of evidence, and while they may have been pervasive they were rarely the only form of evidence presented. My own experience gathering testimonials is that most people give me disappointing assessments of my role in the generation of impact, if at all, and in some cases (such as evidence analysts who are meant to synthesise evidence or NGOs or politicians who want to be able to claim as much impact for themselves rather than having to give credit to others, as described above), it is very difficult to get anyone to go on record to say your work played a role in the impact. While some researchers may be able to “call in favours”, I do not think the evidence presented here supports such a dim view of human nature. It is also worth bearing in mind that the subjective nature of testimonials was recognised by Research England in its guidance for REF2021, asking us to seek evidence-based, rather than opinion-based testimonials.
Conclusion
Richard and I share a dislike and distrust of the impact agenda, and its incarnation in REF. Richard’s root and branch critique of this is well researched and insightful, and we would all do well to heed some of the dark warnings it provides. I hope many researchers and managers read this and take a more critical approach to impact, that reduces the chances of things going wrong for them and the people they seek to help. But I hope that in this review, I have offered some alternative interpretations and an alternative response. I would characterise Richard’s approach as a need to uproot the impact tree because it is rotten to the core. We cannot fix the ills of capitalism with structures that are so deeply rooted in the system that caused the problems we seek to solve. Richard’s post-impact world harks back to a time when we were not required to account for our time or taxpayers’ money the same extent that we do now, and I share that nostalgia. But the impact agenda is here now, and it doesn’t matter how many of us raise our voices with Richard to dissent, because the world has changed.
Rather than fighting to uproot something with such deep and problematic roots, I believe we should be fighting to make impact something that motivates and inspires us, whether that be explaining the beauty of pure mathematics through music or risking our lives to challenge Governments and corporations who are destroying the environment. Instead of uprooting the tree, some of us might be perfectly comfortable with the values that the impact agenda is rooted in, but we have the power to chop down the tree and coppice it, growing our own new stems in ways that please us. Instead of uprooting the tree, some of us might take its seeds and plant them in our own soil, growing impacts that area consistent with our own unique values, which may stand in direct opposition to the politics of those who tell us we need impact.
Let’s not throw the baby out with the bath-water. Let’s make impact our own. When we understand who we are as a researcher, and the values that underpin our approach to life, we can generate impact in ways that are intrinsically motivating, and that enable us to express our academic freedom in new and creative ways.
Although a little unconventional, I thought it would be nice to give Richard the chance to respond to my review, so I could publish his response with my review. This is what follows next.
Response by Richard Watermeyer
In the present time, the excesses of global capitalism have never been so pronounced and pernicious, yet so persistent and seemingly – despite a giant swell of organised protest – inexhaustible. These excesses are both consequence of and contributor to the retreat of social democracy and correspondingly aspirations of an ethical state; moral economy; and democratic accountability. Analogously, these excesses are attributable as route-cause to an escalation of social displacement and inequality, societal fracture, division and polarisation and as Francis Fukuyama has so cogently observed, of human beings becoming lost to themselves.
In the milieu of science and discovery no less is this existential freefall more apparent, as Edward Said has recognised, than with the disappearance of the public intellectual and his/her replacement in universities especially, with professionalised and institutionalised, docile and obedient knowledge technicians. Where once the former would hold ‘power’ to account the latter afford ‘power’ the opportunity to act with impunity. In part this may be explained as an unconscious capitulation to what Louis Althusser called an ‘ideological state apparatus’ and suspension of disbelief in technologies of governance as forms of repressive control. It may also be a case of academics ‘repressive desublimation’ (as Herbert Marcuse would have it) and their indulgence of fabrications of freedom and beguilement by what Wendy Brown has called ‘neoliberalism’s stealth revolution’. Furthermore, it is hard to ignore the various labour-based vulnerabilities endured by academics that engender acts of resilience and survivalism that are emblematic of their displacement from an intellectual ideal.
The performative militarisation of universities by the REF and other aspects of prestige economics, and academics’ collusion with or more often than not coercion to conform with these, and with what I call ‘competitive accountability’, is achieved in an Althusserian sense, through their claims to virtuousness and in the case of REF impact, of a moral conscience, even noblesse oblige, and sense of duty to account for the investment made of them by the ‘public’. Thus, academics involvement in REF-impact is done so on the basis of public accountability being right and proper – an assertion that is difficult to unpick or discredit. However, where the university has become as much a part of the edifice of neoliberal enterprise as industry and government – an unholy ‘triple-helix’ – and is consequently as equally committed to the accumulation of capital, a moral duty of accountability converts into a lever of profit and positional gain (and thus also among academics, humility into hubris and collectivism into individualism). Were it not, the investment made by universities in REF-impact’s (high) production value, would be likely far less lavish and less easily rationalised than the purported millions of pounds and staff hours spent on REF2014 impact submissions – and what we may similarly anticipate for REF2021 where the impact weighting has inflated – and an aim of enhanced QR acquisition and league table status. So too would an incentivisation for those seeking professional advancement through gaming impact, weaken. In total, the substantiation of an impact agenda – and spawning of an impact-industry so integral to the production of ‘spectacle’ – by moral assertion and its dressing up as public accountability has masked or given license to deceit and conceit precipitated by a principle of competitive accountability that undermines so much of what otherwise might be achieved through a more honest, humble and critical reflection and, therefore, also more useful calibration of academics’ public interface. Certainly, than as might be determined by the drivers of money and esteem.
So, in turning directly to Mark’s very generous and concurrently challenging critique of the competitive accountability thesis as applied to REF impact, my concern is that he, much as many other good colleagues, is too much sanguine of what might be achieved through this agenda and too lenient towards and accommodating of its numerous flaws. Mark’s take is that we are stuck with this system – ‘the world has changed’ – and that we ‘cannot fix the ills of capitalism that are so deeply rooted in the system that caused the problems we seek to solve’. His is a voice of political pragmatism that says that ‘rather than fighting to uproot something with such deep and problematic roots . . . some of us might take its seeds and plant them in our own soil’. He speaks of a need to work within the system to change it and effect adaption through more subtle and cautious intervention or appropriation.
My concern is that within this schema of ‘making the best of a bad situation’, we as academics condone, normalise, are desensitized and are at risk of becoming blind to the insidiousness of a system so focused on the accumulation of capital that we become – not as public servants but impact purveyors – infinitely and perhaps irreversibly lost unto any other version of ourselves and (as relates to) our public communities. I worry also that in this instance we make too much of an assumption of the transformative agency of ‘we’ the Academy, or worse ‘we’ as institutions or individual researchers as originators of impact; we invest too much faith in ‘our own soil’ as the foundation of change. Yet what if ‘our soil’ is so far contaminated as to prevent ‘the growing of impacts’, even if it is a richer public interface that might instead more profitably be ploughed? Do we then not need to go in search of new fields to sow – and crucially not of our, or rather the university’s own?
Ultimately, my concern in accommodating an impact-agenda in the way that Mark suggests is that academics will continue to chase shadows of their public selves. And where they persist in ‘bowling alone’ – for the sake of evaluative expediency – the richer rewards of multi-actor and multi-agency collaboration will be sacrificed for the fuller recognition afforded to achievements, exclusively owned or claimed as such. I would argue instead that critical solidarity not creative submission is necessary for the restitution of the university as an authentically public institution, that might produce what Mark calls ‘very different fruits’. And this may be true not just of the university but all such public institutions as cradles of social democracy that have suffered the duress of unbridled capitalism.
The possibility of imagining and implementing a different kind of accountability, freed from the ideological apparatus of the neoliberal Academy, should not be considered so fantastical. Yet, this metamorphosis hinges on trust among all communities that not just the university but the world in which it sits can change and through a common commitment will change. A commitment to change is also as global justice movements are evidencing, contingent upon an investment in the transformative potential of youth and the mobilisation of its energies for a better future, unhindered by the binds of nostalgia, stillborn utopias and ideological cul-de-sacs; one vastly superior to the current intellectual inheritance. The prospect of a future public good beyond the power abuses of global capitalism and the phonyism of ‘moral labour’ that serves as its foil, demands a confrontation with awkward and inconvenient ‘truths’ and a true measure of accountability, which REF-impact is not.
Richard Watermeyer, 23rd September 2019
Buy your copy of Competitive Accountability in Academic Life - The Struggle for Social Impact and Public Legitimacy by Richard Watermeyer.
Conflicts of interest statement: Mark Reed's Chair position is funded by N8 AgriFood, which is 50% funded by Research England. However, he does not report directly to Research England, and they have no say over what he publishes or does. He is CEO of Fast Track Impact, and in that role regularly advises Universities and researchers on impact case studies. His research on impact and REF has been funded by various UKRI Research Councils over the last 15 years, and is currently also funded via Fast Track Impact.