Managing and evaluating impact

How to create a positive research impact culture in your group

Dr Gemma Derrick

What is the impact culture in your institution or research group? Every group has a culture, and you may be able to describe some aspects of this (like your research culture), but can you describe your impact culture? In some places, the impact culture is “we don’t do impact in our discipline”. For others it is part of our DNA, and if we fail to achieve impact then we have failed as applied researchers. For others still, impact is something we do for funders or exercises like the Research Excellence Framework.

In this article, I want to give you the diagnostic tools you need to understand the impact culture you are part of. Once you understand your impact culture, you can begin to do something about it, building on your strengths and tackling issues to build a more healthy culture.

What is an impact culture?

I will start with a definition of research impact culture, and then I’ll explain how you can use this definition as a diagnostic tool to assess the impact culture of your group or institution.

A research impact culture is the shared values, beliefs and norms of an academic community which build trust between researchers and those they seek to benefit, leading to the generation of (significant and far-reaching) non-academic impacts based on robust research, which then define the collective identity of that community and distinguish the strengths and foci of one institution from another.

There are three components to my definition:

First, cultures are based on shared values, beliefs and norms. While we value difference and diversity, cultures are built from the (often limited number of) things we share in common, or at least that work well together. To find values, beliefs or norms that we share, or that work well together, you have to look beneath our differing tastes and preferences to a deeper level, for example, to the ideas we share about what it means to have integrity as a researcher, or values based on the value of knowledge and the role we should play in society. There is no one set of values, beliefs and norms that needs to underpin a healthy impact culture, but they have a huge influence on how people engage with impact, and some foundations are intrinsically more or less stable and nurturing than others.

Second, an impact culture lives and evolves through the communities of people who interact with each other day-to-day around impact. There are many factors that determine the success of a community, but trust is a key factor. There are two ways in which trust and distrust perpetuate and destroy an impact culture.

We have to consider how trustworthy we are perceived to be by those beyond the academy we seek to serve. Do we raise false expectations to get funding and then regularly let people down, or can we be available and relied upon to help those we work with in their time of need? Our perceived trustworthiness is important because we know that people are more likely to learn (and hence gain knowledge) from social networks they trust, especially in instances where risk and uncertainty are high. Impact generation relies on the conversion of data and information from research into knowledge before it can benefit someone or be used, and so trust is integral to impact.
We have to consider the extent to which we trust each other as researchers around impact. Do we work in a culture where we are told that impact is something we should do because we want to or because it is the right thing to do, and yet get told (whether explicitly or via incentives and performance targets) we have to do it in certain ways that get us funding or grades? If my impact doesn’t make the grade, do the people who claim not to be driven by the metrics, celebrate my small successes or make their disappointment clear? Impact cultures that breed trust are typically underpinned by shared values, beliefs and norms that build and maintain trust, so these two parts of an impact culture are closely linked. They in turn, in theory, lead to the generation of impacts.

Finally, my definition idealises these impacts as significant, far-reaching and based on robust research. Clearly, there needs to be room for the insignificant and the small-scale, and not all impact has to be based on peer-reviewed research. For me, significance and reach are optional (unless you are being judged on them, in which case you may put a higher priority on these criteria). However, we cannot compromise the rigour of the research upon which we generate impact if we want to have a healthy impact culture. If we do so, then at best we become lobbyists, and at worst we become responsible for decisions that could lead to people’s deaths.

The second half of my definition suggests that you can see collective identities that distinguish one institution or group from another. This is less important for most researchers, but for those charged with developing and curating an institutional identity, it can be useful to identify what is unique to our institution given our history, location and what we are good at, so we can describe our impact culture in ways that distinguish us from other institutions.

What is YOUR impact culture?

Now we have a definition of impact culture, I want you to use this to interrogate your own impact culture. Think about this at whatever institutional scale is most relevant to you, whether that be your research group, department, faculty or university. Under each of the three components of the definition above, I am going to ask you a series of questions, which should help you understand how you measure up against each component. By the end, when you put these three components together, you should have a pretty good idea of the culture you are in and what you might want to build on or fix.

Questions to ask yourself about the shared values, beliefs and norms that underpin impact

What are the reasons you generate impact?
What are the values that underpin these reasons for engaging with impact?
Why do you think other people in your group typically engage with impact and what are the values that underpin this engagement?
When you hear people talking about impact in this place, what values are implicit in what they say?
For example, do people talk primarily in pragmatic and instrumental terms about getting impact in the context of grants and impact evaluations, or career progression?
Or do you hear people talk about helping others, empowerment, equality, fairness and creativity as they talk about impact, whether or not they use the word “impact” to describe the benefits they are seeking?
How often do you hear people talking about things you might describe as impact?
Do you hear people talking about the benefits of their work beyond the academy regularly in many contexts, or infrequently, in passing or only in relation to funding and administrative processes?

The values, beliefs and norms of any group are practised and rehearsed with every group interaction, face-to-face and via email, but they are deeply implicit, and therefore hard to see unless you are looking. Start looking harder, and you will begin to understand why your group prioritises and behaves the way it does around impact.

Questions to ask about trust around impact

First, we need to ask questions about how trustworthy we, and the groups we are in, are perceived to be. Although you may feel like you are trustworthy, people may perceive the rest of your group, or researchers in general, to be untrustworthy, and different groups may perceive your trustworthiness very differently depending on their previous experience and worldviews. This, therefore, is a question you need to ask deeply of yourself and your group. Even when you look deeply, you may still miss the answer because it is impossible for you to look at yourself through the eyes of others, and so you may want to ask questions of others to get a more balanced understanding of how you are perceived. Question you can ask include:

Do you return emails, calls and messages on social media from those beyond the academy who engage with your work?
Do you make unrealistic promises to non-academic project partners and regularly fail to deliver?
Do you tell people you meet at workshops and events that you’ll get in touch, but bin their business cards weeks later?

Second, we need to ask questions about the levels of trust we see within our groups and institutions:

To what extent do academics and professional services staff working on impact trust each other not to overburden each other and yet to deliver on time?
Do you and your colleagues trust that those at the top of your institution who set agendas, priorities and deadlines in relation to impact have your interests and the interests of those you serve at heart?
To what extent do you trust this of your funders, government and others who demand or reward impact?
Do researchers from different disciplines who need to work together to generate impact trust each other as equals or do some feel relegated primarily to working on engagement and impact rather than the core research questions and outputs?

Questions about the nature of impacts that are generated

Is there a lot of impact, but more limited research, and in particular a lack of robust research to underpin impacts?
Is there a lot of robust (and academically significant and original) research, but limited evidence of impact from the work arising from the group?
Do you regularly see significant and far-reaching impacts emerging from the group’s research?
Do you also see smaller, less significant benefits being prioritised and celebrated (this is more likely to reveal something about values, beliefs and norms though)?

The answers to these questions will vary widely across different types of universities and departments. Some will have head-hunted experts from industry, policy or practice, who are taking their students onto the cutting edge of current practice, without being research active. They are still continuing to generate valuable impact, but typically based on other people’s research or on their practice and experience. If this is a robust basis for the impact, this is good and should be celebrated. However, in systems where impact is only valued when it arises from research conducted by the group, there is a danger that healthy impact cultures can become squashed by the systems they occur in. If the evidential basis for the impact is shaky, however, even if the flawed research was done by you or your group, there is a problem. An impact culture that regularly produces weak, misleading or flawed research, and then pursues impacts uncritically on this basis is not a healthy impact culture. Developing a healthy research culture must become a priority in such groups before it is possible to consider developing a healthy impact culture.

At the opposite end of the spectrum, there are many groups and institutions which are doing world-class research which, despite its applicability to real-world contexts, rarely leads to impact. A culture that values outputs significantly more than impacts and perpetuates this imbalance is just as unhealthy as a culture that generates impact without the necessary underpinning research.

1/2

What can you do about it?

When you put all three components together, what you have is a characterisation of your impact culture. If you are happy with this, then you can identify the strengths you can build on to deepen and broaden your already healthy impact culture:

What aspects of your impact culture should you celebrate and replicate, and what can you do to be part of an impact culture that becomes even more like that, and makes the shared values, beliefs and norms underpinning those successes even more explicit?
What can you do to further build trust and community around impact, bringing others researchers into your impact culture, and consolidating trust in your group and in research beyond the academy?

If you are not happy with the characterisation that has emerged from your analysis of your impact culture, then you need to ask what you can do to change it:

There may be very little that you can do as an individual, but might it be possible to ask these questions at higher levels with people who might have the power to do something?
Can you start conversations with others in your group about impact, bringing it into research group meetings regularly (without using the word “impact”)?
Can you start to discuss more explicitly with your group or institution how you collectively perceive and engage with impact and consider what you might do to improve your impact culture?

How we as individuals relate to and engage with impact influences others and becomes part of a wider conversation that perpetuates or changes culture at different levels. We each have the ability to diagnose strengths and problems with the impact cultures we are in. We each have the ability to ask questions of ourselves, our colleagues and our institutions. Together, we have the ability to find our own basis as a group for pursuing impact, building trust and delivering benefits that people care about.

Two different approaches to building impact cultures

Once we know the issues we are tackling and the kind of culture we want to create around impact, we have a range of motivational levers at our disposal to effect change. The availability of levers will depend on our level of seniority, however, the more levers we have available, the more careful we must be in pulling them. Those with greatest power tend to have a range of extrinsic motivational levers available, which are easy to pull, and can in theory effect change rapidly and at scale. I can incentivise staff by integrating impact into annual reviews or promotion criteria, or offer funding and other rewards for impact-related activity. Although easy to pull, extrinsic motivational levers only work partially. Some of the time for some people, they deliver what you want, and you now have well motivated and rewarded staff pursuing impact. At other times, for other people, these levers produce unintended negative consequences, as staff chase career progression, funding and prestige at the expense of the people they were meant to be helping. I am not saying that we shouldn’t reward people for impact or create incentives. If we did none of these things, then we would inadvertently be sending a message that impact is not valued by our group or institution. However, if that is our sole focus, then we have to accept that we may inadvertently create a culture that encourages game playing around impact to secure rewards.

The alternative is to focus first on the intrinsic motivational levers. These are harder to identify and use, depending as they do on empathy. They are less likely to work at scale. But when done properly, these are the kinds of things that sustain deep motivation and engagement, and build healthy impact cultures over the long term. Some team members may have an intrinsic motivation to make a difference. That’s easy – we can key into that, enable people to achieve their aspirations, equip, inspire, skill-up, and give these people the time and resources they need. But what about team members who are more motivated by creativity or curiosity than they are by impact? If I understand what fundamentally motivates these colleagues, then I can identify opportunities that will pique their curiosity or engage them more creatively. For example, asking a curiosity-driven researcher to attend (not speak at) a policy seminar linked to their research interests to report back to the group may stimulate new thoughts and connections that start them on a journey towards further engagement, even if only with researchers from other disciplines to start with. Giving another researcher an opportunity to give a public lecture or work with an artist in residence as part of a science festival may engage with their intrinsic creative drive, motivating them to engage more with impact in future. If such opportunities are targeted towards opinion leaders, then it may be possible to facilitate culture change at scale, but this approach is not as fast or scalable as the extrinsic levers.

Opening channels of empathy and adaptively identifying opportunities takes time and emotional energy. However, if we are not prepared to invest some of this emotional energy alongside the quick wins, then we are unlikely to succeed in changing our impact culture. Culture change takes time, commitment and deliberate effort.

How to create a positive research impact culture in your group

Move to a new institute

How to move your REF impact case study to a new institution

The REF rules are clear: impacts remain with the institution in which the original research was conducted. If you move institutions, you can’t take your case study with you. Or can you? I am regularly asked by researchers about how they might be able to claim impacts at institutions they have moved to, and often there is little hope. However, there are a number of examples in the REF2014 database of case studies that appeared to follow their authors, and were submitted by both the original and the new institution. So, it must be possible. In this article I will explain how.

It is clear that simply changing your affiliation on publications to your new institution is not enough. Paragraphs 311 and 312 of the REF2021 guidelines (which are currently out for consultation) state that case studies must be underpinned by research produced by staff who carried out the research while working in the submitting institution between 2000 and 2020:

“To be eligible for assessment as an impact, the impact described in a case study must have been underpinned by excellent research produced by the submitting unit, during the period 1 January 2000 to 31 December 2020. [This] means that staff carried out research within the scope of the relevant UoA descriptor, while working in the submitting HEI (even if those staff have since left). This research must be evidenced by outputs referenced in the case study, published between 1 January 2000 and 31 December 2020, while working in the submitting HEI.”

Even if swapping affiliations was an option, you should ask the institution you are leaving, which will almost always request a double affiliation that includes them. If you don’t do this, then most journals will not allow you to do anything other than add a second affiliation. Having said that, changing or adding an affiliation may give you an excuse to add the output to the underpinning research section of your case study, and depending on the quality of output and prestige of publication venue, this may help build the case that your case study is based on research that is 2* or above in quality. However, this should never be a pivotal output or research finding for the impacts being claimed, as there is a strong probability that it may be discounted by the panel based on the guidance above.

A safer approach would involve conducting and publishing new research at the institution you have moved to, which builds on the work you did at your previous institution. The new research would need to add a significant new insight that can take the impact to a new level of significance or to new reaches that would otherwise not have been possible. You will then be able to claim the new and additional impact, which builds on the previous work, with an element of integration between the two.

Worked example

To illustrate, I moved from Leeds to Aberdeen and then Birmingham City University (BCU), arriving at BCU 18 months before the deadline for submitting impact case studies. I finished writing two papers linked to the research at BCU and used my new affiliation, but most of the research had been conducted at Leeds and Aberdeen, so I didn’t want to rely on these as my underpinning research. I therefore applied for four small government-funded projects to conduct some applied economics work that would lead directly to the development of a new policy mechanism (luck was on my side here, both in terms of the timings of the tenders and the fact that I won them all). I now had funding evidence for substantive new research that was initiated and completed entirely at BCU.

The problem was that there wasn’t time to get the work published in peer-reviewed journals. I therefore wrote the final reports to the same standard I would have written a journal article, and got them reviewed by colleagues, who thought they were 2 or 3* quality. Although it was a close call, we got them published on the government website in time for the deadline in 2013. As a result, the case study was underpinned by four eligible research outputs, and if reviewers were concerned about the quality of these final reports, there were two peer-reviewed papers in the list to reassure them and if they read the reports they would find research of a quality that should be eligible.

The case study then clearly stated which impacts had occurred since 2012 when I joined BCU. It was a bit repetitive, but was clear to reviewers what impacts I was claiming since 2012, which were attributable to BCU and which elements were pathways to that impact that had taken place at my former institutions. The silver lining was that I was able to share progress with the University of Leeds who were able to update their case study to include the additional impacts I had achieved at BCU, which wouldn’t have been possible without the original research I had started in Leeds.

So, can you move a case study to your new institution?

The answer is clearly yes – in theory. However, as you can see from my example, doing this without risking your eligibility takes a lot of work. But there is hope for those of us who have moved institutions and are continuing to build on our impacts.

How to get commercially sensitive data to evidence economic impacts from research

If your research has been commercialised in some way, then you may be

asked to evidence the economic benefits arising from your work. Economic claims are harder to evidence than you might think, because many companies are nervous about sharing commercially sensitive data. But this doesn’t mean you shouldn’t try. In this article, I’ve suggested a few things that might get you something useful.

Go in person to your “warmest” contact with a specific data request (e.g. turnover, profit or savings related to the specific product or service that arose from your research). If you don’t ask you won’t get. Rather than pitching your request out of the blue via email, ask to meet up to find out where they are at and if there’s anything you can do to help them, and offer to bring a short summary of some of the latest research in the field. You are more likely to get a reply and a meeting if you offer something of value to them first. In the context of a meeting in which you have added value, you are more likely to get a positive reception to your data request. Crucially, even if you get a “no”, if you are face-to-face with the person, you can quickly move to your plan B so that you don’t leave empty-handed. This is much harder to do if you have received a clear “no” via email, and less likely to work simply via email.

Offer redaction and anonymisation. A number of levels of anonymisation and redaction were available in REF2014 and are expected to be available in REF2021, providing some reassurance to companies that want to help but don’t want to take the risk that the data becomes public. However, for some companies, such reassurances are not enough, and they will not take the chance that something goes wrong and the data gets into the wrong hands.

Move to persuasion if it looks like you are going to get a “no”. This is more likely to work for SMEs and small businesses than large corporations, but it may still be worth trying (GlaxoSmithKline have highlighted their role in REF2014 case studies and contributed a panellist to the process, and so understand the pressure on researchers to evidence their impact). You are conducting an evaluation of the impact of this company’s product or service as an independent academic that will go beyond the economic impact and provide a public record of its benefits that can be used in marketing. Depending on the perceived marketing value of such an independent assessment of the company’s product or service, this may be enough to persuade the company that there is something in this for them, and they may cooperate. In the business world, any deal is possible, and if you treat your request for data as a business deal in which you will give them something in return, you may find you’re shaking hands on a deal that gives you the data you need (possibly with assurances around redaction thrown in to sweeten the deal).

Back to

Continue

Contents

Ask for proxies. If you can’t get access to any of the data you need, then go prepared to ask for proxies that you could use instead. What information might provide those, evaluating your impact with a sense of the significance of economic impacts? If you are working with a large company, can you get global figures for turnover that indicate the significance of the company you are working with? What proportion of this is attributable to the product or service linked to your research? Is this their flagship product representing more than half their revenue or something much smaller? If you can’t find out what proportion of their turnover comes from that product or service, can you find out if it is in their top sellers (top three, top five?) and what proportion of their turnover comes from those top sellers in aggregate? What is their market share in a specific marketplace, and how many products do they sell into that market in addition to the one based on your research? If you know the size of the market and their market share, you can get a sense of the size of your contribution. The answers you get from a conversation like this are necessarily estimates, but you will be able to communicate the order of magnitude, and give readers an overall sense of the significance of your impact. To evidence this, you would need a testimonial on letter-headed paper. If they will not do this, ask if you can name them as the source of the information for someone else to be able to corroborate the evidence in future. If they won’t even do this, then at least you have the information, which you can now communicate, even if it is weaker than you would like. Better than nothing. But there’s still one final option…

Ask their customers. If you can’t get data from the company itself, see if you can get them to tell you who some of their customers are (even better, who is their biggest customer if they are selling business to business) for the product or service based on your research. The customer may be willing to tell you how much they spend on the product or service, and if you know that they are the company’s largest customer then you have the basis for an estimate that would tell you something about the scale of your economic impact. Alternatively, if you can discover a more representative customer who is willing to tell you how much they spend, you may be able to couple that with information or estimates about the number of customers to get a feel for the scale of the economic impact. Bear in mind that these estimates will be commercially sensitive and you will want to ensure these are redacted if you want to retain a good relationship with the company. If these customers can’t give you information about how much they spend, they may tell you how much money the product or service saves them, or the other benefits they get from it, such as improving the health, safety or wellbeing of their staff in measurable ways, or reductions in their environmental impact. Even if you can’t get estimates of the number of other customers, you can use this one customer as an example of the breadth of impact felt by customers using the product or service based on your research.

Evaluating economic impacts is not only challenging, it can get you into trouble. Companies who fear your evaluation may compromise their interests may sue you or your university. Always take advice from your Business Development Manager and, if possible, get them to accompany you to high-profile meetings to make sure you don’t take unnecessary risks.