Linking different types of policy impacts to specific pieces of evidence from research can be a major problem for researchers who want to evidence the impact of their work. This attribution challenge is amplified significantly if you are working with overseas Governments. It is greater still if you think multiple governments around the world may have taken up your work. How do you find out if your work is being used in Government policy, let alone evidence it?
Some types of policy impact may be harder to evidence than others, and you may need to take a different approach to evidencing each type:
Research publication cited in policy-informing document that (might) be used in informing a new or changed policy
Actual new or changed policy
Effect of that new or changed policy being put into practice, with measurable benefit/reduction of harm that has resulted
Most policy impacts in REF2014 focused on the first two types of policy impact. With time on their side, it is likely that the proportion of case studies showing impacts arising from policy implementation will increase in REF2021.
To find references to research in policy documents and collect evidence of actual policy change, the first step is an Internet search:
First, narrow your search down to countries or international organisations (for more global issues) you think are more likely to have used or benefited from evidence from your research (for example based on the prevalence of the issues you address in your work).
If this is a long list of countries, rank according to those you think are likely to be able to benefit most (e.g. prevalence of a disease your work is targeting or share of international Greenhouse Gas emissions from a sector you are targeting).
If there is a phrase or statistic linked to your work (e.g. the name of your method or intervention, or a specific number based on your analysis), use these as search terms.
An alternative way of doing this is to use Altmetrics to identify documents linking to specific research papers...
Using Altmetrics to find evidence of policy impacts
In some cases, Altmetrics can find evidence that your work has been cited in policy documents for you. This is how to find out:
Navigate to your research outputs online
Download the free altmetrics add on (“Boomarklet”)
Highlight the DOI
Click on the Altmetrics bookmarklet in the toolbar and a small window will open showing the Altmetric data
Click on this to obtain full results
Citations in policy documents are indicated in purple in the “Altmetric wheel”
Clicking on the policy tab gives the list and sources of the policy documents identified by Altmetrics
Note that, if you search through Scopus, and click on a result, there is a link on the right hand side of the page to associated Metrics. However at the time of printing, although other Altmetrics data is picked up, the policy metrics are NOT indicated.
Examples of policy sources covered by Altmetrics (last checked March 2017):
UK’s gov.uk policy papers
The Publications Office of the European Union
Intergovernmental Panel on Climate Change (IPCC)
National Institute for Care and Health Excellence (NICE) evidence search (UK)
Médecins Sans Frontières (Doctors Without Borders)
AWMF – Association of Scientific Medical Societies (Germany)
International Committee of the Red Cross
European Food Safety Authority (EFSA)
Intergovernmental Panel on Climate Change (IPCC)
International Committee of the Red Cross (ICRC)
World Health Organization (WHO)
International Monetary Fund (IMF)
Oxfam Policy & Practice
Food and Agriculture Organization (FAO) of the United Nations
National Academies Press, USA
Australian Policy Online
Screenshot of Altmetric results for Reed (2008), showing policy documents citing the paper.
If a targeted search for evidence of impact in these countries does not work via a search engine, then this does not necessarily mean there was no impact. The work may have informed policy but without citing your work or any identifiable number or phrase that might help link the policy to your work obviously. At this point you have two options:
Systematically look through recent legislation published on the Government’s website to find policies that address issues linked to your research that were passed into law after your findings were published, and search for any evidence of potential influence. These clues will need to be corroborated through interviews; and
Conduct telephone interviews with members of the civil service and wider policy community to identify policy impacts. Getting names of relevant people can be tricky given rapid staff turnover in most organisations, and response rates to emails requesting information are often poor, so the most efficient approach may be to pick up the telephone. Go from person to person till you find the most relevant teams and people. In some countries, it can be easier to identify and speak to others from the wider policy community of people who regularly interact with policy-makers, for example charities and think-tanks.
Add value as you go
If this still fails to uncover any evidence of international policy impact, then you can avoid wasting your time by using the contacts you make to start promoting your evidence. If you start your investigation with the goal of helping further inform policy, then you may get better responses to your initial enquiries, compared to just asking for evidence that your work has been used. Consider if there are ways you can add value easily, for example by turning your work into a policy brief, manual or handbook that can make it easy for members of the policy community to act on your evidence and learn from others who have implemented it already elsewhere. If this is difficult to do directly, then research the broader context in which policies are being developed in the country to identify organisations that may be trying to influence policy in areas relevant to your work, and consider whether you could empower them to achieve impact via your research. These organizations can be powerful allies who can affect change on your behalf, but tread carefully. Some organisations you work with may cherry-pick or distort findings to meet their goals. Depending on the context you are working in, aligning yourself with organisations that are trying to influence Government policy can risk your personal safety as well as your academic and personal reputation. Indeed, some researchers take this as far as deliberately not engaging with any external groups who might have an interest in their research, so that they can maintain their independence and academic integrity.
Even if you have identified impacts already, it is always worth asking whether you can add to these impacts, for example by working with agencies that are charged with enforcing policy to ensure that the policy is implemented effectively on the ground in a manner that is consistent with the evidence you have generated. Your research may have been discussed in parliament, a committee or have been cited in a policy document, but rather than leaving it there, you may be able to advise the Government on how they might be able to actually turn this into legislation that can affect real change (especially if you have already learned how other Governments have done this successfully). It is typically easiest to find junior civil servants working in relevant evidence or policy teams, at the bottom of the organisation’s hierarchy. If it is not possible to find details of relevant people online, it may be possible to go via telephone switchboards, or via in-country academic partners who have relevant contacts.
Collecting evidence of impacts arising from policy implementation is much harder. In many countries, it is normal practice for civil servants to conduct policy reviews. These aim to assess whether the policy met its goals or targets were reached. University researchers typically have the credibility and independence necessary to help with these reviews, and if an offer of help is accepted, you may be able to play a key role in obtaining the evidence you need.
Mark Reed is Professor of Socio-Technical Innovation at Newcastle University. He holds a HEFCE/N8 funded chair and is visiting professor at University of Leeds and Birmingham City University.
Dr Sally-Anne Whitman is an Enterprise Officer at University of Leeds, where she is Faculty REF2021 impact lead, with responsibility for identification and development of impact case studies. She helps academics with delivery of research impact, from pre-award support for impact-related parts of grant proposals, to support for early stage commercialisation opportunities and protection of Intellectual Property.