Inside Track | Promoting best practice in research assessment
On the 10th anniversary of the San Francisco Declaration on Research Assessment (DORA), we review progress in the use of responsible metrics at Leeds and the next steps to further support colleagues.
Professor Amanda Bretman and Claire Knowles
Today (16 May) marks a decade since publication of DORA. Leeds became signatories in February 2020, joining a community now numbering more than 23,000 signatories (individuals and organisations) in 160 countries.
DORA was conceived at the end of 2012 by a group of journal editors and publishers at the annual meeting of the American Society for Cell Biology. The group was concerned by the inappropriate use of metrics in research assessment – in particular, the rise in misuse of the Journal Impact Factor (JIF). This is the average number of citations received per paper in that journal during the preceding two years.
The JIF was originally intended as a mechanism to assist university libraries in purchasing decisions, but morphed into a shorthand for quality to assess individual researchers, articles or institutions. Not only was JIF not designed for this purpose, but the San Francisco group pointed to evidence for the ways in which JIF could be ‘gamed’.
Of course, JIF isn’t the only metric used in a problematic way. H-index was designed as a measure of an individual researcher’s academic impact, but is prone to bias, such as being insensitive to very highly cited papers, and is near impossible to compare across disciplines. Moreover, often the metrics available to us simply don’t cover the range of outputs and activity researchers produce.
The DORA group devised a set of recommendations for funders, institutions, researchers and organisations that supply metrics. The three themes running though these recommendations are:
- to remove the use of journal-based metrics in individual assessment (e.g. funding, appointment and promotion)
- to assess research on its merits and not in which journal it’s published; and
- to capitalise on advantages of online publication, such as less strict article formats and new indicators of the reach and impact of research.
DORA continues to explore best practice in research assessment, including embracing the rise of Open Research. Its Project TARA (Tools to Advance Research Assessment) comprises a toolkit of resources to aid in, for example, ways of assessing impact and de-biasing assessment. Nevertheless, there are challenges in DORA being globally accepted, most notably the great variation in the number of signatories across countries, with most situated in Europe and North America.
Changing attitudes
DORA was at the forefront of changing attitudes to research assessment. In 2015, Hicks et al published the Leiden Manifesto, proposing 10 principles for research evaluation. In the same year, Research England published The Metric Tide – a review of metrics in research assessment, which was revisited last year. We’re now seeing funders in the UK, such as the Royal Society, UK Research and Innovation (UKRI) and Wellcome move towards more narrative than metrics-based CVs. However, any method of assessment has its shortcomings, and more narrative approaches might be more prone to subjective biases of reviewers.
As part of our commitment to DORA, we instigated the Responsible Research Metrics Group at Leeds, which brings together academics across disciplines and colleagues from the Library, Organisational Development and Professional Learning (OD&PL) and Human Resources. Our position statement, first published in 2020 and updated in 2021, contains seven guiding principles and considerations to support the use of research metrics. The point is not to entirely remove metrics, but to make sure we use them appropriately. We aim to encourage a considered approach to research assessment wherever we interact with it, whether on either side of the recruitment or promotion processes, during our annual academic reviews, or as leaders assessing the progress of research groups or schools.
Next steps
We will be evaluating our use of research metrics across the institution during the next year and we aim to make some more interactive resources to further support colleagues. We already have evidence of great practice at Leeds, some of which will be presented at a UK Research Libraries workshop on 17 May to celebrate the DORA anniversary.
Further information
Visit the research metrics webpage on the Library website for further information and to access resources about the responsible use of research metrics.
You can also watch the video of Leeds Professor Stephen Curry, Chair of DORA, speaking at a Leeds Open Lunch event in September 2021.
Professor Amanda Bretman, Dean for Research Quality
Claire Knowles, Associate Director: Research and Digital Futures, University of Leeds Libraries
Responsible Research Metrics Group at Leeds Co-chairs