A leading UK university has become mired in a public dispute over how it is assessing researchers’ performance. The evolving situation at the University of Liverpool is being watched closely by concerned academics around the world — and is raising questions about whether more needs to be done to ensure that universities assess their researchers equitably. At the end of last month, the leaders of some of the world’s foremost responsible-research initiatives — the Hong Kong Principles, the INORMS Research Evaluation Group, the Leiden Manifesto and the Metric Tide — wrote a strongly worded letter arguing that the University of Liverpool’s proposals remain “squarely out of line with accepted practice”.
Liverpool wants to cut 32 posts from its Faculty of Health and Life Sciences. To keep their jobs, academics above the grade of lecturer need to demonstrate research income comparable with the average in their discipline for the 24-member Russell Group of research-intensive universities, to which Liverpool belongs. Candidates must also show a “substantial contribution” to two out of four additional categories — “world-leading” publications, commercial or consulting income, teaching, and research impact.
Many scientists are angry. Liverpool’s criteria for assessing its academics do not represent the reality of how research is done. They do not include contributions to peer review, PhD supervision, mentoring or collaborations. Instead, the university is putting weight on criteria that mirror those used in rankings and measurements of research performance — notably the UK’s Research Excellence Framework (REF).
When ranked according to the most recent REF results, from 2014, Liverpool’s position in some categories — for example, the clinical sciences — is below that of a number of other research-intensive universities, including the ‘Golden Triangle’ universities in Cambridge, Oxford and London.
If Liverpool chooses to replace what it sees as underperforming academics with those who have more substantial research profiles, it is likely to attain a higher REF score. That would come with more government research funding — but it would also come at a cost to careers. Moreover, there are aspects of the process that are clearly unfair. The university has said that deans, heads of departments and researchers who sit on external committees — for example, REF peer-review panels — will not be assessed for possible redundancy. In response to the threat of redundancies, researchers took industrial action during May, June and July.
One influential initiative is choosing to negotiate privately with the university. This is the organization behind the San Francisco Declaration on Research Assessment (DORA), an international voluntary agreement through which research organizations vow to conduct research assessment responsibly.
DORA’s signatories pledge not to use metrics such as the Journal Impact Factor to evaluate researchers, and to be transparent in the criteria used to make decisions on matters such as hiring and promotion. Liverpool is one of some 2,200 organizations that have signed the declaration. DORA is in talks with the university, but choosing not to reveal further details. A statement on DORA’s website says that it expects signatories to abide by their pledges, while also reiterating that it is not a regulatory body.
DORA’s approach — to resolve disputes constructively but without publicity — has had some effect. Liverpool initially included the field-weighted citation metric on its criteria for redundancies, but dropped that after consultation with DORA. However, there are conflicting views of whether this puts Liverpool in the clear. The university told Nature its amended criteria are “in keeping with the principles of DORA”. In response, a DORA spokesperson said there are “ongoing concerns”. Such mixed messages show the limitations of quiet diplomacy. DORA and the university should at least agree on their public communications.
If a university insists on a system of assessing its academics that seems both unfair and unrepresentative of how research is done, it begs the question: does the research community need a body, or a mechanism, with the remit and resources to monitor the declaration and act more like a watchdog? Has the Liverpool episode tested the limits of the ‘critical friendship’ approach?
These are important questions because, if Liverpool does not further change its position as a result of the dialogue with DORA, others who have signed DORA might think that they can take similar action. DORA might be able to take the edge off unfair practices, but its leadership and management comprise a small team — just 1.2 full-time-equivalent staff and 29 volunteers. In its current form, it is no match for the resources that universities bring when challenged.
To monitor the more than 2,000 organizations signed up to DORA would require a very different institution, and that, in turn, would pose new challenges. The costs and logistics of setting up such a body could run into seven-figure sums. Such funds would require the involvement of governments or philanthropists, and would pose other risks — for example, to academic freedom.
A compliance-style organization isn’t the only potential model, however. Another option is an approach that is used to recognize gender equality in universities. The Athena SWAN charter started in the United Kingdom and has since been adopted in several countries. Universities submit a self-assessment on their policies on inclusion and equality in hiring, promoting and retaining female staff. The assessments are judged independently, and institutions are awarded one of three grades: bronze, silver or (very rarely) gold. The initiative is funded by the institutions themselves, which pay into a common pool — as they would a membership fee. Each award lasts for a limited time, recently increased from four years to five, which ensures that institutions cannot rest on their laurels if they have achieved recognition once. Athena SWAN is not without its problems, but its underpinning principles could help universities to practise responsible research assessment.
DORA’s team is working tirelessly to persuade more institutions to sign the declaration, as well as trying to resolve the University of Liverpool situation. But in the wake of this episode, a rethink might be needed. And it must not be for a small team to do alone. Everyone who values responsible research — including Nature’s publisher, Springer Nature, a signatory to DORA — has a stake in ensuring that its principles are implemented.
Nature 595, 471-472 (2021)