“The man who moves a mountain begins by carrying away small stones.” Thus wrote Confucius in his Analects about the value of continuous improvement, the philosophy underpinning our “Better Peer Review” work.
The story so far…
Is there a ‘gold standard’ for quality in peer review? By what measures should we assess quality in peer review? What aspects of peer review should we focus on?
It was just about two years ago that we went public with our embryonic work exploring better peer review in seeking to answer these questions. The case studies we solicited at the time, along with a literature review, metamorphosed into a preprint and a peer reviewed article, and from this we generated a self-assessment tool which journals can use to measure, reflect on and improve their peer review performance and processes. We describe the self-assessment in detail on The Wiley Network and encourage as many journals as possible to make use of it. If you’re an editor or publisher of a journal, regardless of publisher, business model or subject area, please feel free to conduct the self-assessment on your journal.
We shared more information about the self-assessment at this year’s 6th World Conference on Research Integrity in Hong Kong, where a panel of experts, representing the perspective of editors, authors and reviewers, shared their views on the five essential areas of better peer review that we have identified: integrity, ethics, fairness, usefulness and timeliness.
What Have We Learned?
To date, 132 journals have conducted the self-assessment across all main subject areas (Figure 1). In time for this year’s Peer Review Week, we have now published our findings based on the completed self-assessments in a preprint and an interactive infographic.
Figure 1: Distribution by subject area of journals which have completed the self-assessment
We hope it’s no surprise that many journals have responded positively to the opportunity presented by the self-assessment to examine their practices. It encourages them to familiarize themselves with aspects of the peer review process that they may be unaware of. There’s a sense of ‘I had no idea this is what we do,’ or ‘I wish we could tighten up this bit of the process.’ And it asks them to reflect on aspects of their editorial policies that they may not otherwise have space and time to do.
The self-assessment asks journals to rate their performance and to explain their reasons for the rating. Examining these qualitative responses, we have identified several obstacles to good practice. These include:
- Lack of technical knowledge or awareness of a process for adopting good practice
- Example: not being familiar with readily available technological solutions
- Inconsistency in the way a process has been applied
- Example: asking authors to comply with reporting guidelines but not asking reviewers to assess manuscripts against these
- Fear of additional workload incurred by implementing new policies
- Example: not offering authors the opportunity to appeal against decisions
- Fear of being exposed for having bad practice
- Example: not encouraging feedback on the review process from authors and reviewers
We also found that journals tend to assess their own performance more positively than is perhaps warranted – although they may be doing good work, there is always a ‘better.’ These may not require much investment of time and effort, but there are opportunities nevertheless. And, journals should focus on ensuring transparency and consistency in the guidelines and processes they provide to authors, editors, and reviewers.
The self-assessment is the beginning of a journey rather than an end in and of itself. It encourages journals to identify strengths and areas for improvement and to receive feedback on how to make adjustments to their processes. But why stop there? Some journals have already turned the results into a specific topic for discussion at editorial board meetings, including, for example, exploring how to make the editorial board more diverse. Other journals are considering providing informal or formal training for editors and reviewers. We also recommend that all journals repeat the exercise after, say, six months or one year, to assess their progress towards better peer review.
Remember: as we aspire to improve peer review processes and quality, it’s all about taking one step at a time, or – as Confucius would say – carrying away the small stones.
About the AuthorMore Content by Michael Willis