Taking Steps Towards Higher Quality in Peer Review: Our Better Peer Review Self-Assessment

September 18, 2019 Michael Willis

 “The man who moves a mountain begins by carrying away small stones.” Thus wrote Confucius in his Analects about the value of continuous improvement, the philosophy underpinning our “Better Peer Review” work.

The story so far…

Is there a ‘gold standard’ for quality in peer review? By what measures should we assess quality in peer review? What aspects of peer review should we focus on?

It was just about two years ago that we went public with our embryonic work exploring better peer review in seeking to answer these questions. The case studies we solicited at the time, along with a literature review, metamorphosed into a preprint and a peer reviewed article, and from this we generated a self-assessment tool which journals can use to measure, reflect on and improve their peer review performance and processes. We describe the self-assessment in detail on The Wiley Network and encourage as many journals as possible to make use of it. If you’re an editor or publisher of a journal, regardless of publisher, business model or subject area, please feel free to conduct the self-assessment on your journal.

We shared more information about the self-assessment at this year’s 6th World Conference on Research Integrity in Hong Kong, where a panel of experts, representing the perspective of editors, authors and reviewers, shared their views on the five essential areas of better peer review that we have identified: integrity, ethics, fairness, usefulness and timeliness.

What Have We Learned?

To date, 132 journals have conducted the self-assessment across all main subject areas (Figure 1). In time for this year’s Peer Review Week, we have now published our findings based on the completed self-assessments in a preprint and an interactive infographic.

Figure 1: Distribution by subject area of journals which have completed the self-assessment

We hope it’s no surprise that many journals have responded positively to the opportunity presented by the self-assessment to examine their practices. It encourages them to familiarize themselves with aspects of the peer review process that they may be unaware of. There’s a sense of ‘I had no idea this is what we do,’ or ‘I wish we could tighten up this bit of the process.’ And it asks them to reflect on aspects of their editorial policies that they may not otherwise have space and time to do.

The self-assessment asks journals to rate their performance and to explain their reasons for the rating. Examining these qualitative responses, we have identified several obstacles to good practice. These include:

  • Lack of technical knowledge or awareness of a process for adopting good practice
    • Example: not being familiar with readily available technological solutions
  • Inconsistency in the way a process has been applied
    • Example: asking authors to comply with reporting guidelines but not asking reviewers to assess manuscripts against these
  • Fear of additional workload incurred by implementing new policies
    • Example: not offering authors the opportunity to appeal against decisions
  • Fear of being exposed for having bad practice
    • Example: not encouraging feedback on the review process from authors and reviewers

We also found that journals tend to assess their own performance more positively than is perhaps warranted – although they may be doing good work, there is always a ‘better.’ These may not require much investment of time and effort, but there are opportunities nevertheless. And, journals should focus on ensuring transparency and consistency in the guidelines and processes they provide to authors, editors, and reviewers.

What’s Next?

The self-assessment is the beginning of a journey rather than an end in and of itself. It encourages journals to identify strengths and areas for improvement and to receive feedback on how to make adjustments to their processes. But why stop there? Some journals have already turned the results into a specific topic for discussion at editorial board meetings, including, for example, exploring how to make the editorial board more diverse. Other journals are considering providing informal or formal training for editors and reviewers. We also recommend that all journals repeat the exercise after, say, six months or one year, to assess their progress towards better peer review.

Remember: as we aspire to improve peer review processes and quality, it’s all about taking one step at a time, or – as Confucius would say – carrying away the small stones.


 

Read the preprint here.

And, view the interactive infographic of our findings here.

If you’re interested in conducting the self-assessment on your journal, check out our Better Peer Review page or email us at betterpeerreview@wiley.com.

 

 

About the Author

Michael Willis

Senior Manager, Content Review // Michael Willis, EMEA Regional Manager of Peer Review, represents Wiley to the PEERE Action. He can be reached at miwillis@wiley.com.

More Content by Michael Willis
Previous Article
How the AGU Is Connecting Members With Communities to Advance Research
How the AGU Is Connecting Members With Communities to Advance Research

Chris McEntee of the American Geophysical Union explains how members are supported through special projects...

Next Article
How Altmetrics Can Help Demonstrate Impact and Support Open Research
How Altmetrics Can Help Demonstrate Impact and Support Open Research

Learn two easy ways tools from Altmetric can help track and promote the online attention around published r...