Demystifying Peer Review—And Finding the Gold Standards

January 1, 2018 Michael Willis

At the Wiley Editor Symposium held in London in December 2017 (a Wiley event which brings groups of editors together in person), I presented a poster (see figure below) describing a project I am currently undertaking with a group of colleagues. You can read more about the project in our article on the Wiley Network. The project seeks to define a ‘gold standard’ in peer review, and we have identified five key elements: integrity, ethics, fairness, usefulness, and timeliness. They encompass both the function of peer review—what does peer review try to achieve?—and the process of peer review—how is it done? Based in part on a set of case studies describing good practice in peer review at a cross-section of journals, we’ve defined a set of criteria for each element, against which the peer review of a given journal can be measured. We aim to describe a spectrum of practice, from good to excellent, and we want to develop resources to equip editors to understand where their journals currently sit on the spectrum, and how they can move their journals along the spectrum in the direction of better practice.

I found it energizing to discuss the poster with Editors and receive their feedback, ranging from wholehearted endorsement of the project to careful thinking about how the criteria might apply to specific subject areas. The gold standards have no value if they cannot be applied to all disciplines, so it’s important that the team re-evaluates and refines the criteria.

In the meantime, I’d encourage you, as Editors, to reflect on and critique the way peer review is practiced at your journal. Here are two examples which I hope will get you thinking.

At the symposium I spoke in a clinic session about ‘demystifying’ the peer review process for authors. A major pain point for authors is the time that peer review can take, but the pain is excruciating if the process is long and the outcome is rejection with next to no constructive feedback and no interaction with the journal along the way. Overcoming this is an ‘easy win’—and may well help ensure that the author sends future papers to the journal—if an Editor explains in detail why the paper has been rejected, as well as exploring strategies to reduce the review time for future papers. One case study submitted to us reported a letter of appreciation and gratitude from an author of a rejected paper. The paper had gone through two rounds of review before being finally rejected. At the end of the process the Editors sent a personalized letter to the author including all the feedback from the reviewers. The author’s assessment of the process was that it had been fair and useful, even though the paper was rejected. It’s quite likely that the author will send future papers to the journal and will recommend the journal to colleagues. High standards of peer review can happily co-exist with high editorial and academic standards.

The second example is a psychology journal which, like many journals in the field, operates a system of double-blind peer review. This is intended to reduce bias on the basis that reviewers are not influenced by the authors' reputation when they are assessing the value and rigor of the research. But with an increasing trend towards pre-registration of clinical trials in a public database, to improve the integrity and transparency of the trial research process, the ability to maintain double-blinding in the review process is becoming nigh-impossible. This journal’s editors faced a dilemma: which should take precedence, double-blinding review or pre-registration of trials? They decided that transparency and integrity in the conduct of trials should take priority.

Particularly helpful in both cases is that the Editors have made deliberate, thoughtful decisions to proceed in the way described. It’s not always easy to reach decisions about good practice, and indeed it may require additional effort; but the pay-off is potentially enormous.

Are there examples of good practice in peer review at your journal that you could share with us to shape the gold standards? Send them to us via editors@wiley.com.

A poster describing our peer review gold standards project.

About the Author

Michael Willis

Senior Manager, Content Review // Michael Willis, EMEA Regional Manager of Peer Review, represents Wiley to the PEERE Action. He can be reached at miwillis@wiley.com.

More Content by Michael Willis
Previous Article
Gender Diversity in Peer Review
Gender Diversity in Peer Review

An investigation into gender balance amongst Wiley authors and reviewers.

Next Article
New Tools and Classic Techniques: Top Tips for Finding Quality Reviewers
New Tools and Classic Techniques: Top Tips for Finding Quality Reviewers

Finding rigorous reviewers can be a challenge. Read on to learn some top techniques that make it easier.