September 10–15 was Peer Review Week 2018, during which a number of publishers and other organizations came together to celebrate the work of reviewers and the crucial role peer review plays in serving the scientific and scholarly literature. The theme of this year’s Peer Review Week was inclusion and diversity. You can read about some of the ways Wiley participated in Peer Review Week here.
One principle at the heart of research, and therefore at the heart of peer review, is impartiality. The scientific method is founded on the idea of the objectivity of the results, that the results would have been the same regardless of who conducted the experiment or observation. The same is true outside the natural sciences. An ad hominem argument is a logical fallacy; an argument is valid if its conclusion follows from its premises, regardless of who wrote it. Some non-scientific disciplines welcome subjectivity, with sub-disciplines actively engaged in considering differing perspectives, including feminist, womanist, and LGBTQI, to name but a few. Even within these subjective research paradigms, an author has the right to have their work assessed for publication on its merits, rather than any demographic identifier.
To what extent conscious and unconscious bias impacts review outcomes is a current research question. Guglielmi (2018) and Tamblyn et al. (2018) report evidence of gender bias in grant peer review. The evidence on gender bias in journal peer review is mixed. Fox et al. (2016) found no evidence of widespread gender bias in reviews but found male reviewers are less likely to respond to review invitations if the authors are female. Teplitskiy et al. (2018) found bias based on how closely connected the reviewer was to the author. Tomkins et al. (2017) found that single-blind peer review (i.e. where the reviewer knows the identity of the author) favors famous authors and authors from high-prestige institutions. Researchers from the PEERE project are evaluating data from one hundred Wiley journals for evidence of gender bias, with results to be published in due course. Regardless of how frequently it occurs, review bias on gender, ethnicity, status, or any other demographic identifier, is both unfair to authors and bad for research.
One way to ensure impartiality and inclusion in the research and scholarly literature is by ensuring the diversity of the reviewer pool. Even those journals operating double- (or triple-)blind peer review, which intuitively would remove some of the risks of reviewer bias, must consider what they lose if their reviewer pool is too monochromatic. It cannot be acceptable for those assessing research for publication to disproportionately represent a narrow demographic grouping. Ideally, one would want the reviewer pool to reflect the general population, and at the very least to reflect the diversity of the specific discipline it serves. Safeguarding the right for all voices to be heard in the literature requires the diversity of the reviewer pool. Research outcomes also benefit from diversity; Nielsen et al. (2017) have argued that there is a “gender-diversity dividend” in science. Furthermore, given that serving as a reviewer leads to other forms of engagement in the research process, including the opportunity to serve as an editor, it can only be right to ensure equal access to the opportunity to review.
Most Wiley journals do not collect the gender of authors or reviewers. However, it is possible to approximate gender identities using first name and country information. This was done for those reviewers for Wiley journals who have registered with Publons. Excluding those reviewers whose gender cannot be confidently determined (e.g. due to ambiguous names), the data shows that only one-third of Wiley reviewers are female (33.7%). There are few journals that have started to collect gender information of authors and reviewers as a way of tracking their gender diversity, and so I was able to obtain accurate gender data for only one journal’s reviewers. Interestingly the results here are comparable to the Wiley average: 29.3% female, 66.4% male, 0.6% other, and 3.7% “I’d rather not say”.
There are a number of factors involved in gender diversity in any area. In part, the underrepresentation of women in peer review may be a feature of their underrepresentation in certain disciplines. It is well known that women are underrepresented in many STEM fields, including Physics and Engineering. It has been estimated that at current rates it will take 170 years to achieve gender equality in STEM, despite the fact that STEM companies with strong gender diversity show 36% higher return on equity (Beijerinck, 2017). This situation is not unique to STEM. The journal referred to above is a philosophy journal. For this journal, the gender split for their reviewers is similar to the gender split for their authors: 26.6% female, 68.8% male, 1.2% other, and 3.4% “I’d rather not say”. So, in part, the question of gender diversity in peer review is part of a much wider question about gender diversity in all areas of scientific and scholarly research.
There is also regional variation. Excluding countries with fewer than 100 registered reviewers in Publons, those countries with the highest representation of female reviewers are Australia (46%), Romania (49%), Serbia (52%). The UK and the US are about the same with 38% female reviewers. There are countries at the other end of the spectrum with very low participation rates for female reviewers, perhaps reflecting broader cultural or socio-economic factors in those regions.
Journal editors also have a role to play here. The evidence is that editors demonstrate homophily when selecting reviewers, being more likely to select reviewers who are like them. Editors are more likely to select reviewers from the same region and country as themselves (Gaston and Smart, 2018) and are more likely to select reviewers who are the same gender as themselves (Helmer et al., 2017). The bias in reviewer selection is, we must assume, largely unconscious. We can speculate that one reason for this bias is the methodology editors adopt for selecting reviewers—which involves picking individuals from their own personal networks. Relying, even in part, on personal networks to select reviewers inevitably imports unconscious (and systemic) biases in the way we (as human beings) network, that is, we form networks with people like us (McPherson et al., 2001). Gender and regional diversity are not the only problems caused by a heavy reliance on personal networks. The recent Global State of Peer Review report found that 10% of reviewers perform 50% of all reviews (Peer 2018), suggesting that editors are depending disproportionately on a small pool of reviewers. This will have adverse implications for opportunities for early career researchers, and for timeliness in peer review—not to mention the workload of those 10%!
Editors have no need to rely solely on personal networks to identify reviewers. It is now easier than ever to use search tools to identify researchers with relevant expertise from their publication record. Since such searches are driven by taxonomy, they are blind to the demographic identities of the researchers, and thus likely to reveal a wide and diverse reviewer pool. While automated searches are still far from perfect, careful checking of the results can ensure the appropriateness of the researchers identified prior to selection and invitation. Within our Content Review department here at Wiley, we have journal administrators—without subject specialism—using such search tools to identify reviewers within the parameters set by the journal’s editor without any depreciation in the quality of review outcomes and with discernible improvement in reviewer selection timelines. Whether undertaken by the editor or by the journal administrator, adopting taxonomic and metric drive selection strategies will help improve the diversity of the reviewer pool. For example, American Geophysical Union (AGU), explored the role of gender in peer review in detail. They found that women do not review papers as often as would be anticipated given their participation as researchers in the field. Rather worryingly, this ‘gap’ seems to stem from the fact that researchers and editors do not suggest or invite women as reviewers as frequently as men. The AGU have put forward some recommendations to address this, including asking researchers to consider diversity when they suggest potential peer reviewers on submission. This simple step had a great impact in terms of increasing gender diversity of author-suggested reviewers, but collecting more data is key.
The future of peer review is better peer review, with impartiality at its core.
Helmer, Markus, Manuel Schottdorf, Andreas Neef, and Demian Battaglia. 2017. “Gender Bias in Scholarly Peer Review.” ELife 6:1–18.
Guglielmi, G. 2018. “Gender bias goes away when grant reviewers focus on the science.” Nature 554, 14-15. doi: 10.1038/d41586-018-01212-0
Gaston, T. and P. Smart. 2018, What influences the regional diversity of reviewers: A study of medical and agricultural/biological sciences journals. Learned Publishing 31: 189-197. doi:10.1002/leap.1155
Fox, Charles W., C. Sean Burns, Anna D. Muncy, and Jennifer A. Meyer. 2016. “Gender Differences in Patterns of Authorship Do Not Affect Peer Review Outcomes at an Ecology Journal.” Functional Ecology 30(1):126–39.
Beijerinck, Herman CW. 2017. “Gender diversity in STEM.” Europhysics News 48/2:16.
Tomkins, A., Min Zhang, and William D. Heavlin. 2017. “Reviewer bias in single- versus double-blind peer review”. PNAS 114(48) 12708-12713 https://doi.org/10.1073/pnas.1707323114
Teplitskiy, M., D. Acuna, A. Elamrani-Raoult, K. Körding, and J. Evans. 2018. “The sociology of scientific validity: How professional networks shape judgement in peer review.” Research Policy.
Tamblyn, R., Nadyne Girard, Christina J. Qian and James Hanley. 2018. “Assessment of potential bias in research grant peer review in Canada”. CMAJ 190(16) E489-E499; DOI: https://doi.org/10.1503/cmaj.170901
Peer, O. F. 2018. “Global State of Peer Review 2018 Who Is Doing the Review?” Publons.
Nielsen, Mathias Wullum, Sharla Alegria, Love Börjeson, Henry Etzkowitz, Holly J. Falk-Krzesinski, Aparna Joshi, Erin Leahey, Laurel Smith-Doerr, Anita Williams Woolley, and Londa Schiebinger. 2017. “Gender diversity leads to better science”. Proceedings of the National Academy of Sciences Feb 2017, 114 (8) 1740-1742; DOI: 10.1073/pnas.1700616114
McPherson, Miller, Lynn Smith-Lovin, and James M Cook. 2001. “Birds of a Feather: Homophily in Social Networks.” Annual Review of Sociology 27:1, 415-444
About the AuthorMore Content by Thomas Gaston