Mid-way through writing last week’s post about whistleblowing and whistleblowers we started a conversation with PubPeer to set up a trial of their alerting service. This service lets journals and publishers know when someone has commented on the PubPeer “online journal club” on one of the articles they’ve published. So we decided to ask Brandon Stell from PubPeer a few questions about the service, and its evolution.

Q. PubPeer… so where did you come from, and what are you all about?

A. PubPeer is a site for discussing any and all research published in scientific journals (and preprints). We make things as simple as possible - every publication has its own page and a text box into which anybody can type their comment and have it appear, either immediately or after moderation. A key and sometimes controversial feature of the site is that we allow users to comment anonymously. As an overall philosophy, at PubPeer we believe that there should be much, much more public discussion of research. Such discussion has the potential to accelerate greatly the clarification of ideas - compare commenting immediately on an internet discussion site with waiting to publish another paper and adding a few discreet lines to its Discussion section.

Q. Didn’t your anonymous commenting give you a reputation for encouraging “vigilantes”?

A. In some quarters, yes, that is the reputation our site has attracted. Allowing anonymity was not a step taken lightly, but we believed (and still believe) that the positive aspects of encouraging commentary and protecting commenters far outweigh the possible negative aspects of anonymity. We have developed commenting guidelines and moderation policies that minimize those negatives. We have written a blog post giving our reasons for allowing anonymity.

Q. Oh, so you’ve addressed that … tell us more about how you make sure all valid comments get through?

A. The key guideline is that comments must be verifiable (or falsifiable) by other readers. That means they must be based upon publicly available information - usually the content of the article being examined, but other articles or resources are also acceptable. Allegations of misconduct, hearsay and insults are forbidden. In other words, what counts on PubPeer is the content of the comment, not who made it. These rules are enforced by moderation and user reporting of abuse. Note however, that the site does not perform a scientific review, so comments can be factual but incorrect, although in our experience the overall accuracy of comments is exceptionally good. Authors receive alerts by email as soon as a comment is posted on their article and they have the possibility of replying with exactly the same prominence as the original comment. We believe that authors should provide an after-sales service for the publications.

Q. Great, so I can see why this would be a useful way for people who see a problem to let off steam, but what about connecting those comments with people who can act on them (like universities, and journal publishers)?

A. To be honest, one of the reasons PubPeer exists is because journals made discussing, commenting or refuting publications so difficult, while neither they nor institutions displayed much appetite for investigating possible misconduct. However, it is true that the situation is now changing and at least some journals and institutions are very proactive and rigorous in resolving questions about their publications. To facilitate this important work, PubPeer is creating "dashboards", a fee-based service available now for journals and soon for institutions. PubPeer journal Dashboards allow journals to keep track of and address comments on their articles with specialized search features, email alerts to new comments, features to track the evolution of commenting statistics for the journal, etc... and all features can be tried out for free:  https://pubpeer.com/journals

Q. Aha… I understand… so it sounds like you’re well on your way to providing a notification service for when whistles are blown (and also for when the discussion about a piece of research is carrying on, whether or not a whistle is being blown). That sounds great! What do you think the next steps are for PubPeer?

A. Our current focus is on getting institutional dashboards out. We'll probably then have a period of consolidation as we try to grow the network of actors using PubPeer.

Thanks, Brandon, for giving us a guided tour of PubPeer. We’ll see how we get on with your alerting service.