In May 2018, we interviewed Matthias Mittner [Associate Professor, Research group for cognitive neuroscience, The Arctic University of Norway] about his experience in submitting a Stage 1 Registered Report to the European Journal of Neuroscience.
Now that the Stage 2 Registered Report has been published, we catch-up again with Matthias to learn how the process went.
Q. Following the advice you received in your initial Stage 1 Registered Report, was it straightforward to execute your experiments as planned?
A. Given the extensive work that went into planning and preparing the study and analyses, the actual execution of the work was very straightforward. We had been going back and forth about various aspects of the study beforehand and tried to anticipate and eliminate potential problems. I think we were much more conscious about that given that we knew we would not be allowed to change the protocol once the study commenced. The feedback we received during Stage 1 certainly helped to streamline our plan and make it more feasible.
Despite all this planning, we did notice some (luckily minor) problems during study execution and analysis that required us to change some aspects of our protocol. Fortunately, we were able to spot those problems when rolling out the study to the participating labs who conducted their own pilots according to our plan, so the changes could be made before the study properly commenced. We considered the changes to be too minor to report them to the journal (or ask for more review) and, instead, collected these adjustments in an appendix. As part of our pre-registration on the Open Science Framework (OSF), we also pre-registered all statistical analyses scripts. When trying to run those unchanged, we realized that several packages had changed during the two years or so that it took us to complete the study and we therefore had to change the scripts to match the new syntax. We handled this issue by uploading so-called “patches” to our preregistration in addition to detailing them in the appendix. All-in-all, execution of the study was straightforward, and we did not encounter any major issues.
Q. How did you find the Stage 2 peer review process?
A. The peer-review at the second stage was quite fast and professional. It was a very good experience to see how our results were treated by experts that had been involved in the whole process of the study. It felt almost like getting feedback from a well-informed co-author because we had been through this extensive Stage 1 peer review round with the same peer reviewers. It was also appropriate that our results or design choices were not challenged (of course, since they were approved at Stage 1) and that the discussion focused wholly on how the results were put into context and which conclusions could be appropriately drawn. Finally, we were very happy to see that the reviews were actually published alongside the final article and that we got to know the names of the reviewers. This is fantastic as it allows everyone to appreciate the full research process, not just the final outcome!
Q. How did the whole end-to-end Registered Report research process compare with your previous experience of regular research and publication?
A. The process was quite different from the process I had been through in other projects. Often, the planning part of a study is constrained by various issues. Perhaps, a PhD student needs to start data-collection in order to finish a project before handing in her thesis. Perhaps funding is only available for a certain time. Such constraints can sometimes lead to taking shortcuts when data collection has to commence in a hurry. For this Registered Report, it was clear from the outset that there could be no compromise at the planning stage. I think that was mostly because we were submitting a manuscript for publication before data-collection. All the effort that goes into preparing the final research paper was invested before data-collection and that meant that we had to take planning much more seriously than we might have otherwise. Also, we had to be open to making changes to our carefully planned- out protocol following peer review. In many ways, acceptance at Stage 1 felt like the real success!
Of course, as a consequence, the planning process took much longer than any other study I had previously conducted and it was much more uncertain whether we would actually conduct the study (since it depended on acceptance at Stage 1). Once our paper was accepted at Stage 1 though, the remainder of the study ran smoothly. It was easy to plan and conduct alongside other ongoing projects. I also fondly remember the day when the last dataset had trickled in and we were pushing the button to run the analysis script for the first time - opening a can of beer in celebration!
Chris Chambers, Registered Report Editor, European Journal of Neuroscience and Professor, Head of Brain Stimulation, School of Psychology, Cardiff University, was very encouraged by the publishing experience and invaluable feedback offered by the team:
“The Registered Report published by Mittner and colleagues is a fine exemplar of the format, and it’s encouraging as an editor to hear that the authors found the process positive and rewarding. I fully agree with Matthias about the value of a cross-journal, machine-readable Registered Report badge, in addition to the current designation as a specific Registered Report article type. One of the emerging challenges of Registered Reports is ensuring that they are readily discoverable and identifiable. This is useful for authors — ensuring that their Registered Reports are distinguished as clearly as possible— and is also valuable for meta-researchers who are examining how Registered Reports differ from regular research articles."
Q. Would you consider the Registered Report format for your future research projects?
A. I will most definitely continue to use this excellent format for many of my future projects. We are, in fact, in the planning stages of our next effort.
Q. And would you recommend the initiative to others?
A. I most definitely would, and I often do - both at the coffee-maker and in more formal settings.
Q. How would you feel about having a pre-registered open research badge on your article?
A. That would certainly help to increase “distinguishability” of pre-registered reports from other article types. Since running this project, I have started looking at the published literature with different eyes. I have become more critical and require more evidence before I am willing to accept a claim in a research finding. I believe that results from pre-registered reports are more trustworthy than other formats and I therefore weigh evidence from such studies more strongly than others. Having a badge clearly indicating that a paper is a pre-registered report would help me get to the important studies more quickly.
About the AuthorMore Content by Jackie Jones