To celebrate Peer Review Week, representatives from ORCID, ScienceOpen, Sense About Science, and Wiley got together (virtually) to discuss why peer review matters – to them and to their organizations – and what they hope this week will achieve.
Many thanks to participants Stephanie Dawson (CEO, ScienceOpen), Chris Graf (New Business Director, Professional Innovations for Associations and Societies, Wiley), Peter Gregory (Editor-in-Chief Advanced Materials, VP and Publishing Director, Wiley), Laure Haak (Executive Director, ORCID), Emily Jesper (Assistant Director, Sense about Science)
.jpg)
(left to right: Stephanie Dawson, Chris Graf, Peter Gregory, Laure Haak, Emily Jesper)
Stephanie Dawson (SD): When I started off as a journal manager at De Gruyter publishers, I used to receive three copies of each manuscript, which I put in envelopes and sent to Japan or Argentina by post and hoped that the chosen peer reviewer would have time to review. Much has changed since then and much not. Some reviewers did an extremely conscientious job in helping authors improve their papers, others sent a rejection in three lines. I really saw firsthand how heterogeneous the process could be and wished I could personally thank each reviewer who put in the effort to really help their colleagues.
With the research and publishing network ScienceOpen, one of our distinct roles besides a Gold Open Access (OA) publisher and content aggregator, is that of a peer review reformer. Our goal is to augment trust in the peer review process by making it entirely transparent. Then everyone can see (and even thank) those reviewers who are doing an exceptional job to improve the quality of scientific communication.
We’ve deployed a novel workflow to demonstrate the efficacy of a different approach to Peer Review and its suitability for the digital future of scholarly communication which we believe will need to speed up and involve the publication of many more digital objects than the current single article unit.
Chris Graf (CG): Speaking of novel workflows, I’m personally interested in how pre- and post-publication peer review in combination will make a better system.
Pre- publication peer review validates and organizes research, and in the main it does a great job. But – let’s be honest – sometimes gets it spectacularly wrong. And that’s where post-publication peer review comes in.
Post-publication peer review can help contextualise, curate, and – excitingly for me – perhaps also gives more accessible entry point for readers (eg, comments in PubPeer, PubMed Commons, metrics and more in Kudos, provide a steer for readers and help surface articles). And – yes – post-publication peer review weeds-out the occasional bad apple. Which is important.
Last, we need to stop and think – and have a public discussion – about whether we should view a peer reviewed article as absolute, definitive, final and 100% correct just because it’s peer reviewed. Might it be more realistic to think about a peer reviewed article as one step on a (fairly tortuous) research journey, upon which we can expect to make missteps, take wrong turns, stumble, and take new directions before we reach our final destination: The Answer. Many steps make up that journey, and many pieces of research make the evidence we use to inform practice and policy.
Laure Haak (LH): I completely agree with your last point but, in terms of validation, is that really what peer review does? Is it supposed to be the filter that says, this paper used the right methodology and statistics (in the case of a science paper) or are we really looking for the peer reviewer to assess the logical flow of the paper? At the end of the day, whose responsibilty is it if the paper is found to be “fraudulent”? Can we really expect/do we want that weight to be on the reviewer?
Emily Jesper (EJ): And to Chris’s point about peer review’s role in informing practice and policy, of course peer review is not just something of significance to scientific researchers. Because it indicates that research has been scrutinized by independent experts in the field, peer review is an important consideration for policy makers, reporters and the public when weighing up research claims and debates about science. Understanding the status of research claims is empowering. It helps us weigh up claims and use evidence to make decisions. Since Sense About Science was set up in 2002, we have been working to popularize an understanding of peer review more widely. Our public guide to peer review I Don’t Know What to Believe encourages people to ask “Is it peer reviewed?” when weighing up claims about science.
Peter Gregory (PSG): Although you’re right that peer review does have a wider significance, it matters to Wiley primarily because it matters to the scholarly community which we serve. In addition to gathering opinion and improving contributions peer review also provides, especially in the sciences and medicine, a safety valve, preventing the distribution of incorrect or even dangerous erroneous articles.
LH: Yes, peer review is a core component of scholarly practice. It is an established set of methods to gather commentary on scholarly works (papers, books, grants, programs, and more) from peers — experts in the field — with the end goal of improving the work. ORCID is interested in peer review because people are involved, both creators of works and the reviewers. We are working with the community to develop digital methods to acknowledge the contributions of reviewers by citing peer review activities. Through this we are hoping to encourage scholars to participate as reviewers, and also to support others who are working on issues of trust in the review process.
SD: I love the idea of making ORCID the place where researchers can log their reviewing activities. ScienceOpen is trying to address this with open reviews that have a registered CrossRef DOI. Publons is also doing a nice job of this as well as supporting reviewers in blind peer review journals. In cases of blind review they are verifying with publishers that a reviewer has indeed done the work that they claim – right now it would be easy for a researcher to add “review for Nature and Cell” to their CV because there is no way to verify this in our current blind review system. Again trust and transparency are key.
PSG: I hope that this Peer Review Week will help intensify discussion of peer review and start sensitizing participants (publishers, authors, editors, referees, funders etc) to the differences in the ways peer review is applied and is useful in different communities. Some areas collect opinions, some are more fact based; some benefit from preprint circulation (prior to peer review and publication i mean) some are damaged by it; for some post publication review is innovative, for some it is dangerous. This leads me to the conclusion that those working to improve/change peer review should carefully consider that one size does not fit all.
SD: I agree that there will not be a “one size fits all” solution but different communities can also learn from each other, especially as an increase in interdisciplinary research draws them together. One great example is the life sciences preprint server BioRxiv which builds on the physics community’s positive preprint experience with arXiv. In the new field of bioinformatics, the computer scientists just put their preprints on arXiv as a matter of fact, and the researcher’s coming more from the life sciences asked “what’s that?” So something really interesting my grow up just at those points of friction. Also the new trend towards megajournals such as PLoS One, SpringerOpen, Scientific Reports, requires rethinking peer review to fit a wide range of communities and all of those journals have come down to asking reviewers to just review the scientific soundness of results. So this may give us some clues to what the common denominator of peer review may be moving forward.
LH: The idea of a common denominator is really key and is exactly the sort of discussion I hope Peer Review Week will generate – stimulating and concentrating the ongoing conversation on peer review and, in so doing, bringing forth what is being done in practice to support improvements.
SD: From ScienceOpen’s perspective, this Peer Review Week, we want to pose a simple question. As the number of research outputs grow and diversify (data sets, negative results, case reports, preprints, posters…) is the research community going to be able to peer-review all these objects prior to publication? We think not. There isn’t enough time in the day, money to pay for it or even appetite for doing this now. Will these outputs be useful none-the-less? Absolutely, if we have a powerful way to find and filter them based on parameters readers find helpful and authors find rewarding. For example:
- What do my peers think of this information?
- Are there any updates to it?
- What impact did it make in the world and who noticed?
- Which work is worth highlighting in a specific field?
- How many times was it cited and where?
- If I took the time to review it, can my contribution be found and cited?
- Will these efforts enhance my career prospects?
- How many times was it cited and where?
None of these valid questions are impacted by an evolution away from blind or double-blind anonymous peer review, apart from the speed with which we can answer them. Transparent processes and simple web tools can filter faster, better and cheaper than journals and pre-publication peer review ever could.
This is why at ScienceOpen we’ve developed systems for Post-Publication Peer Review; Versioning; DOI allocation; Article Metrics; Collections; Open Citation Information and more – to demonstrate a different (and we would argue better) way forwards.
LH: I’m not sure I agree that peer review is unsustainable. I am thinking that if we, the community, provide guidance on what it means to peer review certain types of content, then we may be well on our way to supporting review of a broader set of outputs. We need to address questions such as, who is responsible for content? What is the role of the author/advisor/employer/editor/reviewer? This is fundamental. Some of this “new” content, such as datasets, may require a reviewer to check whether the metadata is sufficiently descriptive and data deposit/access/long-term preservation is accounted for as per the data management plan. And although I like your suggestions for finding and filtering research outputs, none of this is relevant if the (e.g.) dataset is sitting on a datastick in someone’s drawer. A peer reviewer may be a critical component of ensuring that datasets are preserved and the access pathway described in a persistent manner so that it is citable. The same goes for other types of outputs.
SD: There are a lot of publishers and organizations out there who are rethinking what peer review should really accomplish and creating new services to support that – sometimes even too many from the view of the scholar who may feel overwhelmed by new definitions and choices. I also think that it is important during this Peer Review Week to demonstrate how innovations in this field are really built to support reviewers and the fantastic work that they are doing. So I hope this week we’ll see acknowledgement of the new models of Peer Review that exist today, being demonstrated by small start-up organizations, such as ScienceOpen and The Winnower and others such as F1000Research and PeerJ. And that discussions about Peer Review have already moved beyond interesting conversations between academics to involve software developers and publishers working in mostly commercial entities which are good at making innovation a reality.
However, the open question remains “can the scientific community embrace these new opportunities because where a researcher publishes is still coupled to the impact of the research outputs themselves in terms of career progression?”. Breaking the grip of the Impact Factor is no easy feat. But any new system will have to meet the criteria of trust and transparency.
LH: I don’t think the issue is the impact factor. I think the point of this week is to embrace Peer Review in all its diversity, and work to understand its core function in scholarly discourse. To Chris’ point above, peer review is part of a journey. Different disciplines, different topics will benefit from different flavors, all of them good. Let’s focus on the process itself, and not so much on the container (e.g., journal/service platform).
EJ: And let’s also not forget that the public needs to know about peer review and so scientists and publishers should talk about it. Peer Review Week is a great opportunity for researchers to stop and think about the wider role that peer review plays in society in maintaining quality in science and to share that more widely. I especially hope we’ll hear from early career researchers about why peer review matters to them and give greater recognition to this vital, voluntary role that researchers play. It will also give researchers a chance to hear about the fantastic resources, workshops and training opportunities available on peer review.
