Peer review is central to many key research workflows: publications, grant applications, promotion and tenure applications, conference submissions, and more. We are delighted that ORCID member Royal Society Te Apārangi is planning to use ORCID to recognize peer review service for Marsden Fund panel reviewers. Learn more about their progress in this interview with Jason Gush, their Programme Manager for Insights & Evaluation, and watch for more updates in the coming months.
First, please can you tell us a bit about the Royal Society Te Apārangi and the Marsden Fund, and your current integration of ORCID via the New Zealand ORCID Hub?
The Royal Society Te Apārangi is a 150-year old non-governmental organization empowered under an Act of Parliament and responsible for supporting and encouraging scholarship in the sciences and humanities, and encouraging an appreciation and awareness of the same in the New Zealand public. As part of these responsibilities, we’re the home for New Zealand’s national academy, distribute medals and awards for research excellence, administer the New Zealand Journal titles, and act as a fund administrator for government, as well as carrying out a range of promotion, education, and expert advice activities.
The Marsden Fund is the largest of the funds we administer and has been in operation for 24 years. It is now somewhat novel, being both entirely for investigator-initiated research and covering the full gamut of research from the humanities through social sciences, the life sciences, physical sciences, to mathematics and the information sciences. With an annual funding round and success rates typically around 10%, getting a Marsden grant is regarded as carrying a fair degree of prestige.
The Society has had the pleasure of being the lead agency for the New Zealand ORCID Consortium since the Consortium’s launch in 2016. Supported by our government, the consortium is ORCID’s most organizationally diverse and has a goal of representing all of New Zealand’s publicly-funded researchers. The Consortium also supported the development of the New Zealand ORCID Hub to enable our diverse members to interact with ORCID. The Hub is a web application with a simple user interface that allows organizations to read from and write to ORCID records with the record holder’s permission. At present, we are using the Hub to assert affiliations for our staff, funding for grant holders, and have just started looking into how we can properly represent the peer review that’s integral to so much of our operation.
What sort of technical work did the Society undertake to enable peer review recognition in the Hub?
Peer review represented a simple extension of the Hub’s functionality. The main complication was the creation of tools to manage the group id referenced in an ORCID peer review. We’re very fortunate to have an able and dedicated development team at the University of Auckland. Together, Radomirs Cirskis and Roshan Pawar got this up and running with v4 of the Hub which launched in May.
We’re excited that the Royal Society Te Apārangi is the first ORCID member to recognize review contributions for funding. What sort of reviews will be recognized?
The Fund sees around 1,200 expressions of interest a year across a broad range of disciplines. To make assessment practical, the Fund is structured into 10 discipline-based panels. Each panel assesses approximately 120 expressions of interest and selects around 24 to invite to submit a full proposal. Those 24 proposals are sent to a target of three (almost exclusively) international referees for comment. The panels meet again to consider the proposals, referee comments, and the investigator rebuttals, to select the 12 that will be successful.
With referees currently anonymous, the review contributions that we are most interested and able to assert is the service performed by the panellists.
By making peer review recognition available to Marsden Fund Panelists, what are you hoping to accomplish, and what challenges have you faced?
Peer review is such an important part of our processes, so we’re after a clear, clean, and authoritative way of unambiguously asserting that a particular individual has given this service. We’re hoping this is of value to panelists. Given that we’re often approached to confirm that they have served on a panel for other assessment processes, we want to give panelists the ability to share this information themselves.
The socialization of asserting what is already public information therefore is expected to be trivial. Instead, our biggest challenge has been attempting to fit this role into ORCID’s model representation of peer review. Conversely, that model would fit relatively nicely were the Fund, or indeed any of the Society’s processes, to offer referees the opportunity to be open about their identity; however, that is definitely not the case at present.
What approaches did you consider, for example, in terms of recognizing peer reviews versus service affiliation?
Our initial thoughts were to assert these roles in the peer review section of the panelist’s ORCID record, both because we thought we could make it work and, with v2.1 of the ORCID API, this was the only game in town. However, getting to grips with the peer review model showed that this wasn’t really suited to what we wanted to assert, i.e. where an identifiable work is the subject of an identifiable review. After discussions with the ORCID team, we’ve decided instead to wait for v3.0’s service affiliations.
What has the reaction been so far from researchers about the option to have their review work for the Society recognized in their ORCID record?
It is too soon to say as it’s still early days yet. Part of pursuing this was seeking the approval of the Marsden Fund Council which governs the fund, and they have been supportive of both this activity and of funding assertions in ORCID.
What challenges did being the first to work on recognizing peer review in funding pose, and what advice would you give other funders that would like to link and recognize peer review?
Peer review is the newest of the sections of the ORCID record, and, at least in v2.1, is solidly geared around the concept of a reviewer composing a review for a subject on behalf of a review group. As a funder, the challenges were that the subject must be one of ORCID’s work types, while review groups can be one of: publisher; institution; journal; conference; newspaper; newsletter; magazine; or peer review service. Neither really suits a funding organization, where the subject would not be a work but a grant, proposal, or application. The fact that so much of contestable funding review is blinded also makes the strict application of ORCID’s peer review model impractical for us at the moment.
If the Society moved toward open review, or at least more open than currently, the peer review approach would be worth revisiting. For other funders, once ORCID has subject types and organization types which fit, then it is definitely possible if they’re practicing open review; however, service affiliations are looking to be a much more universally applicable approach in the interim.
Looking ahead, is peer review recognition a part of the ORCID roadmap for other New Zealand members using the ORCID Hub?
All major public research funders in New Zealand are part of our consortium, and this is something that they can pursue using the Hub. I’d definitely hope to see this kind of recognition extended to our other funders given the value that we receive from peer reviewers’ service.