This is a guest post by Rupesh Paudyal, Funding Policy Lead, UK Research and Innovation (UKRI)
Demonstrating a Commitment to Peer Review
UK Research and Innovation (UKRI) has recently implemented the ORCID Reviewer Recognition feature in its funding system, Je-S. It provides UKRI reviewers with an option to get formal recognition for their review contribution displayed in their ORCID profiles.
UKRI was one of the world’s first funders to sign the ORCID Open Letter for funders, and is the first funder to implement ORCID Reviewer Recognition in its funding system. This initiative demonstrates our commitment to peer review, and implementing ORCID in our funding systems in accordance with ORCID best practice guidelines for funders.
Here, I’ll take you through our journey in implementing the ORCID Reviewer Recognition and share lessons learnt. In summary, the primary challenge in implementing this feature was not in system development, but rather in the careful considerations for data protection, policy, user and system requirements, and the process map.
Recognizing UKRI reviewers’ contribution without compromising our Principles of Assessment
UKRI values the effort that reviewers contribute towards assessing applications for us, which is an essential part of the funding process. We rely heavily on the time and expertise of reviewers to support our review process and help us make the best funding decisions. Each year, experts carry out tens of thousands of reviews for UKRI.
However, due to anonymity and confidentiality being two of the principles in the UKRI Principles of Assessment and Decision Making, the contribution of our expert reviewers is not formally recognized. Therefore, we are currently exploring several initiatives to better recognize, reward, and incentivize the contribution of UKRI assessors.
ORCID Reviewer Recognition provides us with a visible and verifiable way to publicly credit our reviewers for their work, without compromising the confidentiality and anonymity of the UKRI review process.
ORCID Reviewer Recognition can also act as an effective reference point for reviewers to keep track of the number of reviews that they have carried out for UKRI over time, which will be useful when it comes to applying for a promotion and funding.
Policy considerations
We started with policy questions deciding who should be eligible to get ORCID review credits. After several rounds of engagement with relevant stakeholders across UKRI, we decided ORCID review credits will only be available if:
- The reviewer submits a “usable review” via Je-S
- The review is submitted after 23 November 2020 (the date of implementation)
- A funding decision has been made on the application reviewed
This sounds simple, but there are several details that add to the complexity.
The feature is only available in the Je-S system from 23 November 2020. Therefore, only reviewers who submit reviews via Je-S after that date will be eligible to get ORCID review credits. UKRI is a relatively new organization with nine different Councils that evolved independently until they were brought together in 2018. Because of this, our Councils currently have separate funding systems: the seven Research Councils and other cross-cutting programmes use Je-S; while Innovate UK uses IFS.
As ORCID Reviewer Recognition is only implemented in Je-S, reviews submitted through Innovate UK’s IFS, historical reviews, and off-system reviews are not currently eligible to receive ORCID review credits from UKRI. However, we will review the potential benefits of widening this feature to the applications submitted to Innovate UK in the future.
Over 97% of reviews received between 2014-2018 via Je-S, were classified as “usable.” These are the reviews that are used in the assessment and decision-making process. However, reviewers are unaware of the classification status of their review in the UKRI system. Whilst we still don’t have capacity to inform individual reviewers of the status of their review, we’ve published the list of indicators that our staff use to classify reviews. We hope that this will act as a useful guide for our reviewers in submitting high-quality reviews. We’re also publishing why a review is likely to be classified as “unusable.”
One of the key policy considerations was to ensure that the confidentiality and anonymity of our review process isn’t compromised. Therefore, the transfer of data from Je-S to ORCID doesn’t include details of the individual application, the full date of the review, or the name of the UKRI Council that issued the review. Additionally, we will not issue ORCID review credits until a funding decision has been made on the application. Only the below information will be displayed in the ORCID records:
- Role: Reviewer
- Activity type: Review
- Review date: Year (e.g., 2020)
- Convening organization: UK Research and Innovation (Swindon, GB)
- Source: UKRI Systems
Data protection requirements
UKRI is the data controller for any information that the UKRI system sends to the ORCID profile of Je-S users. Therefore, we are sending the review information to ORCID on behalf of our reviewers, based on their consent. This means that our reviewers decide whether to grant or deny permission to UKRI to send review information to their ORCID account. Our reviewers should be able to turn on or turn off the permission at any time from their Je-S account. This gives our reviewers control to choose the reviews for which they receive an ORCID review credit. Our system will only send ORCID review credits to those reviewers that have explicitly granted permissions in their Je-S profile.
ORCID users have full control in managing the information displayed in their ORCID profile. As such, ORCID users can instantly delete or change the visibility of information from their ORCID profile. As data controller, UKRI also needs to be able to accept this request from ORCID users to delete UKRI-generated information from their ORCID profile. Therefore, we have developed an internal process to manage these requests, and we have advised our reviewers to email our Je-S help team to request any deletion of UKRI data from their ORCID account.
Process map and system development
Any information submitted through Je-S—our front-end system, where users interact with UKRI—is stored in the back-office, internal system, which is not accessible to the users. The internal system includes information on reviews submitted through Je-S. To send the review credits to ORCID, the process must start from the internal system.
We designed the process map so that once the criteria (listed under “Policy Considerations”) are met, the review credit is sent from our internal system to Je-S. Once the review credit is in Je-S, the review credit is only issued to ORCID, if the reviewer gives explicit permission for UKRI to send this review information to their ORCID profile. The reviewer must link their ORCID iD with their Je-S account to receive ORCID review credits.
One thing that we had not considered at the start of the project is the requirement from ORCID to display the review information to an external facing web page owned by UKRI. The idea was that the review activity, as displayed on ORCID, would be linked to the record of the review activity on a UKRI-owned web page. We realized that this requirement would significantly complicate the system development, data protection, and policy requirements. Our aim was for review recognition to be displayed through ORCID and not on the UKRI web page. Therefore, we requested ORCID to consider exempting UKRI from this requirement—and ORCID agreed!
Building of the system started after mapping out all these details. Below, you can see how the review information flows from UKRI systems to ORCID.
Implementation
Finally, implementation of the feature required the development of guidance and information for the members of the research and innovation community, including reviewers. As part of the implementation, we have developed web page content on ORCID Reviewer Recognition, including FAQs aimed at reviewers. Additionally, we also carried out an internal communication campaign within UKRI, creating internal FAQs and slide decks for our staff.
One of the most important things to consider is how we communicate these changes to our reviewers. To do this, we amended our review invite email to mention the option of getting review credits via ORCID and linking to the UKRI web page for more information. We also issued communication to all users who have connected their ORCID iDs to Je-S informing them of ORCID Reviewer Recognition before this feature was implemented.
Whilst the development took place, we noticed that ORCID had updated their API (the code that links ORCID with Je-S) since August 2017, and that the previous API did not have the functionality to send review credits to ORCID. This meant that our users, who linked their ORCID iD with their Je-S account before August 2017, would not be able to receive ORCID review credits. In order for these users to receive ORCID review credits, they would have to remove their current ORCID link with Je-S and reestablish this connection. Therefore, we designed a process to remove an ORCID connection from our system, which Je-S users can do at any time by emailing [email protected]. Additionally, we notified the affected users of this requirement several weeks before the feature was implemented.
Users were also notified via email once the ORCID Reviewer Recognition feature went live in our system.
Conclusion
In this blog post, I have tried to provide an overview of UKRI’s approach to implementing the ORCID Reviewer Recognition. UKRI is committed to implementing ORCID in accordance with the best practice for funders, and we are happy to share lessons learned with other funders who are considering implementing this feature. As I mentioned earlier, the most important things we had to consider in adopting this feature are policy decisions, data protection requirements, system requirements, process mapping, and finally, system building and implementation.
