REPRISE reporting guidelines

Date: 27 March 2020

College of Optometrists reports

Katherine Cowan, Senior Adviser to the JLA, talks about PSPs and the REPRISE reporting guidelines

Priority setting is not always well reported

In a world of complicated health conditions and limited resources, the prioritisation of topics for health research is vital. Prioritisation exercises help ensure that funding is directed towards the most important unmet needs. But how do we know if a prioritisation exercise is legitimate and that the resulting priorities are credible? We can only make that judgement if things are reported transparently and comprehensively – and unfortunately for the world of research priority setting, historically this hasn’t always happened.

This was a problem that Dr Allison Tong, from the University of Sydney School of Public Health, wanted to get to grips with, to improve the quality of reporting of health research priority setting, and to help grow the evidence base for the methods.

Allison Tong

Developing a solution to the problem

“After conducting systematic reviews of research priority setting studies, we found that overall, the reporting was very limited and variable,” Allison told us. “For example, we found that only 60% of research priority setting exercises in childhood chronic conditions described how they collected the initial research topics and the process of prioritisation. The idea of developing a reporting guideline came up in 2015 during a coffee break at the Cochrane Colloquium in Vienna.”

With an international team of colleagues, Allison set about reviewing how past priority setting exercises have been reported, with a view to identifying criteria to include in guidance for future reporting. Multiple frameworks, guidelines, and models for conducting, appraising, reporting or evaluating health research priority setting, and systematic reviews, and primary studies of research priority setting were reviewed. A list of reporting items was identified and piloted with a diverse range of 30 priority setting studies from the records retrieved.

A new reporting guideline

From this, the REporting guideline for PRIority SEtting of health research (REPRISE) was born. It includes 31 reporting items, with examples, that cover 10 domains (described in more detail below).

Allison is keen for this to make a real difference for both reporting and having an impact: “We hope that REPRISE will help to improve the transparency and legitimacy of research priority setting initiatives – to strengthen implementation so that resources and efforts are directed to research questions of high priority to patients, caregivers and health professionals. This could also provide an opportunity to compare and contrast methods used in research priority setting. Given the current climate, there is a need to innovate methods for urgent/rapid research priority setting for an immediate response to, for example, the COVID-19 global pandemic.”

An opportunity for JLA PSPs

So, what does this mean for the JLA? Well, Priority Setting Partnerships already have a great track record of reporting what they did and how their priorities were reached. The REPRISE study found the JLA to be one of the most reported established priority setting methods out there: “This was both in terms of frequency – JLA was the most commonly reported method – but also for quality and comprehensiveness,” adds Allison. There is always room for improvement though: any health research priority setting process risks legitimate criticism if the ‘how’, ‘who’, ‘why’ and ‘what’ isn’t totally transparent. We think that the REPRISE framework is a great resource for PSPs when planning their process, maintaining their records and disseminating their results. When I tweeted about it back in January 2020, I received some really positive feedback from PSP colleagues.

“JLA methods are systematic, comprehensive and robust, and allow for meaningful engagement with key stakeholders including patients,” says Allison. “We hope to receive feedback from JLA PSP leads who use REPRISE to report their study.”

A closer look at the framework through a PSP lens

So you can see what it could mean for PSPs, here’s a breakdown of each REPRISE reporting item as it relates to the JLA’s approach (spoiler alert: you’re collecting most of this information/data already, so it’s not going to generate loads more work or effort).

  REPRISE reporting item REPRISE descriptor and/or examples Implications for JLA PSPs
A Context and scope    
1 Define geographical scope Global, regional, national, city, local area, institutional/organizational level, health service This is described in the PSP Protocol.
2 Define health area, field, focus Disease or condition specific, interventions, healthcare delivery, health system This is described in the PSP Protocol.
3 Define the intended beneficiaries This may include the general population or a specific population based on demographic (age, gender), clinical (disease, condition), or other characteristics who may benefit from the research This is described in the PSP Protocol.
4 Define the target audience of the priorities Policy makers, funders, researchers, industry or others who have the potential to implement the priorities identified JLA asks PSPs to think about this when they complete the readiness questionnaire and consider it further when developing a dissemination plan.
5 Identify the research area Public health, health services research, clinical research, basic science This is described in the PSP Protocol.
6 Identify the type of research questions Etiology, diagnosis, prevention, treatment (interventions), prognosis, health services, psychosocial, behavioral and social science, economic evaluation, implementation; this may not be pre-defined This is initially described in the PSP Protocol and developed further during the data management process.
7 Define the time frame Interim, short-term, long-term priorities, plans to revise and update Not necessarily described in the Protocol but PSPs may want to consider this when developing their priorities into research questions and articulate this in their reporting.
B Governance and team    
8 Describe the selection and structure of the leadership and management team Those responsible for initiating, developing, and guiding the process for priority setting, and examples of structures include; Steering Committee, Advisory Group, Technical Experts This is a description of the set-up of the PSP’s coordinating team and the recruitment of the Steering Group and other supporting roles such as the Information Specialist.
9 Describe the characteristics of the team Stakeholder group or role, institutional affiliations, country or region, demographics (e.g. age sex), discipline, experience, expertise This would simply be a description of the make-up of the coordinating team and the Steering Group members.
10 Describe any training or experience relevant to conducting priority setting Consultants or advisers, members with experience or skills relevant to the conducting priority-setting e.g. qualitative methods, surveys, facilitation This would simply be a description of the make-up of the coordinating team and the Steering Group members.
C Framework for priority setting    
11 State the framework used (if any) James Lind Alliance, COHRED, CHNRI, Dialogue Model, no framework (general research priority setting) JLA!
D Stakeholders or participants    
12 Define the inclusion criteria for stakeholders involved in priority-setting Patients, caregivers, general community, health professionals, researchers, policy makers, non-governmental organizations, government, industry; specific groups including vulnerable and marginalized populations This is described in the PSP Protocol.
13 State the strategy or method for identifying and engaging stakeholders Partnership with organizations, social media, recruitment through hospitals This is described in the PSP Protocol and is further developed as Partners are recruited – this is generally captured in Steering Group meeting notes and communication strategies.
14 Indicate the number of participants and/or organizations involved Number of individuals and organizations, include number by stakeholder group A log of Partners should be maintained. JLA requires the capture of data on participants in both surveys and the priority setting workshop on its Engagement Summary.
15 Describe the characteristics of stakeholders Stakeholder group, demographic characteristics, areas of interest and expertise, discipline, affiliations Demographic details are captured for all survey respondents and summarised regularly for Steering Group members. An overview of the characteristics of workshop participants can be documented by the coordinating team.
16 State if reimbursement for participation was provided Cash, vouchers, certificates, acknowledgement; what purpose e.g. travel, accommodation, honorarium This may apply to workshop participants and can be easily described.
E Identification and collection of research priorities    
17 Describe methods for collecting initial priorities Methods e.g. Delphi survey, surveys, nominal group technique, interviews, focus groups, meetings, workshops; prioritization e.g. voting, ranking; mode e.g. face-to-face, online; may be informed by evidence e.g. systematic reviews, reviews of guidelines/other documents, health technology assessment This is a description of the process of collecting unanswered questions, which is generally an online survey, with additional methods employed for groups for whom a survey may not be appropriate (eg interviews, discussion groups). This is all agreed by and reported to the Steering Group, so is recorded.
18 Describe methods for collating and categorizing priorities Taxonomy or other framework used to organize, summarise, and aggregate topics or questions Categorisation of submitted uncertainties is overseen by the Information Specialist and reported to and scrutinised by the Steering Group, so should be carefully recorded.
19 Describe methods and reasons for modifying (removing, adding, reframing) priorities Based on scope, clarity, definition, duplication, other criteria This pertains to PSP data that is out-of-scope. Decisions about this will be justified by the Steering Group and recorded by the Information Specialist. No data is discarded so again, this is easily captured.
20 Describe methods for refining or translating priorities into research topics or questions Reviewed by Steering Committee or project team Turning the raw data into indicative questions is part of the Information Specialist role and is recorded and reported to the Steering Group. JLA requires PSPs to publish their priorities alongside examples of the original submissions behind them, for transparency about how the data was interpreted.
21 Describe methods for checking whether research questions or topics have been answered Systematic reviews, evidence mapping, consultation with experts JLA requires PSPs to publish their evidence search and checking strategy in a Question Verification Form.
22 Describe number of research questions or topics Number of priorities at each stage of the process This is all captured and is required by JLA in the Engagement Summary document.
F Prioritization of research topics/questions    
23 Describe methods and criteria for prioritizing research topics or questions Methods e.g. Delphi survey, surveys, nominal group technique, interviews, focus groups, meetings, workshops;
Prioritization e.g. voting, ranking;
Mode e.g. face-to-face, online;
Criteria e.g. need, feasibility, novelty, equity
Most PSPs do a shortlisting survey online, and potentially on paper or in group/in-person formats. This is all agreed and easily documented by the Steering Group.
24 State the method or threshold for excluding research topics/questions Thresholds for ranking scores, proportions, votes; other criteria Decisions about the number of priorities discussed at a PSP workshop will be based on logistics (ie what’s manageable for that group), and the top 10 is determined in order to generate a reasonable number of priorities that can be easily communicated. PSPs can note though that all their questions will be published, beyond the top 10.
G Output    
25 State the approach to formulating the research priorities Area, topic, questions, PICO (population, intervention, comparator, outcome) PSPs have an opportunity here to describe how they will translate their priorities into tractable research questions and propositions for funding.
H Evaluation and feedback    
26 Describe how the process of prioritization was evaluated Survey, workshop JLA offers to undertake anonymous evaluation surveys for workshop participants and Steering Group members. The results are returned to the PSP and can be used in reporting.
27 Describe how priorities were fed back to stakeholders and/or to the public; and how feedback (if received) was addressed and integrated Public meetings or workshop, newsletters, website, email, online presentations This would be a description of a PSP’s dissemination activities aimed at patients, carers and clinicians.
I Implementation    
28 Outline the strategy or action plans for implementing priorities Communication with target audience, via policies and funding PSP Steering Groups should be thinking about this during the process: who the audiences for the priorities are, what information they need to implement the priorities and how to reach them. This can be described.
29 Describe plans, strategies, or suggestions to evaluate impact Integration in decision-making, funding allocation, review of relevant documents Again, this may be something that Steering Groups build into their earlier discussions about dissemination and implementation.
J Funding and conflict of interest    
30 State sources of funding Name sources of funding for the priority-setting exercise; if relevant include the budget and/or cost PSPs publish this information in their Protocol and should be transparent about it in all their reporting.
31 Declare any conflicts or competing interests State any conflicts of interest that may be at an individual level and/or at a contextual level (e.g. political issues, controversies) that may affect the process, output or implementation. This is a standard reporting requirement, and worth noting that all Steering Group members (and workshop participants) are required to declare their interests at the start of the project.

Reporting guideline for priority setting of health research (REPRISE)  Allison Tong, Anneliese Synnot, Sally Crowe, Sophie Hill, Andrea Matus, Nicole Scholes-Robertson, Sandy Oliver, Katherine Cowan, Mona Nasser, Soumyadeep Bhaumik, Talia Gutman, Amanda Baumgart & Jonathan C. Craig.  BMC Medical Research Methodology, volume 19, Article number: 243 (2019).

  • Summary:
    How do we know if a prioritisation exercise is legitimate and that the resulting priorities are credible? We can only make that judgement if things are reported transparently and comprehensively – and unfortunately for the world of research priority setting, historically this hasn’t always happened
  • Year: