Stage 5. Verify the uncertainties

Each uncertainty submitted, including indicative uncertainties of combined submissions as described above, needs to be verified as a true uncertainty.  For example, some uncertainties may have already been addressed by research without all patients or clinicians being aware of this.

To check that an uncertainty is a true uncertainty, a search needs to be undertaken for relevant and reliable systematic reviews and guidelines, alongside any ongoing studies which might address the uncertainty.  Large scale registry data may be available for the condition, which can be reviewed.  PSPs which extend their scope beyond ‘treatment uncertainties’ may need to identify a different strategy for evidence checking, and this should be transparent and agreed with the Steering Group and JLA Adviser.

PSPs will already have agreed and noted in their Protocol who will be responsible for verifying uncertainties.

It is suggested that at the same time as verifying the submissions, uncertainties from research recommendations are recorded and added to the list of uncertainties for prioritisation.

Treatment uncertainties submitted by patients and clinicians are identified as true uncertainties by checking them against existing systematic reviews and guidelines.  For each verified uncertainty, the source of verification should be cited (e.g. the systematic review identified as being relevant).

As a minimum, it is recommended that PSPs check the resources listed below to confirm whether or not a submission is an uncertainty:

  • The Cochrane Database of Systematic Reviews
  • NICE guidelines
  • SIGN clinical guidelines
  • Relevant Royal Colleges’ guidance.

When checking an uncertainty against a systematic review, the review needs to be relevant and reliable.  This can be measured by seeing if the authors follow a published methodology for undertaking the review, and if the methodology has made provision for managing bias.  When looking at guidelines, the author needs to have made efforts to identify all relevant and reliable trials or systematic reviews.  Reliability can be further ascertained from the confidence intervals around the main outcomes, enabling an informed reader to make an informed decision about the result.  Narrative reviews, which do not give details or numerical results, may fail the requirements of relevance and reliability.

The Steering Group may wish to discuss how it wishes to approach grading of reliability of evidence.  There are guidelines and ideas about this online, including at www.casp-uk.net/http://gradeing.cochrane.org/ or www.gradeworkinggroup.org/ .

An uncertainty is deemed genuine when a reported confidence interval in a systematic review crosses the line of no effect (the line of unity).  In addition to ascertaining statistical significance of an uncertainty, clinical significance needs to be checked by a clinician or person with relevant clinical knowledge who can confirm that the result obtained from investigating the uncertainty would be clinically relevant.

If a PSP finds that the number of systematic reviews which are relevant to its area is unmanageable within the constraints of its budget and timeframe, it may decide to create limitations.  It should be transparent about these. The size of the evidence base is something to be aware of when setting out the scope for the PSP.

Finally, some apparent uncertainties can in fact be resolved with reference to existing research evidence – ie they are ‘answered questions’ and not uncertainties.  If a question about treatment effects can be answered with existing information but this is not known, it suggests that information is not being communicated effectively to those who need it.  These findings may usefully inform future awareness-raising exercises and education programmes.  Accordingly, the JLA recommends strongly that PSPs keep a record of these 'answerable questions' and deals with them separately from the 'true uncertainties' considered during the research priority setting process.  The JLA suggests incorporating this commitment in the PSP’s Protocol.

Identifying research recommendations

The searched systematic reviews and guidelines may contain research recommendations, indicating uncertainty.  These may be reflected in the dataset already generated by the survey, or they may be unique.  The Steering Group will need to decide whether or not the latter are appropriate to include in the dataset.  Many PSPs have included them, so the questions can be considered at prioritisation.  Some have chosen not include questions that are methodological or too technical and inaccessible to a non-research audience.

When deciding what goes forward for prioritisation, and the associated workload, the PSP’s available resources will need to be taken into account.  It may be that a Steering Group decides not to include all the identified research recommendations, if the number is going to overwhelm the questions generated by patients, carers and clinicians. Steering Groups should be transparent about their decision-making in their reporting.

Ongoing trials and studies

Some PSPs have considered the presence of ongoing trials and studies when searching the literature.  An ongoing study may not mean that an uncertainty will definitely be addressed, and is therefore not worth prioritising.  Indeed, the JLA uses systematic reviews to identify certainty or uncertainty, rather than single trials.  However, identifying ongoing studies may help to avoid waste in research at a later stage by demonstrating that a priority may not need immediate action until a trial is complete and has reported its results.  It is suggested that Steering Group members and the JLA Adviser discuss this and document any decisions made as a result of information found.

Preparing uncertainties for prioritisation and publication on the JLA website

To manage the data throughout the PSP process, please use the data management template supplied by the JLA which includes the minimum data fields to be collected.  PSPs may choose to adapt the spreadsheet and add more fields, depending on how they wish to manage their data.  For example, if a PSP has identified uncertainties from published research recommendations, then it may be helpful to list them and link to the research recommendation in an extra field.

Publication of the completed data template on the JLA website forms a publicly available record of the work of the PSP and allows researchers and funders to see the original data to help them understand what informed the indicative questions.  It is an important part of the dissemination work of the PSP.

The following fields will need to be completed for every uncertainty:

  • Uncertainty - This is the indicative uncertainty, formatted using the PICO question structure where possible.  The advised minimum requirements are ‘Population’ and ‘Intervention’
  • Original uncertainty - This column will contain the original survey submission(s) which informed the indicative uncertainty.  If a number of survey submissions have been combined to form the indicative uncertainty, they can all be listed here, separated by ~.  This allows for transparency of interpretation, so the reader can see both what was originally submitted and the indicative uncertainty that was formed as a result
  • Evidence - This column will contain the evidence checked to ensure that the uncertainty is unanswered.  It should include reference to the most recent relevant systematic review identified by the PSP, plus a maximum of 2 other systematic reviews, including protocols for future systematic reviews, that the PSP considers relevant
  • Source of uncertainty - This column will show what type of people submitted the question.  If there are multiple sources, a PSP may wish to show them e.g. ‘1 x patient, 19 x clinician, 4 x research recommendations’.

For the uncertainties discussed at the final workshop (this is usually a maximum of 30), PSPs should show the following:

  • Final ranking decided on at the workshop for each question
  • An explanatory note for each uncertainty.  This should be a plain language summary of up to 150 words, explaining the key points about what the uncertainty is and why it is important.  PSPs may wish to include examples of the original survey submissions within the note. 

Following the final workshop the PSP should:

  • Update the data sheet to include any agreed changes made to the uncertainties during the final workshop (e.g. merging or rewording submissions)
  • Ensure that the final ranking and explanatory note is included against all of the uncertainties discussed at the final workshop
  • Send the completed data file to the JLA for publication on the JLA website.