Chapter 7: Data processing and verifying uncertainties

The detailed steps involved in the data management for a Priority Setting Partnership (PSP) are summarised in a Data Management Guide (pdf, 295.88 KB) contained in the Templates and useful documents section of the JLA website.  The following paragraphs contain more detail. 

Processing submitted uncertainties

To enable open access to the full list of uncertainties identified, PSPs need to ensure that, on completion of their priority setting exercise, uncertainties are published on the JLA website.

An Excel spreadsheet template (xlsx, 15.63 KB) is available in the Templates and useful documents section for PSPs to use to manage their data and the completed spreadsheet will be published on the JLA website once the PSP is ready to announce its results.

Uncertainties generally fall into four categories:

• individual uncertainties from patients, carers or clinicians
• indicative uncertainties formed by the PSP, reflecting those submitted by the above groups separately but which are similar to or duplicated
• uncertainties from treatment guidelines, research recommendations and systematic reviews – these may be gathered at the same time as the literature is consulted to verify the uncertainties submitted via the survey
• uncertainties being addressed by ongoing research as confirmed by relevant and reliable systematic reviews, in study protocols, or protocols for systematic reviews.

Uncertainties from patients, carers and clinicians collected via the survey can be numerous.  The table below shows examples of the number of submissions received from some previous JLA survey respondents:

Uncertainties must be checked and verified as true uncertainties before prioritisation can begin. This is one of the most labour-intensive stages of the JLA process and the Steering Group needs to identify how it will be resourced and actioned.  There is a basic role description (pdf, 45.35 KB) for the Information Specialist who performs this task in the Templates and useful documents section of the JLA website.

Resources required for this task will vary depending on the number and type of submissions returned.  Uncertainties can sometimes be more detailed than simple research recommendations. This phase of data management and checking can be complex and requires data management skills, critical appraisal skills, clinical knowledge and information retrieval skills.

As part of this process of verifying genuine uncertainties, two criteria must be met: (1) a measure of an uncertainty must be met, which is when a reported confidence interval in a systematic review crosses the line of no effect or line of unity, and (2) a clinician or person with relevant clinical knowledge must confirm that the outcome being reported as statistically significant in (1) is also clinically relevant.

It is essential to adopt a systematic approach to managing and processing the survey submissions in order to create a list of uncertainties for prioritisation, and for publication on the JLA website on the data spreadsheet template (xlsx, 15.63 KB) provided by the JLA.

The suggested approach is divided into five stages.

1. Download the survey data
2. Remove out-of-scope submissions
3. Categorise eligible submissions
4. Format the submissions
5. Verify the uncertainties
   
   

We acknowledge that the JLA’s data management process has evolved significantly, and while we have aimed to simplify it, we are conscious of the complex nature of the task.  We welcome feedback on this section from those who are using, or have previously used, the Guidebook to inform their data management process.