Chapter 6: Data processing and verifying uncertainties

Unlike most surveys, which are designed to collect answers, JLA Priority Setting Partnership (PSP) surveys are designed to collect questions.  Their aim is to generate questions that patients, carers and clinicians want health research to address.  As we are actively consulting with a non-researcher audience, we expect the responses to be a mix of specific questions, personal stories and themes/issues. 

A survey submission may not immediately look like a research question, but it may still contain an unmet evidence need.  The survey responses need to be reviewed, interpreted, sorted and turned into a list of indicative questions for research or ‘evidence uncertainties’.  For this reason, analysing the survey responses (or data) is the most complex and time-consuming part of the process.

To enable open access to the full list of uncertainties identified, PSPs need to ensure that on completion of their priority setting exercise, all uncertainties are published on the JLA website.

An Excel spreadsheet template is available from the Templates and useful documents section of the JLA website for PSPs to use to manage their data. The completed spreadsheet will be published on the JLA website once the PSP is ready to announce its results.  A published example of a completed spreadsheet can be seen from the Alcohol-related Liver Disease PSP.

Evidence uncertainties generally come from the survey responses from patients, carers and clinicians or may be identified within relevant literature in the form of research recommendations.

• Uncertainties submitted to the survey by patients, carers or clinicians may be:
  • unique questions submitted by just one respondent
  • indicative or summary uncertainties, which are formed to combine duplicate or similar responses.
• Research recommendations may be identified in relevant
  • systematic reviews
  • clinical guidelines
  • study protocols or protocols for systematic reviews.

Uncertainties from patients, carers and clinicians collected via the survey can be numerous. The table below shows examples of the number of submissions received from some previous JLA survey respondents:

Uncertainties must be checked and verified as true uncertainties before prioritisation can begin. This is one of the most labour-intensive stages of the JLA process and the Steering Group needs to identify how it will be resourced and actioned. There is a basic role description for the Information Specialist who performs this task.

“I knew from the onset that managing the data components of the PSP would be a major task but I was surprised, once the first survey was completed, at how much data there actually was to process...  All steering group members helped with refining the uncertainties/out of scope/unanswerable questions and the clinicians and some patients checked against published research ensuring that any questions that had already been answered by research were removed.”

From PSP feedback survey to the JLA

Resources required for this task will vary depending on the number and type of survey submissions returned. Uncertainties can sometimes be more detailed than simple research recommendations. This phase of data management and checking can be complex and requires data management skills, critical appraisal skills, clinical knowledge and information retrieval skills.

It is essential to adopt a systematic approach to managing and processing the survey submissions in order to create a list of uncertainties for prioritisation.

However PSPs choose to organise the survey responses, once the priority-setting workshop is complete they need to provide the JLA with a JLA PSP data management spreadsheet, which will be published on the JLA website. We therefore recommend that PSPs use this spreadsheet from the start to help organise the survey responses. A copy of the data management template is available from the Templates and useful documents section of the JLA website.

The suggested approach to managing the data is divided into five stages: 

1. Download the survey data
2. Remove out-of-scope survey submissions
3. Categorise eligible survey submissions
4. Form indicative questions
5. Verify the uncertainties.

We acknowledge that the JLA’s data management process has evolved significantly, and while we have aimed to simplify it, we are conscious of the complex nature of the task. We welcome feedback on this section from those who are using, or have previously used, the Guidebook to inform their data management process.