Stage 4. Format the submissions

Submissions from patients, carers and clinicians may need to be rewritten or rephrased.  This is to clarify the precise uncertainty, which may have been submitted with a lot of narrative text, to ensure consistency in the language used and to make it easier to check the question against the evidence base.  Some PSPs have formatted each individual submission and then combined the duplicates.  Others have created formatted indicative questions based on groups of similar or duplicate submissions.

PSPs may have to interpret what an intervention might be in a question, where the original submission did not indicate one.  PSPs and the JLA Adviser should make best efforts to ensure that decisions made about interventions are transparent and fair.  Any rewording should ensure that the language is accessible to a lay or non-medical audience but also accurate enough to engage clinicians and specialists.  It is suggested that questions may be worded using the PICO format (see below).  This process can be time consuming and labour intensive.

It is advised that this work is carried out by one or more individuals skilled in identifying systematic reviews and who have clinical knowledge, or by an Information Specialist.  However, it is vital that Steering Group members are also offered opportunities to contribute to and comment on the process to ensure they are satisfied that the interpretation of the submissions is fair and neutral.

Rephrasing the submissions

Where possible, it is helpful to re-write submissions using the PICO format, to include:

  • the Patient or Population
  • the Intervention
  • a Comparator or Control
  • an Outcome.

It is important to note that not all the PICO variables will necessarily be available.  The two most difficult variables to identify are the comparator and outcome.

The JLA also recognises that not all submissions are suitable for PICO structure, but PSPs should aim to ensure that the uncertainties are in a format that will ultimately be of value to the research community. 

Combining duplicates

Duplicates and very similar submissions can be combined within one indicative uncertainty. Combining submissions can greatly reduce the volume of data that need to be checked for systematic reviews:

JLA PSPNumber of submitted uncertaintiesNumber of verified uncertaintiesTotal after combining: indicatives
Ear Nose and Throat, Aspects of Balance 669 348 146
Eczema 1,070 718 65
Stroke in Scotland 548 386 226
Diabetes (Type 1) 1,141 890 650


This process is also likely to be repeated after the verification of uncertainties, and the removal of non-uncertainties, thus reducing the data further.

For example, in the Asthma PSP over 100 survey submissions asked about the side effects of both medication to ‘relieve symptoms’ and ‘prevent symptoms developing’ used in asthma treatment.  These 100+ questions were made into two indicative questions:

  • What are the adverse effects associated with long term use of short and long acting bronchodilators (medication to relive symptoms); inhaled and oral steroids (medication to prevent symptoms developing); and combination and additive therapies (combinations of the two types of medication) in adults? (N.B. this includes children aged 12 years old and over)
  • What are the adverse effects associated with long term use of short and long acting bronchodilators; inhaled and oral steroids; and combination and additive therapies in children?

In the Childhood Disability PSP, the indicative question ‘What is the safety and effectiveness of drugs compared to talking therapies (e.g. Cognitive Behavioural Therapy) to treat anxiety in children and young people with Autistic Spectrum Disorders?’ came from the following originally submitted questions:

‘Do medications aimed at reducing anxiety (and thereby aggression) work long term in children with severe autism? / Is the management of anxiety with medication in children beneficial and what are the side effects? / Long term impact and side effect of risperidone vs sertraline vs fluoxetine for associated anxiety, comparative effectiveness and long term side effects, is mono or multiple therapy best? / Which medication helps best for anxiety in autism / CBT is recognised as an effective treatment for ASD especially if it has been specifically modified for them.  Why then can I not access this for my ASD children? / Use of CBT for repetitive behaviours in autism / What CBTs work in children with intellectual disabilities? / What is the evidence for CBT to treat anxiety in adolescents with autism?’

Combining survey submissions is an interpretative and sensitive process.  It is important that the Steering Group can explain why a single submission may be better presented within an indicative question, but that it also communicates that the detail of the original questions will be retained.  Combining is a pragmatic way of managing large datasets.  For example, multiple submissions about the effects of specific foods or drinks (e.g. fruit, coffee, tea etc) on a given condition may be combined into a theme of ‘diet’.

This process is more straightforward if submissions have been allocated to a taxonomy as outlined above.  It is important to keep a record of the original submissions, so that their source (patient/carer, clinician, or both) and the frequency with which that particular uncertainty was submitted are documented.  This detail can be discussed and taken into account at a later stage, when discussing the nature of the priorities with potential research funders.

It should be noted that patients, carers and clinicians, who are not researchers, do not usually formulate precise research questions.  The JLA does not expect them to. Indicative questions capturing the themes and issues raised in the survey are more accessible to a non-research audience that needs to understand them in order to make priority setting decisions.  The process of turning those themes into precise research questions is something the Steering Group is encouraged to work with funders and researchers on after the final workshop. 

PSP Examples:

The Urinary Incontinence PSP gathered a total of 519 submissions, of which 102 came from existing sources and 417 were submitted by partner organisations which had gathered them internally and from their members.

These were refined to remove duplicates and non-uncertainties, leaving a total of 226. Of these:

  • 79 came solely from patients and carers
  • 37 came solely from clinicians
  • six were submitted by patients and clinicians simultaneously
  • two were from patients and research recommendations
  • 102 were derived solely from research recommendations.

For the Diabetes (Type 1) PSP:

  • 583 people submitted 1141 uncertainties
  • 890 were true uncertainties
  • 118 uncertainties came from research recommendations/literature
  • Combining duplicates, there were 350 verified uncertainties.