Finalising the long list of indicative questions

This process can be time consuming and labour intensive. The indicative questions (the evidence uncertainties) that are created at this stage are the ones that may go into the interim prioritisation survey, be discussed at the final workshop, and ultimately go on to become the published PSP results.

It is recommended that the Steering Group meets face to face and spends a considerable amount of time on this, keeping in mind that the questions need to be clearly understood, not overlapping, and accessible to members of the public who will see them in the interim prioritisation survey and at the final workshop.

These questions will form the outputs of the PSP and cannot be changed once they have gone into the interim prioritisation survey so it is important that they have been considered carefully. Steering Group members representing patients and clinicians will need to be content that they faithfully reflect the original survey submissions and that they will also be valuable to the research community. Many Steering Groups create broad questions that encompass more than one issue. It is important however to consider what effect this will have on the questions that researchers and research funders see at the end of the process and how this will affect their understanding of them.

The JLA's experience suggests that participants at the final priority setting workshop may want to debate or even dispute the shortlisted uncertainties, for example the wording, or whether or not any questions could be considered duplicates of each other. It is essential that the Steering Group is confident that it can defend the shortlist that goes out to prioritisation. It is worth the Steering Group reflecting on the following questions:

  • How confident are we that the questions will be understood?
  • Is there scope for confusion?
  • Are any of the questions too similar?
  • Are there any that may be better combined into one question?

It is important that the Information Specialist considers how to make sure that rephrased and reformatted questions can always be traced back to previous versions of the question and to the original submitted uncertainties.

PSP Examples:

In the Asthma PSP over 100 survey submissions asked about the side effects of both medication to ‘relieve symptoms’ and ‘prevent symptoms developing’ used in asthma treatment.  These 100+ questions were made into two indicative questions:

  • What are the adverse effects associated with long term use of short and long acting bronchodilators (medication to relive symptoms); inhaled and oral steroids (medication to prevent symptoms developing); and combination and additive therapies (combinations of the two types of medication) in adults? (N.B. this includes children aged 12 years old and over)
  • What are the adverse effects associated with long term use of short and long acting bronchodilators; inhaled and oral steroids; and combination and additive therapies in children?


In the Childhood Disability PSP, the indicative question ‘What is the safety and effectiveness of drugs compared to talking therapies (e.g. Cognitive Behavioural Therapy) to treat anxiety in children and young people with Autistic Spectrum Disorders?’ came from the following originally submitted questions:

‘Do medications aimed at reducing anxiety (and thereby aggression) work long term in children with severe autism? / Is the management of anxiety with medication in children beneficial and what are the side effects? / Long term impact and side effect of risperidone vs sertraline vs fluoxetine for associated anxiety, comparative effectiveness and long term side effects, is mono or multiple therapy best? / Which medication helps best for anxiety in autism / CBT is recognised as an effective treatment for ASD especially if it has been specifically modified for them.  Why then can I not access this for my ASD children? / Use of CBT for repetitive behaviours in autism / What CBTs work in children with intellectual disabilities? / What is the evidence for CBT to treat anxiety in adolescents with autism?’

Combining survey submissions is an interpretative and sensitive process.  It is important that the Steering Group can explain why a single submission may be better presented within an indicative question, but that it also communicates that the detail of the original questions will be retained.  Combining is a pragmatic way of managing large datasets.  For example, multiple submissions about the effects of specific foods or drinks (e.g. fruit, coffee, tea etc) on a given condition may be combined into a theme of ‘diet’. 

This process is more straightforward if submissions have been allocated to a taxonomy as outlined above.  It is important to keep a record of the original submissions, so that their source (patient/carer, clinician, or both) and the frequency with which that particular uncertainty was submitted are documented.  This detail can be discussed and taken into account at a later stage, when discussing the nature of the priorities with potential research funders. 

It should be noted that patients, carers and clinicians, who are not researchers, do not usually formulate precise research questions.  The JLA does not expect them to. Indicative questions capturing the themes and issues raised in the survey are more accessible to a non-research audience that needs to understand them in order to make priority setting decisions.  The process of turning those themes into precise research questions is something the Steering Group is encouraged to work with funders and researchers on after the final workshop.  

The Urinary Incontinence PSP gathered a total of 519 submissions, of which 102 came from existing sources and 417 were submitted by partner organisations which had gathered them internally and from their members.

These were refined to remove duplicates and non-uncertainties, leaving a total of 226. Of these:

  • 79 came solely from patients and carers
  • 37 came solely from clinicians
  • six were submitted by patients and clinicians simultaneously
  • two were from patients and research recommendations
  • 102 were derived solely from research recommendations.


For the Diabetes (Type 1) PSP:

  • 583 people submitted 1,141 uncertainties
  • 890 were true uncertainties
  • 118 uncertainties came from research recommendations/literature
  • Combining duplicates, there were 350 verified uncertainties.