Approaches used

The Steering Group may design an interim priority setting approach that they feel is appropriate for the communities they are targeting.

It is important to consider the respondents (and their possible health condition) in choosing the method, and to think about what it is reasonable to ask them to do.

Given the varied needs of participants, the JLA does not impose a strict method for this stage. It does however ask PSPs to note a detailed, transparent explanation of how they conducted the interim prioritisation and how rankings were agreed. It may be necessary to offer an alternative to returns by email, such as phoning in ranked uncertainties or postal returns.

Approaches previously used by JLA PSPs include:

  • Asking people to choose the 10 most important questions in their experience
  • Asking people to choose the 10 most important questions and rank them 1-10
  • Asking people to review every question on the list and indicate the importance of each one (Likert scale).

Examples of these are below:

Choose and rank 10
  • Participants are asked to consider the long list of questions, and then to choose and rank 10 of them.
  • This can be done via email and post, using a pro forma produced in Word, or online. 
  • Each ranked question is given a score (rank 1 = 10 points, rank 10 = 1 point) and totals are tallied for each question, keeping patient, carer and clinician responses separate.
  • A rank order for each respondent group is calculated, and each question re-scored according to its position in the list (top ranking gains maximum points).  The totals for each respondent group are added together to generate a combined ranking of all the questions.
ADVANTAGES
  • Participants have to make choices about the questions and enter into a process of priority setting, producing a genuine set of priorities. 
  • The ranking materials can be produced easily and cost-effectively.
  • It also gets respondents into the frame of mind of ranking and choosing a top 10 in the final workshop.
DISADVANTAGES
  • When carried out via email/post, can potentially generate a lot of data that needs to be manually entered into a spreadsheet. 
  • Not all survey software allows for questions to be chosen then ranked.  Alternative or upgraded software may be needed to do the exercise online, taking care not to create a page of questions that is overly-long or difficult to navigate. 
EXAMPLE
See an example of the interim survey ranking form in the Key Documents section of the Childhood Disability PSP and the Type 2 Diabetes PSP on the JLA website.

 

Choose 10
  • Participants are given the long list of questions.  They are then asked to choose 10, but not rank them. 
  • This can be done using email/post, or online. 
  • Each time a question is chosen, it is given one point.  Separate tallies should be maintained for the different stakeholder groups, so the totals for each one are equally weighted when added together.
ADVANTAGES
  • Participants have an opportunity to consider the whole list, but must still make choices that involve them in genuine shortlisting.
  • May be suited to groups that find it hard to rank topics individually, for whom simply choosing 10 would be sufficiently challenging. 
  • May also be useful for those PSPs where the number of questions sent for interim prioritisation is towards the upper end.
DISADVANTAGES
  • When carried out via email/post, can potentially generate a lot of data that needs to be manually entered into a spreadsheet. 
  • When done online, can result in a very long list that may be hard to digest online. 
EXAMPLE
See an example of the interim survey in the Key Documents section of the Anaesthesia and Perioperative Care PSP and the Adult Social Work PSP on the JLA website.

 

Likert scale
  • Participants are asked to rate on a Likert scale how important they think each question in the long list is for research, e.g. not a priority, low priority, medium priority, high priority, very high priority, don't know. 
  • This is most easily done online.  Questions’ categories should be set up to be randomised, so as not to disadvantage questions at the end of the list. 
  • Taking each respondent group separately, the score for each question can be calculated as an average score from the Likert scale, or as a total for all the scores for each question, within each respondent group.  Once ranked in order, each groups’ scores for each question can be added together, creating an overall ranked list of questions. 
ADVANTAGES
  • When conducted predominantly online, this approach creates very little administrative burden. 
  • Particularly suitable for participants who struggle with large amounts of data, or who have limited attention spans (for example due to medication or pain), as each question can be considered separately. 
DISADVANTAGES
  • There is a risk that people rate everything as important and do not think about relative importance or prioritisation. 
  • This can result in very closely scored questions, with very marginal differences between them. 
  • While this is a means of shortlisting, it is not a genuine prioritisation exercise.
EXAMPLE
See the example interim survey form in the Key Documents section of the Mesothelioma PSP and the MS PSP on the JLA website.