Collating and scoring interim priorities

Steering Group members will need to have agreed who will be responsible for coordinating, collating and listing the interim prioritised uncertainties.  This process is closely monitored by the JLA Adviser to ensure transparency and minimisation of bias.

Completed interim prioritisation results should be grouped into patients and/or carers, and clinicians, and separate scores kept to ensure a fair weighting of the constituent groups.  Some PSPs also score responses from organisations separately.  Where people have been asked to choose and rank 10 questions, the most straightforward approach is to apply a reverse scoring system to each submission: 

RankScore
1 10
2 9
3 8
4 7
5 6
6 5
7 4
8 3
9 2
10 1

In order to ensure equal weighting of the responses from all stakeholders groups, regardless of numbers of responses, it is important that the Steering Group looks at the totals for both patients/carers and clinicians, records these for future reference, and then ranks them again and gives them a score based on their position in the ranked list. It is those position scores that are then combined, to work out the final interim prioritised list.  This means that where a low number of patients, for example, has responded compared with clinicians, their scores will still be given equal weighting, in line with the JLA’s principle of equal involvement of patients and clinicians.

However, there have been cases of such disparity between the patient/carer priorities and those of the clinicians that the combined scoring has led to questions that are important to one group being left out of the shortlist for the workshop. In this situation the Steering Group should consider whether or not an alternative working should be applied. For example, the Adult Social Work PSP created a shortlist for the workshop based on including the separate Top 10s for each of its three stakeholder groups which, taking into account the overlaps, led to 21 questions being taken to the final workshop.

The JLA Adviser will work with the Steering Group to agree how many of the prioritised indicative questions to take to the final workshop. Generally, this will be around 25 questions. The JLA advises a maximum of 30, otherwise the workshop process can become unmanageable. However, for some groups it will be more appropriate to have a smaller number nearer to 20. It is important to consider the number of people who will be at the workshop and the ease with which they are likely to be able to review large numbers of questions on the day. There may be an obvious place in the list of questions to draw the line, for example, where scores drop off rapidly. There may be a particular reason for including questions that are on the margins, for example, a question may be about an important aspect of the condition or may have been particularly important to one group of respondents.

The shortlist of around 25 questions should be sorted into random order, i.e. not the ranked order, and each question assigned a letter of the alphabet as a reference.

Here are some examples of the numbers of questions that PSPs have taken to the final workshop: 

PSPNo. of questions at workshop
Adult Social Work 21
Autism 25
Blood Transfusion and Blood Donation 25
Dementia (Canada) 23
Depression 27
Frailty (Canada) 21
Idiopathic Intracranial Hypertension 26
Living With and Beyond Cancer 27
Palliative and end of life care 28
Teenage and Young Adult Cancer 30