Providing evidence on patient aspects: Evaluation of questionnaires

Evaluating questionnaires is one area where patients should play a central role. 

Many PROMs are used repeatedly because of their established presence in clinical research, both for disease-specific and general applications. However, questionnaires should be seen as instruments requiring careful calibration to accurately measure the intended concepts and meet key validity criteria (further discussed in this course). 

Too often, researchers continue using existing tools without revalidating them, which can result in instruments that no longer capture patient-relevant aspects. Additionally, response options or rating scales may feel irrelevant or inadequate from the patient perspective. 

When reviewing questionnaires, patients can consider the following points [8]:  

❓Is the questionnaire appropriate for the specific HTA question? 

❓Has the questionnaire been validated, and according to which criteria? Validity refers to the extent to which the instrument truly measures what it claims to. 

❓Are all questions relevant to the technology under assessment? 

❓Does the questionnaire provide sufficient precision in measuring what it intends to, especially if it was developed for a different purpose? 

❓Is the questionnaire tailored to the patient group or subgroup affected by the HTA decision?  

❓Does any content go beyond what is considered acceptable or appropriate by the target group? 

❓Are all questions clearly understood in the same way by all respondents? 

❓Is the language and style of the questionnaire suitable for the intended audience? 

Further validation criteria can be found in  Course: HTA and Evaluation Methods: Qualitative | EUPATI Open Classroom

Providing evidence on patient aspects: Evaluation of PRO and HRQoL instruments 

Traditional clinical outcome endpoints—such as changes in mortality or morbidity, survival,  risk  or  symptom  reduction—are increasingly complemented by measures reflecting patients’ self-assessed health status following treatment. Instruments used to assess patient-reported outcomes (PROs) or health-related quality of life (HRQoL) are particularly relevant in a health technology assessment (HTA) context, where a comprehensive evaluation of all relevant dimensions is desirable. 

Health status instruments can be either disease-specific or generic. Disease-specific instruments are tailored to particular conditions, offering greater sensitivity in detecting treatment-related changes. Examples include tools developed for arthritis, chronic lung disease, diabetes, and various cancers, as well as instruments targeting single dimensions like pain or depression. However, these tools are limited in that they cannot be used to compare outcomes across different disease areas. 

In contrast, generic instruments are designed for broader application across various diseases and patient groups, facilitating comparisons across conditions. Their limitation lies in potentially lower sensitivity to changes that are specific and meaningful within a particular disease context. 

Using both disease-specific and generic instruments in the same study can enhance the comprehensiveness of an HTA, allowing for both detailed and comparative insights. However, patients often find generic tools less relevant, as they may not fully capture disease-specific experiences or concerns. Thus, the selection of PRO and HRQoL elements in an HTA can significantly influence the relevance and impact of the assessment for patients.

In an HTA context, the assessment of changes in patients’ health and well-being form a central part of the assessment. Patients can play a role when considering below elements: 

Providing evidence on patient aspects: Evaluation of qualitative research 

To assess qualitative research in the context of HTA, patients can use the CASP Qualitative Checklist [9], which offers a structured set of questions for systematically evaluating qualitative studies and understanding why these aspects matter. Developed by qualitative research experts under a Creative Commons license, the checklist supports a transparent and critical appraisal process. 

When applying the checklist within an HTA, patients may consider whether: 

• the research findings are relevant to the patient perspective in the specific HTA context, and 
• the patient population studied is representative for the local patients affected by the HTA-based decision 

Providing evidence on patient aspects: Evaluation of qualitative research 

The effect of a health technology intervention is more than “efficacy”. Effect means how effective a treatment or the application of the technology is. In English there are two terms for effect: “efficacy” and “effectiveness”. “Efficacy” expresses the efficiency under ideal conditions, i.e. under research conditions such as clinical trials, whereas “effectiveness” expresses the efficiency under more normal daily practice.  

Patients can contribute to identifying elements that are most relevant to them, which should be considered in the evaluation of a technology’s effects.   

Providing evidence on patient aspects: Interpretation of effect size 

Note: For more details about Statistics , please see the lesson Statistics in the Clinical Development module. 

When assessing treatment effects, patients should pay close attention to how effect size is presented and the statistical methods used. This is important because the robustness of a statistical outcome depends not only on the methodology but also on the number of patients included in the study. The following sections provide a brief overview of commonly used ways to present effect size, with examples to help non-statisticians understand what to look for. 

Effect size measurements depend largely on the type of endpoints defined in a study—for example, a difference in mean blood pressure (a continuous endpoint) versus a difference in mortality (a categorical endpoint). Additionally, the same effect can be expressed in several different ways depending on the study’s focus. For instance, the same outcome can be described as a relative risk reduction (RRR), absolute risk reduction (ARR), odds ratio (OR), or number needed to treat (NNT). 

The example in the table below shows the same effect as an odds ratio of 0.45, a relative risk reduction of 54%, an absolute risk reduction of 1.6% and number needed to treat of 62. These figures describe the same quantitative effect but framed differently.

The safety of a health technology—including its potential risks—is as important to patients as its expected benefits. This applies equally to pharmaceutical products and medical devices. Positive and negative effects are typically associated with any technology, and a comprehensive risk assessment should always be conducted. 

You can find more information about ‘Therapeutic window’ in Non-Clinical Development module. 

The HTA risk analysis should always include both the patients and the staff or caregivers who are to use the technology. As more and more health technologies are intended for patients' self-management the focus on safety aspects that are important to patients should be thoroughly assessed. Patients can contribute by pointing to the most patient relevant risks and their acceptability for further examinations. 

Patients contributing to the formulation of safety questions for HTA may consider the following aspects: 

The concept of “synthesis” can be understood in various ways. 

This implies that the components being synthesised may differ significantly, requiring careful weighing and interpretation. 

The synthesis phase is arguably the most complex and challenging part of the HTA process, even though it holds a central role. This complexity may stem from the interface between scientific evidence and policy recommendations—particularly when there is limited insight into how HTA results are integrated into decision-making or when methodological frameworks lack clarity and transparency. 

Patients can play a meaningful role in this phase. Their involvement helps ensure that patient-relevant aspects are given appropriate weight in the overall interpretation.  

For further information on synthesising, please refer to HTA Course 2:  HTA Bodies and Principles.