4. Clinical effectiveness assessment in HTA

1. Clinical effectiveness assessment in HTA

(This section is organised in the form of a book, please follow the blue arrows to navigate through the book or by following the navigation panel on the right side of the page.)

Clinical effectiveness assessment

At the heart of HTA lies the evaluation of the effectiveness of a new medicine.

In general, evidence from (multinational) randomised controlled trials is used as a basis, which must then be interpreted in the local context and be supplemented by additional information. It can’t be assumed that the available information desirable for an ideal HTA is comprehensive and contains all aspects. This means that different sources of information must be pieced together to provide a picture for HTA bodies/decision-makers. The required information may be published in scientific journals, available through regulatory assessment reports for marketing authorisation of medicines, or obtained directly from companies that develop and/or market a medicine. While some HTA bodies will commission or conduct their own research, most HTA bodies use work done by others. 

Steps suggested

Clinical effectiveness assessments compare the effect of a new technology on the health of patients in a standard clinical setting to the current standard of care. The impact that a technology has on health is usually analysed through a further examination of health outcomes beyond the assessment of efficacy and safety (benefit-risk) done by regulatory authorities. Assessing the impact of any health technology requires comprehensive information that reflects what is likely to happen in a health system or society. High quality analysis requires the use of expert advice and methods from associated disciplines that are used as sources.

In particular, the evaluation of clinical effectiveness is carried out with the methodology borrowed from epidemiology and medicine .

There are four underlying steps of clinical effectiveness assessment which are frequently followed:

1. Seeking information

2. Asking relevant questions

3. Understanding differences

4. Valuing differences.

1.1. Seeking Information

HTA bodies use clinical information to estimate what health outcomes patients might experience when taking a new medicine. First, though, they must decide how they will gather the information. There are three different ways HTA bodies might obtain clinical information on new medicines:

1. reviewing existing information on the medicine’s performance,

2. conducting a study (e.g., pharmacoepidemiologic)to gather information and evaluate the performance of the medicine in a real-world setting, or

3. asking clinicians and patients (‘experts’) what their experience was in relation to their expectations with the medicine (e.g. surveys).

HTA bodies often use a combination of these approaches. For example:

• They might use information from the medicine’s marketing authorisation holder (MAH) to conduct their own independent reviews and analyses.
• When information is missing, expert opinion may be required – for example, to find out whether changes in short-term outcomes (such as lowering cholesterol) might be used to predict changes in longer-term outcomes (such as avoiding hospitalisation).

New clinical trials are rarely commissioned, because the time required to set up and have a trial approved is typically too long and the costs may be prohibitive. More often, pharmacoepidemiological studies are more practical in terms of design, expenses and time required.  In some cases, the decision-makers have allowed a medicine to be reimbursed conditionally, based on the collection of further information (this would be similar to regulatory authorities granting a conditional marketing authorisation requiring further information to be gathered). The risk of poorer-than-expected performance of a new medicine in a real world setting can then be shared between the MAH and payer(s) through price negotiation mechanisms or other changes to the conditions associated with reimbursed access (such as further restrictions on the patient population eligible to receive reimbursed access) while patients are given immediate access.

1.2. Asking relevant questions

When assessing the clinical effectiveness of a new medicine, the HTA body must carefully consider all outcomes associated with it. It is important to know about these outcomes in order to ask the relevant questions about the medicine’s effectiveness.

There is increasing understanding that the outcomes that may seem relevant to clinicians are not always those considered most important by patients. For this reason, it is advisable for patients to be involved in designing studies in order to make sure that information is collected on the outcomes that matter to them. For instance, in recent years, it has been recognised that quality of life is an important outcome for patients. This has led to the development of specific methodologies to create quality-of-life measures and so-called ‘patientreported outcomes’ (PRO) within in clinical trials and pharmacoepidemiologic studies .

One approach to ensure that all the important outcomes of a particular technology are examined is to use an analytic framework – for instance, following a flowchart. Analytic frameworks are helpful to visualise all of the outcomes associated with an intervention, and to highlight where there are uncertainties.

This is illustrated in the analytic framework shown in Figure 1:

• Cause and effect are depicted by arrows (curved arrows indicate harmful outcomes);
• Health improvement outcomes (such as decreased mortality) are depicted by rectangles (sharp cornered rectangles show clinically relevant endpoints - those that are perceived by the patient, such as reduced chest pain. Round cornered rectangles show intermediate endpoints, including surrogate endpoints, which cannot be perceived by the patient, such as cholesterol level in blood).

Figure 1: Template for an analytic framework. (Source: AHRW) (1)

Key question 1. Is screening for dyslipidaemia in children/adolescents effective in delaying the onset and reducing the incidence of CHD (coronary heart disease) related events?

Key question 2. What is the accuracy of screening for dyslipidaemia in identifying children/adolescents at increased risk of CHD-related events and other outcomes?

Key question 3. What are the adverse effects of screening (including false positives, false negatives)?

Key question 4. In children and adolescents, what is the effectiveness of medicine, diet, exercise, and combination therapy in reducing the incidence of adult dyslipidemia, and delaying the onset and reducing the incidence of CHD-related events and other outcomes (including optimal age for initiation of treatment)?

Key questions 5-8. What is the effectiveness of medicine, diet, exercise, and combination therapy for treating dyslipidemia in children/adolescents (including the incremental benefit of treating dyslipidemia in childhood)?

Key question 9. What are the adverse effects of medicine, diet, exercise, and combination therapy in children/adolescents?

Key question 10. Does improving dyslipidemia in childhood reduce the risk of dyslipidemia in adulthood?

Key question 11 (not pictured). What are the cost issues involved in screening for dyslipidemia in asymptomatic children?

In general, an assessment of clinical effectiveness should address the following questions:

1.     How comprehensive was the information provided for a thorough HTA? provided for a thorough HTA?

• Systematic reviews conducted in a rigorous and transparent manner with extensive information provided by pharmaceutical companies are most likely to be comprehensive and balanced;
• Real world studies should still consider whether what is observed is consistent with previous studies (2).

2.    How accurate is the information?

• Real-world information gathered within the health system for which a decision has to be made is most likely to be relevant;
• Relying solely on reviews or summaries without a clear understanding of the methods used may not be balanced.

3.    Is anything missing?

• Not involving patients or providers to help understand outcomes of importance, not conducting a thorough search or soliciting extensive information from pharmaceutical companies may lead to missing information.

4.    How understandable is the information?

• Information that considers changes meaningful to patients should be interpreted carefully and correctly and presented in a way understandable for lay persons.

1.3. Understanding the differences between outcomes

Once all important outcomes are identified, there may still be several challenges in comparing the effects of a new medicine with the standard of care. Outcomes may be measured in different ways, or two medicines may seem to have similar outcomes until closer inspection shows differences.

In cases where the important identified outcomes are difficult to measure, or have never been measured before, scientists must carefully create a measure that can then be reproduced in a study. For instance, a patient may want to know how a medicine will help them return to work or get out of bed. Scientists may create a numeric pain-rating scale for patients with lower back pain. Where a study measures a change in such a ‘laboratory’ parameter (methodologically reproducible and comparable measurement), this change needs to be transformed into a measure that matters more to patients – such as the ability to return to work.

Some outcomes may seem intuitive but, upon closer examination, may be difficult to interpret. For example, a reduction of the risk of five-year mortality (death within five years) by an average of 50% does not mean that the medicine can prevent premature death in 50% of all cases. It could simply be:

- extending life expectancy from 4.9 to 5.1 years (or worse, 4.99 years to 5.01 years) in some patients, or

Even if differences in measures that are meaningful to patients are observed, these may still be ambiguous to interpret. For example, studies may indicate a new medicine reduced the risk of hospitalisation from infection by 33%. However, this may mean different things. It could mean that:

• The chance of hospitalisation is reduced by 33% relative to the chance of being hospitalised without medicine (this is called a relative reduction in risk). A 33% reduction may seem to be a considerable benefit.

• If the chance of being hospitalised in the absence of the new medicine is 3 out of 1,000, then a 33% reduction reduces this to 2 out of 1,000. This means 1 out of every 1,000 people taking the medicine will benefit. This is called an absolute reduction of risk. A benefit which is for 1 out of 1000 patients may well seem marginal.

A final challenge with orrectly interpreting the differences between a new health technology and the standard of care is the use and misuse of statistical tests. Statistical tests are intended to help researchers know if the differences they have detected are likely to be causally related to taking a medicine or have happened by chance. Often, this is reported in the form of a p-value. However, p-values do not reflect the magnitude (size) of the difference, or whether that difference is clinically meaningful. This means that the smaller the p-value, the higher the probability that the observed effect was not by chance.

Other statistical measures such as confidence intervals are more informative than the p-value, because they also give some sense of the size of the difference between the new health technology and the standard of care. Confidence intervals also reflect any uncertainty about the estimate of the magnitude of difference. For example, a new medicine may be reported to reduce the chance of having a future heart attack by 33% (with a 95% confidence interval of 5% to 45%) relative to the current chance of having a heart attack.

1.4. Valuing differences

The last challenge is to understand how we might perceive and value the differences between outcomes. If a medicine prolongs life by 0.2 years, we still need to know:

• how much a patient would value 0.2 years of additional life expectancy
• if all patients experience roughly the same gains or if there are dramatic differences between patients, and
• if all patients value these gains similarly.

A new medicine that increased life expectancy by an average of 0.2 years would be perceived differently if it worked in some patients but had no effect on others, when compared with a scenario where all patients gained 0.2 years with little differences across patients.

There are several mechanisms that can be used to understand the relative value that patients and providers put on differences in health outcomes. One is qualitative research, such as surveys or focus groups, intended to provide an understanding of which outcomes are most important to patients (you will learn more about qualitative research in Course 4). Another is quantitative research based on scales, which can assign numerical values to specific outcomes in different states of health. In both cases, a proper and thorough validation of the measurement instruments should be performed.