4. Summing it all up: Synthesis of clinical research

Systematic reviews

---

A systematic review is a thorough, comprehensive, and explicit way of exploring the medical literature. It typically involves several steps, including:

• Formulating a clear, answerable research question (often the most challenging step)
• Choosing appropriate databases to search for evidence
• Designing a transparent and reproducible search strategy
• Selecting titles, abstracts, and manuscripts based on explicit inclusion and exclusion criteria
• Extracting data in a consistent, standardised format

Systematic reviews are key inputs to the HTA process and driven by reproducible methods, which provide a structured, comprehensive overview of the published scientific evidence.

While decision-makers may still differ in how they interpret or prioritise the findings, the structured process of a systematic review increases the reliability of the scientific evidence being considered.

Systematic reviews can still result in a biased estimate of clinical impact if:

• Information from relevant studies is not identified
• Key outcomes from the included studies are not reported in a review
• The quality of the studies is low

That said, identifying that 15 out of 20 studies reported positive outcomes is not helpful if those studies were prone to error by design. Systematic reviews should be critically assessed.

When interpreting a systematic review or meta-analysis, HTA bodies and readers must often make a judgement call—a "leap of faith." That leap is smaller when the review clearly follows rigorous, validated methods and is transparent about its process, and much larger when key details are missing.

Meta-analysis

---

To estimate the clinical or economic impact of a new medicine, it is often helpful to combine findings from multiple studies. Meta-analysis does this by pooling data from several studies reporting on the same outcome. This improves the reliability of results by increasing the sample size and reducing random error. It can also help explore heterogeneity—differences between studies that may result from variations in methods or populations.

Some HTA bodies request the marketing authorisation holder (MAH) to submit a meta-analysis, since the MAH may have access to more detailed data. Where individual patient-level meta-analyses are not available, HTA bodies may instead have to rely on already-analysed aggregate (population-level) data. Patient-level data provide more flexibility for analysis and can offer a deeper understanding of how a medicine performs in different subgroups.

Meta-analyses are often used to combine the results of randomised controlled trials based on the assumption that these data share an underlying relationship that allows them to be combined mathematically. However, it becomes more difficult to perform a meta-analysis when combining the results of randomised trials with those from other types of study designs, such as non-experimental studies. In some cases, a meta-analysis is entirely inappropriate. Cost-effectiveness analyses, for example, may include studies that do not share a consistent structure and can’t be combined through meta-analysis.

Meta-analysis can be useful when comparing outcomes or treatments for which multiple studies are available. However, what if none of the studies identified actually compare the two alternatives that a decision maker is interested in comparing? Or, what if a decision maker is interested in understanding the best of many competing alternatives?

To address this, indirect and mixed treatment comparisons have been developed. These methods allow comparison between interventions that have not been directly studied head-to-head. For example, if studies compare medicine A with placebo, and other studies compare medicine B with placebo, indirect methods can estimate how A compares with B. Mixed treatment comparisons go further, combining both direct and indirect comparisons to strengthen the evidence base.

Report checklists for systematic reviews and meta-analysis have been developed to help readers interpret the findings of the analysis. Most recently, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was devised to guide analysts conducting reviews toward proper reporting. It consists of 27 items to be addressed when reporting a systematic review. It encourages the use of a flow chart, so that readers can understand the process that led to the studies’ selection. This helps users of systematic reviews to understand when and where the information came from, and how it was synthesised.

Modelling

---

Systematic reviews coupled with meta-analysis can provide more precise estimates of the relative impact of various interventions. But how can this information be used to support decision-making?

Simulation modelling is a technique that allows the combination of different types of information (clinical, epidemiological, economical, etc.) for an overall picture of the relative costs and effectiveness of medical treatment.

Simulation models are not constrained to a single type of information or a single outcome. These models are useful, particularly when a lengthy wait for more evidence is not feasible.

At their core, simulation models combine probabilities to obtain estimates of expected values. These values can be either clinical outcomes, or costs, making simulation modelling useful for economic evaluation.

These models require assumptions to be inserted regarding various outcomes, which ultimately influence the endpoints calculated by a model. The accuracy and reliability of these assumptions will vary according to the treatment in question and the quantity and quality of evidence that supports the selection of the value inserted for each ‘assumption’.

Many HTA guidelines provide specific guidance on selecting and supporting those model ‘inputs’ to assist reviewers in making judgements regarding the overall quality and ‘probability’ of the model’s predictions.