Bias in Clinical Trials

Site: EUPATI Open Classroom
Course: Interpretation and Dissemination of Results
Book: Bias in Clinical Trials
Printed by: Guest user
Date: Wednesday, 19 June 2024, 7:26 AM

1. Where Bias Can Be Introduced

(This section is organised in the form of a book, please follow the blue arrows to navigate through the book or by following the navigation panel on the right side of the page.)

Bias can occur at any phase of research, including trial design, data collection, as well as in the process of data analysis and publication. The bias that can occur at different stages during a clinical trial are described in the sections that follow.

1.1. During Participant Recruitment

Selection bias
Selection bias can occur at different levels. Some examples are given below:
  1. If the participant eligibility criteria used in a trial are so stringent that only a small percentage of the intended patient population qualifies. For example, those with better physical condition who would be expected to be able to manage toxic treatments.

  2. At treatment assignment, the eligibility criteria used to recruit and enrol participants into the treatment arms may be applied differently. This may result in some characteristics being over-expressed in one arm compared to the other. For example, if participants were selected differently according to their age or health status. The treatment outcomes may be stronger in the arm where the participants are younger and in a better shape. Therefore, any difference in outcome between the two treatment arms can no longer be attributed only to the received treatment.

  3. The use of random participant allocation techniques to two or more treatment arms is key to avoiding this type of bias. Randomisation aims to ensure that the treatment arms are comparable both in terms of known and unknown prognostic factors especially over a large number of participants. Well-performed randomisation will allow the researcher to consider the observed treatment effects (response rate, survival, etc.) to be related to the treatment itself and not due to other factors. The treatment allocation should be independent from researchers’ beliefs and expectations. The optimal way is by using a central randomisation algorithm to distribute characteristics across the treatment arms.
selection bias

Figure 1: Example distribution of trial participants by arm and sex.

Note that in a trial of limited sample size , there is no guarantee that the randomisation will effectively prevent imbalances in important prognostic factors (age or disease status). In this case, the randomisation can be  stratified according to a number of known important prognostic factors. The trial population will be split in groups according to these stratification  factors and the randomisation will ensure that the balance between the treatment arms within these strata is maintained.

selection bias 2

Figure 2: Example distribution of trial participants by arm and sex.

1.2. Information (or Measurement) Bias

This type of bias refers to a systematic error in the measurement of participant characteristics, exposure or treatment outcome, for example:
  • Participants may be wrongly classified as exposed to being at higher risk of disease when they are not. 
  • The disease may be reported as progressing when it is stable.
This might be due to:
  • Inaccuracies in the measurement tool. 
  • Expectations of trial participants – a participant could be more optimistic because they are assigned to the new treatment group .
  • Expectations of the investigators – also called the ‘observer bias’. This occurs when:
    • Investigators are very optimistic about the effect of the new treatment and interpret more favourably any clinical signal.
    • Investigators monitor the adverse effects of the new treatment more carefully than for the standard treatment. 

Blinding  the allocated treatment to the participant and/or the investigators may prevent such biasBlinding is of special interest when the trial outcome is subjective, like the reduction in pain, or when an experimental treatment is being compared to a placebo. However, while a blinded randomised trial is considered the gold standard of clinical trials, blinding may not always be feasible: 

  • Treatments may cause specific adverse effects that make them easy to identify.

  • Treatments may need different procedures for administration or different treatment schedules.

1.3. During Trial Conduct

There are a few common problems that may arise during the course of a trial related to participant adherence (compliance) to the protocol and to the prescribed treatment schedule. For example:

  • Participants may turn out to be ineligible after randomisation.

  • Treatment may have been interrupted or modified but not according to the rules specified in the protocol.

  • Disease assessments may have been delayed or not performed at all.

  • A participant may decide to stop taking part in the trial, etc.

    This can be problematic if not properly accounted for in the analysis. For example, consider the setting of a clinical trial comparing a new experimental treatment to the standard of care. In this trial some participants taking the experimental treatment are too sick to go to the next visit within the allotted time due to side effects. A possible approach would be to include only participants with complete follow-up, so to exclude the sick participants from the analysis. However, by doing so, one selects a sub-group of patients whom, by definition, will present an artificially positive picture of the treatment under trial. This is again an example of selection bias.

    One potential solution to this problem is a statistical concept called an intention-to-treat (ITT) analysis. ITT analysis includes every randomised patient whether they received the treatment or not. As such, ITT analyses maintain the balance of patients' baseline characteristics between the different trial arms obtained from the randomisation. ‘Protocol deviations’ such as non-compliance to the assigned treatment (schedule, dosing, etc.) are part of daily practice. Therefore, treatment-effect estimates obtained from ITT analysis are considered to be more representative of the actual benefit of a new treatment
  • Bias can also happen when measuring the endpoint(s) of interest. For instance, time to disease progression or relapse can be severely affected by the hospital visit schedule. The frequency of disease assessment should be adequate and frequent enough to capture correctly the events that could not be observed by means other than by medical examination at the hospital. The disease assessment schedules should also be the same in both arms.

bias conduct

Figure 3: Example of sample distribution by arm and sex.

1.4. During Analysis and Interpretation of Results

In a clinical trial sample, it is possible to find sub-groups of patients which respond better to treatment. Sub-group analyses involve splitting the trial participants into sub-groups. This could be based on:

  • Demographic characteristics (e.g. sex, age).

  • Baseline characteristics (e.g. a specific genomic profile).

  • Use of concomitant therapy.
Findings from sub-group analyses might be misleading for several reasons:

  • Firstly, sub-group analyses are observational (sub-groups are defined on observed participants’ characteristics) and not based on randomised comparisons. The hindsight bias, also known as the ‘I-knew-it-all-along’-bias, is the inclination to see events that have already occurred as being more predictable than they were before they took place.

  • Secondly, when multiple sub-group analyses are performed, the risk of finding a false positive result (i.e. a type I error) increases with the number of sub-group comparisons. Multiplicity issues are in general related to repeated looks at the same data set but in different ways until something ‘statistically significant’ emerges. With the wealth of data sometimes obtained, all signals should be considered carefully. Researchers must be cautious about possible over-interpretation. Techniques exist to protect against multiplicity, but they mostly require stronger evidence for statistical significance to control the overall type I error of the analysis (e.g. the Holm–Bonferroni method and the Hochberg procedure).

  • Thirdly, there is a tendency to conduct analyses comparing sub-groups based on information collected while on trial. A typical example is looking at the difference in survival between patients responding (yes/no) to treatment. Participants who are responding to treatment are by definition patients who are able to spend sufficient time on treatment to allow a response. Therefore, again by definition, they may simply represent a sub-group of patients of better prognosis, and may therefore bias the analysis. This is an example of what is often referred to as lead-time bias or guarantee-time bias. One way of dealing with this is using a landmark as a starting point for the time-to-event analysis, and creating the categories based on the participants’ characteristics at the time of this landmark (e.g. did a participant respond at three months, yes/no).

1.5. At Time of Reporting

Publication bias refers to the fact that positive results are more likely to be published (rapidly) than negative results. Researchers may not be interested or less motivated in publishing negative trial results, fearing that such findings may negatively reflect on their professional abilities, the image of their company (institution) and perhaps their new medicine. Hence positive results are more likely to be submitted rapidly for publication than negative results.

Publication bias is detrimental since it is preventing access to clinical trial results. Beyond single trial results, the efficacy and safety profile of a treatment needs to be assessed globally taking into account all the data and the results available from clinical trials investigating that treatment. Researchers planning new experimentation may be limited by the information available in published results. Negative results may inform about the lack of efficacy of a treatment and the absence of justification for continuing with further development. The conclusions of a meta-analysis may be flawed if based only on published data.

Initiatives are ongoing for reducing publication bias. One of them is to promote the registration of clinical trials for medicines. For instance, the International Committee of Medical Journal Editors (ICMJE) will not publish trials that are not registered in public registries such as, created and operated by the US National Institutes of Health (NIH). With such registries, researchers know what the existing clinical trials are, even if their results were never published, and may contact the trial sponsor in order to gain access to the results.

Despite these measures, publication bias has not been completely eliminated. While trial registries provide medical researchers with information about unpublished trials, researchers may be left to only speculate as to the results of these trials. Various organisations are currently engaged in initiatives to encourage or require the registration and disclosure of clinical trial information. In Europe, EudraCT, the European Clinical Trials Database of the European Medical Agency collects information on all clinical trials of medicines performed in Europe. As of July 2014, this database also makes trial summary results available to the public. The World Health Organisation (WHO), through its International Clinical Trials Registry Platform (ICTRP), is setting international standards for registering and reporting on all clinical trials. In the US, the registry is doing similarly.

Reducing publication bias will also occur by increasing the willingness of scientific journals to publish trials with negative results. Manuscripts should be reviewed on the basis of the quality of the methodology used, not on the apparent success of the trial. In addition, funding agencies should take a more active role in the dissemination of clinical trials they fund.

2. Correlation vs Causation

When analysing the results from a trial it is important to remember that correlation is not the same thing as causation. Correlation is when two variables are linked in some way however this does not mean that one will cause the other. An example of this involves hormone replacement therapy (HRT) and coronary heart disease (CHD) where women taking HRT were at less risk from CHD. This however was not due to the actual HRT process but rather due to the fact that the group of people receiving HRT tended to belong to a higher socio-economic group, with better-than-average diet and exercise regime. This is why it is important to record as much information as possible about the subjects participants of trials.

Therefore: To avoid being misled, approach correlations between variables with skepticism by looking for confounding factors. Humans like neat, causal narratives, but that’s usually not what the data is telling us.