Assessing the Validity of a Clinical Trial

Not all clinical trials are of equal validity. In deciding how much weight to give to the results of a trial, it is worth asking a few key questions:

1. How well designed was the trial?

There is no one ‘correct’ design for a clinical trial – it is more a question of whether the design used was appropriate to the circumstances. While large trials are generally more reliable than small ones, this must be interpreted with common sense. For example, a trial of a rare inherited enzyme deficiency is never going to include the 5,000 patients which are frequently seen in trials for heart-attack medicines. A follow-up period of a few weeks is perfectly adequate for a pneumonia trial, but would be inappropriate for a contraceptive pill. Placebo control groups, while very helpful in the interpretation of results, are clearly unethical in some situations (e.g. life-threatening illnesses for which effective treatments exist). While comparative trials are the best way of assessing efficacy, larger and longer open-label trials may offer more insight into real-life safety of a medicine.

Each trial design must be approached from the standpoint of ‘Was this the best way to do things in these circumstances?’

2. Does the population studied correspond to the one of interest?

Information from a trial conducted in adults aged 18 to 65 may be of limited relevance to very elderly patients and will almost certainly be inadequate for guiding treatment of small babies. Similarly, people with severe or very advanced disease may respond quite differently to those with milder or in an earlier stage of illness.

3. How relevant are the endpoints?

Some diseases and symptoms lend themselves more readily than others to study in a clinical trial. If a new cancer medicine is increasing median survival by a year, there can be little doubt that this is a relevant measurement. A new painkiller used to treat the same patients will be much more difficult to assess because there are no clear ‘standard units of pain’. Again, all that can be done is to ask whether the approach taken is appropriate to the circumstances.

4. Were the effects of the medicine clinically relevant?

Generally the bigger the effect of the medicine the better. All medicines come with costs in terms of both money and side effects. We are looking for the greatest possible benefit in return for those costs. It is worth remembering, however, that a result that is modest overall may be made up of a dramatic improvement in some patients and no change in others. If further research can help to identify the sub-group likely to do especially well, the new medicine may have much to offer this target population.

5. How do the results fit into the pattern of previous knowledge?

It is very unusual for a clinical trial to stand alone as the only information available in a particular area of medicine. When this does happen, it usually represents the first use of a radically new approach to treatment and all one can do is to note the results with interest and wait to see whether subsequent trials support them. Much more commonly, there will have been previous trials with the same medicine or others of the same class in the same illness or in related diseases. One can then view the new results in the light of the previous body of knowledge. Findings that mesh well with what is already known are generally easier to accept than those which directly contradict earlier results. However, it is important to keep an open mind.