Critical Reading

1. The risk of misuse and misinterpretation of statistical tests and their outcomes, due to the confusion about the meaning of the numbers (estimates) and the interpretation of hypothesis tests (p-values, power).
   
   
2. Data dredging (data mining) or testing large numbers of hypotheses in a single data set in the search for a positive effect. When numerous hypotheses are tested, it is virtually certain that some will falsely appear statistically significant. This is because almost every data set with any degree of randomness is likely to contain some coincidental correlations. If they are not cautious, a researcher using data mining techniques can be easily misled by these apparently significant results.
   
   
3. Bias. In research, bias occurs when systematic error is introduced into data sampling or hypothesis testing by selecting or encouraging one outcome or answer over others. Of note, bias is not always introduced intentionally, for example calibration error, or unknown confounding variables may introduce bias. Bias may affect the results of a clinical trial if the effect of interest deviates from its true value: estimates of association can be systematically larger or smaller than the true association. In extreme cases, bias can cause a perceived association which is directly opposite to the true association. Bias may also take the form of systematic favouritism in the way results are reported or in the way they are interpreted in the discussion and conclusion on clinical trial results.

Critical reading checklist

• Are the objectives and the hypotheses clear?
  
• Can the results of the trial be generalised to the broader population? The reader needs to consider to whom the results of the trial can be applied. The characteristics of the recruited population sample need to be described.
  
• Are all treatments used in the trial clearly detailed, would the treatments under investigation be relevant to the reader?
  
• What are the patients’ likely benefits and harms from the treatment?
  
• Does a conflict of interest exist?
  

• Is the control treatment a fair comparator that corresponds to current practice? Placebo, available therapy or best supportive care, historical control group?
  
• The trial population should be clearly defined, as well as whether the whole population or a sub-set has been studied and whether there is any possible selection bias. Consider the relevance of any participants who have dropped out of the trial, the reasons for dropping out and the relevance for the results and conclusions of the research.
  
• Was the control group well matched? Are the exclusion criteria valid?
  
• Are the trial endpoints well defined and meaningful?
  
• Is it clear how the trial was powered for the primary endpoint?
• Was the trial long enough for the outcome measure to occur and for capturing enough events?

• The results should be clearly and objectively presented in sufficient detail.
  

• Identify the rate of loss to follow-up and how non-responders have been dealt with.

• Check for any bias. Consider the possibility of any confounding variables. 

• The discussion should include all the results of the trial and not just those that have supported the initial hypothesis.
  
• Have the initial objectives been met, and has the research question been answered?
  
• Have the authors taken into account possible bias and acknowledged the possible limitations of the trial?
  
• Check whether any incorrect generalisations have been made by inappropriately applying the trial results to a different type of population.
• Check whether the discussion fits with existing knowledge and opinion (always look for other publications on the same topic).