Fundamentals of Statistics
6. Significance Level
In everyday language significant means important, but when used in statistics, ‘significant’ means a result has a high probability of being true (not due to chance) and it does not mean (necessarily) that it is highly important. A research finding may be true without being important.
The significance level (or α level) is a threshold that determines whether a study result can be considered statistically significant after performing the planned statistical tests. It is most often set to 5% (or 0.05), although other levels may be used depending on the study. It is the probability of rejecting the null hypothesis when it is true (the probability to commit a type I error). For example, a significance level of 0.05 indicates a 5% risk of concluding that a difference exists when there is no actual difference.
The probability value (p-value) is the likelihood of obtaining an effect at least as large as the one that was observed, assuming that the null hypothesis is true; in other words, the likelihood of the observed effect being caused by some variable other than the one being studied or by chance.
- a large p-value suggests that the observed effect is very likely if the null hypothesis is true.
- a small p-value (equal to or less than the significance level) suggests that the observed evidence is not very likely if the null hypothesis is true – i.e. either a very unusual event has happened or the null hypothesis is incorrect.
This is more easily illustrated by an example:
We have a medicine ‘A’ which decreases blood pressure. We therefore set our null hypothesis as being: ‘Medicine A will NOT decrease blood pressure’. We also decide that the significance level should be 0.05. We run a study using medicine A and observe an average decrease in blood pressure of 20%. Was this due to chance? (null hypothesis true) Or due to the effect of medicine A?`(null hypothesis false). Now we calculate the p-value (the probability) that this result occurred if the null hypothesis was true (the results occurred by chance). The p-value was calculated to be 0.03. Since the p-value (0.03) is less than the significance level we set (0.05) we are able to reject the null hypothesis and conclude that that the measured decrease in blood pressure is likely to be due to the effects of medicine A, rather than being due to chance.
The p-value is compared with a pre-defined cut-off for the test (significance level). If it is smaller than this value, the estimated effect is considered to be significant. Often a p-value of 0.05 or 0.01 (written ‘p < 0.05’ or ‘p< 0.01’) are chosen as cut-offs. These are called the ‘significance levels’ of the experiment.