Significance level and decision
The significance level alpha is the cut-off you set before the test, often 0.05. If the p-value is at most alpha, you reject H0. Otherwise you fail to reject it, which is not the same as proving H0 true.
Last updated
Analogy
It is the bar of evidence you set in advance, like deciding how convincing a witness must be.
Example
p = 0.0077, alpha = 0.050.0077 < 0.05: reject H0If p were 0.20:fail to reject H0(not enough evidence)
In practice
Drug trials and quality checks fix alpha first to avoid choosing a bar after seeing the result.
Practice
Flashcards
- When do you reject H0?When the p-value is at most alpha.
- Does a non-significant result prove H0 true?No. It only means the evidence against H0 was not strong enough.
Interview questions
- A machine should fill bottles with a mean of 500 ml. A sample of 16 bottles has a mean of 504 and s of 8. Test H0: mean = 500 at the 5% level.
- SE = 8 / 4 = 2
- t = 4 / 2 = 2.0 with 15 df
- Critical value 2.131 (two-sided)
- Fail to reject H0
- State the conclusion in words
Questions people ask
What happens when I run many tests at once?
Each test at alpha 0.05 carries a 5 percent false alarm chance, and the chances pile up. Across 20 independent tests, the chance of at least one false alarm is 1 minus 0.95^20, about 64 percent. Corrections such as Bonferroni tighten the threshold.
Why say fail to reject instead of accept the null?
A non-significant result may simply mean the sample was too small to detect a real effect. Absence of evidence is not evidence of absence. We say we lack enough evidence against H0, leaving open that a real effect exists but was too small to detect with this sample.