Post-hoc comparisons
After a significant ANOVA, a post-hoc test such as Tukey's finds which pairs of groups differ. It adjusts for the many comparisons so the overall false alarm rate stays near 5 percent.
Last updated
Analogy
After the school says one class is different, you check each pair of classes, with a stricter bar each time.
Example
3 groups = 3 pairsChance of a false alarm in at least one:1 - 0.95^3 = 0.14310 groups = 45 pairs:1 - 0.95^45 = 0.90
In practice
Without correction, comparing many pairs almost guarantees a false positive somewhere.
Practice
Flashcards
- Why use a post-hoc correction?To stop many comparisons from inflating the false alarm rate.
Interview questions
- Why use ANOVA instead of several t tests, and what does the F statistic compare?
- Many t tests inflate the false alarm rate
- ANOVA tests all means at once
- F compares between-group and within-group variation
- A large F suggests real differences
- A post-hoc test finds which groups differ
Questions people ask
What is the multiple comparisons problem?
Each test has a chance of a false alarm, and running many tests multiplies the chances. With 45 pairwise tests, a false discovery is almost certain. Corrections such as Tukey or Bonferroni raise the bar for each comparison to compensate.
What is the Bonferroni correction?
Divide the significance level by the number of tests. For 5 comparisons at an overall alpha of 0.05, use 0.01 for each. It is simple and safe but conservative, so real effects may be missed. Tukey's method is usually tighter for pairwise comparisons.