ANOVA types in a nutshell

Type I, II, and III. What do I need?

Apr 20, 2022 9 min read stats, R

Yes, there are different types of ANOVA. And yes, they can give different outputs depending on the data.

ANOVA is a widely used statistical method used to test the effect of one or more factorial variables (independent terms) on a continuous numerical variable (dependent term). This method aims to split the total variation of the dependent variable into one or more sources of variation using the Sum of Squares, and unveiling if independent variables have a significant effect on the dependent variable. There are three basic assumptions in ANOVA:

Independence of samples
The distribution of the residuals should be normal
Homocedasticity: homogeneity of the variances

In case of testing the effect of only one independent variable, we will be using the One-Way ANOVA. However, if we want to test two independent variables, we will use the Two-Way ANOVA. In the same way, if we want to test the effect of three or more factorial independent variables, we will call it X-Way ANOVA.

For example, we want to assess the effects of diet and exercises on weight loss in 10 sedentary males. Fortunately, the data for this experiment can be found in the datarium R package. Here, our factorial independent variables are diet and exercises, and our dependent numeric variable is the weight loss score measured at the end of each trial (t3). For more info about the dataset see ?datarium::weightloss.

library(tidyverse)
library(datarium)
data(weightloss)

head(weightloss)

# A tibble: 6 × 6
  id    diet  exercises    t1    t2    t3
  <fct> <fct> <fct>     <dbl> <dbl> <dbl>
1 1     no    no         10.4 13.2   11.6
2 2     no    no         11.6 10.7   13.2
3 3     no    no         11.4 11.1   11.4
4 4     no    no         11.1  9.5   11.1
5 5     no    no          9.5  9.73  12.3
6 6     no    no          9.5 12.7   10.4

Now, we can try to answer different questions given this dataset and the above concepts.

One-Way ANOVA

What is the effect of the diet on the weight loss score without regard to exercises?

summary(aov(t3 ~ diet, data = weightloss))

            Df Sum Sq Mean Sq F value Pr(>F)
diet         1   0.09   0.091   0.017  0.895
Residuals   46 239.61   5.209

Seems that diet alone doesn’t have a significant effect on weight loss.

What is the effect of the exercises on the weight loss score without regard to diet?

summary(aov(t3 ~ exercises, data = weightloss))

            Df Sum Sq Mean Sq F value   Pr(>F)    
exercises    1  116.7  116.66   43.61 3.58e-08 ***
Residuals   46  123.0    2.67                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

On the contrary, seems that exercises have a significant effect on weight loss.

Two-Way ANOVA

Is there a unique effect of the diet or exercises on the weight loss score (t3)?

summary(aov(t3 ~ diet + exercises, data = weightloss))

            Df Sum Sq Mean Sq F value   Pr(>F)    
diet         1   0.09    0.09   0.033    0.856    
exercises    1 116.66  116.66  42.697 5.01e-08 ***
Residuals   45 122.95    2.73                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Given these results, there seems to be an effect of exercises but not diet on weight loss.

Does the effect of the diet depend on the exercises? Here we need an interaction term (*). Interaction effects represent the combined effects of factors on the dependent variable. When an interaction effect is present, the impact of one factor depends on the level of the other factor.

summary(aov(t3 ~ diet * exercises, data = weightloss))

               Df Sum Sq Mean Sq F value   Pr(>F)    
diet            1   0.09    0.09   0.064    0.801    
exercises       1 116.66  116.66  82.493 1.19e-11 ***
diet:exercises  1  60.73   60.73  42.943 5.18e-08 ***
Residuals      44  62.22    1.41                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

As we can see here, the effect of the interaction is significant (diet:exercises). This means that our two independent variables depend on each other and that the interaction between them has an effect on our dependent variable, i.e., weight loss (t3).

So far, all of this probably sounds pretty familiar.

The “problem” comes when we realize that there are different ways to calculate the Sum of Squares, and therefore to compute these ANOVA models. Often, this is an unknown fact, but not only that, many widely used software for statistical analysis use different types of ANOVA by default (this is even less known), so we need to be careful about which ANOVA model we want to use, what software we are using, etc. Moreover, this can be even more confusing because if the data is balanced, the different types of ANOVA will give the same result, but they don’t in case of unbalanced data.

There are three types of ANOVA depending on how the Sum of Squares is done: ANOVA type I, II, and III.

ANOVA Type I

The fact that characterizes this type of ANOVA is that the Sum of Squares is calculated sequentially, so the variation will be allocated to the different variables in order.

Given our Two-Way ANOVA model (independent variables diet and exercises), this type of ANOVA will assign the maximum variation to the diet variable, and the remaining variation to the exercises variable and the residuals, in that order.

summary(aov(t3 ~ diet + exercises, data = weightloss))

            Df Sum Sq Mean Sq F value   Pr(>F)    
diet         1   0.09    0.09   0.033    0.856    
exercises    1 116.66  116.66  42.697 5.01e-08 ***
Residuals   45 122.95    2.73                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

summary(aov(t3 ~ exercises + diet, data = weightloss))

            Df Sum Sq Mean Sq F value   Pr(>F)    
exercises    1 116.66  116.66  42.697 5.01e-08 ***
diet         1   0.09    0.09   0.033    0.856    
Residuals   45 122.95    2.73                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We can see that the result doesn’t change if we flip the factors in the formula. This is because our data is balanced. Let’s remove some random samples and repeat the process…

weightloss_unbal <- weightloss %>% 
  dplyr::sample_n(floor(nrow(.)*0.9)) # remove random 10%
  
summary(aov(t3 ~ diet + exercises, data = weightloss_unbal))

            Df Sum Sq Mean Sq F value   Pr(>F)    
diet         1   0.06    0.06   0.022    0.884    
exercises    1 116.34  116.34  44.526 5.37e-08 ***
Residuals   40 104.51    2.61                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

summary(aov(t3 ~ exercises + diet, data = weightloss_unbal))

            Df Sum Sq Mean Sq F value   Pr(>F)    
exercises    1  116.4  116.39   44.55 5.34e-08 ***
diet         1    0.0    0.00    0.00     0.99    
Residuals   40  104.5    2.61                     
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Now, we can appreciate differences in the results when we flip the order of factors! Remember that in ANOVA type I, the Sum of Squares is sequential, so the order matters!

This type of ANOVA is used by the R software (by aov() and anova() functions). Therefore, if you use R in your analysis, keep in mind that the order of the independent variables in your formula matters, because if your data is not perfectly balanced, your result will change according to the order of variables. Finally, if your data is not balanced, maybe you should consider another type of ANOVA, and if you’re interested in computing other type of ANOVA in R, you can use the function Anova() from the car package, or anova_test() from the rstatix package.

ANOVA Type II

In ANOVA type II, the variation assigned to the first independent variable takes into account the second and vice versa. In addition, the Sum of Squares of ANOVA type II have no interaction effects. Therefore, we will only use this type of ANOVA when we have no interaction between our independent variables.

Thus, when we are facing unbalanced data (and no interaction), this type of ANOVA is much better than the first type as we do not have to worry about the order of variables in our model. We see below (in R) as if we exchange the order of factors the result doesn’t change (as it did in the previous ANOVA type).

mod <- lm(t3 ~ diet + exercises, data = weightloss_unbal)
car::Anova(mod, type = "II")

Anova Table (Type II tests)

Response: t3
          Sum Sq Df F value    Pr(>F)    
diet        0.00  1  0.0002    0.9903    
exercises 116.34  1 44.5264 5.369e-08 ***
Residuals 104.51 40                      
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

mod2 <- lm(t3 ~ exercises + diet, data = weightloss_unbal)
car::Anova(mod2, type = "II")

Anova Table (Type II tests)

Response: t3
          Sum Sq Df F value    Pr(>F)    
exercises 116.34  1 44.5264 5.369e-08 ***
diet        0.00  1  0.0002    0.9903    
Residuals 104.51 40                      
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

This type of ANOVA is the default type in Python (statsmodels library), so if this is your software, well for you and less headaches!

ANOVA Type III

The last one is the ANOVA type III, also called partial Sum of Squares.

Like ANOVA type II, ANOVA type III Sum of Squares is not sequential, so the order of the independent variables doesn’t matter. However, unlike type II, type III allows an interaction effect between independent variables.

See below how ANOVA type III allows interaction and the result doesn’t change if we flip the independent factors.

mod3 <- lm(t3 ~ diet * exercises, data = weightloss_unbal)
car::Anova(mod3, type = "III")

Anova Table (Type III tests)

Response: t3
                Sum Sq Df  F value    Pr(>F)    
(Intercept)    1412.74  1 1005.911 < 2.2e-16 ***
diet             25.34  1   18.040 0.0001298 ***
exercises       159.89  1  113.850 3.950e-13 ***
diet:exercises   49.74  1   35.414 6.058e-07 ***
Residuals        54.77 39                       
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

mod4 <- lm(t3 ~ exercises * diet, data = weightloss_unbal)
car::Anova(mod4, type = "III")

Anova Table (Type III tests)

Response: t3
                Sum Sq Df  F value    Pr(>F)    
(Intercept)    1412.74  1 1005.911 < 2.2e-16 ***
exercises       159.89  1  113.850 3.950e-13 ***
diet             25.34  1   18.040 0.0001298 ***
exercises:diet   49.74  1   35.414 6.058e-07 ***
Residuals        54.77 39                       
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

This type of ANOVA is the default type in the SAS and SPSS software. Although I mostly use R, this should be the default ANOVA type in my opinion.

Take home ideas

If the data is balanced, the type of ANOVA doesn’t matter
ANOVA type I should be used when there is a serious theoretical reason for it. Put the most important independent variable first and it will receive the maximum amount of variation
ANOVA type II should be used when we don’t expect a true interaction between independent variables
ANOVA type III should be used when a true interaction between independent variables is expected
Don’t be alarmed if your results do not match those of your colleague who is using other software, there is always an explanation

stats R