30 AIC, BIC, and Deviance

Author

Luke Miratrix

In this section, we briefly walk through how to find AIC, BIC, and Deviance to compare models. We have a simple multilevel dataset (we generate through a utility package, blkvar, that Miratrix and the C.A.R.E.S. lab has used to explore how multilevel modeling works in practice), and generate a few variables that we will use as predictors. Only the fourth variable is actually useful for prediction! Let’s see if our AIC, etc., measures identify which model is superior.

To install a “working package” we use devtools:

devtools::install_github("https://github.com/lmiratrix/blkvar" )

library( blkvar )
dd = generate_multilevel_data( J = 40 )
head( dd )

    sid         Y0        Y1 Z       Yobs        W
1     1 -0.4227620 0.2354816 0 -0.4227620 1.597797
1.1   1  0.9677406 1.6259842 0  0.9677406 1.597797
1.2   1  0.4381904 1.0964339 1  1.0964339 1.597797
1.3   1  0.6008868 1.2591304 1  1.2591304 1.597797
1.4   1  1.7714616 2.4297051 0  1.7714616 1.597797
1.5   1  0.5488574 1.2071010 1  1.2071010 1.597797

dd$X1 = rnorm( nrow(dd) )
dd$X2 = rnorm( nrow(dd) )
dd$X3 = rnorm( nrow(dd) )
dd$X4 = dd$Yobs + rnorm(nrow(dd))
 
M1 = lmer( Yobs ~ 1 + (1|sid), data=dd )
M2 = lmer( Yobs ~ 1 + X1 + X2 + X3 + (1|sid), data=dd )
M3 = lmer( Yobs ~ 1 + X1 + X2 + X3 + X4 + (1|sid), data=dd )

library( arm )

Loading required package: MASS


Attaching package: 'MASS'

The following object is masked from 'package:dplyr':

    select


arm (Version 1.14-4, built: 2024-4-1)

Working directory is /Users/lmiratrix/Dropbox/MLM Course F2024/MLM textbook

display(M1)

lmer(formula = Yobs ~ 1 + (1 | sid), data = dd)
coef.est  coef.se 
    0.06     0.11 

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.68    
 Residual             0.62    
---
number of obs: 400, groups: sid, 40
AIC = 859.5, DIC = 848.4
deviance = 851.0

display(M2)

lmer(formula = Yobs ~ 1 + X1 + X2 + X3 + (1 | sid), data = dd)
            coef.est coef.se
(Intercept)  0.06     0.11  
X1           0.02     0.03  
X2          -0.04     0.03  
X3          -0.04     0.03  

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.68    
 Residual             0.62    
---
number of obs: 400, groups: sid, 40
AIC = 876.7, DIC = 829.7
deviance = 847.2

library( texreg )

Version:  1.39.3
Date:     2023-11-09
Author:   Philip Leifeld (University of Essex)

Consider submitting praise using the praise or praise_interactive functions.
Please cite the JSS article in your publications -- see citation("texreg").

screenreg( list( M1, M2, M3 ) )


=====================================================
                      Model 1   Model 2   Model 3    
-----------------------------------------------------
(Intercept)              0.06      0.06      0.04    
                        (0.11)    (0.11)    (0.08)   
X1                                 0.02      0.05    
                                  (0.03)    (0.03)   
X2                                -0.04     -0.02    
                                  (0.03)    (0.03)   
X3                                -0.04     -0.00    
                                  (0.03)    (0.03)   
X4                                           0.33 ***
                                            (0.02)   
-----------------------------------------------------
AIC                    859.54    876.74    703.94    
BIC                    871.52    900.69    731.88    
Log Likelihood        -426.77   -432.37   -344.97    
Num. obs.              400       400       400       
Num. groups: sid        40        40        40       
Var: sid (Intercept)     0.46      0.46      0.20    
Var: Residual            0.38      0.38      0.25    
=====================================================
*** p < 0.001; ** p < 0.01; * p < 0.05