30  AIC, BIC, and Deviance

In this section, we briefly walk through how to find AIC, BIC, and Deviance to compare models. We have a simple multilevel dataset (we generate through a utility package, blkvar, that Miratrix and the C.A.R.E.S. lab has used to explore how multilevel modeling works in practice), and generate a few variables that we will use as predictors. Only the fourth variable is actually useful for prediction! Let’s see if our AIC, etc., measures identify which model is superior.

To install a “working package” we use devtools:

devtools::install_github("https://github.com/lmiratrix/blkvar" )

library( blkvar )
dd = generate_multilevel_data( J = 40 )
head( dd )

    sid         Y0         Y1 Z       Yobs         W
1     1 -0.3378507 0.50562804 0 -0.3378507 0.4600557
1.1   1 -0.5208864 0.32259232 0 -0.5208864 0.4600557
1.2   1 -0.6337652 0.20971350 1  0.2097135 0.4600557
1.3   1 -0.7818651 0.06161364 0 -0.7818651 0.4600557
1.4   1 -0.5771846 0.26629419 0 -0.5771846 0.4600557
1.5   1 -0.8134147 0.03006407 0 -0.8134147 0.4600557

dd$X1 = rnorm( nrow(dd) )
dd$X2 = rnorm( nrow(dd) )
dd$X3 = rnorm( nrow(dd) )
dd$X4 = dd$Yobs + rnorm(nrow(dd))
 
M1 = lmer( Yobs ~ 1 + (1|sid), data=dd )
M2 = lmer( Yobs ~ 1 + X1 + X2 + X3 + (1|sid), data=dd )
M3 = lmer( Yobs ~ 1 + X1 + X2 + X3 + X4 + (1|sid), data=dd )

library( arm )

Loading required package: MASS

Attaching package: 'MASS'

The following object is masked from 'package:dplyr':

    select

arm (Version 1.14-4, built: 2024-4-1)

Working directory is /Users/lmiratrix/Dropbox/MLM Course F2025/MLM textbook

display(M1)

lmer(formula = Yobs ~ 1 + (1 | sid), data = dd)
coef.est  coef.se 
    0.15     0.14 

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.84    
 Residual             0.58    
---
number of obs: 428, groups: sid, 40
AIC = 877.8, DIC = 867.5
deviance = 869.7 

display(M2)

lmer(formula = Yobs ~ 1 + X1 + X2 + X3 + (1 | sid), data = dd)
            coef.est coef.se
(Intercept)  0.15     0.14  
X1           0.02     0.03  
X2           0.00     0.03  
X3          -0.02     0.03  

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.85    
 Residual             0.58    
---
number of obs: 428, groups: sid, 40
AIC = 898.8, DIC = 850.9
deviance = 868.9 

library( texreg )

Version:  1.39.4
Date:     2024-07-23
Author:   Philip Leifeld (University of Manchester)

Consider submitting praise using the praise or praise_interactive functions.
Please cite the JSS article in your publications -- see citation("texreg").

screenreg( list( M1, M2, M3 ) )

=====================================================
                      Model 1   Model 2   Model 3    
-----------------------------------------------------
(Intercept)              0.15      0.15      0.10    
                        (0.14)    (0.14)    (0.09)   
X1                                 0.02      0.04    
                                  (0.03)    (0.02)   
X2                                 0.00      0.02    
                                  (0.03)    (0.02)   
X3                                -0.02     -0.00    
                                  (0.03)    (0.03)   
X4                                           0.32 ***
                                            (0.02)   
-----------------------------------------------------
AIC                    877.82    898.83    755.32    
BIC                    889.99    923.19    783.73    
Log Likelihood        -435.91   -443.42   -370.66    
Num. obs.              428       428       428       
Num. groups: sid        40        40        40       
Var: sid (Intercept)     0.71      0.72      0.31    
Var: Residual            0.34      0.34      0.25    
=====================================================
*** p < 0.001; ** p < 0.01; * p < 0.05