30 AIC, BIC, and Deviance

Author

Luke Miratrix

In this section, we briefly walk through how to find AIC, BIC, and Deviance to compare models. We have a simple multilevel dataset (we generate through a utility package, blkvar, that Miratrix and the C.A.R.E.S. lab has used to explore how multilevel modeling works in practice), and generate a few variables that we will use as predictors. Only the fourth variable is actually useful for prediction! Let’s see if our AIC, etc., measures identify which model is superior.

To install a “working package” we use devtools:

devtools::install_github("https://github.com/lmiratrix/blkvar" )

library( blkvar )
dd = generate_multilevel_data( J = 40 )
head( dd )

    sid         Y0         Y1 Z       Yobs         W
1     1  1.7596309  2.3091809 1  2.3091809  1.018444
1.1   1  0.9473675  1.4969176 1  1.4969176  1.018444
1.2   1  1.5897352  2.1392852 0  1.5897352  1.018444
1.3   1  1.3973794  1.9469295 0  1.3973794  1.018444
2     2 -0.5235175 -0.4486207 1 -0.4486207 -1.376848
2.1   2 -1.1383334 -1.0634366 1 -1.0634366 -1.376848

dd$X1 = rnorm( nrow(dd) )
dd$X2 = rnorm( nrow(dd) )
dd$X3 = rnorm( nrow(dd) )
dd$X4 = dd$Yobs + rnorm(nrow(dd))
 
M1 = lmer( Yobs ~ 1 + (1|sid), data=dd )
M2 = lmer( Yobs ~ 1 + X1 + X2 + X3 + (1|sid), data=dd )
M3 = lmer( Yobs ~ 1 + X1 + X2 + X3 + X4 + (1|sid), data=dd )

library( arm )

Loading required package: MASS


Attaching package: 'MASS'

The following object is masked from 'package:dplyr':

    select


arm (Version 1.14-4, built: 2024-4-1)

Working directory is /Users/lmiratrix/Dropbox/MLM Course F2025/MLM textbook

display(M1)

lmer(formula = Yobs ~ 1 + (1 | sid), data = dd)
coef.est  coef.se 
    0.21     0.15 

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.92    
 Residual             0.64    
---
number of obs: 434, groups: sid, 40
AIC = 973.7, DIC = 963.7
deviance = 965.7

display(M2)

lmer(formula = Yobs ~ 1 + X1 + X2 + X3 + (1 | sid), data = dd)
            coef.est coef.se
(Intercept)  0.22     0.15  
X1          -0.06     0.03  
X2           0.00     0.03  
X3          -0.04     0.03  

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.92    
 Residual             0.64    
---
number of obs: 434, groups: sid, 40
AIC = 989.4, DIC = 943.3
deviance = 960.4

library( texreg )

Version:  1.39.4
Date:     2024-07-23
Author:   Philip Leifeld (University of Manchester)

Consider submitting praise using the praise or praise_interactive functions.
Please cite the JSS article in your publications -- see citation("texreg").

screenreg( list( M1, M2, M3 ) )


======================================================
                      Model 1   Model 2    Model 3    
------------------------------------------------------
(Intercept)              0.21      0.22       0.14    
                        (0.15)    (0.15)     (0.10)   
X1                                -0.06 *    -0.01    
                                  (0.03)     (0.03)   
X2                                -0.00      -0.04    
                                  (0.03)     (0.03)   
X3                                -0.04      -0.01    
                                  (0.03)     (0.03)   
X4                                            0.33 ***
                                             (0.02)   
------------------------------------------------------
AIC                    973.68    989.39     796.25    
BIC                    985.90   1013.83     824.76    
Log Likelihood        -483.84   -488.69    -391.12    
Num. obs.              434       434        434       
Num. groups: sid        40        40         40       
Var: sid (Intercept)     0.85      0.85       0.40    
Var: Residual            0.41      0.41       0.26    
======================================================
*** p < 0.001; ** p < 0.01; * p < 0.05