30 AIC, BIC, and Deviance

Author

Luke Miratrix

In this section, we briefly walk through how to find AIC, BIC, and Deviance to compare models. We have a simple multilevel dataset (we generate through a utility package, blkvar, that Miratrix and the C.A.R.E.S. lab has used to explore how multilevel modeling works in practice), and generate a few variables that we will use as predictors. Only the fourth variable is actually useful for prediction! Let’s see if our AIC, etc., measures identify which model is superior.

To install a “working package” we use devtools:

devtools::install_github("https://github.com/lmiratrix/blkvar" )

library( blkvar )
dd = generate_multilevel_data( J = 40 )
head( dd )

    sid         Y0         Y1 Z       Yobs          W
1     1 -0.8491674  0.1793707 0 -0.8491674 -0.3047711
1.1   1 -0.7800720  0.2484662 1  0.2484662 -0.3047711
1.2   1 -2.2845610 -1.2560229 1 -1.2560229 -0.3047711
1.3   1 -1.6637536 -0.6352155 0 -1.6637536 -0.3047711
2     2  1.0160661  2.1959872 0  1.0160661  1.4354998
2.1   2  1.2583262  2.4382473 1  2.4382473  1.4354998

dd$X1 = rnorm( nrow(dd) )
dd$X2 = rnorm( nrow(dd) )
dd$X3 = rnorm( nrow(dd) )
dd$X4 = dd$Yobs + rnorm(nrow(dd))
 
M1 = lmer( Yobs ~ 1 + (1|sid), data=dd )
M2 = lmer( Yobs ~ 1 + X1 + X2 + X3 + (1|sid), data=dd )
M3 = lmer( Yobs ~ 1 + X1 + X2 + X3 + X4 + (1|sid), data=dd )

library( arm )

Loading required package: MASS


Attaching package: 'MASS'

The following object is masked from 'package:dplyr':

    select


arm (Version 1.14-4, built: 2024-4-1)

Working directory is /Users/lmiratrix/Dropbox/MLM Course F2024/MLM textbook

display(M1)

lmer(formula = Yobs ~ 1 + (1 | sid), data = dd)
coef.est  coef.se 
    0.05     0.14 

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.84    
 Residual             0.61    
---
number of obs: 423, groups: sid, 40
AIC = 912.7, DIC = 902.5
deviance = 904.6

display(M2)

lmer(formula = Yobs ~ 1 + X1 + X2 + X3 + (1 | sid), data = dd)
            coef.est coef.se
(Intercept)  0.05     0.14  
X1           0.03     0.03  
X2          -0.01     0.03  
X3           0.00     0.03  

Error terms:
 Groups   Name        Std.Dev.
 sid      (Intercept) 0.84    
 Residual             0.62    
---
number of obs: 423, groups: sid, 40
AIC = 933.1, DIC = 886.3
deviance = 903.7

library( texreg )

Version:  1.39.3
Date:     2023-11-09
Author:   Philip Leifeld (University of Essex)

Consider submitting praise using the praise or praise_interactive functions.
Please cite the JSS article in your publications -- see citation("texreg").

screenreg( list( M1, M2, M3 ) )


=====================================================
                      Model 1   Model 2   Model 3    
-----------------------------------------------------
(Intercept)              0.05      0.05      0.03    
                        (0.14)    (0.14)    (0.09)   
X1                                 0.03      0.03    
                                  (0.03)    (0.03)   
X2                                -0.01     -0.01    
                                  (0.03)    (0.03)   
X3                                -0.00      0.00    
                                  (0.03)    (0.03)   
X4                                           0.29 ***
                                            (0.02)   
-----------------------------------------------------
AIC                    912.74    933.13    788.79    
BIC                    924.89    957.42    817.12    
Log Likelihood        -453.37   -460.57   -387.39    
Num. obs.              423       423       423       
Num. groups: sid        40        40        40       
Var: sid (Intercept)     0.71      0.70      0.30    
Var: Residual            0.38      0.38      0.28    
=====================================================
*** p < 0.001; ** p < 0.01; * p < 0.05