Week 11

Prediction in MLM

Week Learning Objectives

By the end of this module, you will be able to

• Describe the role of prediction in data analysis
• Describe the problem of overfitting when fitting complex models
• Use information criteria to compare models

Task Lists

1. Review the resources (lecture videos and slides)
2. Complete the assigned readings
   • Yarkoni & Westfall (2017)
3. Attend the Thursday session and participate in the class exercise
4. Post progress of your project to the Discussion Board on Blackboard for peer review

Lecture

Slides

Note that in some of the videos below the Bayeisan analyses were used; however, for the class this year we will stay with frequentist analyses. The results and interpretations are basically; just note some differences in the terminology.

You can view and download the slides here: PDF

Predictive Models in MLM

---

Think more: Think about a prominent theory in your area of research. What predictions does it make? Does it give precise predictions?

---

Example

---

Check your learning: In a multilevel model with students nested within schools and with student math achievement as the outcome variable, what is a cluster-specific prediction?

The predicted achievement for a student from one of the schools included in the sample.
The predicted mean achievement of a new school.
The predicted achievement for a student from a new school that are not part of the sample.

---

Prediction Error

Some information has been updated since the video was recorded. Check out the updated slides

Overfitting

---

Check your learning: Which of the following growth curve model would show the largest degree of overfitting, given a sample of 15 participants across 5 time points?

A linear growth model.
A piecewise growth model with two phases.
A piecewise growth model with four phases.

---

Out-Of-Sample Prediction Errors

---

Check your learning: Why shouldn’t we just choose a model with the lowest in-sample prediction error?

Prediction error should not be a criterion for choosing among models.
One should use a likelihood ratio test instead.
A model that fits one data set too well usually does not fit well with other data sets.

---

Cross Validation

---

Check your learning: Why does cross-validation, compared to in-sample MSE, give a better estimate of the out-of-sample prediction error?

It uses different subsets of the data for estimation and for prediction.
It is developed in the machine learning literature.
It uses less data for estimation, so the results are more stable.

---

Information Criterion

---

Check your learning: Which of the following two models are nested?

M1: mathach ~ meanses + ses_cmc + sector + (1 | ID)

M2: mathach ~ sector + (1 | ID)

M3: mathach ~ meanses + sector + (ses_cmc | ID)

M1 and M2
M2 and M3
M1 and M3

---

Model Comparison

---

Check your learning: Which of the following model is the best based on AIC?

M1: mAIC = 1203, cAIC = 1037

M2: mAIC = 1202, cAIC = 1000

M3: mAIC = 1210, cAIC = 1055

M1 M2 M3

---