Week Learning Objectives

• Describe the importance of having sufficient sample size for scientific research

• Describe conceptually the steps for sample size planning: precision analysis and power analysis

• Perform power analysis for MLM using the PowerUpR application and the simr package

• Understand the effect of uncertainty in parameter values and explore alternative approaches for sample size planning

Small Sample Size is a Problem Because . . .

Low power

Misleading and noisy results¹

• When coupled with publication bias (statistical significance filter)^{2 3}

Nonreproducible findings

Review: Sampling distributions

Test yourself! -- Week 13 Quiz (ungraded)

What is the null distribution?

• Suppose we examine the effect of a therapy on eating disorder
• We test against the null hypothesis $H_0:{\gamma }_{01}=0$, where ${\gamma }_{01}$ is the fixed effect of the therapy on eating disorder

What is the alternative distribution?

• Assume that the true effect of this therapy is ${\gamma }_{01}=.1$

Sampling Distribution as a Function of Sample Size

Assume true effect is ${\gamma }_{01}=0.10$

Let's say

• when $N=20$, $p<.05$ when $\hat{\gamma }\ge 0.82$
• when $N=200$, $p<.05$ when $\hat{\gamma }\ge 0.26$

Steps for Sample Size Planning

1. Write down your model equations

2. List out all parameters in the model

3. Determine if you want to achieve a desired level of

a. Power, or

b. Precision

Step 1: Write down model equations

Group-based therapy for eating disorder (cluster-randomized trial)

Step 1: Write down model equations

Group-based therapy for eating disorder (cluster-randomized trial)

Level-1 $$Y_{ij}={\beta }_{0j}+{\beta }_{1j}X\_{\text{cmc}}_{ij}+e_{ij}$$ $$e_{ij}\sim N(0,\sigma )$$ Level-2 $$\begin{array}{cc}{\beta }_{0j}& ={\gamma }_{00}+{\gamma }_{01}{W}_j+u_{0j}\\ {\beta }_{1j}& ={\gamma }_{10}+{\gamma }_{11}{W}_j+u_{1j}\\ \left[\begin{array}{c}{u}_{0j}\\ u_{1j}\end{array}\right]& \sim N(\left[\begin{array}{c}0\\ 0\end{array}\right],\left[\begin{array}{cc}{\tau }_0^2& \\ {\tau }_{01}& {\tau }_1^2\end{array}\right])\end{array}$$

• ${\gamma }_{10}$: $X$ (purely level-1 with ICC = 0)
• ${\gamma }_{01}$: $W$ (level-2)
• ${\gamma }_{11}$: $W\times X$ (cross-level interaction)

Step 2: List out all parameters

Sample Size and SE/Post. SD

In the previous graph, when $N=20$, the sample estimate is likely to be anywhere between -0.4 and 0.6

$$SE\propto \frac{1}{\sqrt{N}}$$

One goal of sample size planning is to

Have sufficient sample size to get precise (low SE) sample estimates of an effect

Analytic Formulas of SE

$J$ = Number of clusters; $n$ = Cluster size

• E.g., $J=100$ schools; $n=10$ students per school

Assuming ${\tau }_{01}=0$

$$\begin{array}{cc}SE\left({\gamma }_{01}\right)& =\sqrt{\frac{1}{S_W^2}(\frac{{\tau }_0^2}{J}+\frac{{\sigma }^2}{Jn})}\\ SE\left({\gamma }_{10}\right)& =\sqrt{\frac{{\tau }_1^2}{J}+\frac{{\sigma }^2}{Jn{S}_X^2}}\\ SE\left({\gamma }_{11}\right)& =\sqrt{\frac{1}{S_W^2}(\frac{{\tau }_1^2}{J}+\frac{{\sigma }^2}{Jn{S}_X^2})}\end{array}$$

Precision Analysis

Group-based therapy for eating disorder (cluster-randomized trial)

• Intervention at group level

• 10 participants per group

• Outcome standardized (i.e., SD = $\sqrt{{\tau }_0^2+{\sigma }^2}=1$)

  • $\gamma$ = Cohen's $d$

• ICC = .3 (i.e., ${\tau }_0^2=.3$)

• Goal: estimate $J$ such that $SE\left({\gamma }_{10}\right)\le .1$

  • E.g., if we estimated the sample effect size to be $d=.25$, the 95% CI would be approximately [.05, .45].

Calculating $J$

When the predictor is binary (e.g., treatment-control), if half of the groups is in one condition, $S_W^2=0.25$

• Otherwise, if 30% in one condition, $S_W^2=0.3\times 0.7$
• ${\tau }_0^2=0.3$, ${\sigma }^2=0.7$, $n=10$

E.g., if $J=30$ $$SE\left({\gamma }_{01}\right)=\sqrt{\frac{1}{S_W^2}(\frac{{\tau }_0^2}{J}+\frac{{\sigma }^2}{Jn})}=\sqrt{\frac{1}{0.25}(\frac{0.3}{30}+\frac{0.7}{\left(30\right)\left(10\right)})}=0.2221111$$

Keep trying, and you'll find ...

When $J$ = 148, $SE\left({\gamma }_{01}\right)=0.1$

So you'll need 148 groups (74 treatment, 74 control)

Tools for Power Analysis

1. Stand-alone programs

   • Optimal Design
   • PinT

2. R packages

   • simr

3. Spreadsheet/Webapp

   • PowerUp!

See more discussion in Arend & Schäfer (2019)

Monte Carlo Simulation for Power Analysis

• Simulate a large number (e.g., $R$ = 1,000) of data sets based on given effect size, ICC, etc

• Fit an MLM to each simulated data

• Power $\approx$ Proportion of times $p<\alpha$

See sample R code for using simr

Uncertainty in Parameter Values

In the PowerUpR demo, to calculate the number of clusters $J$ need to achieve 80% power, we determined

1. Type I error rate = .05
2. Two tailed test = TRUE
3. g2, r21, r22 = 0, as we did not include any covariates
4. p = .5, for a balanced design (half treatment, half control)

However, we need to guess the values of

1. Effect size = .3?
2. ICC = .3?

The Effect of Uncertainty in Power

Hybrid Classical-Bayesian approach

• Incorporates uncertainty for sample size planning

• Instead of plugging in a point value of a guess, we can specify how much uncertainty we have (e.g., standard error of ${\gamma }_{01}$ from a previous study)

$$\delta \sim N(.3,.1)\rho \sim \text{Beta}(a,b)$$

• where $a$, $b$ can be calculated by $\hat{\rho }=.3$ and ${\sigma }_{\rho }=.1$ (estimate and uncertainty about $\rho$)

Additional Notes on Power

• Increasing $J$ usually leads to higher power than increasing $n$

• Balanced designs generally have higher power than unbalanced designs

• Larger sample size required for testing level-2 predictors

• Testing an interaction requires a much larger sample size

  • E.g., 16 times larger than for a main effect