A Multilevel Weighted Bootstrap Procedure to Handle Sampling Weights

class: center, middle, inverse, title-slide

# A Multilevel Weighted Bootstrap Procedure to Handle Sampling Weights
## 2021 AERA Virtual Meeting
### Wen Luo and Mark Lai
### Texas A&M University and University of Southern California
### 2021/04/12

---

# Overview

### Background

### Multilevel weighted bootstrap

### Simulation

### Sample Code

---

# Multistage Survey Data

Reason: Cost-effective and convenient

However, it leads to **nested data** with (usually) **unequal selection probabilities**

&zwj;Example: 2000 PISA (US data; OECD, 2000)

- First stage: probability sample of schools
    * Oversampled schools with > 15% minority students

- Second stage: probability sample of students
    * Oversampled minority students

---

# Nested Data

--
### Multilevel Modeling (MLM)

- Decompose variance into between-cluster and within-cluster components

- Model varying intercepts and slopes across clusters

### To incorporate sampling weights

Multilevel pseudo maximum likelihood (MPML)

---
# MPML (e.g., Asparouhov, 2006; Rabe-Hesketh & Skrondal, 2006)

- Maximize the weighted likelihood function

- Standard errors obtained with the sandwich estimator

### Assumptions/requirements

- Large sample (both within-cluster and between-cluster)

* Different options of scaling of level-1 weights (cf. Pfeffermann et al., 1998; Stapleton, 2002)
    
--

- Distributional assumptions

---

# Multilevel Bootstrap

- Parametric (functional form + distribution)

- Residual (functional form)

- Case (only conditional independence)
    * But requires larger samples

They are implemented in the `bootmlm` package (https://github.com/marklhc/bootmlm)

The multilevel residual bootstrap has a good balance of robustness to assumption violations and efficiency (e.g., Lai, 2020)

---
background-image: url(images/types_bootstrap.png)

# Parameteric, Residual, and Case Bootstrap

---

# Multilevel Weighted Residual Bootstrap

Builds on previous work

- Pseudopopulation (Wang & Thompson, 2012)
    
- Resampling and rescaling of weights (Kovacevic et al. 2006)

Algorithm:

1. Obtain MLM parameter estimates and residuals with unweighted ML/REML

2. Reflate residuals

3. Sample with replacement and **weights**:

- Lv 1: using lv-1 unconditional weights
    - Lv 2: using lv-2 weights

4, 5, 6. Form new responses, refit with unweighted ML, obtain bootstrap distributions

---

# Simulation

Superpopulation: `$Y_{ij} = \beta_0 + \beta_1 X1_{ij} + \beta_2 X2_j + u_{0j} + e_{ij}$`

Finite population: `$J_\text{pop} = 500$` clusters and `$n_\text{pop} = 100$` observations for each cluster

Informative sampling

- Two lv-2 strata: 70% for `$u_{0j} > 0$`, 30% for `$u_{0j} < 0$`

- Two lv-1 strata: 70% for `$e_{ij} > 0$`, 30% for `$e_{ij} < 0$`

---

# Design Factors

| Factor   | Levels   |
| -------- | -------- |
| ICC | 0.05, 0.2, 0.5 |
| Sampling fraction (both lv 1 and lv 2) | 0.1, 0.5 | 
| Distributions of random effects/errors | normal, `$\chi^2$` with df = 2 |
| Selection at lv 2 | non-informative, informative |
| Selection at lv 1 | non-informative, informative |

Total = 48 conditions

Analyses: Unweighted ML, MPML (effective weights), bootstrap

Evaluation: bias, coverage rates of 95% CI

---

# Results

Point estimates of fixed effects of `$X1$` and `$X2$` `$(\beta_1$` and `$\beta_2)$` were close to unbiased

Coverage for `$\beta_1$` was close to 95%

---
background-image: url(images/wspboot_table2.png)
background-size: contain

### Relative Bias, Normal Data

---
background-image: url(images/wspboot_table3.png)
background-size: contain

### Relative Bias, Skewed Data

---
background-image: url(images/wspboot_table4.png)
background-size: contain

### Coverage, Normal Data

---
background-image: url(images/wspboot_table5.png)
background-size: contain

### Coverage, Skewed Data

---

# Summary of Results

- Bootstrap performed similarly or better than MPML (intercept, level-2 effect)

- Bootstrap more robust for nonnormal data (level-2 variance component)

### Random slopes model

- Bias also found in level-1 effect when unequal selection was not accounted for

- No convergence issue for bootstrap; MPML has low convergence rate (0.59 to 0.76) with small samples and small ICC

- Bootstrap slightly better for lv-2 predictor; MPML slightly better for lv-1 predictor

- Bootstrap gave better variance components estimates

---

# Sample Code

```r
# Install developmental version of the bootmlm package
remotes::install_github("marklhc/bootmlm", ref = "weighted_boot")
# Load required packages
library(bootmlm)
library(boot)
library(lme4)
# Unweighted ML
m1 <- lmer(SC17Q01 ~ ISEI_m + male + (1 | Sch_ID), data = PISA, REML = FALSE)
# Weighted residual bootstrap
boo <- bootstrap_mer(
  m1, 
  FUN = function(x) {
    c(x@beta, c(x@theta ^ 2, 1) * sigma(x) ^ 2)
  }, nsim = 999L, type = "residual_cgr",
  w1 = PISA$W_FSTUWT,  # unconditional student weights
  w2 = unique(PISA[c("Sch_ID", "WNRSCHBW")])$WNRSCHBW  # school weights
)
# Print the output
boo  # bootstrap results
colMeans(boo$t)  # parameter estimates
apply(boo$t, 2, sd)  # bootstrap SE
# Percentile intervals
boot.ci(boo, type = "perc", index = 1L)
boot.ci(boo, type = "perc", index = 2L)
boot.ci(boo, type = "perc", index = 3L)
boot.ci(boo, type = "perc", index = 4L)
boot.ci(boo, type = "perc", index = 5L)
```

---

# Conclusion and Implications

- Multilevel weighted bootstrap is a good alternative to MPML to handle sampling weights

* Especially when MPML does not converge (usually with small sample and ICC)
    
    * when normality may not hold

- With bootstrap, statistical inference for `$Cov(u_0, u_1)$` is not trustworthy (reason not clear)

- Researchers should conduct sensitivity analysis with different methods (ML, MPML, weighted bootstrap)

---

# References

<small>

Asparouhov, T. (2006). General multi-level modeling with sampling weights. Communications in Statistics—Theory and Methods, 35(3), 439-460.

Kovacevic, M. S., Huang, R., & You, Y. (2006). Bootstrapping for variance estimation in multi-level models fitted to survey data. ASA Proceedings of the Survey Research Methods Section, 3260-3269.

Lai, M. H. C. (2020). Bootstrap confidence intervals for multilevel standardized effect size. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2020.1746902

Rabe‐Hesketh, S., & Skrondal, A. (2006). Multilevel modelling of complex survey data. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(4), 805-827.

Organization for Economic Co-operation and Development (2000) Manual for the PISA 2000 Database. Paris: Organization for Economic Co-operation and Development. Retrieved from http://www.pisa.oecd.org/dataoecd/53/18/33688135.pdf

Pfeffermann, D., Skinner, C. J., Holmes, D. J., Goldstein, H., Rasbash, J. (1998). Weighting for unequal selection probabilities in multi-level models. Journal of the Royal Statistics Society: Series B (Statistical Methodology), 60(1): 23–56.

Stapleton, L. (2002). The incorporation of sample weights into multilevel structural equation models. Structural Equation Modeling, 9(4): 475–502.

Wang, Z., & Thompson, M. E. (2012). A resampling approach to estimate variance components of multilevel models. Canadian Journal of Statistics, 40(1), 150–171. https://doi.org/10.1002/cjs.10136

</small>

---

class: center, middle

# Thanks!

Slides created via the R package [**xaringan**](https://github.com/yihui/xaringan).

For questions, please email Wen (wluo@tamu.edu) or Mark (hokchiol@usc.edu).