A Multilevel Weighted Bootstrap Procedure to Handle Sampling Weights2021 AERA Virtual MeetingWen Luo and Mark LaiTexas A&M University and University of Southern California2021/04/121 / 20

OverviewBackgroundMultilevel weighted bootstrapSimulationSample Code2 / 20

Multistage Survey Data

Reason: Cost-effective and convenient

However, it leads to nested data with (usually) unequal selection probabilities

3 / 20

Multistage Survey Data

Reason: Cost-effective and convenient

However, it leads to nested data with (usually) unequal selection probabilities

‍Example: 2000 PISA (US data; OECD, 2000)

First stage: probability sample of schools
- Oversampled schools with > 15% minority students

3 / 20

Multistage Survey Data

Reason: Cost-effective and convenient

However, it leads to nested data with (usually) unequal selection probabilities

‍Example: 2000 PISA (US data; OECD, 2000)

First stage: probability sample of schools
- Oversampled schools with > 15% minority students

Second stage: probability sample of students
- Oversampled minority students

3 / 20

Nested Data4 / 20

Nested DataMultilevel Modeling (MLM)4 / 20

Nested Data

Multilevel Modeling (MLM)

Decompose variance into between-cluster and within-cluster components
Model varying intercepts and slopes across clusters

4 / 20

Nested Data

Multilevel Modeling (MLM)

Decompose variance into between-cluster and within-cluster components
Model varying intercepts and slopes across clusters

To incorporate sampling weights

4 / 20

Nested Data

Multilevel Modeling (MLM)

Decompose variance into between-cluster and within-cluster components
Model varying intercepts and slopes across clusters

To incorporate sampling weights

Multilevel pseudo maximum likelihood (MPML)

4 / 20

MPML (e.g., Asparouhov, 2006; Rabe-Hesketh & Skrondal, 2006)

Maximize the weighted likelihood function
Standard errors obtained with the sandwich estimator

5 / 20

MPML (e.g., Asparouhov, 2006; Rabe-Hesketh & Skrondal, 2006)

Maximize the weighted likelihood function
Standard errors obtained with the sandwich estimator

Assumptions/requirements

Large sample (both within-cluster and between-cluster)
- Different options of scaling of level-1 weights (cf. Pfeffermann et al., 1998; Stapleton, 2002)

5 / 20

MPML (e.g., Asparouhov, 2006; Rabe-Hesketh & Skrondal, 2006)

Maximize the weighted likelihood function
Standard errors obtained with the sandwich estimator

Assumptions/requirements

Large sample (both within-cluster and between-cluster)
- Different options of scaling of level-1 weights (cf. Pfeffermann et al., 1998; Stapleton, 2002)
Distributional assumptions

5 / 20

Multilevel Bootstrap

Parametric (functional form + distribution)
Residual (functional form)
Case (only conditional independence)
- But requires larger samples

6 / 20

Multilevel Bootstrap

Parametric (functional form + distribution)
Residual (functional form)
Case (only conditional independence)
- But requires larger samples

They are implemented in the bootmlm package (https://github.com/marklhc/bootmlm)

6 / 20

Multilevel Bootstrap

Parametric (functional form + distribution)
Residual (functional form)
Case (only conditional independence)
- But requires larger samples

They are implemented in the bootmlm package (https://github.com/marklhc/bootmlm)

The multilevel residual bootstrap has a good balance of robustness to assumption violations and efficiency (e.g., Lai, 2020)

6 / 20

Parameteric, Residual, and Case Bootstrap7 / 20

Multilevel Weighted Residual Bootstrap

Builds on previous work

Pseudopopulation (Wang & Thompson, 2012)
Resampling and rescaling of weights (Kovacevic et al. 2006)

8 / 20

Multilevel Weighted Residual Bootstrap

Builds on previous work

Pseudopopulation (Wang & Thompson, 2012)
Resampling and rescaling of weights (Kovacevic et al. 2006)

Obtain MLM parameter estimates and residuals with unweighted ML/REML

8 / 20

Multilevel Weighted Residual Bootstrap

Builds on previous work

Pseudopopulation (Wang & Thompson, 2012)
Resampling and rescaling of weights (Kovacevic et al. 2006)

Obtain MLM parameter estimates and residuals with unweighted ML/REML
Reflate residuals

8 / 20

Multilevel Weighted Residual Bootstrap

Builds on previous work

Pseudopopulation (Wang & Thompson, 2012)
Resampling and rescaling of weights (Kovacevic et al. 2006)

Obtain MLM parameter estimates and residuals with unweighted ML/REML
Reflate residuals
Sample with replacement and weights:
- Lv 1: using lv-1 unconditional weights
- Lv 2: using lv-2 weights

8 / 20

Multilevel Weighted Residual Bootstrap

Builds on previous work

Pseudopopulation (Wang & Thompson, 2012)
Resampling and rescaling of weights (Kovacevic et al. 2006)

Obtain MLM parameter estimates and residuals with unweighted ML/REML
Reflate residuals
Sample with replacement and weights:
- Lv 1: using lv-1 unconditional weights
- Lv 2: using lv-2 weights

4, 5, 6. Form new responses, refit with unweighted ML, obtain bootstrap distributions

8 / 20

Simulation

Superpopulation: $Y_{i j} = β_{0} + β_{1} X 1_{i j} + β_{2} X 2_{j} + u_{0 j} + e_{i j}$

9 / 20

Simulation

Superpopulation: $Y_{i j} = β_{0} + β_{1} X 1_{i j} + β_{2} X 2_{j} + u_{0 j} + e_{i j}$

Finite population: $J_{pop} = 500$ clusters and $n_{pop} = 100$ observations for each cluster

9 / 20

Simulation

Superpopulation: $Y_{i j} = β_{0} + β_{1} X 1_{i j} + β_{2} X 2_{j} + u_{0 j} + e_{i j}$

Finite population: $J_{pop} = 500$ clusters and $n_{pop} = 100$ observations for each cluster

Informative sampling

Two lv-2 strata: 70% for $u_{0 j} > 0$ , 30% for $u_{0 j} < 0$
Two lv-1 strata: 70% for $e_{i j} > 0$ , 30% for $e_{i j} < 0$

9 / 20

Design Factors

Factor
Levels


ICC
0.05, 0.2, 0.5

Sampling fraction (both lv 1 and lv 2)
0.1, 0.5

Distributions of random effects/errors
normal, χ2χ2 with df = 2

Selection at lv 2
non-informative, informative

Selection at lv 1
non-informative, informative

10 / 20

Design Factors

Factor	Levels
ICC	0.05, 0.2, 0.5
Sampling fraction (both lv 1 and lv 2)	0.1, 0.5
Distributions of random effects/errors	normal, $χ^{2}$ with df = 2
Selection at lv 2	non-informative, informative
Selection at lv 1	non-informative, informative

Total = 48 conditions

Analyses: Unweighted ML, MPML (effective weights), bootstrap

Evaluation: bias, coverage rates of 95% CI

10 / 20

Results

Point estimates of fixed effects of $X 1$ and $X 2$ $(β_{1}$ and $β_{2})$ were close to unbiased

Coverage for $β_{1}$ was close to 95%

11 / 20

Relative Bias, Normal Data12 / 20

Relative Bias, Skewed Data13 / 20

Coverage, Normal Data14 / 20

Coverage, Skewed Data15 / 20

Summary of Results

Bootstrap performed similarly or better than MPML (intercept, level-2 effect)
Bootstrap more robust for nonnormal data (level-2 variance component)

16 / 20

Summary of Results

Bootstrap performed similarly or better than MPML (intercept, level-2 effect)
Bootstrap more robust for nonnormal data (level-2 variance component)

Random slopes model

Bias also found in level-1 effect when unequal selection was not accounted for

16 / 20

Summary of Results

Bootstrap performed similarly or better than MPML (intercept, level-2 effect)
Bootstrap more robust for nonnormal data (level-2 variance component)

Random slopes model

Bias also found in level-1 effect when unequal selection was not accounted for
No convergence issue for bootstrap; MPML has low convergence rate (0.59 to 0.76) with small samples and small ICC

16 / 20

Summary of Results

Bootstrap performed similarly or better than MPML (intercept, level-2 effect)
Bootstrap more robust for nonnormal data (level-2 variance component)

Random slopes model

Bias also found in level-1 effect when unequal selection was not accounted for
No convergence issue for bootstrap; MPML has low convergence rate (0.59 to 0.76) with small samples and small ICC
Bootstrap slightly better for lv-2 predictor; MPML slightly better for lv-1 predictor

16 / 20

Summary of Results

Bootstrap performed similarly or better than MPML (intercept, level-2 effect)
Bootstrap more robust for nonnormal data (level-2 variance component)

Random slopes model

Bias also found in level-1 effect when unequal selection was not accounted for
No convergence issue for bootstrap; MPML has low convergence rate (0.59 to 0.76) with small samples and small ICC
Bootstrap slightly better for lv-2 predictor; MPML slightly better for lv-1 predictor
Bootstrap gave better variance components estimates

16 / 20

Sample Code

# Install developmental version of the bootmlm package
remotes::install_github("marklhc/bootmlm", ref = "weighted_boot")
# Load required packages
library(bootmlm)
library(boot)
library(lme4)
# Unweighted ML
m1 <- lmer(SC17Q01 ~ ISEI_m + male + (1 | Sch_ID), data = PISA, REML = FALSE)
# Weighted residual bootstrap
boo <- bootstrap_mer(
  m1, 
  FUN = function(x) {
    c(x@beta, c(x@theta ^ 2, 1) * sigma(x) ^ 2)
  }, nsim = 999L, type = "residual_cgr",
  w1 = PISA$W_FSTUWT,  # unconditional student weights
  w2 = unique(PISA[c("Sch_ID", "WNRSCHBW")])$WNRSCHBW  # school weights
)
# Print the output
boo  # bootstrap results
colMeans(boo$t)  # parameter estimates
apply(boo$t, 2, sd)  # bootstrap SE
# Percentile intervals
boot.ci(boo, type = "perc", index = 1L)
boot.ci(boo, type = "perc", index = 2L)
boot.ci(boo, type = "perc", index = 3L)
boot.ci(boo, type = "perc", index = 4L)
boot.ci(boo, type = "perc", index = 5L)

17 / 20

Conclusion and Implications

Multilevel weighted bootstrap is a good alternative to MPML to handle sampling weights
- Especially when MPML does not converge (usually with small sample and ICC)
- when normality may not hold

18 / 20

Conclusion and Implications

Multilevel weighted bootstrap is a good alternative to MPML to handle sampling weights
- Especially when MPML does not converge (usually with small sample and ICC)
- when normality may not hold

With bootstrap, statistical inference for $C o v (u_{0}, u_{1})$ is not trustworthy (reason not clear)

18 / 20

Conclusion and Implications

Multilevel weighted bootstrap is a good alternative to MPML to handle sampling weights
- Especially when MPML does not converge (usually with small sample and ICC)
- when normality may not hold

With bootstrap, statistical inference for $C o v (u_{0}, u_{1})$ is not trustworthy (reason not clear)
Researchers should conduct sensitivity analysis with different methods (ML, MPML, weighted bootstrap)

18 / 20

References

Asparouhov, T. (2006). General multi-level modeling with sampling weights. Communications in Statistics—Theory and Methods, 35(3), 439-460.

Kovacevic, M. S., Huang, R., & You, Y. (2006). Bootstrapping for variance estimation in multi-level models fitted to survey data. ASA Proceedings of the Survey Research Methods Section, 3260-3269.

Lai, M. H. C. (2020). Bootstrap confidence intervals for multilevel standardized effect size. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2020.1746902

Rabe‐Hesketh, S., & Skrondal, A. (2006). Multilevel modelling of complex survey data. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(4), 805-827.

Organization for Economic Co-operation and Development (2000) Manual for the PISA 2000 Database. Paris: Organization for Economic Co-operation and Development. Retrieved from http://www.pisa.oecd.org/dataoecd/53/18/33688135.pdf

Pfeffermann, D., Skinner, C. J., Holmes, D. J., Goldstein, H., Rasbash, J. (1998). Weighting for unequal selection probabilities in multi-level models. Journal of the Royal Statistics Society: Series B (Statistical Methodology), 60(1): 23–56.

Stapleton, L. (2002). The incorporation of sample weights into multilevel structural equation models. Structural Equation Modeling, 9(4): 475–502.

Wang, Z., & Thompson, M. E. (2012). A resampling approach to estimate variance components of multilevel models. Canadian Journal of Statistics, 40(1), 150–171. https://doi.org/10.1002/cjs.10136

19 / 20

Thanks!

Slides created via the R package xaringan.

For questions, please email Wen (wluo@tamu.edu) or Mark (hokchiol@usc.edu).

20 / 20

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help

A Multilevel Weighted Bootstrap Procedure to Handle Sampling Weights

2021 AERA Virtual Meeting

Wen Luo and Mark Lai

Texas A&M University and University of Southern California

2021/04/12

Overview

Background

Multilevel weighted bootstrap

Simulation

Sample Code

Multistage Survey Data

Multistage Survey Data

Multistage Survey Data

Nested Data

Nested Data

Multilevel Modeling (MLM)

Nested Data

Multilevel Modeling (MLM)

Nested Data

Multilevel Modeling (MLM)

To incorporate sampling weights

Nested Data

Multilevel Modeling (MLM)

To incorporate sampling weights

MPML (e.g., Asparouhov, 2006; Rabe-Hesketh & Skrondal, 2006)

MPML (e.g., Asparouhov, 2006; Rabe-Hesketh & Skrondal, 2006)

Assumptions/requirements

MPML (e.g., Asparouhov, 2006; Rabe-Hesketh & Skrondal, 2006)

Assumptions/requirements

Multilevel Bootstrap

Multilevel Bootstrap

Multilevel Bootstrap

Parameteric, Residual, and Case Bootstrap

Multilevel Weighted Residual Bootstrap

Multilevel Weighted Residual Bootstrap

Multilevel Weighted Residual Bootstrap

Multilevel Weighted Residual Bootstrap

Multilevel Weighted Residual Bootstrap

Simulation

Simulation

Simulation

Design Factors

Design Factors

Results

Relative Bias, Normal Data

Relative Bias, Skewed Data

Coverage, Normal Data

Coverage, Skewed Data

Summary of Results

Summary of Results

Random slopes model

Summary of Results

Random slopes model

Summary of Results

Random slopes model

Summary of Results

Random slopes model

Sample Code

Conclusion and Implications

Conclusion and Implications

Conclusion and Implications

References

Thanks!

Overview

Background

Multilevel weighted bootstrap

Simulation

Sample Code

Help