Outline

Single-Level Reliability

Multilevel Data

• Types of composites
• Reliability indices

Longitudinal Data?

Reliability

• Psychological scales are not perfect

• Certain level of reliability needed

  • Statistical analyses are not trustworthy when the numbers are not consistent

APA Journal Article Reporting Standards (JARS)

• In the Psychometrics section (Appelbaum, et al., 2018), researchers were asked to

Estimate and report values of reliability coefficients for the scores analyzed (i.e., the research's sample) (p. 7)

Classical Test Theory

Reliability $\rho =\frac{{\sigma }_T^2}{{\sigma }_Y^2}=\frac{{\sigma }_T^2}{{\sigma }_T^2+{\sigma }_E^2}=\left[Corr\right(Y,T){]}^2$

Composite Reliability

(Essential) Tau Equivalence

$\alpha$ Reliability

Variance of unweighted composite: $$V\left(Z\right)=p^2\psi +\sum _k{\theta }_k$$ Reliability $\alpha =\frac{p^2\psi }{V(p^2\psi +{\sum }_k{\theta }_k)}$

Alternative form: $\alpha =\frac{p}{p-1}(1-\frac{\mathrm{tr}\left(\Sigma \right)}{1^{\prime }\Sigma 1})$

Composite Reliability

Congeneric

$\omega$ Reliability

• True Score Variance $V\left(T\right)=({\sum }_k{\lambda }_k{)}^2\psi$

• Error Variance $V\left(E\right)={\sum }_k{\theta }_k$

Reliability $\omega =\frac{V\left(T\right)}{V\left(T\right)+V\left(E\right)}$

Note: Reliability is a property of observed test scores $\left(Z\right)$, not the latent scores $\left(\eta \right)$

Example

2007 Trends in International Mathematics and Science Study (TIMSS; Williams et al., 2009)

• 7,896 students (4th grade) from 515 schools

Positive attitudes toward math (PATM)

Multilevel Reliability Not Consistently Reported

Kim et al. (2016): Only 54% reported reliability, among 39 articles using multilevel confirmatory factor analysis (MCFA)

• Usually only one reliability reported for one scale

Reliability Options in the Literature

Ignore clustering

• Omit uncertainty (SE too small/CI too narrow)

  • Can use multilevel factor analysis (Raykov, et al., 2005; Raykov, et al., 2006)

• No level-specific information

Level-Specific Reliability (Geldhof, et al., 2014)

For the TIMSS data

• Est ${\tilde{\omega }}^w$ = .857, 95% CI [.849, .863]

• Est ${\tilde{\omega }}^b$ = .977, 95% CI [.964, .987]

Not uncommon in the literature . . .

• Positive and negative affects: ${\tilde{\omega }}^b$ = .94 to .97 (Rush, et al., 2014)

• Instructional Skills Questionnaire: ${\tilde{\alpha }}^b$ between .90 to .99 (Knol, et al., 2016)

What Went Wrong?

What Are the Observed Scores With Multilevel Analysis?

• Raw/Overall composite $\left(Z_{ij}\right)$
• Composite school means of (cluster mean; $Z_j^b$)
• Composite student deviations (cluster-mean centered; $Z_{ij}^w=Z_{ij}-Z_j^b$)

Sampling Error of School Means

Variance Components in Reliability Coefficients

Within $$\frac{V\left(T^w\right)}{V\left(T^w\right)+V\left(E^w\right)}$$

Between $$\frac{V\left(T^b\right)}{V\left(T^b\right)+V\left(E^b\right)+\underset{\text{sampling error}}{\underset{}{\left[V\right(T^w)+V(E^w\left)\right]/n}}}$$

Raw $$\frac{V\left(T^w\right)+V\left(T^b\right)}{V\left(T^w\right)+V\left(E^w\right)+V\left(T^b\right)+V\left(E^b\right)}$$

Extensions of $\alpha$

First, obtain estimated covariance matrices at the between $\left({\Sigma }^b\right)$ and the within levels $\left({\Sigma }^w\right)$

$$\begin{array}{cc}{\alpha }^{2l}& =\frac{p}{p-1}\left(\frac{{\sum }_{k\ne k^{\prime }}({\sigma }_{k{k}^{\prime }}^b+{\sigma }_{k{k}^{\prime }}^w)}{1^{\prime }{\Sigma }^{b}1+1^{\prime }{\Sigma }^{w}1}\right)\\ {\alpha }^b& =\frac{p}{p-1}\left(\frac{{\sum }_{k\ne k^{\prime }}{\sigma }_{k{k}^{\prime }}^b}{1^{\prime }{\Sigma }^{b}1+1^{\prime }{\Sigma }^{w}1/\tilde{n}}\right)\\ {\alpha }^w& =\frac{p}{p-1}\left(\frac{{\sum }_{k\ne k^{\prime }}{\sigma }_{k{k}^{\prime }}^w}{1^{\prime }{\Sigma }^{w}1}\right)\end{array}$$

$\tilde{n}$ = harmonic mean of cluster sizes (e.g., number of students in each school in the sample)

Extensions of $\omega$

Other Model Considerations

Which One(s) to Report?

• If a variable is partitioned in a multilevel model (most likely an individual construct), all three $({\omega }^{2l},{\omega }^b,{\omega }^w)$ should be reported

  • Cluster means and cluster-mean centered predictors

  • Outcome variable

R Function multilevel_reliability()

multilevel_reliability(
  timss_usa[c("AS4MAMOR", "AS4MAENJ", "AS4MALIK", "AS4MABORr")],
  id = timss_usa$IDSCHOOL
)

## Parallel analysis suggests that the number of factors =  NA  and the number of components =  1 
## Parallel analysis suggests that the number of factors =  NA  and the number of components =  1

## $alpha
##   alpha2l    alphab    alphaw 
## 0.8675110 0.6527865 0.8592387 
## 
## $alpha_ci
##              2.5%     97.5%
## alpha2l 0.8604497 0.8741798
## alphab  0.5802825 0.7046562
## alphaw  0.8517564 0.8663573
## 
## $omega
##   omega2l    omegab    omegaw 
## 0.8674844 0.6233126 0.8605125 
## 
## $omega_ci
##              2.5%     97.5%
## omega2l 0.8602266 0.8739374
## omegab  0.5501061 0.6761630
## omegaw  0.8529576 0.8672256
## 
## $ncomp
##  within between 
##       1       1

Midlife in the United States

Data from MIDUS 2: Daily Stress Project, 2004-2009 (Ryff, et al., 2009)

• 2,022 participants, 8 days each

• Target construct: Positive affect

• Type of scores: raw composite, person means, person-mean centered

However, the model assumes no time-specific variances and covariances

Incorporating Time

Many Questions Remain

1. Discrete indicators?

2. Should we treat within-person and between-person constructs as parts of one construct?

   • Local homogeneity assumption (Ellis, et al., 1993)

1. Time-varying reliability?

   • Martin & Rast (2020, PsyArXiv)

1. Are there "shared" constructs at the person level?

   • E.g., response bias --> item scores more correlated at the between level

Summary

Computing and reporting reliability information is important for multilevel data

• Both cross-sectional and longitudinal

Reliability information is needed for raw, cluster means, and cluster-mean centered scores

Previous approach to between-level reliability is an overestimate when cluster size is small

Nature of target construct should be considered, and it has implications on reliability computation

• E.g., "shared" constructs

Acknowledgement

• Collaborators: Feng Ji (UC Berkeley) and Shi Chen (Northern Arizona University)

• Oi-man Kwok and Hio Wa Mak for suggestions on the paper

References

Appelbaum, M. et al. (2018). "Journal article reporting standards for quantitative research in psychology". In: American Psychologist 73.1, pp. 3-25. DOI: 10.1037/amp0000191.

Asparouhov, T. et al. (2016). "General random effect latent variable modeling: Random subjects, items, contexts, and parameter". In: Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications. Charlotte, NC: Information Age, pp. 163-192.

Ellis, J. L. et al. (1993). "Local homogeneity in latent trait models. A characterization of the homogeneous monotone irt model". In: Psychometrika 58.3, pp. 417-429. DOI: 10.1007/bf02294649.

Geldhof, G. J. et al. (2014). "Reliability estimation in a multilevel confirmatory factor analysis framework". In: Psychological Methods 19.1, pp. 72-91. DOI: 10.1037/a0032138.

References (cont'd)

Jak, S. et al. (2014). "Measurement bias in multilevel data". In: Structural Equation Modeling: A Multidisciplinary Journal 21.1, pp. 31-39. DOI: 10.1080/10705511.2014.856694.

Jeon, M. et al. (2012). "Profile-likelihood approach for estimating generalized linear mixed models with factor structures". In: Journal of Educational and Behavioral Statistics 37.4, pp. 518-542. DOI: 10.3102/1076998611417628.

Knol, M. H. et al. (2016). "Measuring the quality of university lectures: Development and validation of the Instructional Skills Questionnaire (ISQ)". In: PLOS ONE 11.2. Ed. by D. S. Courvoisier, p. e0149163. DOI: 10.1371/journal.pone.0149163.

Lai, M. H. C. (2021). "Composite reliability of multilevel data: It’s about observed scores and construct meanings." In: Psychological Methods 26 (1). DOI: 10.1037/met0000287.

References (cont'd)

Mehta, P. D. et al. (2005). "People are variables too: Multilevel structural equations modeling". In: Psychological Methods 10.3, pp. 259-284. DOI: 10.1037/1082-989X.10.3.259.

Raudenbush, S. W. et al. (2002). Hierarchical linear models: Applications and data analysis methods. 2nd ed. Thousand Oaks, CA: Sage. ISBN: 076191904X.

Raykov, T. et al. (2006). "On multilevel model reliability estimation from the perspective of structural equation modeling". In: Structural Equation Modeling: A Multidisciplinary Journal 13.1, pp. 130-141. DOI: 10.1207/s15328007sem1301_7.

Raykov, T. et al. (2005). "Estimation of reliability for multiple-component measuring instruments in hierarchical designs". In: Structural Equation Modeling: A Multidisciplinary Journal 12.4, pp. 536-550. DOI: 10.1207/s15328007sem1204_2.

References (cont'd)

Rush, J. et al. (2014). "Differences in within- and between-person factor structure of positive and negative affect: Analysis of two intensive measurement studies using multilevel structural equation modeling.". In: Psychological Assessment 26.2, pp. 462-473. DOI: 10.1037/a0035666.

Ryff, C. D. et al. (2009). Midlife in the United States (MIDUS 2): Daily Stress Project, 2004-2009: Version 2. type: dataset. DOI: 10.3886/ICPSR26841.V2.

Stapleton, L. M. et al. (2019). "Models to Examine the Validity of Cluster-Level Factor Structure Using Individual-Level Data". In: Advances in Methods and Practices in Psychological Science 2.3, pp. 312-329. DOI: 10.1177/2515245919855039.

Stapleton, L. M. et al. (2016). "Construct meaning in multilevel settings". In: Journal of Educational and Behavioral Statistics 41.5, pp. 481-520. DOI: 10.3102/1076998616646200.

Influence of ICC and Cluster Size

Assume cross-level invariance holds and ${\omega }^{2l}=.8$

$ICC\left(\eta \right)=\frac{{\psi }^b}{{\psi }^b+{\psi }^w}$

Relation to Generalizability Theory

Most meaningful with systematic time effects

Cranford, et al. (2006): generalizability coefficients for diary studies

‍Differences:

• Fixed vs. Random item facet (in estimation)

• Relax the essential parallel test assumption

  • Item-specific loadings and uniqueness

• Test invariance assumptions

• Flexible SEM modeling