| Jeanine Molock
University of Delaware
Selection bias in multilevel linear models
FINAL REPORT:
For nearly four decades, the study of what makes school effective has provided lively debates in the United States. Since the publication of the infamous "Coleman Report" (Coleman et al., 1966), researchers have tried to understand better the differences that exist between schools. Soon after its publication, several articles were written that addressed some of the methodological flaws discovered in the Coleman Report. Despite these attempts, statistical modeling issues continue to pervade school effectiveness research.
The purpose of this dissertation is to simultaneously address two methodological problems, accurately modeling both the hierarchical structure of the data and the nonrandom allocation of student to schools. It proposes an extension of propensity score methodology within a hierarchical linear model of student achievement as a way to accomplish this task. It was hypothesized that the combination of propensity score adjustment and hierarchical linear modeling would provide a more detailed analysis for the study of achievement differences between public and private school students. To demonstrate the impact that common methodologies can have on the interpretation of results, four models of tenth grade student achievement were specified. A separate model for each of the three achievement domains (reading, math, and science) was specified. These models then were analyzed using NELS:88 data with the following methodologies 1) ordinary least squares regression, 2) standard hierarchical linear modeling (HLM), 3) two-stage estimation and 4) HLM with propensity score adjustment.
Overall, the results of the first three methodologies suggested that there was an achievement advantage for private school students. In contrast, the results of the last methodology (HLM with propensity score adjustment) generally suggested that there was no academic achievement advantage for private school students in four of the five strata. This could be interpreted as suggesting that the private school achievement advantage found with the other methodologies may have been the result of hidden bias.
The potential presence of hidden bias can jeopardize the causal interpretation of the results from any model-dependent analysis of treatment effects. To determine how sensitive the results of the HLM analysis with propensity score adjustment were to hidden bias, a sensitivity analsis was conducted within each stratum. The results of the sensitivity analysis revealed that, overall, the results of the HLM analysis with propensity score adjustment relatively were insensitive to hidden bias.
Back to Funded Dissertation Grants Page |