| Xuejun (Ina) Shen
Stanford University
Helping or harming: Do unintended effects of high-stakes testing hit disadvantaged schools harder?
FINAL REPORT
The purpose of the study was to test the hypothesis that disadvantaged schools are more likely to engage in narrowly-targeted test preparation activities in response to high-stakes testing. We hypothesized that such "teaching to the test" would result in larger gains over time on some test items than others. In particular, performance on items that teachers judged most "teachable" would show bigger improvements over time if the same high-stakes test was used year after year. Data for the study came from California, which used the same high-stakes test from 1998 to 2002. We first looked for the hypothesized pattern in each elementary school in California, then checked to see if this pattern was more prevalent in disadvantaged (quintile 1) schools versus advantaged (quintile 5) schools. The disadvantaged schools were indicated by nine variables in three dimensions: high proportion of low SES students, high proportion of minority students, and low teacher quality. To the end of testing the hypothesis, we developed an experimental approach to investigate the probability of school engagement in narrowly targeted test preparation activities, mainly drawing upon the framework for validating gains under high-stakes conditions, Q-sort survey methodology, SIBTEST-based Differential Bundle Functioning (DBF) method, and False Discovery Rate (FDR) statistical controlling procedure. This approach was applied to the data of a 3rd grade math problem solving subtest on a California high-stakes testing program. First, we used the Q-sort method to interview nine classroom teachers, and asked them to sort the test items into piles according to how hard versus easy it would be to teach the knowledge and skills assessed by the test items. Next, we used the SIBTEST-based DBF method to test whether the subtest of the easy to teach items has anomalously larger gains over time than the subtest of the hard to teach items at the school level. The DBF analysis reached a probability level for each school. Next, we linked the calculated school-level test preparation probabilities with the school characteristic variables from the National Center for Education Statistics and California Department of Education's datasets. The significances of school-level probabilities were then tested using the FDR controlling procedure. Then we sorted schools by respective school characteristic variables, and compared the true proportions of schools with evidence of narrowly targeted test preparation between the most and least disadvantaged quintile schools. The findings support the hypothesis, and challenge the validity of gap closing for disadvantaged students on state tests.
Back to Funded Dissertation Grants Page |