Imagine it’s October, and there’s a struggling first grader just getting to know his classmates, the teacher, which bus number is his, etc. Then his unemployed father gets a well-paid job in another state. They pack up and leave in a week.
Research tends to show (though this is not as consistent as you might think!) that when students switch schools, grades go down. But, as often is the case, it isn’t clear whether switching schools is a cause of the observed deterioration in academic performance, or just a symptom of general familial instability that causes both.
I recently had the opportunity to explore this in the ECLS-K, and after prodding the data, I seemed to find essentially no average negative effect of switching schools on child outcomes, for grades K through 5, even without the inclusion of controls. I thought this was crazy. Maybe I made a programming mistake?
It turns out I’m not crazy. Three researchers from the University of Michigan, David Burkam, Valerie Lee, and Julie Dwyer, have a 2009 working paper floating on the web, titled “School Mobility in the Early Elementary Grades: Frequency and Impact From Nationally-Representative Data.” Their research asks some simple but important questions about student mobility in grades K through 3:
- Who changes schools?
- Are changes usually during or between school years? Are they usually for structural (that is, the school’s grade span is K through 1st) or family (all other) reasons?
- What’s the impact on math & reading achievement?
- Are these impacts different for different types of kids?
The dataset they use is the ECLS-K, a longitudinal, nationally-reprsentative school-clustered survey of students, their teachers, and their parents, produced by the NCES.
Caveats of the ECLS-K
It is not a trivial matter to use the ECLS-K for this analysis, since students who switch schools are very likely to drop out of the survey. The details regarding how they and the NCES handle this serious problem are contained in their paper. It suffices to say here that inference is largely dependent on the NCES-provided sample weights being accurate.
The ECLS-K also suffers because it is a “spotty survey,” that is, students are surveyed in Fall K, Spring K, Fall 1st (though that’s a 30% subsample only!), Spring 1st and Spring 3rd. (They are also surveyed in Spring 5th and Spring 8th, but these researchers do not use this data.) So, we can only talk about certain moves.
Results
For the moves they can talk about, here are the results in a nutshell (see the paper for more details, including quantitative findings):
Percent of American student population who switches schools, by grade interval of the switch
Generally the results suggest that switching schools is not on average harmful, and moreover, if any subgroups are harmed from switching schools in these early grades, they are likely to be: (a) students who switch schools over and over again, (b) students from low SES families, (c) disabled students (students receiving special education services).
Clustering
To briefly speak of clustering, the ECLS-K is a clustered survey, so it is important to cluster standard errors at the school-level. They talk about so-called "design effects"--another jargon for clustering--in their endnote 8. They suggest using the conservative p<0.01 test for hypotheses in order to adjust for the clustering of the data in schools, since they don't do it for us. I think the problem they have in their case is that it's difficult to write down a coherent clustered OLS model (with a "school effect") when students switch schools--and I think this might be what they are saying on page 21 when they talk about hierarchical linear models, but I'm not sure.
References for that interactive bar chart
I learned from Mike Bostock's "Let's make a bar chart", Frank Guerino's "Multiple D3 Vertical Bar Charts Mixed In With Common HTML Layout Constructs", and of course the many contributors to StackExchange.
No comments:
Post a Comment