MSPnet Blog: “Test scores: What do they really tell us?”
posted December 14, 2016 – by Brian Drayton
Back when high stakes tests were the Big New Thing, and Massachusetts was bringing in its MCAS tests, researchers noted early on that the strongest predictor of school performance was demography (including things like median household income, educational attainment, etc.). This was a finding that was not new, and not unique — indeed, similar results were widespread. The notion at the time was that the test data would lead to the identification of low-performing schools (leading to interventions that would improve them), and of schools outperforming their demography (possibly indicating the presence of contributing factors that could be replicated elsewhere).
Well, as it turns out, the story has not changed that much. Here we have now a recent study from New Jersey by Christopher Tienken of Seton Hall Univ. and colleagues, which finds that a model built on just 3 demographic factors provides the most accurate predictor of middle school student results on the statewide standardized tests. (h/t to Curmudgucation again, which see for more commentary). The 3 factors are: [a] Percentage of families in a community with incomes over $200,000/year; [b] percentage of people in a community in poverty, and [c] percentage of people in a community with bachelor’s degrees. Just as Gaudet in the MCAS paper mentioned above found,the fundamental equation remains:
DEMOGRAPHY + school = results
Tienken et al. have a very insightful discussion of what middle school is all about — the subjects targeted by the standardized tests are hardly the most important things young adolescents are learning during these years.
They also point out that demography is a proxy — that it stands in for things like summer learning opportunities, enriched after-school opportunities, and homes and communities that can provide cognitive and affective advantages, especially for children whose parents can afford them. Despite all the qualifications and caveats one can make about how various public and private institutions can address some of these issues, the fact is that there has been over the past few decades a fairly steady retreat from such equalizing of opportunities, and in any case they are rarely enough, for enough children.
I think of this as I watch the news of the past few months and years, and see school choice and similar measures gain in popularity again, and as market thinking continues to consolidate its over-extended hold on American thinking about just about anything. Suppose I can choose to send my child to a more opulent school, thanks to vouchers from heaven. This will not help me purchase tutors, or summer camp, or enable me to work fewer hours so I can be at home reading books with my children, or playing music with them, or engaging in chores and crafts…. No wonder some advocates of the mainstream “reforms” of the past few decades are feeling a bit blue about how their big experiment (conducted on our children and teachers and parents and…) is turning out.
How does this look from where you are?
Note: The opinions expressed in this blog are those of the writer alone. Do not blame MSPnet, TERC, or the National Science Foundation for them. In fact, don’t “blame” — post a comment and build a conversation!
Demography as a predictor
posted by: David Carraher on 12/23/2016 11:30 am
Great, thought-provoking post! The recent findings of Tienken and colleagues that you draw attention to are truly remarkable. (inter-school variation in student achievement on state tests can be largely accounted for by (a) the percentage of B.A. degrees, (b) the percentage of households having an income greater than $200k per year and (c) the percentage of households having incomes less than $35K per year.)
Strong associations between demography and academic success at first might seem to suggest that little can be done, until major economic disparities are eradicated, to overcome the circumstances that students from disadvantaged backgrounds find themselves in. For example, according to Curmudgucation, to whom you also refer, "...Tienken et. al. have demonstrated that we do not need to actually give the Big Standardized Test in order to generate the "student achievement" data, because we can generate the same data by looking at demographic information all by itself."
Curmudgucation's interpretation is potentially misleading. The "student achievement data" refers only to variation in inter-school averages (as large as they may be). There is a substantial amount of variation within schools that depends on teacher effectiveness, as the Measures of Effective Teaching project (Kane et al., 2012) has revealed, using random assignment of students to teachers within schools. Using a combined measure of prediction, the difference between a teacher at 25th percentile and 75th percentile of effectiveness corresponded to almost 8 months in state-mandated tests and 4.5 months on Balanced Assessment in Mathematics. The striking difference arose after only one year of teaching!
This still leaves room for inter-student variability after the first two-sources (community demographics and teacher effectiveness) have been accounted for. It would be useful to know the relative weights of these three "sources of variance".
And demography can be deceiving. ( "Demography is a proxy" is Tienken (and you) stated.)
Several decades ago Wells (1981) carried out an important investigation of pre-school factors and behaviors that predict student achievement in the first years of schooling. Although parental background (based on parents' occupational level and terminal level of education) correlated (r=0.58) with success in early schooling, certain habitual behaviors of parents and their preschool children play an important role. The pre-school child's "concentration on literacy"-proclivity to read-was even more strongly related to early school success (r= 0.69). And the number of books owned by the child was moderately associated (r= 0.59) with success in school. Remarkably, whether or not the mother worked, and how much time the mother spent with her child were not associated with success in school. But the parents' interest in literacy was.
Treisman (1992) found that ethnic differences in achievement at the college level were closely linked to the social support available to students:
"It was interesting to see how the Chinese students learned from each other. They would edit one another's solutions. A cousin or an older brother would come in and test them. They would regularly work problems from old exams, which are kept in a public file in the library. They would ask each other questions like, "How many hours did you stay up last night?" They knew exactly where they stood in the class. They had constructed something like a truly academic fraternity..."
It would be a mistake to regard class and ethnic differences in academic success as the inevitable outcome of "demography". Clearly, there are demographic differences in opportunities for learning. But even within restrictive conditions, it matters immensely what teachers, parents, students (and their peers) actually do.
Kane, T. J., & Staiger, D. O. (2012). Gathering Feedback for Teaching: Combining High-Quality Observations with Student Surveys and Achievement Gains. Research Paper. MET Project. Bill & Melinda Gates Foundation. 68pp.
Treisman, U. (1992). Studying students studying calculus: A look at the lives of minority mathematics students in college. The College Mathematics Journal, 23(5), 362-372.
Wells, G. (1981). Some Antecedents of Early Educational Attainment. British Journal of Sociology of Education, 2(2), 181-200.
post updated by the author 1/3/2017
Teacher Effectiveness and Demography
posted by: David Carraher on 12/26/2016 2:24 pm
One more thing: I leave aside, for the moment, the important issues of the validity and desirability of state-mandated tests.
post updated by the author 1/3/2017
Teacher demographics
posted by: Larry Suter on 12/27/2016 8:52 am
Actually, analyses of survey data shows not much difference in teacher demographics (race particularly, age might be different, six is also perhaps different) and student performance. But the interactions are somewhat complex because in specific circumstances it does make a difference.
Teacher effectiveness and community demographics
posted by: David Carraher on 12/27/2016 11:47 am
Thanks. So, is your point that the *personal demographics of teachers* are not significantly associated with their effectiveness as teachers? (Were you referring to MET research?)
In my second posting I was wondering about the association between *community demographics* and teacher effectiveness--"community" being the town or neighborhood of the school's student population. Put simply, are the students from advantaged communities more likely to have effective teachers--effective in terms of teaching practices highlighted by classroom observation measures such as RTOP, UTOP, CLASS, Framework for Teaching, or MQI?