Lesson for Teach for America in statistical versus practical significance

Image: Teach for America founder Wendy Kopp (source)

A recent study has many touting the benefits of TFA. The most overheated has been Dylan Matthews (Teach for America is a deeply divisive program. It also works.), but Matthew Yglesias is similarly condescending (Since empirical evidence doesn't change anyone's mind about this issue...), though Yglesias is at least more reasoned in his assessment (see below). No, the study didn't show that TFA works. It showed that it works for one subject in economically disadvantaged schools versus inexperienced teachers, provided you assume that TFA volunteers weren't teaching to the test. And, just because something's statistically significant doesn't mean it has any real world value. I'll let Dana Goldstein take it from here:

TFA math teachers outperformed non-TFA math teachers in their schools by .06 standard deviations in middle school and .13 standard deviations in high school. The talking point will be that this is the equivalent of an additional 2.6 months of learning per schoolyear. But it's important to realize this represents a relatively modest improvement in student achievement. For the average child in this study, who scored in just the 27th percentile in math compared to her peers across the country, having a TFA teacher will help her move up to the 30th percentile--still a long way off from grade-level math proficiency.

Right. TFA "works," in that it can improve student performance in one subject from well below grade level to...still well below grade level. Also worth noting:

The students of [non-TFA] second-year teachers outperformed the students of [non-TFA] first-year teachers by .08 standard deviations--a larger gap than the average one (.07) between the students of TFA and non-TFA teachers.

And teachers with over five years of experience were even better. Remember, this is only for math in low-income schools. Goldstein also argues that the researchers have no way of controlling for TFA volunteers teaching to the test:

After observing TFA's summer training institute this July, I'd guess that there are two major factors...Second, TFA's training emphasizes data tracking of student outcomes and the importance, specifically, of raising standardized test scores. That could lead to the students of TFA teachers getting more test-prep and hearing more messages about why performing well on tests is important...

The researchers tell Dylan Matthews that although they used the results of high-stakes state exams to measure student outcomes in the middle-school grades, at the high school level, the tests they used were completely new to the teachers, so they couldn't have prepared students for them. I'd still make the point that the students of TFA teachers may be more likely to take testing seriously, for the reasons I outline above.

There are two points I want to make here:

First, TFA volunteers may well outperform traditional public school teachers (or at least the inexperienced ones) in one subject in low-income schools in a statistically significant way. But the improvement corresponds to improving three percentiles, which is practically meaningless. It could be that the effects of having a TFA teacher accumulates three percentiles every year, but the study doesn't show that; moreover, it seems ludicrous to assume that TFA would have no diminishing returns.

Those three percentiles in just one subject (prior studies have have found no evidence TFA is better in subjects other than math) should argue persuasively that if we think we can solve this problem by improving teachers, TFA is most certainly not the answer. Yglesias astutely makes the converse of this argument:

The basic upshot is that the current [teacher] certification process has no meaningful validity.

Which is absolutely true; underpaid, undertrained volunteers are probably as good as first year teachers. If you want to argue that teachers are grossly underqualified for their jobs and generally incompetent, be my guest. They are, and that's exactly my point. Finland has the highest quality teachers in the world, and their adherence to evidenced-based practices for teaching and classroom management is second to none. That's the way teaching should be done.

The difference is that Finland requires all of it's teachers to have master's degrees in education. Most American teachers just have bachelor's degrees. I think it's generally unfair to expect people without a graduate degree to review--and critically appraise--current research, and rigorously follow evidence-based practices. If you want higher quality teachers, you must give them higher quality education. If you want graduate-level work, then your workers had better have graduate degrees. If it works in other countries, why wouldn't it work here? But graduate degrees are extremely expensive, and apparently to expensive for us to consider the investment worthwhile. Instead of paying for master's degrees, we're paying TFA volunteers next to nothing for a few years before they move on to their intended career, then wondering why Finnish teachers outperform our own.

Second, there is no evidence suggesting that the gulf in performance between rich and poor students has anything to do with teacher quality. The PIRLS test is a standardized international comparison of student reading performance. This small table speaks volumes:

Students in schools attended by very few poor children easily outperformed the rest of the world. But the more poor children in a school, the worse the school performs, until schools that are predominately poor are well below the world average. Poverty would seem to play an utterly dominant role in student performance. Also:

Similar calls for more high-quality schools, however defined, issue from multiple quarters. Many critics cite the performance of American students on international comparisons of mathematics and science. The most often used comparison comes from rankings on the Programme for International Student Assessment (PISA), from the Organization for Economic Cooperation and Development (OECD). Most recently (2006), American students ranked 24th of 30 OECD nations in mathematics and 17 of 30 in science. Errors in test booklets prevented the reporting scores for American students in reading...

A publication from OECD itself observes that if one examines the number of highest-scoring students in science, the United States has 25% of all high-scoring students int eh world (at least in "the world" as defined by the 58 nations taking part int he assessment--the 30 OECD nations and 28 "partner" countries). Among nations with high average scores, Japan accounted for 13% of the highest scorers, Korea 5%, Taipei 3%, Finland 1%, and Hong Kong 1%. Singapore did not participate. The picture emerging from this highest-scorer comparison is far different than that suggested by the frequently cited national average comparisons; it is a picture that suggests many American schools are actually doing very well indeed.

This could be construed as misleading, so I'll clarify. The US is actually underrepresented in the tally of highest-scoring students; for example, the US is 2.5 times the size of Japan, but only has twice the number of top scorers; the US is six times the size of South Korea, but only has five times the number of top scorers; etc. Nevertheless, it's clear that many US schools are cranking out very smart students. It's a feast or famine situation; like in the above table, schools with very few impoverished students perform very well, and schools with high incidence of student poverty perform very poorly.

(If you're interested in how America's poor perform in comparison to the world's poor, there's more here)

In case you're concerned about ecological fallacies,

The income gap in academic achievement is not growing because the test scores of poor students are dropping or because our schools are in decline. In fact, average test scores on the National Assessment of Educational Progress, the so-called Nation’s Report Card, have been rising — substantially in math and very slowly in reading — since the 1970s. The average 9-year-old today has math skills equal to those her parents had at age 11, a two-year improvement in a single generation. The gains are not as large in reading and they are not as large for older students, but there is no evidence that average test scores have declined over the last three decades for any age or economic group...

It may seem counterintuitive, but schools don’t seem to produce much of the disparity in test scores between high- and low-income students. We know this because children from rich and poor families score very differently on school readiness tests when they enter kindergarten, and this gap grows by less than 10 percent between kindergarten and high school. There is some evidence that achievement gaps between high- and low-income students actually narrow during the nine-month school year, but they widen again in the summer months.

That isn’t to say that there aren’t important differences in quality between schools serving low- and high-income students — there certainly are — but they appear to do less to reinforce the trends than conventional wisdom would have us believe...

It’s not clear what we should do about all this. Partly that’s because much of our public conversation about education is focused on the wrong culprits: we blame failing schools and the behavior of the poor for trends that are really the result of deepening income inequality...

And here are some of the conclusions from a report comparing Milwaukee's voucher schools versus traditional public schools:

This report documents findings from our analysis of the school performance data released through the Wisconsin Department of Public Instruction (DPI) Report Cards...The data show:

• Higher DPI Report Card scores have a significant correlation with lower economically disadvantaged (ED) enrollment.
• Almost half of the variation from school to school in DPI Report Card scores can be explained by the variation from school to school in level of ED enrollment.
• [...]

In this must-read piece, David Sirota argues that the obsession with scapegoating teachers when poverty is clearly the issue conveniently deflects the national conversation away from inequality and policies to address poverty. No problem with poverty or inequality here! Just some teachers who need to be replaced! Full speed ahead on tax cuts, entitlement cuts, and the sequester!