UPDATED
The District of Columbia’s closely-watched system for evaluating teachers and providing bonus pay appears to have motivated weak teachers to make improvements, and to spur already-effective teachers to even higher levels of performance, concludes.
Teachers on the cusp of dismissal under D.C.'s IMPACT evaluation system improved their performance by statistically significant margins, as did those on the cusp of winning a large financial bonus, according to the study, which was published as a working paper Thursday by the Cambridge, Mass.-based National Bureau of Economic Research.
The provocative study raises new questions about how recently revamped teacher-evaluation systems—and pay schedules linked to them—shape teacher behavior. It is among the first research studies to look at teacher evaluation empirically.
As an actual policy rather than a pilot, IMPACT was somewhat different in scope from other programs aimed at boosting teacher skill. Teachers with lower initial ratings also received access to instructional coaches to improve areas of their practice, one possible explanation for the gains.
“All of the actors involved are taking this system seriously, and it’s in those situations where you might expect greater responses to the incentives,” said James H. Wyckoff, a University of Virginia professor of education and policy, and one of the study’s authors.
The study does not look at whether the improvement in teachers’ scores translated into more learning for their students, though, and the researchers cautioned that the results apply only to the populations close to the performance thresholds studied. They do not depict an “overall” effect of IMPACT.
Dividing Line
Established in the 2009-10 school year, D.C.'s IMPACT evaluation system relies on a complex mix of factors to score each teacher, including both multiple observations and measures of student achievement. Teachers deemed ineffective under the system can be dismissed, while those scoring at the “minimally effective” level, the second lowest, get one year to improve. Those teachers who earn the “highly effective” rating are eligible for of up to $25,000. Earning successive “highly effective” ratings also permits teachers to on the salary scale.
Since its rollout, IMPACT has led to the dismissal of several hundred teachers. The system, instituted by then-Chancellor Michelle Rhee, began operation just as the federal Race to the Top initiative rocketed teacher evaluation to the top of policymakers’ agendas. The performance-bonus system, initially funded through the support of several private foundations, began a year later, the product of contract negotiations with the Washington Teachers’ Union.
Though IMPACT has won accolades in policy circles, some teachers have been less happy with the system. In surveys and focus groups, they have complained about feeling monitored, and that it’s harder for teachers in the most challenging schools to get top scores on the evaluation.
For the study, Wyckoff and Stanford University Professor Thomas Dee analyzed records for D.C. teachers and their students from 2009-10 through 2011-12, including more than 2,600 teachers each year.
The researchers examined those teachers falling just above or just below the dividing line separating “minimally effective” from “effective” teachers and compared how they fared in subsequent rounds of the evaluation. Similarly, they analyzed the performance of those teachers just above and just below the decision point for awarding the financial bonuses, and compared how they did in the next year’s evaluation scoring.
The idea behind this methodology, known as a regression-discontinuity design, is that there aren’t major differences in characteristics between these groups, so their subsequent performance can be analyzed as a function of how they responded to their initial IMPACT scores. The design helps to rule out other factors that could have affected the results, though it is not as conclusive as a true randomized experiment.
Teachers with two “minimally effective” ratings first were fired in the 2010-11 school year. The researchers found that in the 2011-12 year, those teachers at risk of losing their jobs pulled up their performance by about 13 IMPACT points. Teachers on the cusp of winning a performance bonus in 2010-11, the first year they were offered, also improved by about 11 IMPACT points.
To put those gains in perspective, they’re about 50 percent and 41 percent, respectively, of the growth the average D.C. teacher makes in his or her first three years on the job.
Both effects are relative to the population of teachers that just made it over the IMPACT bar.
D.C. schools Chancellor Kaya Henderson said in an interview that the study validates IMPACT’s theory of action, which was to reward the best teachers, give assistance to struggling ones, and dismiss those not able to perform.
“The simple fact that this study tells us that the things we set out do to on our evaluation system are getting done makes me super proud, and hopeful about where we’re going,” she said.
Adding to the Literature
The study also found that in 2010-11, teachers deemed minimally effective were also less likely to stay, with their retention rate falling by about 11 percentage points—an increase of 50 percent over the population of teachers that just made the “effective” cut.
Rates of teacher turnover in Washington have also earned the district its , although the paper shows that highly effective teachers had high overall retention rates. Also, the study notes, teachers newly hired in 2011-12 had average IMPACT scores that were more than 25 points higher than the teachers who left.
Henderson was unapologetic about those findings.
“When I came to the job as the deputy chancellor of human capital, our goal was to reset the standard about what it meant to teach in DCPS. There were a lot of people who rose to the challenge, and some who didn’t, or couldn’t or wouldn’t,” she said. “I want to know that the best teachers are sticking around, and if the minimally effective teachers are choosing to go elsewhere, I’m encouraged by that, too.
“This is not every other district in America; it is a really tough place, and we are asking our teachers to do amazing things.”
The study’s suggestion that a high-stakes evaluation system linked to pay decisions and professional development can be a motivating factor for teachers to improve directly contrasts with a run of recent studies on bonus-pay programs showing little or no effects overall on student achievement.
Teacher evaluation and its effects have not been as widely scrutinized. Just one study indicates that improvements in teachers’ evaluation scores may have an effect on student learning. That 2011 paper found that on Cincinnati’s teacher-evaluation system correlated to increases in achievement among their students.
But the new study did not examine in detail whether Washington teachers’ improvements translated to higher student performance, a criticism local union officials highlighted. “The goal of IMPACT was to improve student achievement, but in fact, [overall] student achievement has been flat,” Washington Teachers Union President Elizabeth Davis said.
Still others praised the study’s design but pushed back against interpretations that teacher evaluations of the sort used in the District of Columbia will lead to widespread improvements in teaching quality.
“Put simply, what this study says is that if we take a group of otherwise similar teachers and randomly label some as ‘OK’ and tell others they suck and their jobs are on the line, the latter group is more likely to seek employment elsewhere,” wrote Bruce D. Baker, a professor in the graduate school of education at Rutgers University in New Brunswick, N.J., “No big revelation there, and certainly no evidence that D.C. impact ‘works.’ ”
UPDATED 1:51 p.m. with more reaction.