A decade ago, Harvard鈥檚 Jon Fullerton and I penned 鈥�,鈥� arguing that educational data systems focus too much on metrics useful to policymakers and too little on the numbers that are useful to educators and families. We observed that the data most useful to policymakers 鈥渁re often simple, straightforward data on assessment results and graduation rates, whereas the key data for district officials shed light inside the black box of the school and district鈥攊lluminating why those results look like they do and what might be done about them.鈥�
Those same concerns are still with us today. And the disruptions of the pandemic have only made the need for useful, actionable data even more pressing. Well, in a recent report, 鈥�,鈥� Fullerton, the executive director of Harvard鈥檚 Center for Education Policy Research, tries to make sense of why dissatisfaction with data remains so high and what we can do about it. Fullerton has worked intimately with state and district data over the years, as part of CEPR鈥檚 , and he鈥檚 a wealth of wisdom on this topic. (Full disclosure: The analysis was published by AEI Education, which I direct.)
Fullerton argues that we never seem to have enough data for two reasons: technical constraints and normative disagreements about what schools should do and how to measure this effectively. To make sense of all this, he walks through the 鈥渄ata gaps鈥� that bedevil early childhood, school spending, postsecondary outcomes, and tutoring interventions.
For instance, when it comes to determining whether tutoring is effective, he notes that we frequently don鈥檛 know how much is spent on particular interventions or even which students receive which interventions. Fullerton writes that, so long as this is the case, neither school systems nor evaluators will 鈥渂e able to determine whether tutoring worked,鈥� 鈥渢he cost of tutoring relative to student growth,鈥� or 鈥渨hether tutoring is more or less cost-effective than other interventions.鈥�
These kinds of problems can be solved. If we want to understand whether and when tutoring (or any other intervention or program) actually works, Fullerton says schools need to get consistent about how they collect and report the essential data. Districts, he explains, do not currently standardize or even collect basic information: Is a student being tutored in math or history? During or after the school day? Does the student even attend each session? Fullerton suggests that schools track program and intervention participation and integrate that into student-information systems. So, for instance, 鈥淎 student receiving high-dosage tutoring might get tagged with 鈥榟igh-dosage tutoring, [provider name], math, two hours per week, in-school, in-person.鈥欌��
Even if we get such things right, though, data dissatisfaction will persist because of broader technical shortfalls, especially with how we measure outcomes. For instance, he notes that 鈥済rades are not particularly reliable in demonstrating students鈥� actual skills,鈥� while standardized tests omit important skills and competencies. As Fullerton puts it, 鈥淢easures, particularly of educational outcomes, almost never capture the richness of what we want to measure efficiently and in time to be useful.鈥�
And then, of course, Fullerton points to the role of normative disagreement about what data we should be collecting and why. As he writes, technical debates about how and when to collect data often skip past the fact 鈥渢hat we don鈥檛 agree, at least in any deep way, on the specifics about what education is for and what we are even trying to measure.鈥�
In the end, Fullerton wisely cautions, 鈥淲e must also maintain an appropriate sense of humility and realize that data will not answer all our questions or make our core disagreements go away.鈥� We shouldn鈥檛 ask data to do what they can鈥檛. But we should also ensure that we鈥檙e doing what we can to ensure that the data do what they can to help us understand the impact of our dizzying array of programs, practices, and interventions.
