Amid shrinking budgets, multiplying federal and state requirements for school improvement, and rapidly changing educational technology, it鈥檚 critical for education leaders to identify whether a given program will really improve learning for their students.
While leaders have more access than ever to education program evaluations and research clearinghouses, experts say it鈥檚 easy to overlook red flags in studies and evaluations of particular programs and interventions.
鈥淚t鈥檚 very challenging, and we鈥檙e asking educators whose job isn鈥檛 evidence to apply this to making their work more effective,鈥� Nancy Madden, co-founder of the Success for All Foundation and an education professor at the Center for Research and Reform in Education (CRRE) at Johns Hopkins University, said during an Education Week webinar last week. 鈥淓ducators are just now beginning to get down into the weeds enough to figure out how to make that evidence matter.鈥�
Madden; Robert Peters, the senior vice president and chief academic officer for the nonprofit education research clearinghouses the Successful Practices Network and the National Dropout Prevention Center; and Amanda Neitzel, CRRE鈥檚 deputy director of evidence research, highlighted ways education leaders can find evidence-based interventions more effectively.
What does 鈥榚vidence based鈥� mean?
The Every Student Succeeds Act formally defines indicating how promising researchers have found a particular intervention to be. (Some states also allow a fourth tier for early studies exploring a new program with a strong rationale.) These tiers are used mostly to identify appropriate programs for school improvement under the law. To fit in any tier, the study must be well-conducted and completed by a researcher who is independent of the program, and it must show statistically significant findings.
- Tier 1 (strong) studies track the outcomes of a large group鈥�350 or more students, teachers, or other participants. These studies randomly assign participants to the intervention or a control group in an effort to show cause and effect. This practice is considered the gold standard for causal research.
- Tier 2 (moderate) evidence comes from a study that also includes 350 or more participants tracked over time, but may use non-randomized, quasi-experimental designs in situations in which it is not ethical or possible to randomly assign participants.
- Tier 3 (promising) evidence may have a smaller sample group, use correlational versus causal designs, or be retrospective (with analysis performed after the intervention) rather than follow the sample over time.
Madden noted that terms like 鈥渞esearch-based鈥� and 鈥渆vidence-based鈥� are often used interchangeably to mean the same thing. However, different research groups and clearinghouses often disagree about exactly what is needed to prove that an intervention is effective.
The federal Institute of Education Sciences鈥� and CRRE have adopted the ESSA tiers when evaluating research. Neitzel said the highest-quality studies also follow a standardized intervention for at least 12 weeks, and include treatment and control groups that are similar and have had relatively few participants drop out.
Will an evidence-based intervention actually work in the classroom?
Many programs and interventions also sound great on paper or in the lab, but fizzle when they get to the classroom.
It鈥檚 important, experts said, to look for red flags in a study that suggest it could be hard to implement in a new class or school.
For example, no intervention will work if students don鈥檛 receive it, so an evaluation should state how much time students actually participated in a program that produced particular results.
鈥淲e see often, especially in cases of education technology, studies will compare students who used the program a lot to students who didn鈥檛 use it at all, and drop out all those kids in the middle who had low usage,鈥� Neitzel said. 鈥淲hile that will tell you how this works for kids who use it five days a week, consistently for the whole year, it doesn鈥檛 necessarily tell you how it鈥檚 going to work for every one of your students.鈥�
Even if a program shows strong evidence of effectiveness, it鈥檚 important for leaders and educators to consider how easily they could replicate the model with their own students.
鈥淐hoosing something with great evidence that doesn鈥檛 address the problem in your situation is not going to make the progress that you want to make,鈥� Madden said.
She recommended that education leaders try to look for insight into the following questions when reading education research.
- What training do teachers or other intervention staff need to implement the program effectively?
- What aspects of the program may be harder to implement in your district? For example, rural districts may have less access to staff or resources used to implement a program in an urban school.
- How will the school monitor the effectiveness of the intervention with its own students?
Peters, who works with districts on improvement plans, said the highest-performing school systems also analyze the cost-effectiveness of their programs at the end of each school year to understand how to prioritize them.
鈥淎 lot of times we see large school districts across the country spend millions of dollars on programs that are not even connected to their strategic planning,鈥� Peters said.
