More K-12 districts are moving toward 1-to-1 device programs, but determining whether such initiatives are having a positive impact on student learning might take a while.
A researcher who studied a well-known 1-to-1 program in the Mooresville Graded School District in North Carolina said it may take more than a handful of years to determine whether ed-tech projects are working. Educators and researchers who do evaluations after just one or two years鈥攁s much existing research on ed tech currently does鈥攎ay be tempted to scrap programs that don鈥檛 seem to be having a significant impact before they have a chance to show their effectiveness.
鈥淲e shouldn鈥檛 necessarily give up on programs that have null findings in the short term,鈥� said Marie Hull, an assistant professor of economics at the University of North Carolina Greensboro who co-authored the 2018 paper with social scientist Katherine Duch, a principal at One Minus Beta Analytics.
The researchers looked at Mooresville鈥檚 program over eight years of planning and implementation and compared the district鈥檚 progress to similar, neighboring North Carolina districts. In the report, first released online in the journal of Educational Evaluation and Policy Analysis in September of 2018, the researchers examined test scores, along with survey data.
They examined math and reading achievement for students in grades 4 to 8, but also looked at the impact of the program on student behavior, such as time spent using digital learning devices and student absentee rates. The researchers found that in the short term, test score improvements in math and reading were statistically insignificant compared with other districts. But in the 鈥渕edium term"鈥攆our and five years after implementation鈥攎ath scores in Mooresville improved in 鈥渕eaningful鈥� ways compared with neighboring districts: the equivalent of about 3.2 extra months of math learning. 69传媒 was more of a mixed bag: in the fourth year, the 1-to-1 program had no significant impact compared to other districts, but in the fifth year there was a slightly positive impact on reading compared with other districts over that span of time.
As more districts move toward such programs鈥� from the Consortium for School Networking found that 40 percent of K-12 technology leaders said their school systems had 1-to-1 computing programs, and 43 percent said they expected to reach that level within three years鈥擧ull鈥檚 emphasis on taking the long view becomes even more important.
Education Week Contributing Writer Michelle Davis recently interviewed Hull to help educators, policymakers, and researchers better understand what it will take to evaluate ed-tech programs in truly meaningful ways.
Why is this research different?
The data I used covered eight years鈥攆ive years of post-implementation data. That鈥檚 unique. When I did my literature review, the most I saw was a look at a program over three years, but most of them were one or two years. The longer horizon does make a difference.
In our findings, we saw that the short-term impacts are modest, even statistically insignificant. But when we go out four or five years, we see positive impacts that are meaningful. The evidence on reading scores is mixed, but for math scores we find in the first couple years after the program was implemented, math scores improved, but not by much鈥攊t wasn鈥檛 statistically significant. But in a medium-term horizon, four or five years post implementation, we see an improvement in math scores by .13 standard deviations. Anything at or above .10 is noteworthy. 69传媒 was similar to math in that the first couple of years after the program was implemented the positive impacts were small and not statistically significant. But when we look at the medium horizon, there was one year when there was a positive effect and one year where it was statistically insignificant.
You examined factors beyond test scores. Why is that important?
Yes. We had some results on time use. The first was how often students reported using a computer, and that went up a lot, which was good. If you give students a computer you want them to be using it. We also looked at the time spent on homework and the time spent reading for fun. The homework results were interesting, because we didn鈥檛 find any new time spent on homework overall. The students were keeping their homework time constant, but they were spending more of that time using a computer. In terms of free reading, we found a decrease in time students reported reading for fun. That could explain why we see impacts in math, and not as much in reading, but those negative impacts on free reading were small in size鈥攁mounting to about eight fewer minutes per week.
What lessons can other districts draw from this research?
The positive impacts were really encouraging. Even more positive is that the program was essentially cost neutral for the district. The district did get a donation to start, but they mostly did it by moving around items in the budget. Also, in Mooresville, it wasn鈥檛 just about the laptops. They used the 1-to-1 program as a means to an end. They did a lot of professional development with teachers and tried to accompany the initiative with a culture change.
Other school districts shouldn鈥檛 think they can just pass out laptops or tablets and that鈥檚 going to make a difference. You have to be working ahead with the teachers and getting buy-in from them and the community.
Other districts, notably the Los Angeles district, have not had the same success with 1-to-1 programs. What made Mooresville stand out?
The Mooresville case is a best-case scenario, kind of a proof of concept. A 1-to-1 program has the potential for this upside, but when you dig into the details, it wasn鈥檛 just the laptops in Mooresville. They did a lot of work before and continue to support it.
What can the research community learn from this report, particularly around educational technology?
I鈥檇 go back to this short-term versus medium-term or longer-term horizon idea. It shows that we shouldn鈥檛 necessarily give up on programs that have null findings in the short term. I get why people do鈥攂udgets are limited and you want to invest more resources in things that are working initially. If I had only written this paper based on two years of results, I鈥檇 say the program wasn鈥檛 very effective and even if it was cost neutral budget-wise it took up a bunch of people鈥檚 time. It was really only in four or five years post implementation that I saw positive effects.
Are there other ed-tech areas researchers should be taking a closer look at?
With technology it鈥檚 just that the effects can be all over the place because it depends on what the computer is replacing or how people are using the computers. What we still don鈥檛 know much about is mechanisms. I鈥檇 like to see more work done surveying teachers on the practices they鈥檙e using, surveying parents on the rules around using a computer at home. That way we can uncover the key elements of what鈥檚 working and what isn鈥檛.
See also: