Evaluating the usage of ed-tech products is tricky, complicated, and oftentimes confusing. But it can be done.
Consider the case of the Granite County school district in Utah. It partnered with a company called LearnPlatform to measure whether time spent on three particular pieces of software led to a bump in student achievement.
The district found that one program had great results for English-language learners and Native American students. Another seemed to get results when students used it as often as the manufacturer suggested, but going beyond that didn鈥檛 lead to better outcomes. A third was barely used at all, and the district is considering nixing it.
But most districts aren鈥檛 nearly as sophisticated as Granite.
Report after report cites low usage numbers for software, a problem driven largely by districts not thinking through how an ed-tech product should be used prior to buying it and then setting realistic expectations for usage, experts say.
Complicating matters is the fact that there isn鈥檛 a clear consensus on just what constitutes a 鈥済ood鈥� usage rate: Is it 25 percent, 50 percent, or 100 percent? And experts and educators are divided on whether education technology leaders should even be worried about low usage rates.
This special report鈥攖he first in a series of three special reports for the 2019-20 school year that Education Week is producing for K-12 ed-tech leaders鈥攅xamines how schools track tech usage and what steps they should take to make better use of educational technology tools. Read the full report here.
To solve this problem, experts say the best way to get the best bang for your education technology buck is to set expectations for how often a particular piece of software should be used. Then keep careful track of how close you come to your goal. If a program isn鈥檛 coming close to meeting that goal, it鈥檚 probably time to jettison it.
But that鈥檚 easier said than done.
There鈥檚 no silver bullet when it comes to evaluating which software to keep and which to ditch, said Joshua Patchak, the executive director of education technology and innovation for the Green Bay Area public school district in Wisconsin.
鈥淚f it is magic, I would love for someone to teach me those spells,鈥� Patchak said. 鈥淚 don鈥檛 know the secret to it. It鈥檚 difficult, unless you have a very disciplined staff from the classroom all the way up to the top.鈥�
鈥榃aste of Resources鈥�
Most software licenses districts buy never get used, according to a November 2018 report by BrightBytes, an education data-management and -analytics platform. In fact, the study found a median of 30 percent of ed-tech licenses are never used at all.
What鈥檚 more, nearly all鈥�98 percent鈥攐f licenses aren鈥檛 used intensively, meaning a student spends 10 hours or more with the product between assessments. And most teachers aren鈥檛 following vendors鈥� recommendations when it comes to 鈥渄osage"鈥攈ow often and for how much time a student uses a particular program or product.
We clamor all the time about how we don鈥檛 have enough resources to meet the needs of our children, and yet if we鈥檙e investing in resources and they go unused, I think that鈥檚 unconscionable.鈥�
Those findings are based on a set of data from 48 school districts, 393,000 students, and 177 apps鈥攚hich BrightBytes defines as browser-based online curricular and learning tools. The districts studied were of different sizes, with enrollments from more than 30,000 students to fewer than 1,000.
Similarly, a study released earlier this year by the ed-tech company Glimpse K12 that looked at $2 billion in school spending found that, on average, 67 percent of educational software product licenses go unused.
Glimpse K12 tracked 200,000 curriculum-software licenses purchased by 275 schools during the 2017-18 school year. The analysis found educational software was the biggest source of wasted spending in K-12 districts. And it estimated that, overall, districts are losing a whopping $2 million each on these products every year.
Those data depress Anton Inglese, the chief financial officer for Batavia Public School District 101, in a Chicago suburb.
鈥淚 think it鈥檚 a waste of resources,鈥� he said. 鈥淲e clamor all the time about how we don鈥檛 have enough resources to meet the needs of our children, and yet if we鈥檙e investing in resources and they go unused, I think that鈥檚 unconscionable.鈥�
But Richard Culatta, the CEO of the International Society for Technology in Education, said looking just at how often a particular product is used is the wrong way to think about the problem.
鈥淚 actually don鈥檛 like using usage as a metric,鈥� he said. 鈥淚t implies that we want everybody to use the software that we bought. What we should be thinking about is: Are the right kids and the right teachers using the right software at the right time? There鈥檚 rarely a case where the answer should be 100 percent, or having more is better.鈥�
Instead of asking how often a particular piece of software is being used, Culatta said, district officials should ask: 鈥淚s it meeting the learning need?鈥�
Reasons for Low Usage
One potential reason for low usage rates: District leaders in charge of buying software can鈥檛 always consult with every teacher to figure out what meets his or her needs.
I actually don鈥檛 like using usage as a metric. It implies that we want everybody to use the software that we bought. What we should be thinking about is: Are the right kids and the right teachers using the right software at the right time?鈥�
In many cases, 鈥渟chool districts have to make one-size-fits-all decisions,鈥� said Dan Carroll, the chief product officer of Clever, an education technology company that has helped track usage rates in the past. Given that reality, it 鈥渁ctually feels a little bit inevitable that you鈥檙e not going to get 100 percent of people using your software.鈥�
Of course, in his years working with districts, Carroll, a former tech director, said he鈥檚 also 鈥渟een a lot of dysfunction鈥� where software or an application is purchased sometime over the summer and not a single teacher uses it for the entire school year, sometimes because they don鈥檛 know it even exists.
But many education technology leaders will allow teachers or principals to buy software that not everyone may be interested in because they don鈥檛 want to close the door on experimentation with new applications.
鈥淭hat鈥檚 obviously a tricky situation when the tech evolves so fast. If you lock the door and don鈥檛 let anything in, you miss opportunities,鈥� said Hal Friedlander, the founder and CEO of the nonprofit Technology for Education Consortium and a former chief technology officer for the New York City schools. 鈥淚t鈥檚 a tough job.鈥�
Another contributing factor that district officials point to: aging workers who might not be quick to adapt to technological advances and some who might resist using them altogether.
鈥淲e have a much older workforce than a lot of other organizations and industries,鈥� said Matthew Lentz, the chief financial officer for the Upper Moreland district, near Philadelphia. That means some educators aren鈥檛 as 鈥渘imble鈥� on 鈥渉ow you can adapt in a changing environment.鈥�
What Products to Jettison
Part of the Batavia district鈥檚 strategy: classifying curriculum materials into three categories, including core materials that get widespread use, supplemental materials for remediation or enrichment, and materials that might just meet an individual school, student, or teacher鈥檚 needs. The latter category gets the most scrutiny, with the goal of shifting materials in it to one of the other categories.
Districts spend millions on educational technology products and services. But many of those products and services are not used by teachers or students as much as you might expect. Here鈥檚 some advice from ed-tech experts and educators on how to pump up those usage rates:
1. Give educators meaningful opportunities to offer feedback on potential ed-tech purchases.
2. Listen carefully to educators鈥� critiques of the ed-tech products and services currently used in the district.
3. After you buy an ed-tech product or service, establish a detailed plan for showing all educators why you purchased it and how to use it. Have a usage plan or goal in place and continually check in and measure progress against the goal.
4. If you expect the software or application to be used for a certain amount of time, or for a specific purpose, make sure that expectation matches what鈥檚 actually going on in the classroom. For instance, if a teacher is only going to use a piece of software as an option in a lesson rotation and some students won鈥檛 have access to it for weeks, but the vendor is recommending an hour of use per week for each student, it might not be the best fit.
5. Pilot software and structure those pilots in a way that district leaders and teachers can see what the student outcomes are likely to be, perhaps by using a control group and a treatment group. Districts should make sure that they are testing out a particular piece of software or service in an environment similar to where it will eventually be used, since some tools might work well with some types of students but not so well with others.
There鈥檚 no secret sauce in trying to decide what to jettison, Inglese, the district鈥檚 CFO, said. 鈥淭here aren鈥檛 universal criteria you can apply to every situation. You have to delve into the particulars of each situation to make decisions about whether or not these things work.鈥�
And then there鈥檚 giving school leaders direct responsibility for deciding which technology will work best for their schools. In the IDEA charter network in Louisiana and Texas, principals can choose to buy software using discretionary funds. But that money doesn鈥檛 have to go to technology鈥攊t could go to other priorities.
That means if a particular product isn鈥檛 being used or isn鈥檛 effective, principals are going to hear about it.
鈥淭hey would get that feedback from their teachers and the staff,鈥� said Cody Grindle, the senior vice president of information systems at IDEA, which serves 53,000 students in 97 schools. 鈥淭hat鈥檚 a lot easier and [more] successful than me sitting in my ivory tower saying, 鈥榊ou鈥檙e only using 20 percent.鈥� We do share data around that stuff, but we鈥檙e not the ones slapping the wrist.鈥�
What鈥檚 more, the amount of time a student spends on a particular application may not actually say much about whether it is effective, said Karl Rectanus, the co-founder and CEO of LearnPlatform. For instance, one district that partners with LearnPlatform found its students were getting much better results by using a product once a week, then supplementing with other materials, as opposed to the five times a week recommended by the vendor.
Some products are designed to save time, so you don鈥檛 necessarily want students spending huge amounts of time using them, he said. In his mind, 鈥済ood usage is the usage that matches instruction and gets the best outcomes for students.鈥�
