69传媒

Special Report
School & District Management

Beta Testing Ed. Products Can Get Tricky for 69传媒

By Sean Cavanagh 鈥 April 22, 2013 8 min read
  • Save to favorites
  • Print
Email Copy URL

Beta testing is one of the most basic steps on the path to getting education products and ideas into the classroom鈥攁nd, researchers and developers say, one of the trickiest to get right.

As applied to K-12, the term generally refers to the early stage of testing almost any product in schools鈥攁 game, an assessment, a software package, a personal computer鈥攁nd then refining it, based on how well it works.

School districts typically agree to take part in beta testing because they will get access to a product or service they believe will help them鈥攚hich could be a new learning tool for their students or a professional-development program for their staff members.

Yet beta testing can also pose challenges for both schools and product developers. When superintendents and school boards agree to beta-test a product, they know that they may be creating extra work for teachers and administrators who need to be trained in how to use it, and that their schools may be forced to carve out class time for trying it out.

Those concerns, among others, sometimes lead districts to reject offers to stage beta tests, to the frustration of developers desperate to take products for test flights in schools.

To complicate matters, beta testing can mean very different things within the research community, and within schools.

As in other fields, like medicine, education beta tests are often defined as efforts to test something through a series of pilots or experiments to see if a product does what it鈥檚 supposed to before it goes live on the market, said Grover J. 鈥淩uss鈥 Whitehurst, a former director of the Institute of Education Sciences, the main research arm of the U.S. Department of Education.

In other cases, a beta test can include a far more structured process to evaluate something in a scientifically rigorous fashion, such as through a randomized control trial, said Mr. Whitehurst, who now directs the Brown Center, a research division at the Brookings Institution in Washington focused on school improvement. In those experiments, one group of students might be exposed to a product, while a second one is not, and results are compared between the two groups.

In any case, the process is 鈥渃ritical for any product that has any degree of innovation in it,鈥 said Mr. Whitehurst, who conducted beta tests on reading and assessment products he helped develop before joining the federal government. 鈥淚f the developer intends for it to accomplish something that hasn鈥檛 been accomplished before, it鈥檚 absolutely necessary,鈥 he said.

Strengths and Shortcomings

Finding a district willing to act as a partner in beta testing is one thing; staging a successful pilot test that leads to a product breaking into the K-12 market is something else.

Elliot Soloway, a professor of computer science and engineering at the University of Michigan in Ann Arbor, who has conducted several beta tests as a product developer, has experienced that reality firsthand.

Several years ago, Mr. Soloway said, he worked with a company that was beta-testing pocket personal computers鈥攎eant to serve as 鈥渕obile learning environments"鈥攁mong a relatively small group of teachers and about 150 students in a school district in an Eastern state. (He declined to name the school system or the state.) The teachers in the group quickly became adept at using the tool. 69传媒鈥 test scores improved. So did their record in turning in homework on time.

鈥淚t went swimmingly,鈥 Mr. Soloway recalled.

The trouble started when the product was scaled up for use by more than 50 teachers and 1,500 students. Costs rose. While the system鈥檚 software and hardware worked well in the original test group, unexpected problems emerged when teachers in the larger group used the devices, probably because those educators lacked the tech-savvy to resolve glitches on their own, Mr. Soloway speculated. Teachers confused by the technology tended to stop using the devices 鈥渄ead in their tracks,鈥 without providing developers with enough information to diagnose the problem, he said.

Another complication: While teachers in the original test group were able to craft learning activities to make use of the technology鈥檚 features, teachers in the larger group were unable to figure out how to make the technology work for their curricular needs鈥攁nd believed doing so simply added to their workload, he added.

The pocket personal computer system never made it to the marketplace.

The project unraveled because of a computer 鈥渂ug rebellion鈥 and a 鈥渢eacher rebellion,鈥 Mr. Soloway said. 鈥淚t was horrible. ... We got lured into thinking we had a successful pilot; we can roll it into the rest of the school. Not a chance.鈥

The most obvious temptation among developers is to structure the beta test in a way that will heighten the chances a product will succeed, which can undermine the process, Mr. Soloway said.

In those circumstances, 鈥渙f course it鈥檚 going to succeed,鈥 he said, but the developer finds 鈥測ou don鈥檛 learn enough鈥 about the product鈥檚 true strengths and shortcomings.

Mr. Soloway still consults with education companies, and he still does beta testing, but he says he has learned from earlier setbacks. Today, he says he pays more attention to providing teachers with tech support, and help crafting lessons that mesh with the technology. He adds that experience has taught him that developers and district officials need to work closely together to choose a range of teachers and students with different backgrounds, who are likely to have different degrees of willingness to accept the technology, rather than cherry-picking participants.

Michael Casserly, the executive director of the Washington-based Council of the Great City 69传媒, said that when district leaders resist allowing beta testing in their schools, it鈥檚 often because they worry that process isn鈥檛 transparent, and that the results may not be portrayed accurately. Concerns about creating new administrative burdens and training tend to be less important, he said.

The council will agree to help coordinate or recommend beta tests among its 67 member districts only if developers agree to make the results of those tests public鈥攅ven if they produce undesirable results, Mr. Casserly said.

His group also expects developers to choose independent evaluators, rather than people predisposed to cast the results in a positive light. In addition, the council recommends that the individual districts it represents follow those same guidelines when considering allowing beta testing, independently of the organization.

When developers are told of the council鈥檚 expectations, 鈥渁ll of a sudden they鈥檙e not interested,鈥 Mr. Casserly said. 鈥淭he last thing they want is to test a product, then have all of the nation鈥檚 major school districts know that it doesn鈥檛 work.鈥

Evidence-Based Decisions

Developers sometimes conduct beta tests in different phases, with different goals in mind.

That was the case with an experiment being conducted by Melanie Stegman鈥攖he director of the learning technologies program at the Federation of American Scientists, in Washington鈥攚ho is conducting a beta test of Immune Defense, a video game designed to teach students about immunology in an interactive and entertaining way.

Ms. Stegman is testing the game among students studying video-game design and other subjects at McKinley Technology High School, a public school in Washington. Those students are relatively adept with technology, but that鈥檚 OK, Stegman says鈥攖he goal of this part of the test is to understand on a basic level whether the game works without glitches, whether students understand how to play the game, and whether they enjoy it.

She said she also asks those teenagers for suggestions about its design.

Ms. Stegman is now recruiting teachers鈥攕ome tech-savvy, some not鈥攖o participate in the next phase of her research, a randomized control trial which will ask them to use the game over three separate days, for one class period each day. The experiment will help determine whether students鈥 knowledge of cell biology increases as a result of playing the game, compared with other classroom activities considered to be best practices, she said.

At that point, 鈥渨e鈥檒l be able to test whether the game is really working,鈥 she said. 鈥淚s it teaching kids biology?鈥

Chris Dasenbrook, a teacher who is overseeing the classes where Immune Defense is being tested, said the project appeals to him because it gives students insights on the challenges faced by actual game developers. He has carved out time for students to test the game in three different classes, including ones on interactive media and 2-D concepts.

鈥淎 lot of my kids play video games,鈥 Mr. Dasenbrook said, but 鈥渋t鈥檚 different when you鈥檙e pilot-testing. You鈥檙e chasing down bugs. You鈥檙e putting the program through its paces.鈥

Ultimately, he said, the beta test shows students the challenges in transforming an idea for a game into a 鈥渃ommercially viable product.鈥

Not all developers and researchers are able to find willing partners in K-12 to help them test their products.

When he was the director of the Institute of Education Sciences, Mr. Whitehurst said, members of the research and development community occasionally asked him to consider supporting a requirement that schools take part in beta testing in exchange for federal funding, such as Title I money鈥攁 step he considered a major overreach.

Still, Mr. Whitehurst said, federal officials should consider providing incentives to districts, possibly through grants provided for education research, to encourage beta testing.

Developers, he added, could make the process more attractive to districts by agreeing to carve out time during their beta tests to conduct research specific to the needs of those school systems and their personnel. Mr. Casserly of the Council of Great City 69传媒, meanwhile, believes districts and developers would benefit if school systems set clearer procedures for how beta tests are to be conducted and managed.

Encouraging more schools to take part in beta testing would have the side benefit of creating a new, more sophisticated class of consumers of educational products, Mr. Whitehurst added.

鈥淚t would be a way to get school leaders to buy into the importance of using evidence to make decisions,鈥 he said.

Coverage of the education industry and K-12 innovation is supported in part by a grant from the Bill & Melinda Gates Foundation.
A version of this article appeared in the April 24, 2013 edition of Education Week as Beta Testing Ed. Products Can Get Tricky for 69传媒

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Artificial Intelligence Webinar
AI and Educational Leadership: Driving Innovation and Equity
Discover how to leverage AI to transform teaching, leadership, and administration. Network with experts and learn practical strategies.
Content provided by 
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School Climate & Safety Webinar
Investing in Success: Leading a Culture of Safety and Support
Content provided by 
Assessment K-12 Essentials Forum Making Competency-Based Learning a Reality
Join this free virtual event to hear from educators and experts working to implement competency-based education.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide 鈥 elementary, middle, high school and more.
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.

Read Next

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Whitepaper
Future-Driven Leadership: Five Goals for Dynamic School Leaders in 2024
This guide offers practical strategies for district leaders to foster innovation, empower staff, support wellness, amplify student voices...
Content provided by BookNook
School & District Management What the Research Says Four Ways to Stop Teacher Turnover From Hamstringing School Improvement
Staffing instability can unravel the social fabric of schools, experts say, unless leaders work to keep connections strong.
6 min read
Woman of color exiting out of a door.
iStock/Getty Images Plus
School & District Management Spooked by Halloween, Some 69传媒 Ban Costumes鈥擝ut Not Without Pushback
69传媒 are tweaking Halloween traditions to make them more inclusive to all students.
4 min read
A group of elementary school kids sitting on a curb dressed in their Halloween costumes.
iStock/Getty
School & District Management 69传媒 Take a $3 Billion Hit From the Culture Wars. Here鈥檚 How It Breaks Down
Culturally divisive conflicts in schools have led to increased legal and security costs, as well as staff time spent on the fallout.
4 min read
Illustration of a businessman with his hands on his head while he watches dollars being sucked down into a dark hole.
DigitalVision Vectors