69传媒

Federal

69传媒, Math Software Found to Have Little Effect on Scores

By Debra Viadero 鈥 March 13, 2009 4 min read
  • Save to favorites
  • Print
Email Copy URL

For the second year in a row, a controversial $14.4 million testing the effectiveness of reading and math software programs has found few significant learning differences between students who used the technology and those taught using other methods.

Of the 10 commercial software programs tested at various grade levels, only one鈥擫eapTrack, a supplemental-reading program for 4th graders that is published by LeapFrog Schoolhouse, of Emeryville, Calif.鈥攑roduced significant improvements in students鈥 test scores across both years of the study.

Although not large, the test-score boost that the program provides is considered enough to move a typical student from the 50th percentile to the 54th percentile on a national standardized reading test, according to the report.

The two Algebra 1 products tested鈥擟arnegie Learning鈥檚 Cognitive Tutor Algebra 1 and Houghton Mifflin Harcourt鈥檚 Larson Learning Algebra 1鈥攍ed to similar-size test-score gains, but only among students taught by a subset of teachers who had used the same products for two years in a row.

Publishers, researchers, and federal officials called the findings disappointing, but also raised cautions about relying too heavily on the results to compare effectiveness among products and choose which ones to buy.

鈥淚f you already have the hardware in the classroom and you want one of these products, this would not dissuade you,鈥 said Mark Dynarski, the lead researcher on the project for Mathematica Policy Research Inc., the Princeton, N.J.-based company that conducted the study.

鈥淚f you鈥檙e quite skeptical of the software and very budget-pinched, I think you would feel this is evidence in favor of your position,鈥 he added. 鈥淎nd if you鈥檙e really right in the middle, I think it comes down to how much you want to move test scores, because you鈥檙e really not going to see that happen with these products.鈥

Study Draws Criticism

Despite a quiet release in January, the study met with criticism from independent researchers and software publishers.

鈥淭here鈥檚 nothing really here that superintendents or state policymakers or corporations could use that would be a strong basis for decisionmaking,鈥 said Christopher J. Dede, a professor of learning technologies, innovation, and education at the Harvard Graduate School of Education and a critic of the study. 鈥淚 feel the methods used were more flawed in the second year than the first.鈥

Product Testing

Ten computer-based reading and math products were evaluated in the 2005-06 school year as part of a major federal research project.

Grade 1 Early 69传媒
鈥 Destination 69传媒, Riverdeep Inc.
鈥 Headsprout, Headsprout Inc.
鈥 The Waterford Early 69传媒 Program, Waterford Institute Inc.
鈥 Plato Focus, Plato Learning

Grade 4 69传媒 Comprehension
鈥 Academy of 69传媒, AutoSkill International Inc.
鈥 Leaptrack, LeapFrog Schoolhouse

Grade 6 Prealgebra
鈥 Plato Achieve Now, Plato Learning Inc.
鈥 Larson Prealgebra, Houghton Mifflin Harcourt

Grade 9 Algebra
鈥 Cognitive Tutor, Carnegie Learning Inc.
鈥 Larson Algebra, Houghton Mifflin Harcourt

Source: Mathematica Policy Research Inc.

Note: Some of the developers and companies have since sold their product lines or been involved in corporate acquisitions.

The findings don鈥檛 mean that products that seem to be ineffective in one school or district won鈥檛 work better in another, the report concludes, nor should educators and policymakers use the results to make head-to-head comparisons between products. In some cases, Mr. Dynarski said, too few schools were using the individual products studied to make those kinds of comparisons.

Involving roughly 13,000 students, the study was ordered by Congress in the No Child Left Behind Act. The report on the first round of findings, which looked at 16 products, came out in 2007. (鈥淢ajor Study on Software Stirs Debate,鈥 April 11, 2007.)

The new report, the last one for the project, evaluates 10 commercial software programs that are widely used in the 1st, 4th, and 6th grades, as well as in Algebra 1 classes, which can be taught at several grade levels.

Unlike its predecessor, the final report gives product-by-product results for all 10 programs studied. Over the 2005-06 school year, researchers tested the programs in 23 districts around the country, most of which served high numbers of low-income students, and 77 schools. In each school, and for each product used in those schools, researchers included at least one control classroom and one experimental classroom.

鈥淭he control classrooms are generally using only products for Internet browsing or practicing on state assessments,鈥 Mr. Dynarski said. 鈥淭hey weren鈥檛 using the other software products.鈥

A subset of teachers鈥115鈥攕tuck with the same products for a second year, allowing researchers to see whether the programs became more effective as teachers grew more familiar with them. The additional experience only seemed to matter for the Algebra 1 software, though; for the other programs, students fared about the same in both study years.

The study also found that the average amount of time that students spent using the programs fluctuated from year to year. Yet the researchers could find no correlations between programs鈥 effectiveness and the amount of time that students spent using them.

Questions on Method

Some experts said the study may raise more questions about the usefulness of experimental research designs in education than about the findings themselves. The software study was among the first to reflect the then newly formed Institute of Education Sciences鈥 early emphasis on large-scale randomized studies.

鈥淭hese studies are intended to wash out all the variation in school environments, teacher quality, resources鈥攁ll the things that we, in fact, know make a difference when it comes to student learning,鈥 said Margaret A. Honey, a technology expert who is the president of the New York Hall of Science.

Mr. Dynarski said such concerns stem from the belief that the study had failed to pick up actual learning gains. 鈥淚鈥檓 not sure that the right answer isn鈥檛 zero,鈥 he said.

A version of this article appeared in the March 18, 2009 edition of Education Week as 69传媒, Math Software Found to Have Little Effect on Scores

Events

School & District Management Webinar Crafting Outcomes-Based Contracts That Work for Everyone
Discover the power of outcomes-based contracts and how they can drive student achievement.
School & District Management Webinar EdMarketer Quick Hit: What鈥檚 Trending among K-12 Leaders?
What issues are keeping K-12 leaders up at night? Join us for EdMarketer Quick Hit: What鈥檚 Trending among K-12 Leaders?
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Artificial Intelligence Webinar
Teaching 69传媒 to Use Artificial Intelligence Ethically
Ready to embrace AI in your classroom? Join our master class to learn how to use AI as a tool for learning, not a replacement.
Content provided by 

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide 鈥 elementary, middle, high school and more.
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.

Read Next

Federal How Trump Could Roll Back Access to Free School Lunches
Project 2025 and a GOP budget proposal call for axing a federal rule that allows public schools to serve free meals to all students.
5 min read
Cafeteria workers serve student lunches at Firebaugh High School in Lynwood, Calif. on Wednesday, April 3, 2024. Demand for school lunches has increased after California guaranteed free meals to all students regardless of their family's income.
Cafeteria workers serve lunches at Firebaugh High School in Lynwood, Calif., on Wednesday, April 3, 2024. Demand for school lunches has increased after California guaranteed free meals to all students regardless of their family's income. A federal school lunch provision that makes it easier for public schools to provide universal free meals may be a target for elimination in President-elect Donald Trump's upcoming term if some conservative activists and lawmakers get their way.
Richard Vogel/AP
Federal A Bill to Kill the Education Department Is Already Filed. Here's What It Says
The bill represents another attempt at a long-term Republican goal.
6 min read
People walk outside the U.S Capitol building in Washington, June 9, 2022.
People walk outside the U.S Capitol building in Washington, June 9, 2022. Legislation has been introduced in the Senate to abolish the Department of Education.
Patrick Semansky/AP
Federal Video Linda McMahon: 5 Things to Know About Trump's Choice for Education Secretary
President-elect Donald Trump plans to nominate former pro-wrestling CEO Linda McMahon to lead the education department.
1 min read
Federal The K-12 World Reacts to Linda McMahon, Trump's Choice for Education Secretary
Some question her lack of experience in education, while supporters say her business background is a major asset.
7 min read
Linda McMahon, former Administrator of Small Business Administration, speaks during the Republican National Convention on July 18, 2024, in Milwaukee.
Linda McMahon speaks during the Republican National Convention on July 18, 2024, in Milwaukee. McMahon has been selected by President-elect Trump to serve as as the next secretary of education.
J. Scott Applewhite/AP