It started with a simple question from a parent: Why was his 11th grade son being scheduled for Spanish when he still couldn鈥檛 read competently in English? Wouldn鈥檛 it be better to enroll him in a reading class?
The query sent Superintendent Richard A. Ross off in search of more data. How many students in grades 7-12 in the Reynoldsburg city schools, he wondered, were reading below the 4th grade level?
Within 15 minutes, the Ohio superintendent had his answer: Eighty-four students fit that description. Based on the information, the district designed a more targeted reading program for those students. Now, almost a year later, 鈥渨e鈥檙e seeing grade-level growth, on average, in excess of a year and a half in those kids,鈥 Mr. Ross said.
鈥淲ould I have had that information if I had to go through 3,000 paper files to see who was reading at the 4th grade level?鈥 speculates the superintendent of the 6,300- student district. 鈥淚t might have taken weeks and weeks and weeks.鈥
Instead, thanks to a Web-based data-analysis and decision-support system, known as Schoolnet, 鈥渋n 15 minutes, we were able to drill down and get those names,鈥 he said.
Such experiences are making Mr. Ross and other school leaders like him believers in the power of analyzing data to improve public education. Although the practice goes by a variety of names鈥攂usiness intelligence, data warehousing, decision-support systems, data-based or evidence-based decisionmaking鈥攖he idea is the same. By knitting together the thousands of data elements now collected by districts and states into a centralized computer bank, educators can look for patterns that could help improve both the management and productivity of schools.
鈥淚 think people within education have realized that there鈥檚 a tremendous amount of value they can apply to helping kids learn that鈥檚 embedded in the data they already have, once they can look at it all in one place,鈥 said Shawn Bay, the founder of eScholar, a data- warehousing company.
Mr. Bay, who was a management-information director at the Procter & Gamble Co. before starting eScholar, said businesses have been using data warehousing and data-based decisionmaking for more than a decade to enhance their bottom lines. But the practice is just now taking hold in education. His client list has grown from about 10 districts to more than 700 in the past two years.
A similar transformation is taking place at the state level, where states ranging from Georgia to Michigan to Mississippi are creating huge warehousing systems to link everything from student test scores and enrollment figures to teacher information and financial data. Michigan and Pennsylvania, meanwhile, have contracted with Standard & Poor鈥檚 School Evaluation Services to provide the public with Web-based access to information about the performance of individual districts and schools.
The passage of the 鈥淣o Child Left Behind鈥 Act of 2001 significantly increases the pressure on states, districts, and schools to collect, analyze, and report data in a timely fashion. Under the federal law, for example, states must report annually on student performance by race, income, gender, English fluency, migrant status, and disability. They also must report on the percentage of students not tested, by the same categories. And they must track the professional qualifications of teachers, including the percentage of classes taught by those lacking certification in their subjects.
Chester E. Finn Jr., the president of the Washington-based Thomas B. Fordham Foundation, describes the law鈥檚 鈥渆arnest efforts to force better information from the education system鈥 as perhaps its greatest virtue.
What鈥檚 Working, What鈥檚 Not
Beyond the requirements in the revised Elementary and Secondary Education Act, observers point to a number of factors that are fueling the hunger for data-based decisionmaking. The decades-long push for accountability, for one, has focused attention on test scores and other results.
鈥淚 think what鈥檚 happened is the accountability demands are now forcing us to look much more deeply into the granular data and to do more sophisticated analyses,鈥 said Philip A. Streifer, an associate professor for educational leadership at the University of Connecticut in Storrs. 鈥淭hat鈥檚 because just looking at how School A versus School B did, in the aggregate, was not revealing enough to tell us what to improve.鈥
Studies of high-performing schools and districts have found that they rely heavily on data to inform their practices. Increasing state expenditures on education also have prompted calls for more and better information about what that money is buying and whether it鈥檚 being used effectively.
Georgia lawmakers mandated the creation of a comprehensive education-information system in 2000 as part of the A-Plus Education Reform Act. While the system could cost $100 million to develop, estimates Thomas M. Wagner, the executive director of the State Data and Research Center at the Georgia Institute of Technology, which has responsibility for the program, 鈥渨e鈥檙e spending in excess of $10 billion a year on public education in Georgia.鈥
Equally important, noted Ted Sanders, the president of the Denver- based Education Commission of the States, 鈥渨ith the new electronic technologies, it becomes more and more possible to have at your fingertips useful data to answer important questions.鈥
Last year, the ECS joined with the University of Texas at Austin and Just for the Kids, an Austin-based nonprofit organization, to form the National Center for Educational Accountability. The center鈥檚 goal is to improve states鈥 knowledge of how to use data to monitor, analyze, and improve student and school performance.
The center currently has data for six states鈥擜rkansas, Florida, Minnesota, Tennessee, Texas, and Washington鈥攖hat together represent 19 percent of the student population of the United States. It will help states compare schools with similar populations and identify the high fliers so that others can learn from them. The Council of Chief State School Officers, the National School Boards Association, and the American Association of School Administrators also have launched projects to work with districts on the collection, analysis, and use of data.
鈥楬undreds of Pools鈥
While many see huge potential in mining data to improve public education, most states and districts have a long way to go before the rhetoric matches the reality.
鈥淰ery few states really have the kind of data that would enable ease of reporting either at the federal level or for policymakers at the state level,鈥 said Brad Duggan, the president of the national accountability center and the executive director of Just for the Kids. 鈥淧robably no more than 10 states have a substantial amount of the information necessary.鈥
Only 16 states, for example, maintain a unique identification number for each student that enables them to link student-test data to other student and school characteristics or to follow the progress of individual students over time. Only 17 include breakdowns of information on student achievement鈥攕uch as by race or family income鈥攐n school report cards.
The same holds true at the district level. 鈥淢y experience is that probably the large majority, if not the supermajority of districts, have very little capacity to do this,鈥 said Mr. Streifer of Connecticut, who is also the president of EdSmart, a company that provides decision-support systems to some 20 districts, primarily in New England.
鈥淭hey don鈥檛 have the technology,鈥 he said. 鈥淭hey don鈥檛 have the money for the technology. They don鈥檛 have the human resources to get this done.鈥
For instance, a report this spring on New Jersey鈥檚 management of education data, commissioned by the Institute on Education Law and Policy at Rutgers University, lambasted the state for its lack of a coherent information-management system. The report, prepared by education consultant Philip E. Mackey, found that the state education department 鈥渉as no centralized repository鈥攊ndeed has no record鈥攐f all the data it has available.鈥
Instead, Mr. Mackey found, mid-level managers tended to set up their own small-scale, project-specific databases, which numbered in the thousands. 鈥淭hat was the revelation to me,鈥 he said, 鈥渨hen I realized it鈥檚 not that they鈥檝e got information that they鈥檙e not sharing with us. They can鈥檛 find the information themselves, and they don鈥檛 even know what they鈥檝e got. They鈥檙e expressing just as much frustration as the users out in the field.鈥
Since the release of the report, a new state administration has formed a task force to work on the development of a student-level information system.
Similarly, in California, a report this spring by the Center for the Future of Teaching and Learning and SRI International decried the lack of a coherent data system about the teacher workforce. 鈥淲e frequently have been asked important questions about the teacher workforce that simply cannot be answered due to the inadequacy of state-level data,鈥 the authors write.
鈥淲hen we started doing this work,鈥 said Margaret Gaston, a co-director of the center for teaching and learning, 鈥渢here were these hundreds of pools of information, and they were so scattered and so diffuse that policymakers just couldn鈥檛 get to them.鈥
Often, the most basic information in states is either missing or reported in ways that make it only minimally useful鈥攊nformation such as detailed attendance and discipline data, graduation rates, student-coursetaking patterns by race and ethnicity, and teacher qualifications.
鈥淚 think the reason is that the way the public school system has been created over time, data were collected for compliance purposes, not for diagnostics, and it鈥檚 an entirely different mind-set required to look at data in a way that illuminates what鈥檚 actually going on,鈥 said Leslye A. Arsht, the president and co-founder of StandardsWork, a nonprofit organization based in Washington. The group last year produced 鈥淭he Results Card,鈥 which compares student performance and nontest indicators across Arizona, Colorado, Georgia, Florida, Maryland, Ohio, Pennsylvania, Texas, and Virginia.
鈥榁ery, Very Messy鈥
Even when data do exist, observers note, such information is often of poor quality, because too little attention has been paid to validation.
鈥淪chool data is generally very, very messy,鈥 said Alvin Crawford, the vice president for Internet sales and marketing for Schoolnet, a New York-based company whose products are now used in about a dozen districts in five states. 鈥淭ypically, the lowest-paid employee in the district is the one who鈥檚 responsible for adding and updating the data.鈥
鈥淗aving the data open and available to users makes districts realize how bad it is and to do something about it,鈥 he argued. 鈥淛ust having a system that lets them analyze the data means districts are now working toward fixing it.鈥
In 1997, for instance, the Oregon legislature mandated a central database for schools that includes funding, staffing, and student information. Previously, the state鈥檚 200 districts had submitted in excess of 200 forms to the state education department, allowing numerous opportunities for redundancy and errors, said Michael J. Hess, a managing director with KPMG Consulting, which helped set up the system. The new system now includes fewer than 100 indicators, as well as a detailed process for validating the data.
Developing a robust system of data collection, analysis, and use in education can be a long, complicated, and costly process.
Ohio, for instance, spent an estimated $79 million on its education management-information system between 1989 and 1999. Georgia lawmakers have appropriated $50 million since 2000.
The various data-warehousing, -reporting, and -analysis systems marketed to districts can also be expensive. The products offered by eScholar range from $5 to $15 per pupil, said Mr. Bay, depending on the size of the district and the scope of services offered. Schoolnet costs between $2 and $5 per pupil.
Broward County, Fla., one of the first districts to establish a data warehouse, did so with a $2 million grant from IBM鈥檚 Reinventing Education program, along with $2 million of its own money.
鈥淐ompanies clearly see an opportunity,鈥 said Richard J. Wenning, the president of the education performance network for the Arlington, Va.-based New American 69传媒, a nonprofit group that promotes comprehensive school reform.
鈥淭here are probably over 100 companies that provide these types of services,鈥 he noted, 鈥渂ut there are only a handful of providers that have the ability to work at any scale. And many of them are relatively young companies as well.鈥
鈥淲e really think a very careful review of the marketplace and understanding of the different offerings is essential,鈥 said Mr. Wenning, whose organization is hoping to position itself to provide such advice to clients and, potentially, handle some data management itself.
Peter J. Stokes, the executive vice president for EduVentures Inc., a Boston research company that specializes in the education industry, said the best measure of the business sector鈥檚 growing interest in data-based decisionmaking in education 鈥渋s the fact that some of the largest and best-established education companies are thinking more seriously about getting into the decision-support and data-warehousing business.鈥 That includes such companies as the publishing giant NCS Pearson and Chancery Software, a leading publisher of information-management systems and Internet learning communities.
鈥淚t鈥檚 not the most glamorous or sexy technology market going,鈥 Mr. Stokes said, 鈥渂ut it has the long-term potential to be, in some ways, the 眉ber-market. It will help administrators realize which technologies really impact education and which technologies really improve student outcomes. And that鈥檚 the real benefit.鈥
Companies like Schoolnet, for example, offer a suite of applications for data-based decisionmaking that, in addition to providing analysis and reporting, enable teachers to align their instruction with state and district academic-content standards.
鈥淭here鈥檚 a tremendous opportunity here, and we鈥檙e going to discover lots and lots of competition,鈥 said Denis P. Doyle, the vice president and chief academic officer of Schoolnet. 鈥淚t鈥檚 the one area in which the private sector can make a significant contribution to lasting school reform.鈥
Others worry, however, that in the short run, there could be more data than clarity.
鈥淲hat I鈥檓 seeing is a growing interest in the public reporting of data,鈥 said Brian Stecher, a senior social scientist at the RAND Corp., a think tank based in Santa Monica, Calif. 鈥淭here seems to be the assumption that those to whom it鈥檚 reported will be able to make wise use of it. But it isn鈥檛 clear to me that anyone has a specific plan for use in mind.鈥
Ultimately, the hope is that the push to data-based decisionmaking will also help teachers and principals. This school year, for example, every teacher in the 262,000-student Broward County district received a detailed analysis of how each of their students performed on every subtest of the state assessment system. If a student performed below the passing grade in an individual area, such as fractions or statistics, the report flagged it.
鈥淭he whole idea is that all this information is there,鈥 said Phyllis Chasser, the senior data-warehouse analyst for the Florida district, which includes Fort Lauderdale, 鈥渂ut if we don鈥檛 get it out to teachers, who can then change the way they teach, ... it鈥檚 useless to have the information.鈥
In fact, one of the hardest tasks, observers say, is not technical but cultural. 鈥淵ou have to build a trust and comfort level with schools,鈥 said Mr. Wagner of Georgia. 鈥淲e鈥檙e dependent on them to input a large amount of this information. It鈥檚 really, I鈥檝e told superintendents, about providing them with tools they need to be effective.鈥
鈥淚 think what it does, with the ease of use, is allow us to move away from voodoo education, where we鈥檙e just shooting in the dark about what the issues are with students,鈥 said Superintendent Ross of Ohio鈥檚 Reynoldsburg district, 鈥渁nd to become much more professional in our delivery of instruction, assessment of instruction, and description of interventions for kids.鈥
