69传媒

With the end of the six-year period of 69传媒 First on the horizon, no clear empirical picture has emerged of how well the federal program is doing at a national level in bringing struggling readers to proficiency.

Preliminary findings from the long-awaited suggested that the $1 billion-a-year funding for the program has had no impact on students鈥� reading comprehension.

But those findings did not answer many of the research questions that lawmakers required the evaluation of the program to address when they created it under the federal No Child Left Behind Act. Those questions, which many in the field say are critical, include whether the program and the research base undergirding it have been effective, and which approaches, programs, and assessments for teaching struggling readers are the most promising.

Dueling analyses by critics and advocates of the program have tapped the report鈥檚 contents to claim, alternately, that 69传媒 First is a wholesale failure and a qualified success. Flaws in the research design, proponents say, may skew the findings against the program, which has delivered improvements on local- and state-level tests and gained fans among many school officials and educators.

鈥楢n Economic Question鈥�

Federal officials stand by the study, saying it is the best national evaluation possible, though they caution that it is too early to come to the kinds of conclusions that program critics have drawn. But while the final report later this year is expected to include a deeper analysis of key aspects of 69传媒 First, and its impact on student achievement over a longer period, even in its final form it is unlikely to paint a comprehensive picture of the program鈥檚 effectiveness. Nor is it likely to clarify how best to translate research findings on effective instruction into classroom practice.

鈥淚 would say you have to wait for the final report before it would be reasonable for people to draw conclusions about the 69传媒 First study,鈥� said Grover J. 鈥淩uss鈥� Whitehurst, the director of the Institute of Education Sciences, the research arm of the U.S. Department of Education that commissioned the congressionally mandated study. One difficulty in doing the study, he said, is that the treatment is not clearly defined, and implementation of the program varies from site to site.

鈥淭he 鈥榠t鈥� [what is being measured] is more ambiguous than it might be in certain other impact studies,鈥� he added. 鈥淭here鈥檚 not a manual that you can get on the Web and order that is 69传媒 First.鈥�

The 69传媒 First legislation provides a detailed list of 10 analyses required by the law, most of which will not be included in the five-year, $35 million impact study. Four other studies that have already been released or are pending will address some of those questions.

Previous federal reports on the program determined through surveys of state and local officials that the program has changed classroom practice or raised test scores in many places. But no experiments have been conducted to find out the extent of the progress or reasons behind it.

The latest study, released May 1, was expected to provide some answers, and the final report later this year may include deeper insights into the program鈥檚 effectiveness. (鈥�69传媒 First Doesn鈥檛 Help Pupils 鈥楪et it鈥�,鈥� May 7, 2008.)

But it is not an evaluation of the policies and practices put into place in 69传媒 First schools.

G. Reid Lyon, who helped write the 69传媒 First legislation as chief of the branch of the National Institute of Child Health and Human Development that financed reading research, said people were misreading 鈥渆xactly what is being studied here.鈥�

鈥淚t is an economic question: Do schools that get 69传媒 First money have a greater impact on reading comprehension than eligible schools that didn鈥檛 get the money?鈥� he said. 鈥淏ut people are interpreting that it鈥檚 about what the money buys, the [instructional] programs or packages.鈥�

Mr. Lyon said the evaluation outlined in the law would have included reviews of the commercial reading programs and assessments used by grantees, as well as teacher preparation and professional development in the subject.

The final impact study will include a second year of test results on reading comprehension, and will also compare students鈥� reading fluency in 69传媒 First schools with nonparticipating schools in the same districts. Research on the relationship between 69传媒 First instruction and student achievement will also be analyzed.

Little Evidence

The design has generated confusion, as well as criticism about the quality and accuracy of the study.

Mr. Lyon said it was well designed to measure the financing impact, but the comparison sample is 鈥渃ontaminated鈥� because of the likelihood that those schools were following many of the principles and practices of the participating schools. He also suggested the limited results should not be used to draw sweeping conclusions for or against the program.

Many people, however, are doing just that, and are using the study to make their case for continuing, reconstituting, or killing 69传媒 First.

鈥淎 whole bunch of my colleagues who have been against this forever are choosing to jump on any piece of evidence that supports their case,鈥� said Michael Kamil, a reading researcher at Stanford University and a member of the National 69传媒 Panel. 鈥淭his study represents an aberrant data point. It鈥檚 the only piece of evidence we have that it doesn鈥檛 work, so you鈥檝e got to explain why [other studies] found significant improvements in 69传媒 First schools.鈥�

Much of the other evidence Mr. Kamil referred to does not include data on reading comprehension, a skill that is difficult to measure in young children.

Other researchers, however, argue that it was predictable that the program would not improve reading comprehension.

鈥淚鈥檝e always believed that the way that 69传媒 First was implemented in many states put a premium on paying attention to a relatively narrow body of research about kids鈥� learning and their acquisition of reading skill,鈥� said P. David Pearson, a professor of language and literacy and the dean of the graduate school of education at the University of California, Berkeley. 鈥淣o one, including the National 69传媒 Panel, ever meant for the big five to be all there was for teaching reading, but that鈥檚 what we got in 69传媒 First.鈥�

Mr. Pearson was referring to the elements of reading instruction that the influential reading panel, in its 2000 report, determined should be taught explicitly to students. Those elements鈥攑honics, phonemic awareness, fluency, vocabulary, and comprehension鈥攁re required in 69传媒 First schools.

Several former officials with the Education Department say it is how the study was conducted and the size of the sample, not the program, that are faulty.

The nonparticipating schools that were compared with those in the program, for example, were likely to have adopted many of the practices required in 69传媒 First schools鈥攊ncluding scientifically based instruction as defined by the National 69传媒 Panel鈥攎aking it less likely that they would have significant differences in reading performance, those officials say. Most states, they argue, have reported dramatic improvements in 69传媒 First schools on measures of basic reading skills鈥攕uch as decoding and fluency鈥攖hat are the foundations of early instruction.

鈥淲hen the 69传媒 First schools make bigger gains than the rest of the state on state reading tests, that鈥檚 telling you something,鈥� said Sandi Jacobs, who was an administrator of the 69传媒 First program at the Education Department until last year.

She said that she and her colleagues raised concerns that the study, as designed, would not properly gauge the program鈥檚 effect. 鈥淚f you look at the implementation study [released by the Education Department last year], it shows dramatic change in teacher practice and significant differences along every reported measure.鈥� (鈥�69传媒 First 69传媒: More 69传媒 Going On, Study Finds,鈥� Aug. 9, 2006.)

That implementation study was based mostly on survey data and state-reported test gains and would not have met the rigorous standard for evidence the reading panel had required.

Gains on state tests occur more rapidly in low-performing schools once efforts kick in to raise them, said Robert E. Slavin, the founder of the Baltimore-based Success for All Foundation.

Mr. Slavin agreed that the study would be more useful if it measured a broader range of skills, but said the most troubling aspect of it is that it has taken more than six years to produce quantitative data on the program.

鈥淚 do think it鈥檚 sad that they rolled the dice on this one study, and that there couldn鈥檛 have been a variety of things going on that started earlier, so that we could have taken things into account to improve the program,鈥� he said. 鈥淣ow, we鈥檙e almost seven years into this, and we don鈥檛 have a whole lot to say about how the program is working.鈥�

Unjustified Criticism?

Mr. Whitehurst, however, said that by research standards, the study sample is large and valid. A representative national sample of 69传媒 First schools would have been impossible to study, he said. Other criticisms of the design, he added, are not justified either.

鈥淭he schools in this study and the students served by them are very similar to schools served by 69传媒 First nationally and the children served by 69传媒 First nationally,鈥� he said. 鈥淚s it possible that other schools and districts participating in 69传媒 First are producing stronger impacts on achievement. Yes.鈥�

The findings, Mr. Whitehurst said, do not support the arguments made by some critics that the 69传媒 First principles, or the research-based approach to instruction overall, are ineffective.

鈥淪cientifically based reading instruction has to work, because by definition it is based on practices that have been shown to work,鈥� he said. 鈥淪o this almost gleeful conclusion that because of this report we can ignore cumulative evidence on effective reading instruction is simply inappropriate.鈥�