The latest lackluster results from a on the impact of the 69传媒 First program offer little insight into which parts of the program are worth saving and which need revamping鈥攙ital questions, experts say, that could help guide any future federal endeavor to address the nation鈥檚 reading woes.
One of the largest and most rigorous studies ever undertaken by the U.S. Department of Education found that the $6 billion funding for 69传媒 First has helped more students 鈥渃rack the code鈥� to identify letters and words, but it has not had an effect on reading comprehension among 1st, 2nd, and 3rd graders in participating schools.
But the study鈥檚 limited scope鈥攖he impact of the federal funding on reading comprehension鈥攄oes not offer enough information to craft better plans or policies aimed at reform.
鈥淢y one complaint about this evaluation is that its design was powered to ask a limited question (Does RF work on average?) rather than the contextual question of: Under what conditions does RF work and why?鈥� Barbara R. Foorman, a prominent reading researcher at Florida State University in Tallahassee, wrote in an e-mail.
Ms. Foorman, who served as commissioner of education research at the Education Department鈥檚 Institute of Education Sciences in 2005 and 2006, said she is reassured by the findings that 69传媒 First schools are spending more time teaching and practicing basic skills such as decoding, and seeing improved student achievement in those areas. She is disappointed, however, in the lack of improvement in students鈥� understanding of text鈥攐utcomes, she said, which are to be expected given the program鈥檚 priorities of teaching the fundamentals.
The final impact study does not provide much hope to 69传媒 First advocates at a time when Congress has slashed funding鈥攆rom $1 billion a year since the program was launched in 2002 to $353 million in fiscal 2008鈥攁nd has proposed eliminating it altogether. Proponents of the program point to other findings from state data that show greater gains on several measures among participating schools.
鈥淚t鈥檚 not such a bad piece of work, but there are no conclusions you can draw from it,鈥� Stanford University researcher Michael L. Kamil said of the study. 鈥淏ut as you look at the whole set of data that we have on 69传媒 First, it is much more encouraging. There鈥檚 lots of evidence that this has worked.鈥�
Those state-reported testing data and surveys, Mr. Kamil acknowledges, were not gathered using rigorous methods or compared against results at similar schools that are not in the program.
The impact study, released by the IES Nov. 18, is the only scientifically rigorous review of the grant program. Some of its findings are consistent with those state reports and independent surveys. More time is spent on structured reading instruction and teacher professional development, for example, in schools that received 69传媒 First grants than in comparison schools.
The $36 million study may have left more questions than answers, many observers say.
鈥淭here is very little in the report that鈥檚 useful,鈥� said G. Reid Lyon, who as then-chief of the reading-research arm of the National Institute of Child Health and Human Development helped draft the 69传媒 First legislation and the requirements for evaluating the program under the No Child Left Behind Act. 鈥淭he only way for it to be helpful was in an effectiveness study, which was supposed to look at which kids did respond [to 69传媒 First-style instruction] and why, and which kids did not respond and why.
鈥淭his evaluation,鈥� he added, 鈥渄id not help us to improve the program in any way.鈥�
Missing Elements
The study was reconfigured to fit a reduced budget for the evaluation, and also because it was started after implementation of 69传媒 First began, making it unfeasible to assign schools randomly to the treatment or comparison groups, according to federal officials.
Broader questions, however, are the ones that policymakers and researchers say could help refocus a federal reading initiative. It is unclear whether 69传媒 First will survive鈥擟ongress鈥� fiscal 2009 budget proposal provides no funding, but the program will receive continuation funding at least through next March鈥攐r if Congress or the new administration will push for a new plan for addressing the nation鈥檚 reading woes.
Congressional staff members debated future directions for reading policy last month at a briefing on the impact study, according to Grover J. 鈥淩uss鈥� Whitehurst, the director of the institute, the Education Department鈥檚 research arm, until Nov. 21.
Those present at the briefing, he said, were not suggesting that the federal government abandon its efforts, but rather try to discover what is missing from the current reading program. Stronger pre-K literacy programs, adolescent literacy, efforts to improve students鈥� background knowledge, and teacher preparation were all discussed, Mr. Whitehurst said.
The impact study shows clearly, he said, that the program needs improvement.
鈥淎dvocates for the program will be pleased that it鈥檚 shown a positive correlation on [improved] decoding skills ... the focus of the program,鈥� said Mr. Whitehurst.
Among both students in the 69传媒 First and non-69传媒 First schools examined in the study, reading achievement was low and had not improved significantly over the course of the three-year study. Fewer than half of 1st graders and less than 40 percent of 2nd and 3rd graders showed grade-level proficiency in their understanding of what they read. Yet, on a basic decoding test, 1st graders in 69传媒 First schools scored significantly better than their peers in the comparison schools.
鈥淚 don鈥檛 think anyone should be celebrating the fact that the federal government invested $6 billion in a reading program that has shown no effects on reading comprehension,鈥� Mr. Whitehurst added.
Critics of 69传媒 First say the lack of results are because of the program鈥檚 narrow focus on explicit skills and the inadequate attention to the complex set of knowledge and skills children need to learn to read.
鈥淵ou can鈥檛 get a much more concentrated effort to make something positive happen than this program has,鈥� said David Reinking, an education professor at Clemson University in Columbia, S.C., and the vice president of the National 69传媒 Conference, a research organization based in Oak Creek, Wis. 鈥淭he whole [approach] rested on the assumption that improved reading fluency and decoding would translate somehow directly into gains in comprehension.鈥�
The study, he added, 鈥渃ertainly isn鈥檛 a ringing endorsement for 69传媒 First, and by extension ... of scientifically based evidence, as being the magic bullet.鈥�
Similar or Not?
Some observers argue, though, that the study鈥檚 complex design makes it difficult to draw positive or negative conclusions overall. It compares 69传媒 First schools with similar ones in the same districts that are not part of the program to determine the impact of the extra funding on instruction, reading proficiency, and the relationship between instruction and students鈥� comprehension.
Between 30,000 and 40,000 students in grades 1, 2, and 3 were given a reading-comprehension test four times from fall 2004 to spring 2007. The students attended nearly 250 schools in 17 districts and a statewide jurisdiction.
The study also includes extensive classroom observations to identify the instructional practices in both types of schools, as well as surveys of teachers, principals, and reading coaches.
An interim report on the findings, released in May, drew scathing criticism from supporters of the program, who suggested that the design of the study was flawed because it did not consider the likelihood that 69传媒 First principles and practices had spread to schools that were not in the program. (鈥�69传媒 First Doesn鈥檛 Help Pupils 鈥楪et it鈥�,鈥� May 7, 2008.)
Other studies have found that a significant proportion of schools serving struggling students have incorporated explicit instruction in the basic reading skills found to be essential in learning to read: phonemic awareness, phonics, fluency, vocabulary, and comprehension.
But Mr. Whitehurst dismissed those claims, saying that although there may be some 鈥渂leed over鈥� into non-69传媒 First schools, the classroom observations and survey data show that the schools are not so similar.
鈥淭he schools were not doing the same thing,鈥� he said. 鈥淭here were differences in professional development, there were differences in their use of reading coaches, ... and there were significant differences in classroom practices.鈥�
Some experts, however, disagree.
鈥淭he schools in the study started out as different as one could expect,鈥� Mr. Lyon said. 鈥淏ut as time went on there was more districtwide adoption of 69传媒 First methods. I鈥檓 not at all persuaded.鈥�
