The organizer of a plan to review all 1,400 schools of education and publish the findings in U.S. News & World Report is altering several key aspects of its methodology—an attempt, officials for the group say, to respond to mounting complaints about the study.
The National Council on Teacher Quality posted its grading criteria online Wednesday morning to address concerns among education school deans that the review wouldn’t be transparent or accurate. It also plans to supplement the content-based analysis at the heart of its methodology with information on candidate classroom performance culled from “value added” data.
Finally, the organization will not, as previously planned, issue a failing grade to institutions that don’t provide the information requested.
Much about the ambitious study, due out next year, is still in flux—including whether such changes will be enough to persuade schools of education to participate in it. The situation underscores the tensions brewing as conversations about teacher accountability gradually permeate higher education.
“I think it’s good that they are addressing some of these methodological issues,” said Mary Brabeck, the dean of the Steinhardt School of Education at New York University. Ms. Brabeck, along with dozens of other deans, signed letters protesting the study. “Nobody’s against rankings, nobody’s against evaluation, nobody’s even against high-stakes evaluations. But if the methodology is flawed, how does that serve the public,” she asked.
The president of NCTQ indicated that the changes were meant to encourage participation, but she was adamant that the study continue.
“These are folks for whom this is their life’s work. It’s a very emotional issue. We totally understand why people are inclined to get upset about this,” Kate Walsh of the Washington-based advocacy institution said in an interview. “But, on the other hand, we’re asking folks to put that to the side and recognize what we all recognize, that there are many institutions in the U.S. not preparing teachers adequately, in addition to many doing a great job.”
Robert Morse, the director of data research for U.S. News & World Report, which will publish the NCTQ’s findings, confirmed that the outlet will continue the review.
“We’re used to people having negative opinions about what U.S. News does,” he said.
Review Details
The two organizations officially launched the project on Jan. 18, but it has been in the works for nearly a year. Its main purpose, according to the NCTQ, is to provide school districts and other education consumers with more information about the training teacher-candidates receive.
The report’s methodology is largely based on pilot reviews conducted last year of education schools in Illinois and Texas, and as such, its methodology differs markedly from that used in other U.S. News projects.
Rather than rankings, the council will rate all programs on an A-to-F scale on up to 17 standards. They include whether teachers are adequately trained in the science of reading and math; the length and coherence of their field experiences; and whether their programs include training on working with English-language learners.
After the council sent initial requests to education schools in late January, two distinct groups of deans organized and publicly objected to the review in letters to U.S. News.
One of them was signed by about 35 education school deans belonging to institutions within the Association of American Universities, an organization of elite research institutions.
A smaller group of education school deans at institutions that have received teacher-training grants from the Spencer Foundation, sent another complaint. Additional letters could be forthcoming from state-oriented groups of deans in New Jersey and Illinois, but none was confirmed at press time.
Among other complaints, the deans alleged incomplete standards, a flawed research methodology, and scoring criteria that were not transparent.
“The data-collection process must itself be transparent and clear, the assessments must be reliable, and the presentation of findings must be honest and fair. Without these characteristics, the rating will be meaningless,” the deans wrote in one of the letters.
The AAU deans compared NCTQ’s standards unfavorably with teacher standards crafted by the National Council for Accreditation of Teacher Education and the Council of Chief State School Officers.
Both groups of deans were galled at the NCTQ’s plan to give institutions an F grade on the standards if reviewers could not obtain information on which to base a rating.
“We worry that this implied coercion will cast doubt on the results of the entire evaluation,” the AAU deans wrote in the Feb. 3 letter.
Ms. Walsh said Tuesday that the council will address some of the criticism. Instead of issuing failing ratings, it will give an “incomplete” rating while seeking the information through other means.
In addition, she said, the council has now placed its scoring criteria on its website so it is accessible to all.
“We’re making those changes because we want institutions to feel they can trust this process. It will work so much better,” Ms. Walsh said.
She defended her group’s standards, saying the alternatives presented in the deans’ letters were too vague to serve as guidelines.
“It’s an external review. These are not standards developed by the teacher education community, and that itself is a challenge for them,” she said. “Previous reviews [in Texas and Illinois] have shown high rates of failure against these standards, and that creates nervousness.”
In a webinar on Wednesday, NCTQ representatives said that, once the ratings were computed, the council would show how its scorers arrived at their determinations and allow schools to provide additional evidence to contest them, possibly leading to a ratings change. Such details would be publicly posted on the council’s website. During the Illinois and Texas reviews, some deans charged that the council wasn’t responsive to additional evidence.
Bad Blood?
Previous NCTQ reports have been poorly received by education schools, perhaps explaining why their appeals have been directed at U.S. News and not the NCTQ.
A 2006 study of reading preparation in a sample of teacher colleges, for instance, led to complaints that the reviews were flawed or incomplete. (“Teacher Ed. Faulted on 69ý Preparation,” June 7, 2006.)
More recently, findings from the pilot reviews conducted last year were met with a boycott from Texas education school deans and created consternation among deans in Illinois.
The American Association of Colleges for Teacher Education, a national membership group based in Washington, has not taken a public stand on the report because it does not want to influence member institutions’ decisions about participating. But it is closely watching the process, and e-mails from AACTE officials indicate that the organization has expressed concerns about the review both to its members and to the editor of U.S. News.
The president of AACTE, Sharon P. Robinson, said that the changes were overall a “very gratifying” response to the concerns of members.
“I’ll be interested in seeing what our members think of the indicators,” she said.
Some of the indicators, especially the ones on content training, could prove difficult to respond to because of education schools’ varying approaches to delivering such courses, she cautioned.
Not all education schools have condemned the study. The NCTQ’s technical advisory panel includes some from inside education schools. One such panel member, the dean of Southern Methodist University’s education school, in Dallas, said he didn’t agree with all the standards and results in his school’s review last year, but the content analysis provided some useful information overall.
“I believe there should be other ways to evaluate the effectiveness of our programs as well, but that doesn’t negate NCTQ’s efforts to review programs for their content,” David L. Chard wrote in an e-mail.
As a result of the review, Mr. Chard added, SMU will put more of an emphasis on math-content instruction and on special education.
Shift to Outcomes
Some of the deans’ objections are also rooted in the tricky issue of how to balance the value of the content of education school programming with measures of what teacher-candidates have learned and can do. The AAU institutions contended in their letter that the review was too focused on teacher-preparation “inputs,” rather than whether teachers coming out of the programs are effective in the classroom. It suggested several alternative methodologies.
Ms. Walsh said that the NCTQ will now also examine information generated from value-added systems where available. States such as Florida, Louisiana, and Tennessee have already begun to examine their education schools based on such systems, which track newly placed teachers and gauge their ability to raise their students’ achievement scores.
The AAU letter also suggested that the council consider a project by institutions in more than a dozen states to devise a performance-based licensing test. (State Group Piloting Teacher Prelicensing Exam,” Sept. 1, 2010.)
But Ms. Walsh said that pass rates on the California Performance Assessment for Teachers, on which the new assessment is based, are too high to provide useful information for distinguishing program quality.
Tough Sell
Education school deans acknowledged that their protests are coming at a time in which public perception of teacher education’s strengths is tenuous.
“I fear that if anyone protests or refuses to participate in the review, they are sending a message that they are not responsive to the public,” said Mr. Chard of Southern Methodist University.
Deborah L. Ball, the dean of the University of Michigan’s education school, said in a statement that protesting the study would distract from conversations about how to improve the quality of teacher training.
“Spending our time fighting about a survey of our syllabi and requirements is a distraction,” she said in a statement. “Claiming over and over that we know what we are doing and that we should control training looks foolish to our critics and, in the face of weak or nonexistent evidence, only discredits our claim to expertise.”
Many deans are in the position of Richard De Lisi, the dean of the education school at Rutgers University, in New Jersey, who is still weighing whether to take part in the review.
“It’s unfortunate that this can be painted as shying away from evaluation,” Mr. De Lisi said. “But that’s not what it’s really about. You have to have judgment criteria that are clear and evidence everyone can see.”
Ms. Brabeck of NYU echoed that sentiment.
“I don’t know what NYU will do,” she said.
