The 53 teachers gathered around tables here have been called to a new kind of jury duty. But they won’t be deciding whether a fellow citizen is guilty of a crime: Their charge is to pass judgment on stacks of instructional materials.
Amid papers and coffee cups, they pore over a 90-page curricular unit on constitutional freedoms. In Socratic rounds of discussion, they explore the high school unit from dozens of angles, looking for fidelity to the common core.
How clearly does the unit state its purpose? Does it expect students to read texts that are rich and complex enough? Does it offer sufficient support for students who are struggling? Does it provide good, clear ways to assess how well students are learning as they go along?
These teachers are trying to answer one of the most vexing questions in the age of common-core instruction: Which materials fully reflect the new standards for English/language arts and mathematics? They’ve come to this suburb of the nation’s capital from more than 20 states to learn and practice a new rating system for lessons and units that purport to be “fully aligned” with the Common Core State Standards.
The new system, called EQuIP, represents one way that teachers are trying to make sense of the flood of curricular offerings that’s been unleashed by the nearly nationwide adoption of the common standards.
There are other tools or processes, too, that are designed to evaluate instructional materials for common-core alignment. The two national teachers’ unions have launched free portals where teachers can post and comment on lessons. Student Achievement Partners, whose founders led the writing of the common standards, has a set of free online tools that can be used to judge the fidelity of instructional materials to the standards.
Reviewing Materials
For-profit groups—like the Austin, Texas-based Learning List, which uses panels of judges to size up instructional materials—are also wading into the alignment-evaluation business. The Business Roundtable is talking with partners about creating a group to do “Consumer Reports-type reviews” of common-core materials.
And a group of experts led by Maria M. Klawe, the president of Harvey Mudd College, is creating a nonprofit to review the most widely used common-core math materials.
Educators who review curricular units and instructional lessons as part of Achieve’s “EQuIP” program evaluate four aspects, or “dimensions,” of the materials submitted. Below are highlights of some of the English/language arts criteria they consider.
Do They:
I. Aligning to the Common Core State Standards.
• Target a set of grade-level standards?
• Include a clear and explicit purpose for instruction?
• Choose texts that measure within students’ grade-level band?
II. Reflecting key shifts of the standards.
• Require students to read text closely for evidence and deep meaning?
• Facilitate rich, rigorous evidence-based discussion and writing through thought-provoking, text-dependent questions?
• Expect students to draw evidence from texts to produce clear, coherent writing that informs, explains, or argues?
III. Responding to students’ varied needs for instructional support.
• Cultivate student interest and engagement?
• Integrate appropriate supports in reading, writing, listening, and speaking for students who read below grade level, are English-learners, or have disabilities?
• Provide extensions and/or more advanced text for students who read well above grade level?
IV. Regularly assessing whether students are mastering the content and skills in the lesson/unit.
• Elicit direct, observable evidence of degree of mastery?
• Provide sufficient guidelines for interpreting student performance?
Note: A full version of this evaluation rubric and a companion document for mathematics are on the EQuIP website at www.achieve.org/EQuIP.
Sources: Achieve; Educators Evaluating Quality Instructional Products
The EQuIP program was created by Achieve, a Washington-based nonprofit organization that played a key role in launching the common-standards initiative in 2009. Achieve has worked for many years with states on academic expectations and accountability.
The approach for EQuIP originated with the work of three states—Massachusetts, New York, and Rhode Island—that designed a set of criteria to use in judging lessons and units for alignment with the standards.
The Tri-State Rubric, as it came to be known, created a buzz as other states tried it out and sought training to spread its use among teachers and curriculum writers. Achieve renamed the project EQuIP—for Educators Evaluating Quality Instructional Products—and began training for states, districts, teachers’ unions, colleges of education, and others interested in using it.
Achieve selected its first round of teacher-jurors last June. It chose a second group in November. Both groups came together late last month for training and practice using the rubrics—checklists of criteria—to evaluate math and English/language arts lessons and units that had been submitted by states, districts, groups of teachers, and nonprofit curriculum developers.
Jurors have reviewed about 40 of the 125-plus lessons and units that have been submitted so far, and of those, only nine have been deemed sufficiently aligned to be posted as resources on the EQuIP website, according to Alissa Peltzman, a vice president of Achieve who leads the project.
Curricular materials produced by large publishing houses—which dominate tens of thousands of classrooms—haven’t been submitted to EQuIP for review, Ms. Peltzman said. EQuIP doesn’t anticipate evaluating much from that sector, either, in part because of licensing restrictions that limit those materials’ use, she said, and in part because it lacks the capacity to analyze such a big volume of materials.
At the session here in Arlington, Va., reaching consensus on whether a lesson was aligned wasn’t easy or quick. It took all day to evaluate and rate the constitutional-freedoms unit, with rounds of detailed analysis and areas of disagreement.
One table of teachers, for instance, was deeply divided on whether the unit lived up to a key criterion in the rubric: stating a “clear and explicit purpose for instruction.” The teachers also disagreed about whether the unit met a criterion that required materials to demand a good deal of writing based on evidence in a text.
One middle school teacher criticized the unit for providing “scaffolding,” or support, for students in the assignments but not in the readings. A group of elementary school teachers at a nearby table got into an animated debate about whether the unit’s reading material from primary and secondary sources was challenging enough, yet still accessible to students.
Terri King Hunt, a teacher from Atlanta, told the group that she thought the unit fell short on that criterion.
“You could say you threw a lot of material at them, but what did they get out of it?” she said.
Next to Ms. Hunt, Kay Dugan, an assistant superintendent from Bensenville, Ill., pressed her colleagues to “be hard” in sizing up how well the unit provides ways to assess student learning day to day. “We need to get better at giving students feedback through formative assessment,” she told them.
Cautionary Notes
Since this was a sample exercise, votes on rating each “dimension” of the materials were taken by a show of hands. Collectively, the teachers voting showed they thought the unit needed significant revision.
Had it been a real evaluation, three or more reviewers would have graded each dimension on a scale of 0 to 3, and written explicit feedback to guide developers in revising the materials. Then a “lead reviewer” would have written one evaluation, summarizing the feedback and assigning an overall rating.
The materials rated “exemplar” or “exemplar if improved” would be posted on EQuIP’s website. Those needing more revision, or not yet ready for review, would not be posted, but feedback would be returned to the developers.
Guiding curriculum developers is a central aim of the EQuIP project. In fact, a section of the training was devoted to giving effective feedback.
“It’s easy to get cranky, like ‘Why didn’t you do this?’ and ‘If I were teaching this, I’d do it this way,’ ” Judson Odell, one of the facilitators of the training, told the participants. “Try to stay positive.”
The first round of EQuIP judging returned individual reviewers’ comments to developers. That feedback could be “conflicted and confusing” because of the differences in each reviewer’s comments, Ms. Peltzman of Achieve said. So another layer—the lead review—has been added to the process in an attempt to send clear, overall messages about the revisions needed.
Evaluating instructional materials for quality or alignment is inherently thorny and subjective, and reaching consensus can be tricky. Assembling juries of experts to make those choices is hardly new.
In 19 states, panels comb through submitted materials and decide what to include on an “adoption list” from which districts must choose if they purchase materials with state funds. Elsewhere, districts are left to shop for themselves.
Even those who welcome the evaluations by EQuIP and other organizations voice cautionary notes about the process.
Sandy Hayes, the immediate past president of the National Council of Teachers of English, said the EQuIP guidelines are “terrific tools for conversation” as teachers write lessons together in their schools. But she worries that a rating system could subtly work to narrow the concept of “good” materials.
“In talking about what’s good, sometimes people can be silenced, or there’s this ‘groupthink’ that happens, when you find yourself compromising to get consensus,” Ms. Hayes said. “I just wonder what’s lost in that process.”
Another risk in creating panels to evaluate materials is that their findings can be viewed as silver bullets.
Jim Burke, an English teacher at Burlingame High School in California, said it’s important to avoid mistaking any panel’s approved materials for a “total curriculum solution.”
“It does give you some assurance of quality about the ‘what,’ but it doesn’t deal with the heart of the work—the ‘how'—how you are teaching your students,” he said.
Educators who had come to the Virginia training and EQuIP jurors were eager, however, to see how the process could help them, and cadres of colleagues back home, move forward with the common standards. They seemed particularly interested in its value as a tool for creating their own instructional materials.
In Idaho, a small team of literacy coaches at the state department of education will use the EQuIP process this spring to help 250 teachers write model units, said Christopher Butts, one of the state coaches, who is serving as an EQuIP juror.
Suzanne Snider, a curriculum coordinator, said many of the teachers in her districts in California’s San Bernardino County “haven’t had a lot of direction” in figuring out what constitutes real alignment to the common core. “Teachers have been going on Pinterest, for goodness’ sake,” she said. “We really need this.”
Different Uses
States, districts, and vendors have been using the EQuIP tools in a variety of ways. The Maryland education department submitted sample lessons and units, and found the reviewers’ feedback “explicit and valuable,” said Ava B. Spencer, the state’s English/language arts coordinator. Teachers are working on revisions based on that feedback, she said.
In North Carolina, the state education department has used the EQuIP rubrics in training to help teachers think about how to design their own curriculum materials, said Julie Joslin, who oversees English/language arts. Eight regional programs in the fall of 2012 proved so popular that the department added seven more the next winter and spring, she said.
“We would get phone calls all the time, ‘Is this a common-core-aligned lesson?’ ” Ms. Joslin said. “Teachers were struggling to understand what that is, and the rubric does help with that.”
Washington state has used EQuIP’s evaluation criteria as it searches for open education resources to build a digital library that the state legislature mandated in 2012. But “one of the big questions people had about open education resources was ‘It’s free, but is it good?’ ” said Barbara Soots, the state’s program manager for such resources.
In scouring instructional materials from such sources as the Library of Congress, the National Endowment for the Humanities, and those developed by other states, including New York and Utah, Washington state’s panel of reviewers is employing a handful of evaluation tools as guidance, Ms. Soots said.
Along with EQuIP, which evaluates lessons and units, it is also using Student Achievement Partners’ Instructional Materials Evaluation Tool, which is more focused on full-course content, she said.
‘A Wild West Moment’
Expeditionary Learning has been both a consumer and a subject of the EQuIP process. The New York City-based nonprofit company, which runs 170 regular and charter schools, used the criteria to shape an English/language arts curriculum for grades 3-8 in New York state, and then submitted the curriculum to EQuIP reviewers for evaluation.
Scott Hartl, its chief executive officer, said he viewed participation in the evaluation process as important because of the way the curricular landscape is evolving in the common-core era.
“It’s a Wild West moment, with lots of people saying their materials are aligned to these new expectations,” he said. “We wanted our curriculum to go through the rigors of smart folks with a common vision looking at our stuff in relationship to what was out there.”
No curriculum developer can rightfully draw conclusions yet about what’s good, Mr. Hartl said. That will be left to time and experience.
“There has been a tremendous wave of innovation and new-product creation that eventually will get sorted out by real-life market forces,” he said. “That’s what will show us the results.”
