The first independent review to weigh whether new science curriculum series are truly aligned to a set of national standards .
Four of the series—Discovery’s Science Techbook, Carolina Biological Supply Company’s Science and Technology Concepts, and two versions of Teachers’ Curriculum Institute’s Bring Science Alive!—were deemed insufficiently aligned to the Next Generation Science Standards. One series, Houghton Mifflin Harcourt’s Science Dimensions, was considered partially aligned. Only one series, Amplify’s Amplify Science, got top marks for alignment, coherence, and usability, according to the nonprofit EdReports, which conducted the reviews.
Those texts represent a sample of middle school science curricula; others will be reviewed in the future. All six of the series were designed for grades 6-8, and they represent a range of print and digital materials.
Already, some of the publishers of the series are disputing the way EdReports carried out these reviews. Let’s dig into the results a bit and see what they reveal about curriculum development for the NGSS and the shape of the current science-materials market.
Who are these folks, and how did they conduct these reviews?
EdReports is a nonprofit that tries to gauge whether published learning materials align to states’ expectations for students, including the NGSS and the Common Core State Standards. It’s mostly been supported by philanthropies, including the Bill & Melinda Gates Foundation, which has given the group more than $15 million over the past decade.
As it has in the past, EdReports relied on teachers to use an in-house framework to judge each curriculum. This time around, about 40 reviewers, mostly practicing teachers or science specialists, participated.
Each series goes through a succession of gateways. Only those learning materials that get a high enough score on the first gateway, which is focused on how well they embody key features of the NGSS, were assessed on the next two, which focus in more detail on the specific coherence of lessons and their usability for teachers.
In the past, not everyone has liked the gateway concept. EdReports’ very first math reviews, several years ago, were criticized by publishers and math groups partly for this reason. (The organization changed its methodology slightly after that.)
Remind me. What’s in these standards?
The NGSS were rolled out in 2013 and have since been adopted in 19 states and the District of Columbia. Even proponents acknowledge that the standards are incredibly complicated: They expect students to master science and engineering practices, such as analyzing and interpreting data. 69ý are also supposed to recognize themes that cut across biology, earth and physical science, and chemistry, such as how energy and matter flow into and out of systems. And both the practices and the crosscutting themes are layered on top of core science content, including weather patterns, natural selection, and ecosystems.
(There’s rather a lot of jargon and acronyms in the standards to explain all this.)
Publishers have struggled to figure out how to embody all those demands in their materials—for example, should they attempt to put every science practice or crosscutting theme in every unit? Or can they be parceled out over the course of two or three units?
Still, most of the major publishers—including the three biggies, Pearson, HMH, and McGraw-Hill—have put out series supposedly aligned to the NGSS in the past three years.
See also: Teachers Scramble for Texts to Match Science Standards
Where did EdReports think these materials fell short?
The series that reviewers thought fell short didn’t make it through Gateway 1 on overall design criteria for the NGSS; they received fewer than half of the 26 points available for that section. (HMH’s series, which reviewers deemed partially aligned, got exactly half.)
That doesn’t mean they’re terrible, but it should be something for teachers to think about, said Morgan Martin, a teacher on special assignment from the Los Alamitos district in California and one of the reviewers.
“I don’t think not passing a certain gateway is the red flag [signaling] ‘This is the worst thing ever,’ she said. “It’s just really good for teachers to have information about where the strengths are in the program, and then to know your teaching team and how qualified are you in understanding certain components and whether you’re capable of filling in some gaps.”
And what are those gaps? They fall into three main buckets.
One of the big problems was that most of the series apparently didn’t consistently measure all three dimensions (I warned you that there was gonna be jargon!) of the standards: the science and engineering practices, crosscutting concepts, and disciplinary content. Discovery’s series Discovery Science, for example, had some lessons with objectives that didn’t include any of the crosscutting concepts or science and engineering practices, reviewers said.
Another problem has to do with phenomena. Basically, the standards indicate that scientific phenomena should undergird lessons and units. (A phenomenon could be something like why, in dry weather, you get a shock when you shuffle rubber-soled shoes on a woolly carpet and touch a metal doorknob, or why certain organisms in a particular ecosystem appear to be dying out.)
Then, each sub-unit is supposed to help students learn more about what’s causing that phenomenon, to use scientific practices to record data about and make sense of it, and to connect it to the crosscutting themes. By the end of each unit, students should be able to generate a hypothesis for what caused the initial phenomenon and back it up with evidence.
EdReports’ reviewers felt that some of the series appear to have gotten this a little backwards: The phenomena they included were more for illustrative purposes. For instance, here’s what the reviewers wrote about one of the TCI series: “The ‘Anchoring Phenomena’ are most often used as examples of the content topic or concept as opposed to a driving mechanism for student questions and sensemaking.”
Finally, some of the series ran into problems with assessment. HMH’s Science Dimensions, for example, had some classroom-based assessments that didn’t give teachers enough helpful information to change their teaching. Discovery Science’s end-of-unit tests didn’t always match the objectives given at the beginning of each unit. And Carolina’s Science and Technology Concepts’ tests didn’t adequately measure all three of the dimensions.
“There was a huge range of quality and types of assessment, and some of the concerns were the ones that were multiple- choice content only, and didn’t really look different or new,” said Martin.
What do the publishers say about these findings?
In an interview, Carolina Biological Supply Company officials agreed that their series’ presentation of the crosscutting themes wasn’t as explicit as it could be, and promised they’d change that in future versions of the curriculum.
David Heller, director of product development for Carolina’s curriculum division, also felt that the the findings reflect how interpretations of the standards have evolved in the K-12 science field. In the early days of NGSS, curriculum writers knew that phenomena were supposed to get kids questioning and thinking, but “being so explicit about how [lessons] relate back wasn’t as much an understood point,” he said.
Discovery Education officials claim the review process suffered from some serious flaws. Their main dispute is how EdReports gauged a section of the Techbook’s curriculum that allows teachers several choices of how to proceed. EdReports, they say, interpreted the whole section as optional, even though Discovery says it isn’t.
Overall, said Marty Creel, the chief academic officer at Discovery Education, the review doesn’t reflect the current version of the Techbook.
“The reality in classrooms today is that teachers are pulling from materials all over the place, so to take a snapshot that’s 10 months old we think is fundamentally unfair,” he said. “We’ve been making a lot of improvements since and the version they are now reporting on is not one we have actively going into classrooms. So we’re kind of scratching our heads about why you’d put out a review on a version that basically no longer exists.”
Asked about product updates, EdReports officials said they chose to review these series only after publishers assured them they wouldn’t change radically, and that they did keep up-to-date with additions.
“It’s kind of a merry-go-round with some curricula that are digital and it’s hard to know when to jump on and off,” said Eric Hirsch, the executive director of EdReports. “But we would have called off a review if we noticed the merry-go-round was starting to go around too fast. We also know that these materials will change, and that’s why we stand ready to re-review.”
Two other publishers with low marks both sent statements.
”... There are numerous instances in their report where we believe that EdReports overlooked or disregarded evidence that we shared or where EdReports reviewers fundamentally misunderstood our program,” TCI President Bert Bowers said.
HMH’s assessment was even harsher. The rating, the company said, “does not reflect errors or problems with alignment on HMH’s part, but rather reveals the EdReports’ rubric’s lack of depth of engagement with NGSS and a philosophical difference in approach to the standards integration.
“We believe the rubric is limited by a disconnection from the research base of NGSS, its writers, and the community of teacher practitioners implementing the standards,” it concluded.
Publishers also pointed out that the grading framework changed midway through the process, though EdReports officials said they rescored everything when those revisions were made.
All of the publishers get to submit formal responses to the findings, and those should be uploaded now to the
Educational publishing is a tough business, and publishers are definitely concerned about these early findings, though none would say that on the record. But some did acknowledge the results would make marketing the products more difficult, challenging, and also potentially confusing for consumers.
In past reviews, publishers have made revisions to their products after a low-scoring review and earned higher scores on later ones.
Where do these findings fit in with other science reviews?
That’s a good question. This is the first independent review of science materials. Only a handful of states, notably California and Oregon, have issued comprehensive reviews of at least a half dozen series.
Louisiana has rolling curriculum reviews which, like EdReports, are considered pretty tough; so far, they’ve given just one set of materials a green light, for 4th grade science.
In a way, the lack of thumbs-ups for science learning materials by both EdReports and Louisiana represents the opposite problem from what happened in California, which reviewed last fall. Even California officials say that districts will need additional help in narrowing down the list and making good choices, and there are some projects underway to help them do that.
It’s also a good reminder that alignment is a bit in the eye of the beholder: Different people can set up different criteria about what constitutes alignment to standards, and reach different conclusions about the results.
And what does the science education field think of these results?
That’s for you to tell me! I look forward to hearing from you about your thoughts, comments, and critiques on EdReports’ findings. Leave a comment, or email me directly at ssawchuk@epe.org.
Clarification: This story has been updated to underscore that EdReports will review additional science series in the future.
For more on NGSS curriculum: