Where states set the bar for what counts as “proficient” performance on state tests differs dramatically across states and even across subjects and grades within states, a new study reveals.
The analysis by the Portland, Ore.-based Northwest Evaluation Association examines where 14 states set their proficiency levels in reading and mathematics, by comparing how students did on the state tests with how they performed on tests produced by the NWEA and taken within a month of the state exams.
Read from the (requires free registration). Or, read the (Requires .)
Under the federal No Child Left Behind Act, schools must meet annual targets for the percent of students who perform at or above the proficient level on state tests in reading and math in order to make “adequate yearly progress.” But the study’s findings suggest that whether schools are identified as doing a good job and students are deemed on track academically hinges as much on geography as on actual performance.
“The problem is we have a federal law with very uniform language, rules, and consequences being applied to very disparate criteria,” said Allan Olson, the president and executive director of the evaluation association.
The nonprofit organization provides computerized adaptive tests and paper-and-pencil assessments to more than 1,000 member districts in 38 states. More than 3 million students in grades 2-10 take the tests, and their performance is scored on a common measurement scale across grades.
See Also... | |
| See the accompanying table, “Performance Standards for Math,” | |
By seeing where students in each state fell on the NWEA scale, and where they fell on their state tests, the test-maker was able to compare state proficiency levels against a common yardstick. In each state, at least 1,000 students in each grade and subject took part in the study.
‘Inappropriate Impressions’
While it’s not surprising that proficiency levels differ across states, subjects, and grades, said Mr. Olson, the degree to which they differ and the potential for misinterpretation are surprising. For example:
- In 8th grade math, students who perform at the “proficient” level on state tests in Montana score at the 36th percentile on the NWEA’s exam, while students who score at the “proficient” level in Wyoming score at the 89th percentile.
- In Arizona, 3rd graders who perform at the proficient level in math on the state test score at the 46th percentile on the NWEA exam. But 8th graders who perform at the proficient level score at the 75th percentile.
That suggests Arizona has set a much higher bar in grade 8 than in grade 3, a common occurrence across states, said Mr. Olson. As a result, a large proportion of students who are identified as on track academically in elementary school could be categorized as below proficient in grade 8, even if their learning trajectory has stayed the same. Meanwhile, states could mistakenly think that their middle schools are not doing as good a job as their elementary schools at educating children.
- In Washington state, the proficiency level on the 4th grade reading test is at the 53rd percentile on the NWEA exam, while the proficiency level on the 4th grade math test is at the 76th percentile. That means more students will be identified as proficient in reading than in math, even if students’ performance is the same relative to that of their peers. Such results could lead the state to spend more money to help students in mathematics without needing to do so.
“You may create impressions that are inappropriate,” Mr. Olson warned. While most states do not have proficiency levels that are as divergent as the previous examples, he noted, almost all states have noticeable differences that will affect some students.
No Agreed-Upon Solution
Stuart R. Kahl, the president and chief executive officer of Measured Progress, another testing company, based in Dover, N.H., agreed that proficiency levels can vary greatly even across grades and subjects within the same state. One reason, he said, is that states often set proficiency levels, or cutoff scores, independently for each grade and subject.
But the problem is becoming more apparent as states add tests in more grades to comply with the No Child Left Behind law, Mr. Kahl said.
“What you don’t want kids doing as they go through the grades is jumping from ‘proficient’ to ‘not proficient’ just because of the way the standards are set,” he argued.
Testing experts have proposed a number of solutions to the problem, he noted. Those range from statistical techniques, such as “smoothing,” to being clearer about what students should know and be able to do in each grade in state content standards, to putting all tests on the same vertical scale, as the NWEA does.
But, Mr. Kahl cautioned, “there are strong debates over which is the best way to go.”
Regardless, he said, “states are going to have to do something.”
