As a boy in Ireland, I attended Mass every Sunday during which priests with their backs to the congregation mumbled in incomprehensible Latin. None of us knew what it meant, but we were assured these mysteries were good for the soul.
In education today, measurement experts occupy the exalted status of Irish priests. With their figurative back to the schools, these prelates genuflect at the altar of Item Response Theory and mumble confidently amongst themselves in a language known as psychometrics. No one in the school congregation understands a word of it, but we are assured these mysteries are based on science.
It鈥檚 a troubling situation. As I , assessment results are but the tip of an iceberg. All the assumptions that produce the numbers the public sees are buried beneath the waterline. Discussion between the measurement experts who develop the assessments and front-line educators who must act on them is almost non-existent. It鈥檚 a broken conversation.
of the history of the governance of international large-scale assessments introduces a new element that makes the situation even more problematic. Researchers once dominated the governance mechanisms of ILSAs. That鈥檚 not true today. Nearly three-quarters of the members of the policy governance committee of the International Association for the Evaluation on Education Achievement (IEA), for example, are from government or the bureaucracy. It is a safe bet that many of these people don鈥檛 possess even a rudimentary understanding of the 鈥渧oodoo that we do so well,鈥� as described the arcana of psychometrics at a 2012 conference sponsored by Teachers College鈥檚
Pizmony-Levy鈥檚 history reminds me of Torsten Hus茅n鈥檚 , which I served as deputy director. Hus茅n, one of the towering figures in the early history of IEA, made a plea to the commission: Don鈥檛 interpret international large-scale assessments as a horse race between nations. ILSAs weren鈥檛 developed by bookies to handicap horses but by researchers to help clarify the goals each nation pursued for itself through its schools.
Unfortunately, the politically appointed members of the excellence commission could not resist the catnip of ranking nations by mean results. In its startling 1983 report, , the commissioners thundered: 鈥淥n 19 [international] academic tests, American students were never first or second and, in comparison with other industrialized nations, were last seven times.鈥�
What would Hus茅n make of where we are today? It is not just that the horse-race mentality of mean scores on ILSAs dominate the conversation, domestically and internationally. It is also that bureaucrats in both individual nations and international associations have demonstrated remarkable skill and cunning in providing more of the horse-race catnip politicians crave. And, truth be told, in the quiet of their university and think-tank offices many measurement experts acknowledge that, although troubled by some directions in assessment today, they have to follow the money.
So, where does this leave us? Several things might be said. First, no responsible educator denies the need for assessment. Second, accountability needs to run in both directions. It is clear that teachers and administrators are today at the mercy of how public officials interpret ILSA results. This puts a special onus on measurement leaders to live up to the best practices of the profession. They shouldn鈥檛 be in the business of providing catnip. Third, it鈥檚 time for another look at the governance structures of these assessments.
Finally, we should openly acknowledge that what鈥檚 鈥渂elow the waterline鈥� is simply incomprehensible to policymakers and the lay audience. It鈥檚 time to clarify communications among the three sectors (measurement experts, politicians, and the educators). After all, even in the churches of Ireland, priests long ago abandoned Latin and turned their faces to the congregation.
James Harvey
National Superintendents Roundtable