We all know about gerrymandering—the process by which politicians in a majority party sit down with a map and carve out electoral boundaries to maximize their party’s electoral advantage. Gerrymandering upends the electoral process. Rather than allow voters to choose their representatives, incumbent politicians choose their voters. More than 90 percent of US congressional elections are now noncompetitive, largely due to gerrymandering.
Some scholars employ a similar process in their research. Rather than allow all of the research “literature”—the full expanse of all relevant evidence on a topic—to lead them to a research conclusion, they reference only that part that supports their preferred conclusions.
I call these scholars “dismissive reviewers,” because they ignore or declare nonexistent (i.e., dismiss) relevant evidence. When a group of dismissive reviewers cooperate they form a “citation cartel”—citing only each other’s research and dismissing all the rest.
Readers of this blog already know that much—perhaps most—mainstream US education research is cherry picked. In part, that is how easily disprovable education myths persist. Education journals publish study after study that purports to consider all the relevant evidence on a topic but, in fact, references only that part of the evidence supportive of the myth. That proportion may be tiny, but it is still “evidence.”
This is why I find little reassurance in the phrase “evidence-based research.” All research is “evidence-based.” But, some is based on only part of the evidence available. Moreover, some of that is fraudulent. Education hosts a gargantuan quantity of research evidence. But, much is of poor quality. And, much more than most people realize is simply dishonest, with fabricated or doctored data, surreptitiously altered definitions of terms, selective references, and dismissive literature reviews.
Despite its reputation as the most trustworthy of US education research sub-fields, the research conducted in education testing, or “psychometrics,” is no different. Some of it is poorly done. Some is biased. And, some cherry-picks its evidence to reach preferred conclusions.
To my observation, honest, objective scholars still run things in the more technical realms of education testing research. In the realm of education testing policy, however, cherry pickers have run the show for over three decades. Moreover, they have managed to “capture” the testing policy research function at the National Research Council, the National Academy of Education, the World Bank, and the National Council of Measurement in Education (NCME), the primary US professional association of testing and measurement scholars in education.
Indeed, just recently, NCME announced the names of the scholars who will write the testing policy section—”Accountability in K-12 Assessment”—of the next edition of the organization’s primary reference publication, Educational Measurement. NCME appears to have chosen a group that will assure continuity with past versions that reliably use cherry-picked evidence to advantage authors’ citation cartel. Furthermore, all four current authors and “reviewer-collaborators” have participated in Common Core promotion efforts and done work for the Gates Foundation.
By avoiding mention of rival evidence, and profusely referencing each other, citation cartel members can boost their own professional profiles, at the expense of other scholars’.
I have long been a strong advocate for education testing in general and standardized testing in particular. Yet, I would agree with many readers of this blog that US education testing policy has been sub-optimal since 2001 and remains so today. I do not share the conclusion that testing itself is responsible, however. Rather, responsibility lies with our country’s policymakers, in both major parties, who continue to rely on the advice of a relatively small group of policy analysts who limit their perspective to a pinhole of the available research evidence.
Un-Rig the Research!
Niki Hayes says
Thank you, thank you! I keep remembering that old saying in statistics: “Figures don’t lie, but liars do figure.”
However, I had come to the point of accepting any education research that had been peer reviewed. Then I learned about the “old boy network” within the review circles. Now, I believe education research is like a bowl of jello. There’s plenty of shine and movement but no consistent direction. It offers little nourishment; kids like it because it’s fun to play with, teachers like it because the kids like it. And, very important, it does provide money to a vendor for providing its “substance.”
How to determine what to believe on education issues today comes down to my gut feeling after 38 years in the field: “What are the proven results?” That means success must have happened across all identified subgroups for 3+ consecutive years in specified locations. Those on the ground involved—the schools, administrators, teachers, parents and students— must be willing to provide personal and clear evidence, if asked for such unexpectedly.
If results have not been successful after three years, someone must pay a price for experimenting—again—with our children’s lives.