A Shocking Statistic About the Quality of Education Research

by Valerie Strauss, Washington Post: http://www.washingtonpost.com/blogs/answer-sheet/wp/2014/08/21/a-shocking-statistic-about-the-quality-of-education-research/

 

In 1968, an article in Kappan magazine titled “The Need for Replication in Education Research” by education Professor Robert H. Bauernfeind said:

The principle of replication is the cornerstone of scientific inquiry. This principle holds that under similar conditions, one should obtain similar results. Replication has long been an essential aspect of research in the natural sciences, where science findings are not published until their repeatability has been demonstrated. In the natural science, the investigator may repeat his experiment 10 or 20 times, cross-comparing all results, prior to publishing his “findings”… Yet the process of replication is much more vital in our field than in the natural sciences or even the biological sciences. The reason, simply, is that more things can go wrong in a behavioral research project than a physical research project. There is a higher probability that the findings of a single behavioral study might be in serious error, or might not be generalizable beyond the specific circumstances of the specific study …

That is why the findings of a new research study about research studies are so important - and cautionary.

Veteran teacher Larry Ferlazzo alerted me to “Facts Are More Important Than Novelty: Replication in the Education Sciences,” published in the peer-reviewed Educational Research, a journal of the American Educational Research Association, and written by Matthew C. Makel of Duke University and Jonathan A. Plucker of the University of Connecticut.

Their study analyzed the “complete publication history of the current top 100 education journals ranked by 5-year impact factor.”

The impact factor of an academic journal speaks to the average number of times an individual article has been cited in the Journal Citation Report in a year; a five-year impact factor would mean the average number of times pieces from a specific journal published in the previous five years were cited in the JCR.

Journals with a higher number of citations are considered to be more important in their field than those with fewer citations.Their study says that despite the fact that replication of education research findings is vital to policymakers and educators, only 0.13 percent of education articles were replicated.

That’s 0.13 percent. Less than one percent. Less than half of one percent.

There’s more: While nearly 68 percent of the replicated studies were successful in reaching the same conclusions as the original studies, it was also true that replications “were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles,” according to an abstract of the report.

When an entirely new team of researchers did a replication, they were successful in duplicating the results of the original 54 percent of the time. “The results,” the abstract said, “emphasize the importance of third-party, direct replications in helping education research improve its ability to shape education policy and practice.”

For more than a decade, school reformers have said that education policy should be driven by “research” and “data,” but there’s a big question about how much faith anyone should have in a great deal of education research.

This is so not only because the samples are too small or because some research projects are funded by specific companies looking for specific results, but because in nearly all cases, it appears that nobody can be certain their results are completely accurate.

It is true that replication studies that confirm the original findings are not necessarily definitive and cannot resolve all issues about rigor, reliability, precision and validity of educational research. The report cites eight reasons that explain the lack of replication in education research, including the notion that “novelty equals creativity.” But it goes on to say this:

However, implicitly or explicitly dismissing replication indicates a value of novelty over truth… and  a serious misunderstanding of both science and creativity. If education research is to be relied upon to develop sound policy and practice, then conducting replications on important findings is essential to moving toward a more reliable and trustworthy understanding of educational environments.

Inside Higher Ed quotes Plucker in this excerpt from its story on the Mackel-Plucker report:

Neighboring fields in the social sciences - psychology, sociology, criminology - also suffer from a dearth of replications. But whereas psychology has weathered a number of fraud cases, the world of education research has had not a single fraud accusation in years, Plucker said. That’s a remarkable statistic, considering that conservative estimates place the number of educational researchers at 50,000 in the U.S. and 100,000 worldwide, according to the American Educational Research Association.
Education research’s spotless record is no accident, he said. It’s the result of scholars who aren’t checking each other’s work. “I would love to believe that every single person doing education research around the world has ethics that are as pure as the driven snow,” Plucker said. “[But] the law of averages tells us there’s something out there.”

Valerie Strauss covers education and runs The Answer Sheet blog.