Edublogger: Research that needs work

I was skimming Darren Kuropatwa's A Difference, which led me to Do The Math, which led me to this devastating (IMO) article by the late Herf Wilf. Below is Wilf's abstract. This is a cautionary tale about education research.

We examine a number of papers and a book, all of which have been cited, by people who are knowledgeable in the field, as being good examples of “research in mathematics education.” We find specific serious flaws, indeed fatal flaws, in all of them, so that no conclusions of any interest follow as a result of any of the “research” that is reported in these works. We have found no evidence that the research paradigm, involving test and control groups, randomized trials, etc., which is invaluable in the life sciences, is of any use whatever in studying mathematics education and we urge that it be abandoned, in favor of human-to-human discourse about how we can improve curricula and teaching.

Wilf isn't doing this just to skewer papers for the fun of it. And his arguments are not complex: "In fact, the shortcomings of these papers are colossal, and no mathematical or statistical training is prerequisite to perceiving them."

He begins by examining three papers that summarize research indicating that "students in reform-based classes do have significantly better achievement in mathematics than those in traditional instruction." This quote is from the NTCM's Research in Mathematics Education, and the summarized research supports the NTCM curriculum and pedagogy proposals. He dispatches the three articles quickly and succinctly. The first is based on the findings of a group of volunteer teachers. Not randomly selected! Unfortunately, reforms will be adopted by non-volunteers. One down. The second article, using randomized teachers found no statistical difference in achievement. But that was what we were trying to show. You third article makes no claim, and does not address achievement at all.

There's more. But I'll let you read it! When I someday write educational research, I will imagine Wilf looking over my shoulder.