Comments on: Lott v. Levitt III

By: Bill Barth

Bill Barth — Fri, 21 Apr 2006 09:56:45 +0000

I’m not an experimentalist. I do computational work (computational fluid dynamics). As such, I take “replicate” to mean that the author should provide enough data about the methods, algorithms, numerical tolerances, etc and the problem setup (boundary and initial conditions, etc.). that another researcher could reproduce the author’s work by doing a separate implementation of the aforementioned methods, algorithms, etc. and then running the same numerical experiments. I also expect the original author to provide some evidence (convergence analysis, comparison to physical experiment, or both) that the method described in a published article is likely to have been a good simulation of nature.

The engineering community has not yet embraced the idea that computer codes and input datasets must be placed on file with the publishing journal or a standardized third-party repository. I understand that this is de rigueur in the social sience these days. I doubt that it will ever become necessary in the engineering community because the claims that are made from the results of engineering numerical or physical experiments are not about society in general, and are therefore not nearly as controversial. When the results are controversial, more scrutiny is required (with assistance and openness from the original authors).

As to Pons and Fleischmann, it’s my understanding that either they didn’t provide sufficient information to make replication easy, or that in the face of conflicting data they failed to do a publicly scrutinized demonstration of thier own original work. (Using a pressre release rather than a peer-reviewed journal article to announce their work, didn’t help either). Being able to repeat your own work is an absolute requirement for controversial conclusions in the physical sciences.

It seems to me that when it comes to the sort of statistics and statisical modeling that is so prevalent in the social sciences, that it is incumbent upon researchers to make their exact computer codes, data, and assumptions available for their colleagues to verify that the original authors did not make programming or (data) coding errors that signficantly change the results upon which the societal conclusions are drawn. This, in my mind, constitutes replication. Furthermore, other researchers should attempt de novo examination of the problem in question with thier own data, models, and computer codes to determine the the work done is definitive, or is simply one way of looking at the data (and by looking I mean in the social scientific sense).

In the physical sciences it is often possible to simply look at some result and conclude that it must be in error, since it is completely outside the realm of ordinary scientific possibility. Whenever statistics and statistical reasoning are involved (in the physical and social sciences), you can throw your intuition out the window (see the early history of quantum mechanics). At that point, only careful rexamination of the reported data, repitition of the undertaken experiments, and completely new experiments may verify that a result is certain (or even believable).

Finally, the inanimate portions of nature don’t lie (can’t lie), so only error in the design of apparati, conduct of experiment, or interpretation by the experimenter may lead to error in research into natural phenomena. Suvery respondents lie all the time, and so do law enforcement departments who collect crime data (i.e. there is frequent reclassification of certain crimes to skew the statistics to make a department look good). This means that in the social sciences even if a researcher’s process, assumptions, and models are perfect, the result can be wrong. This is considerably less likely in the physical sciences.

OK, that’s enough babble from me, I’m going to stop now.

By: John Quiggin

John Quiggin — Thu, 20 Apr 2006 23:27:45 +0000

As you’re a rocket scientist, Bill, how do you understand “replicate” in the physical sciences? Do you have to follow identical experimental protocols, same sample size, same test statistics and so on. Or do you just do the same experiment with what is, in effect, a new data set? I suggest the latter.

I read lots of articles about people who had failed to replicate the “cold fusion” results and in every case Pons and Fleischman (sp) blamed deviations from the original protocols. But no-one suggested that they could sue on this basis.

By: Bill Barth

Bill Barth — Thu, 20 Apr 2006 19:08:36 +0000

roy:

I’m not a professional economist, but I am a rocket scientist (i.e. I have a PhD in Aerospace Engineering). I was pretty taken aback by the replication claim as I read it in the more technical and narrow sense. Do I count as a layman? I certainly know little about economics and econometrics, but I do know a thing or two about publishing in academic journals (in engineering and physics).

I seems to me that many scientists and engineers may share my (and Lott’s) reading. Might not other types of non-economists as well?

By: roy

roy — Thu, 20 Apr 2006 12:42:19 +0000

The precise definition of “replicate” in question is only understood as such by professional economists, yes? Laymen readers — who are responsible for Freakonomics‘s astronomical sales — understand or perhaps misunderstand “replicate” to mean “reaching the same conclusion”. So will damages awarded be based only upon how many professional economists read Levitt’s claim, or how many total readers?

By: Overlawyered

Overlawyered — Thu, 20 Apr 2006 08:10:47 +0000

Lott v. Levitt IV

David Glenn, in the Chronicle of Higher Education, has the definitive MSM reporting on the affair. (Permanent link here after Apr. 24.) He finds a mixture of scholars who agree and disagree with Lott on…