Open Science: the Case for Preserving Raw Data

Everyone speaks well of the idea that the results of scientific research should be open for all to see, although there are obvious caveats to complete openness: Proprietary research, human subjects research, preliminary results, the pace and timing for releasing results, all come to mind.  But when it comes to research funded by the taxpayer, open science is almost a truism.  And again, while there are practical and principled reasons why complete openness is sometimes restricted, the readers of the OSTI blog will be familiar with the arguments for openness; the principle of reproducibility is a fundamental tenant of science, the possibility of accelerating the pace of discovery by making scientific results readily and easily accessible, these are just two critical pieces of the argument.  There is another reason for openness connected to both these points that was highlighted recently in Jonah Lehrer’s always interesting Head Case column in the Wall Street Journal (6/25/11).

            Here Lehrer points us to a fascinating controversy sparked by Stephen Jay Gould’s decades old criticism of work by the much maligned Samuel Morton on the varying sizes of people’s skulls.  Gould checked Morton’s data and concluded that Morton had cooked the books as it were and that his conclusions weren’t really supported by the actual measurements at hand.  Gould argued that Morton’s racial theories of intelligence biased his work.  And so the business stood until recently when a group of anthropologists reanalyzed all this data, going back and doing their own measurements of the skulls Morton used in his study 170 years ago (now that’s preserving samples!) and discovered, to everyone’s surprise you have to imagine, that Morton’s original measurements were correct.  They found his theories related to race and intelligence to be nonsense, but the data itself were solid.  Lehrer quotes the anthropologists as concluding that “ironically, Gould’s own analysis of Morton is likely the stronger example of a bias influencing results.” 

            “The larger lesson of the Gould-Morton affairs,” Lehrer notes, “is that bias is everywhere....”  “We assume that our judgment isn’t affected by financial incentives or personal opinion. But we’re wrong.”  Lehrer goes on to site recognized problems of bias in tightly controlled randomized, double-blind trials of drug efficacy, considered as “the gold-standard of medical evidence.”  And it’s not just science, of course, as some recent studies of independent auditors apparently reveals, according to Lehrer.

            The point for us is not to resolve the Gould-Morton spat or to become cynical about the quest for objectivity; for one, I happen to think objectivity is frequently, sometimes remarkably, achieved in all fields.  (The cynics, who thrive off the glib response, will say I’m biased toward objectivity.)   Whatever, the point to consider here, and one that’s framed by the Lehrer piece, is not only the need for us to understand how easily bias can creep into studies, but to make sure that the evidence that supports scientific and other studies is available for validation and reproducibility, just as Morton’s samples were available over 150 years after his work was completed.  This does not mean that all data needs to be retained for centuries, some aren’t worth saving for days, I suppose.  But where it matters, and can’t commonsense guide us here, raw data should be available in the spirit of open science.   It’s one of the best ways to answer cynics who say objectivity is impossible, “so go ahead, check my work.”