Beyond lab folklore and mythology

Editors note: this post is part of our ongoing investigation into synthetic biology and bioengineering. For more on these areas, download the latest free edition of BioCoder.

Over the last six months, Ive had a number of conversations about lab practice. In one, Tim Gardner of Riffyn told me about a gene transformation experiment he did in grad school. As he was new to the lab, he asked two more experienced scientists for their protocol: one said it must be done exactly at 42 C for 45 seconds, the other said exactly 37 C for 90 seconds. When he ran the experiment, Tim discovered that the temperature actually didnt matter much. A broad range of temperatures and times would work.

In an unrelated conversation, DJ Kleinbaum of Emerald Cloud Lab told me about students who would only use their lucky machine in their work. Why, given a choice of lab equipment, did one of two apparently identical machines give good results for a some experiment, while the other one didnt? Nobody knew. Perhaps it is the tubing that connects the machine to the rest of the experiment; perhaps it is some valve somewhere; perhaps it is some quirk of the machines calibration.

The more people I talked to, the more stories I heard: labs where the experimental protocols werent written down, but were handed down from mentor to student. Labs where there was a shared common knowledge of how to do things, but where that shared culture never made it outside, not even to the lab down the hall. Theres no need to write it down or publish stuff thats obvious or that everyone knows. As someone who is more familiar with literature than with biology labs, this behavior was immediately recognizable: were in the land of mythology, not science. Each lab has its own ritualized behavior that works. Whether its protocols, lucky machines, or common knowledge thats picked up by every student in the lab (but which might not be the same from lab to lab), the process of doing science is an odd mixture of rigor and folklore. Everybody knows that you use 42 C for 45 seconds, but nobody really knows why. Its just what you do.

Despite all of this, weve gotten fairly good at doing science. But to get even better, we have to go beyond mythology and folklore. And getting beyond folklore requires change: changes in how we record data, changes in how we describe experiments, and perhaps most importantly, changes in how we publish results.

30-odd years later, we have experiments that throw off terabytes and even petabytes of data per day, and we have the tools to analyze that data. But we have problems with reproducibility, arguably more problems than we had years ago. I dont believe the problem is that we dont sacrifice enough chickens. Rather, we havent been radical enough about what data to collect. Weve automated some of the data collection, but we still dont collect all (or even most) of the data thats potentially available: intermediate results from each step, calibration data from each piece of equipment, detailed descriptions of the process, and the configuration of every piece of equipment. Kleinbaum told me that some experiments are sensitive to whether you use glass or plastic test tubes. That makes sense: its easy to scratch plastics, and microbes can hide in scratches that are invisible to the human eye. Plastics can also release trace amounts of gasses long after theyre manufactured, or absorb some of the compounds you want to measure; for some experiments, that matters, for others, it doesnt. Few scientists would consider the test tubes used, the pipettes, and so on, as part of the experimental data. That must change if were going to solve our problems with reproducibility.

In addition to the data, we also have to record exactly how experiments are performed, in detail. Everybody I talked to had stories about protocols that were part of their labs oral culture: you did things a certain way because thats how you did it. It worked, no need to belabor the point. A recent article asks, in frustration, Never mind the data, where are the protocols? Having the data means little if you dont have the methods by which the data was generated. The best way to record the protocols isnt by scribbling in lab notebooks (or their virtual equivalents), but by implementing the experiment in a high-level programming language. The program is then a complete description of how the experiment was performed, including setup. The importance of this step isnt that the experiment can be run on lab robots, though that is important in itself; programming forces you to describe the process precisely and completely, in a standardized language that is meaningful in different contexts, different labs. Thinking of an experiment as a program also allows the use of design tools that make it easier to think through the entire process, and to incorporate standard protocols (you might think of them as subassemblies or subroutines) from other experiments, both in your lab and in others.

Thinking of an experiment as a program allows the use of design tools that make it easier to think through the entire process.

Were still missing an important component. Science has always been about sharing, about the flow of ideas. For the first few centuries of scientific research, publishing meant paper journals, and those are (by nature) scarce commodities. You cant publish a terabyte of data in a journal, nor can you publish a long, detailed, and extremely precise description of an experiment. You cant publish the software you used to analyze the data. When youre limited to paper, about all that makes sense is to publish a rough description of what you did, some graphs of the data, and the result. As our experiments and analyses get more complex, thats no longer enough. In addition to collecting much more data and describing the experiments as detailed programs, we need ways to share the data, the experimental processes, and the tools to analyze that data. That sharing goes well beyond what traditional scientific journals provide, though some publications (notably F1000Research and GigaScience) are taking steps in this direction.

See more here:
Beyond lab folklore and mythology