Reproducibility

A fundamental principle of empirical (experimental) science is reproducibility. Scientific results should not be flukes, they should be based on documented and replicable processes. When we report the results of an experiment, we typically present a written description of the methods, as well as written and graphical reports of the results. In principle, these descriptions should be sufficient for a reader to reproduce your experiment, and hopefully get similar results. Of course, in neuroscience and psychology research, each experiment typically involves a new sample of participants, so even if the experiment is reproduced exactly, and the data analyzed identically, we can expect some variability in the results because we sampled a different set of individuals. However, if we take a copy of the original data, we should be able to produce the same results by following the documented procedures. This is one of the fundamental principles of open science, as discussed in the previous chapter.

In practice, this is harder than it sounds — especially if we are using the manual spreadsheet approach described above. Unless the analysis was documented very closely, it’s possible that methodological differences will arise. For example, a Methods section might state that the mean RT was calculated for each participant and then averaged across participants, but this doesn’t specify that this was done by a lab volunteer cutting and pasting numbers while simultaneously watching YouTube and chatting with another student in the lab. Even if the procedures were precisely documented, replicating the process would still be as tedious and error-prone — and very likely, even the errors would be different since they are random occurrences.

Science should not be this way; science should be accurate, precise, reliable, and reproducible. For this to be true, we need high standards of control and automation to ensure that data is handled consistently and reproducibly. If only we could replace those flaky lab volunteers with machines that did precisely what we intended…