Skip to main content
  • Poster Presentation
  • Open access
  • Published:

Challenges and solutions in replicability and provenance tracking for simulation projects

Reproducibility of experiments, one of the foundation stones of science, ought to be easy in computational neuroscience, given that computers are deterministic, and do not suffer from the problems of inter-subject and trial-to-trial variability that make reproduction of biological experiments more challenging.

In general, however, it is not easy, neither for the gold-standard case of reproduction by an independent researcher using independent code [1], nor for the much simpler case of an individual scientist or team being able to replicate their own results some months or years later. For this second case, the reasons include the complexity of our code and our computing environments, and the difficulty of capturing every essential piece of information needed to reproduce a computational experiment using existing tools such as spreadsheets, version control systems and paper notebooks.

In other areas of science, particularly in applied science laboratories with high-throughput, highly-standardised procedures, electronic lab notebooks are in widespread use, but none of these tools seem to be well suited for tracking simulation experiments. In developing something like an electronic lab notebook for computational science, there are a number of challenges:(i) different researchers have very different ways of working and different workflows: command line, GUI, batch-jobs (e.g. in supercomputer environments), or any combination of these; (ii) some projects are essentially solo endeavours, others collaborative projects, possibly distributed geographically; (iii) as much as possible should be recorded automatically; if it is left to the researcher to record critical details there is a risk that some details will be missed or left out, particularly under pressure of deadlines.

I present here a tool, Sumatra, for simulation project management and for automated recording of detailed provenance information: (i) the code that was run, (ii) any parameter files and command line options, (iii) the platform on which the code was run. Sumatra consists of a core library, implemented as a Python package, Sumatra, together with a series of interfaces that build on top of this: a command-line interface, a web interface, and a desktop interface. Each of these interfaces enables (i) launching simulations with automated recording of provenance information; and (ii) managing a simulation project: browsing, viewing, deleting simulations. Alternatively, modellers can use the Sumatra package directly in their own code, to enable provenance recording, then simply launch simulations in their usual way. Sumatra is distributed as open-source software (http://neuralensemble.org/sumatra/), so its functionality can be incorporated in other tools, and developed using a community model: anyone is welcome to get involved with its development.

References

  1. Nordlie E, Gewaltig M-O, Plesser HE: Towards Reproducible Descriptions of Neuronal Network. PLoS Comput Biol. 2009, 5: e1000456-10.1371/journal.pcbi.1000456. [http://0-httpdx-doi-org.brum.beds.ac.uk/10.1371/journal.pcbi.1000456]

    Article  PubMed Central  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Andrew P Davison.

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Davison, A.P. Challenges and solutions in replicability and provenance tracking for simulation projects. BMC Neurosci 11 (Suppl 1), P76 (2010). https://0-doi-org.brum.beds.ac.uk/10.1186/1471-2202-11-S1-P76

Download citation

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/1471-2202-11-S1-P76

Keywords