Research Reproducibility

In Nature, How to make replication the norm, the authors discuss several ways in which scientific research can – or more particularly in this case, cannot – be reproduced by later researchers.

One phrase from the paper struck me particularly hard:

“In addition, for a random sample of papers, journals should attempt to reconstruct the code from scratch or search the executable code for errors.”

A lot of my day-to-day job is spent searching code for errors. We have many tools for this:

  1. visual analysis of source code – imagine that I have plenty of experience of the language and software frameworks used, and of the domain that we’re working in. I simply read through the code, work out what it’s going to do, and compare that against what I understand the authors’ requirements were.
  2. discussion of potential errors with original authors; either before, during or after the previous step, I talk to the original authors, and discuss what their requirements were, and how they think the code is achieving them.
  3. many different kinds of dynamic testing; this is ultimately just executing the code, possibly – and more helpfully if so – with additional dedicated software tools to let me monitor the code as it’s running to see if it’s doing what we expect.
  4. many different kinds of static analysis that are pretty effective at spotting likely errors without even executing executable code – there is some really clever stuff out there that uses correlations between certain code snippets, and things that turned out historically to be errors.

Ultimately, then, I’m quite surprised that code provided as the basis for published research might not already have had this analysis done: as a software engineer, it’s pretty much unthinkable to provide software for any purposes where this hadn’t been done already.

In line with some work I’m doing with Tom Crick and Chris Wild, I’m keen to explore the cross-over between existing practices in software development and other fields that use software, or software-like human practices. I’ll be writing more on this soon.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s