On the importance of curating models

Most cardiac action potential models are published in a machine-readable markup language called CellML. This is a very good thing, because it unambiguously describes the system of equations that makes up the model, allows people to exchange and use them without the chance of introducing typos every time they translate it to a new language (or indeed publish it in print), and allows you to use whatever software for simulations you care to choose.

Back in 2011, Jonathan Cooper, Steve Niederer and I published a paper that proposed/developed a similar language for describing simulated experiments, in a format so that they could be applied to a wide variety of relevant action potential models.

One of the main reasons for this was to allow us to examine the functions, or behaviours, of the models in certain scenarios, and to compare them against experimental data from the same scenario. We termed this concept “functional curation“.

In the paper we took cardiac electrophysiology models in the CellML repository, ran the same protocols on all of them, and highlighted how quite a few of them failed to replicate some fairly basic electrophysiology properties.

In particular, we examined how the length of time a cell is electrically active (its Action Potential Duration, or APD) depends on how frequently it is stimulated. As you might expect, a cell that is stimulated more frequently needs to be electrically active for a shorter amount of time, to be ready for the next heart beat; whilst it makes sense to be active for longer at slow stimulation rates to avoid being susceptible to ectopic beats. This is known as APD restitution, and studying it on the first beat of a new pacing regime is called ‘S1-S2 restitution‘, as opposed to steady-state restitution.

Experimentally you tend to record a nice smooth ln(x) shaped graph, which would [perhaps?!] suggest some fairly simple underlying mechanism for this behaviour. Anyway, whatever the mechanism, the Beeler-Reuter model from 1977 replicated (qualitatively) this experimental behaviour very nicely:

S1-S2_curve_BR

The Beeler-Reuter 1977 restitution curve.

So we were quite surprised and dismayed to find that a lot of the newer models didn’t do anything nearly so sensible – see the paper for full details. A particularly odd example was the Decker 2009 dog model:

S1-S2_curve_old

The original Decker 2009 CellML restitution curve.

We didn’t think anything of this really, as a lot of the models were and are doing funny things, most of them because the model equations really do behave like that. Fast-forward to 2013 and Prof. Yoram Rudy, an author on the Decker 2009 paper, came to visit our group.

I was explaining our work on the functional curation system, and showed him the results of the S1-S2 study. Prof. Rudy was surprised the model performed like that, not least because the title of their paper was “Properties and ionic mechanisms of action potential adaptation, restitution, and accommodation in canine epicardium“! Sure enough we checked the paper, and the results in there were much more sensible than those shown above.

Prof. Rudy suggested that we compared our simulations with the code he publishes on his lab website. So I spent a couple of days tracking down the reason for differences between their code and the model as published on the CellML website. To be fair to the CellML curation process, there was a note in the repository that the file  “does not yet recreate the published results – the action potential is close but the calcium dynamics still require some fixing” – something we missed when including it in our paper! To cut a long story short, there were quite a few differences in the end – detailed here on the tracker item to replace the original CellML file, most of them small, but one mistake in a CellML calcium equation that was clearly a typo, and fixing this gave us a curve almost identical to the one in the paper (compare with Figure 3B in their paper, cycle length S1 = 1 second):

S1-S2_curve_new

The amended Decker 2009 CellML restitution curve.

So the Decker CellML file has now been updated, and there are a few morals to the story:

  • Be careful that the equations you are using are an accurate representation of the model.
  • Be careful that the model you are using behaves appropriately for the study you are performing.
  • I hope you haven’t been using it for arrhythmia simulations!

We reckon that the best way to prevent these problems is to replicate the results from a paper using a CellML file, and a similar unambiguous description of the protocol that is being applied. Ideally you’d do this for each figure in a paper. To this end we’re proposing a few enhancements to the Simulation Experiment Description Markup Language (SED-ML), that will enable it to do complicated protocols like the above. We’ve called this Functional Curation for now, it is open source and you can play with it if you like. There are lots of future plans, but we’re planning to publish a more systematic comparison of CellML models under a wider range of cardiac protocols on a dedicated website soon.

EDIT: I’ve used the new comments on PubMed to make a note about this next to the paper.

Advertisements
This entry was posted in Action Potential Models and tagged , , , , , . Bookmark the permalink.

4 Responses to On the importance of curating models

  1. Hitesh says:

    Hi Gary,

    A nice example showing how important it is to check the code and mathematical equations are in agreement. Quite a tough ask though given the size of these models!

    This is the first time I became aware of the paper by Cooper et al. that you mention in this post and it made me think about how should someone choose one of the many cardiac models that do exist for their particular problem. I wonder if you could comment on the criteria your group has used for choosing certain models when looking at drug effects on these models from a safety perspective. In particular why move away from the human model by Grandi et al.? Looking at the article by Cooper et al.i’m assuming it has something to do with the restitution curve?

  2. Gary Mirams says:

    Hi Hitesh,

    It’s an interesting quesiton, it seems certain models have risen to the top of the “re-use” pile (Luo-Rudy ’91, ten Tusscher ’04/’06) because they exhibit sensible basic properties across a range of different experimental situations/protocols. I suspect each researcher is testing a handful of models for relevant behaviour before using it in a study (hopefully), or possibly just re-using the one that was used last time someone did a similar study (also fairly likely!).

    For most of our drug safety studies to date, we’ve been mainly interested in APD behaving sensibly under block of a handful of different ion channels, so this has been our primary selection criterion. We haven’t moved away from Grandi (although TT and Shannon models were almost as good in the original Torsade paper). Although as you say, the Grandi cellML file’s S1-S2 curve doesn’t look great, so I would avoid using that CellML file for re-entry simulations. Note though: their paper shows different results for the restitution. It isn’t quite clear whether this is a possible bug in the CellML – as was the case above for Decker – or whether the steady-state restitution curve was put in their original paper in place of the S1-S2 curve.

    It is becoming more and more important to be able to characterise the wide variety of available action potential models, against a wide variety of experimental protocols (and in the near-future corresponding experimental data). Martin Scharm (@binfalse) is working with us at the moment on a prototype web interface for doing this, and we should have something publicly available to play with in early 2014.

  3. Pingback: Initial conditions and steady states | Mathematical Matters of the Heart

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s