Gnuritas

Tips on designing, writing and debugging individual based simulations

This post was originally written in 2012, and was slightly updated in 2017.

When I first started writing stochastic, individual-based computer simulations in 2005, I thought it would be a pretty straightforward job. Although I’m technically a biologist, I already had quite some (self-taught) experience in C-programming and knew about Object Orient programming. Also, my history of hand-optimising assembly code for the Atari Falcon had taught me a thing or two about writing fast code. Or so I thought. Of course I then proceeded to walk into just about every trap that a naive non-professional software developer can walk into… Here are some of the things I learned over the past years, quite a few of these by trial and error…

Debugging

It is often said that a software developer spends 50% of his time designing code, 50% of his time implementing it and 400% of his time on debugging*. This is probably an understatement. Moreover, bugs are especially problematic in scientific software, as they may invalidate all your results… (*: Thanks Bastiaan, for that quote! ;) Here are some points that may significantly help you reduce both the risk of serious bugs, and the time spent debugging:

• Test your code frequently in the design-phase. Write dedicated testing code for the different parts of the simulation, if needed (unit-testing).
• Compile with all warnings enabled (e.g. -Wall in GCC), and read the warnings!
• Insert plenty of debugging code and sanity checks. More is better! Use #ifdef to enable/disable such code-blocks if needed (for speed).
• Use a family of status/error logging functions or macros, so you can specify verbosity.
• Always compile test-runs with debugging information (-g and -ggdb on GCC) and perform your test-runs in GDB orDDD. Disable optimisation if needed.
• Learn the basics of how to use the GDB/DDD debugger: backtrace, frame and print are your friends. Use watchpoints for hard-to-catch bugs (e.g. dangling and uninitialised pointers). Don’t be afraid to insert breakpoints or throw exceptions in your debugging code.
• Regularly use Valgrind to check your program for memory leaks and corruptions. Really.
• Consider using safe pointers and other proxy-constructs that can aid debugging, but can also be disabled for speed. In C++, consider using std::safe_ptr / std::tr1::shared_ptr (or go with the boost variants), in combination with #define macros.

KISS, design and maintenance

As any professional software developer knows, design is important. However, most scientists have had neither training nor have experience in software design. This is a problem. A bad design will generally get you into trouble further down the road. No design will mostly result in one big mess. Spend some time on reading about software design and on designing your program before you start to code. Also, Keep It Simple, Stupid.

• Design your class-structure beforehand, try to use established design patterns such as encapsulation, but don’t over-design things. Realise that your program will evolve, which means that your design *will* get messed up sooner or later. Don’t be afraid to re-factor your code when needed, and don’t wait too long with that.
• Start by implementing data-dump (and restore) functionality. This will not only allow you to save/restore simulations, it also greatly helps data analysis and debuging.
• Limit the simulation code to performing the actual simulation. Read input, perform simulation and dump output. Try to do as much as possible (e.g. data preparation and analysis) in higher level languages, like Python, Perl/PDL, R or Matlab. This also allows you to re-analyse the output without having to re-run the entire simulation.
• Keep the simulation code simple. Don’t make functions any more complicated than they need to be.
• Use a version-management system! My life has become much easier since I started using git. Regularly commit changes, and document your commits. Also, make regular backups of your code, notes, git repository and input data. You’d be surprised at how many people have lost months, if not years of work by harddisk-crashes, power-surges or even accidentally typing rm -R * in the wrong terminal…
• Provide embedded documentation of functions and data structures in your source-files, using something like Doxygen, Naturaldocs or RoboDOC.
• Provide a framework to store (at the very least) the parameters of all your runs, so you can reproduce earlier results. Also keep notes on your simulation-runs and important code developments, a lab-journal if you will. Make regular backups of both. I use small shell-scripts for launching runs, archiving output and parameters and making backups of everything. I also use Evernote for at least part of my notes.

Performance

If your program only runs for a few minutes, speed is not that important. If on the other hand you have a simulation that needs to run for several days, every cycle may count. A good design helps: keeping performance in mind from the start saves you having to redesign the entire thing to squeeze out some extra speed. However, the bottlenecks are not always where you think they will be during design, so profiling your code is essential.

• If you mostly know Java, consider learning C++. The speed difference is immense.
• Use the compiler optimisations that are available! (e.g. -O3 on GCC) This can easily speed up things by a factor 2 or more. Also provide the target-architecture if possible (e.g. -march=core-avx-i on GCC for newer Intel Core processors, -march=bdver2 for newer AMD processors, -march=amdfam10 for older AMD processors such as the Phenom I, etc.), to make maximum use of processor-specific optimisations.
• Profile your code, from the beginning! A good starting-point is Valgrind/Callgrind (valgrind --tool=callgrind ./[Your binary]) together with kcachegrind.
• Make your code modular (to ease maintenance and debugging), but consider inlining functions you call very often. Note that the compiler can do this for you, have a look at the GCC documentation and the inline keyword in C++.
• Use a decent pseudo Random Number Generator (e.g. the SFMT RNG), and implement buffering of random numbers if needed (I can provide an example of that).
• Some mathematical operations are expensive (e.g. exp() and log()). Consider using a processor-optimised implementation (e.g. sse_mathfun for SSE capable CPUs), a lookup-table (e.g. for exponentiation) or an appoximation (e.g. fastexp()) if this is a bottleneck, especially if precision is not that important.
• Don’t try to gain performance by saving on debugging code, just use compile-time macro’s to enable/disable code blocks when needed!
• Consider implementing paralellisation, but try to avoid locking. In other words, independent parts of your program can run in parallel, but when threads need to write in each other’s data structures, the locking overhead will significantly reduce the speed benefits of multithreading. Again, a good design is important, as is profiling to figure out where the performance bottlenecks are. Note that OpenMP is really useful for incorporating multithreading into existing programs, as it uses compiler #pragma’s.

Acknowledgements

Thanks to Patrick van der Willik for providing me with advice on decent programming techniques, Ben Gras for many a fruitful discussion and for introducing me to git and -many years ago- C, Max Baak for -among other things- insisting I try C++, Daniel Weise for yet more fruitful discussions and getting me to look into multithreading, Nobuto Takeuchi for showing me How Things Should Be Done and introducing me to SFMT, and Paulien Hogeweg and Rob de Boer for supervising and teaching me, and for introducing me to this entire subject in the first place…