Code coverage rocks
4th March, 2019
5 min read
As a group, developers are a funny lot. We have a rosy view of ourselves. We like to think we are proud of our craft. We tell ourselves that we study best practices, write tests, seek reviews from our colleagues, practice self-reflection, and regularly refocus how we work as individuals and as a team. We fool ourselves into thinking we write the best code we possibly can.
In reality we fail all the time. We generate technical debt, we add bugs, we create confusion - what the heck is wrong with us?
Clearly, we need help.
Picking up rocks
Even monkeys know that it is easier to crack a nut by banging it with a big rock. It's questionable that we code monkeys are as clever, too often we throw away the metaphorical rock and set about cracking nuts using only the power of our 'highly evolved' brains, wondering why we are hangry all the time.
As my colleague Attila Vágo points out at HMH we recognised the power of the metaphorical rock long ago and have adopted a pretty standard, but very powerful toolset. I'm lucky enough to work with a great team that puts in the work to achieve the rosy-tinted ideal. But we are very aware that hard work is not enough, and is certainly not sustainable at the required level - we are always looking for a better rock.
insight
One of the hardest parts of coding is understanding exactly what your code is doing. This goes hand in hand with writing the simplest code possible and sticking to code style agreed by your teammates - no alarms, no surprises! Clever coders realised this simple technique a long time ago.
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Kernighan’s law
As they stand, unit tests and code reviews are pretty good at catching issues in your code. However, they can promote a false sense of security.
Unit tests are usually written by the developer that wrote the code, they might think they have tested all the variations of paths through the code - they probably have not.
A code reviewer is likely to notice deviation from coding standards or point out some code that is difficult to grok but they are much less likely to notice that unit tests fail to test all logic branches or even notice that some code will never be reached.
Code coverage reports make such problems impossible to miss.
Do your tests spark joy?
I've created a small repo that helps illustrate the points I'm making : https://github.com/mlennox/code-coverage-rocks
The code is a contrived example and is definitely not something that would pass a review in our team!
The tests pass and look reasonable as first glance. Great, we're done - pass the pizza, hand me a beer!
Hang on, if we look at the code coverage we can see something is up - uh oh...
Oh noes! The code coverage report spots the flaws straight away - you can't miss them!
Clearly it is not pizza time yet.
100% code coverage make you and your team better developers
Having a report that so obviously points out the flaws in tested code makes it easy to quickly open the code, add a test, refactor the code a little and achieve 100% code coverage before the pizza and beer reach thermal equilibrium. Best of all you don't even have to think very hard to figure out what to do - the report shows you exactly where the issue occurs.
Fixing issues highlighted by the coverage reports makes me happy every time, and I notice that my colleagues feel the same. The drive to achieve and maintain 100% code coverage is addictive, and dare I say it, fun!
Code coverage does not shame you into fixing your code. I mean yeah, code coverage reports are like giant, floodlight posters of your embarrassingly shoddy code with all the bad parts highlighted in neon yellow, nobody wants that hung out in public! It's the joy of removing the junk and replacing it with something simpler that drives you. That and learning what silly mistakes you unknowingly make.
Yes, code coverage reports teach you things you'd never have learnt - they make you a better developer.
Without code coverage reports, it is difficult to see where there are problems in the codebase. As pressure to deliver features builds, your code standards slip, your tests lag behind and it gets harder to fix each sprint. Constant, small, simple, and ideally daily test improvements and code refactors are easy to do when you know where to look, and will mean your codebase should never experience the technical debt it once did.
Caveats
Now, I imagine some of you are saying (screaming?) that 100% code coverage is a white elephant the pursuit of which will cause Ragnarok, and you are right. Sort of. Some frameworks require boilerplate code, and plenty of it. In these cases, there is no point in testing the framework. I'll delve into more details of strategies for refactoring and testing based on code coverage reports in another article, but for now:
Test early, test often.