About feedback (Irrational Software)

During the weekly project meeting this morning I came to think about the importance of speed when it comes to feedback. I’ve been thinking about this before, when working on my now hibernating test-first programming book project (now that Kent Beck is working on one; mailinglist).

When unit testing, you get feedback in the matter of seconds, or else test-driven development wouldn’t work. For other testing approaches, feedback is slower. I’ve never been fortunate to do automated acceptance/customer testing, but that’s the next speediest feedback after unit tests. When it comes to releasing software to a group of testers, feedback is even slower – although it doesn’t compare to the slowness of feedback when you release to the public.

When you release the software to its final users, feedback might be quick (like if there’s a very serious bug that occurs frequently) or it might take months or years. And you can’t say that the software is okay just because you haven’t yet gotten any feedback to prove that it isn’t – although you can be more confident that it’s okay the longer it has been since it was released, and the more heavily it has been used.

So, speed matters when it comes to feedback. You want feedback as soon as possible. Therefore it’s important to do smaller releases – at least internally to the testers. The longer the feedback takes to arrive, the more overhead you’ll get in fixing bugs (since the matters isn’t fresh in your head). But there’s something more that matters besides speed: quality.

Think about it: let’s say that you release your software and after three months you get a post-it tacked to your display, saying that the program crashes all the time, asking you to fix it. Let’s also say that the person reporting the bug has gone on vacation for five weeks. This is an example of low quality feedback, while high quality feedback would tell you in which circumstances the bug occurs, or basically everything you need to do to reproduce it.

So, you want to do everything you can to maximize the speed and quality of the feedback – but unfortunately, there’s also a cost issue: building something that would ensure rapid high-quality feedback probably would be insanely expensive. The result is that you have to balance the speed, quality and cost of feedback.

When doing test-driven development, thinking about the cost and quality of the feedback (and sometimes the speed, although unit tests generally are very fast) is really important. Often, your first idea for testing something is more expensive (in terms of time to code the tests) than it would have to be. There’s often cheaper approaches that provide roughly the same test quality. Sometimes, lesser test quality isn’t that big a deal, since many tests cover the code in different ways. The measure for low test quality is how easily you can break code without tests detecting it. (See my post Testing in layers for more on this.)