Let’s assume (this obviously a much simplified setting) that you are planning to release a new product version that will generate $1mn in sales. There may or may not be one critical bug left in the version. If there actually is a critical bug, this will cost to you in terms of product replacements, customer supports and customer disloyalty total of $2mn; if not, there is no extra cost. Let’s say that you estimate that the probability of having a critical defect in the product is 40%.
If you do not ship the product at all, your total future net profits are zero. If you ship it now, the expected net profits will be $1mn – (40% * $2mn) i.e. $0.2mn. This can be presented by the following payoff matrix:
| ACTION \ HAS DEFECT? | No | Yes |
|---|---|---|
| Ship | $1mn | $-1mn |
| Do not ship | $0 | $0 |
In this table, P(No) = 60% and P(Yes) = 40%.
Let’s say now that you can spend $0.2mn in testing the product before you ship it, and that the testing process has 50% change of finding the critical defect if it’s there. If you find the defect, you will fix it and then ship (we can assume here for simplicity that the cost of fixing the bug is zero); if you don’t find a defect, you need to make a judgement call. You have now the following options:
- Do nothing
- Ship without testing
- Test and fix, and ship only if a defect was found
- Test and fix, and ship always
The corresponding payoff matrix looks now like this:
| YOU \ NATURE | No defect | Defect, not found | Defect, found |
|---|---|---|---|
| Ship without testing | $1mn | $-1mn | $-1mn |
| Do not ship | $0 | $0 | $0 |
| Test and ship if a defect was fixed | $-0.2mn | $-0.2mn | $0.8mn |
| Test and ship in any case | $0.8mn | $-1.2mn | $0.8mn |
In this case, P(no defect) = 60%, P(defect exists but not found) = 40% * 50% = 20%, P(defect exists and found) = 20%, so we can calculate the expected outcomes for the four different strategies:
- Do not ship: $0 (as before)
- Ship without testing: $0.2mn (as before)
- Test and ship if defect was fixed: 60% * -$0.2mn + 20% * -$0.2mn + 20% * $0.8mn = $0
- Test and ship in any case: 60% * $0.8mn + 20% * -$1.2mn + 20% * $0.8mn = $0.4mn
If you order these strategies in the order of diminishing profits, the topmost (best) strategy is the last one: test the product, and ship it in any case.
What to Take Home
Ok, so the point is now that if you invest in testing and you do not find any defects, that has created you net economic value of $0.2mn (in this example), because shipping after testing that does not find defects leads to $0.4mn profits (including the cost of testing), whereas shipping before testing leads to only $0.2mn profits. (If your testing campaign finds the defect, then you know that the total value generated by the product minus cost of testing will be $0.8mn, but that does not really matter here.)
The key observation hence is that tests that pass do generate real, economic value, if they are tests that have actually a good chance in spotting potential defects. From this it is easy to draw the further conclusion that what really matters is the coverage of your tests (what is the chance of spotting defects) than the actual number of defects spotted. And that makes perfect sense. Think of a mine sweeper. The economic value of sweeping a field for mines and not finding them is huge if you are planning to walk over!