Metriculation - The Faulty Assumption

Metrics can be used to garner a lot of feedback that’s valuable to the software development team. And they can also be used as a convincing argument to push an agenda. You have to be careful that metrics are used legitimately, and avoid metriculation. Metriculation is a term I use to describe how metrics can lie. One form of metriculation is the faulty assumption.

A faulty assumption occurs when we draw some conclusion based on the occurrence of simultaneous events, but have no substantive evidence correlating the events. Faulty assumptions are based on the logical fallacy of correlation proves causation. In software development, we have to be very careful not to make decisions based on faulty assumptions. Let’s take a couple of simple examples.

On any software development team, the number of developers who eat lunch while working at their desks may be related to the number of developers that get stuck in the elevator. The conclusion then is that you should not eat lunch at your desk if you want to avoid getting stuck on the elevator. Even the less than astute individual recognizes the absurdity of that conclusion.

But what about this more plausable scenario? The number of developers using Fancy Framework X is related to the number of developers with fewer defects in their code. Our conclusion now is that if we want to reduce software defects, then all developers should use Fancy Framework X. Possibly. But is there a causal relationship between using Fancy Framework X and code with fewer defects? Or have we falsely assumed that there is?

Proving causal relationships is difficult, if not impossible. The slightest possibility that there may not be a causal relationship means the conclusion can always be challenged. Proving causality requires statistical analysis, which may not always be feasible. Sometimes the best we can do is apply inductive reasoning to arrive at our conclusion based on the likelihood of causality. The moral here is that we should not be awestruck by metrics. Nor should we be naive. Instead, leveraging metrics requires critical thinking.

Comments