Measure and Benchmark - Part 2: Science of Measurements & Common Errors in Software Metrics Benchmarking

Measures and Metrics in Software World is an interesting story of Love and Hate. “If you can’t measure it, you can’t improve it” is oft- quoted in presentations selling metrics. It is logical and Improvement Methodologies like 6-Sigma build heavily on metrics. However, before we go further into Metrics used in Software Engineering, a peek into the Science of Measures and Metrics.

• Parameters like speed of light, frequency of Cesium are universal constants 
• Standard definition available for units like Meter, Kilogram like “distance travelled by light in a defined time-frame” or “Mass of a reference material”
• Derived Quantities like “Volume”, “Density” have standard definition in terms of base units like “Meter³” and “Mass per unit volume”
• For Units other than SI Units e.g. Mass measured in Pounds or distance in Miles, there are standard conversions factors available to convert to Kilograms, Kilometres etc.

In the world of Software Engineering,

• No constant number available for “Person-hours per day” or “Person-hours per person-month” but is widely used across
• No standard definition of units like Size. Not even a universally approved list of units used like Function points, Feature point, Use Case Points etc.
• No universal dictionary of Derived quantities like “Defect Density”, “Productivity”
• No Industry Standard Conversion factors e.g. “Function point to Use Case Point” etc.

Because of the above, discussions around comparison with benchmarks or verification for capability to meet Contractual Expectations are error-prone. Some of the aspects frequently missed out are as below:

Some Common Error	Description
Compare the values of a metric with another company’s or with benchmark assuming the Definition to be the same	For example, a company may have a semi-formal Unit-testing and hence defects not logged/ counted. While comparing Defect density do check if the other metric comes from a process that includes inspection process and, if so, if defects are counted and added here
Comparing two values of a metric ignoring possible errors in conversion factor	Comparing one company’s “Function points per person-month” with another’s without checking “How many person hours is one person month”
Unit of Measurements that defy Dimensional Analysis	In physical science, if velocity is “Distance per time”, multiplying velocity with time can give only Distance. A “%” can come only if numerator and denominator have the same units and is multiplied by 100. Look for this dimensional analysis in Software. For e.g. “# of Defects detected” divided by “# of Days” can’t be a metric with “%” as Unit of Measurement.
Tagging a Metric as “Good” and “Bad”.	Every metric has something to tell and has things that it can’t. A metric by itself is not “Good” or “Bad”. Understanding every metric  as it is, is wiser than tagging it as “Good” or “Bad”
Labelling a Metric as “Leading” or “Lagging”	In the complex web of Cause-Effect, there’s nothing which is purely a “Cause” or an “Effect”. A project that slipped schedule may be a lagging indicator for a Project manager but a leading indicator for a Process owner who monitoring “% of projects delivered on-time for the year”

 

In addition to the above conceptual aspects, Metrics in Software Engineering is also subject to the below behavioral issues:

• It involves resources to gather measures or calculate metrics. So overdoing on metrics can impact human motivation
• Many measures like Effort consumed, Defects caught are declared by employees. If metrics are not used or not used constructively (e.g. used against individuals), management may start getting more “Good news” than “Accurate News”
• Taking decisions based on metrics need a good understanding of the metric, it’s correlation with other metrics or impact on behavioral aspects. If not it can lead to sub-optimal decisions. For e.g., the Cost of Quality may be high in a Financial or Healthcare application.

So, to summarize,

• Identify the measures & metrics needed based on information need. Don’t under-do or overdo
• Define every metric clearly at least at the org-level  and communicate, educate the stakeholders about the same
• Beware of the definitions, conversion factors etc. while discussing contracts or bench-marking
• Be sensitive to the behavioral aspects and it’s impact on metric system while using metrics for various reasons