Measure & Benchmark - Part 1: Why & What

When we make any purchase e.g. a car, there are lot of comparisons that happen pre-purchase to make the selection and post-purchase to validate the purchase decision. Most of these are numbers. In the car example, few evaluation parameters are as below.

• Budget?
• Which Features (ranging from airbags to sunroof)
• Fuel efficiency in Kilometres per Litre
• What’s the Maintenance cost (per year, roughly)
• Car characteristics like Boot Space, legroom and technical features like Weight (safety), Ground clearance, Turning Radius etc. to the extent we understand

We know that more the features, more expensive is the car. For the color we chose, there was a waiting time too. Similarly in a software project, there are parameters needed to be traded-offs with each other. These are best expressed by “Iron Triangle” – Quality/ Scope, Cost & Schedule (QCS).

• Scope/ Features. Size measures like Function points, Feature points or Story points are used as indicators. Features/ User Stories/ User Points  committed and delivered are measured for these
• Effort is measured in Person-hours. Going by many contracts, Effort is the Cost Indicator in Software projects. Effort measures are Planned Effort, Actual Effort etc.
• Schedules are about the dates e.g. Planned delivery date, Actual Delivery date etc. Going by nature of contract, schedule may be the primary cost indicator than effort. Depending on the business plan, Schedule slippage may mean huge opportunity cost to the client as well
• Defects indicate Quality. Measures include count of defects based on “Where they are caught” (internally or externally), severity of the defect (Show-stopper/ critical) etc.

Above measures are combined in different ways to provide various information e.g.

• Did I get everything for which I paid? à “Scope Expected vs. delivered” (%)
• How good was the Quality of the Product? à Count of Defects leaked. This is normalized (divided) by Project Effort or Size to enable comparison across projects
• Did I get it on-time? à Delay in delivery. This can be expressed as a % of duration of Project
• How efficient was the process? Efficiency or Productivity is “Output/ Input” e.g. Function Points per person-month (similar to Kms/Litre of the car)
• Cost/ Effort Overrun – Planned Vs. Actual Effort

In addition to the above measure of a "Project", "Product Quality Measures" are also done using Static Code Analyzers. These are similar to Weight, Ground Clearance and Turning Radius of the car. In software measures like Cyclomatic Complexity, Comment Ratio etc. are used which indicate many things including the future Maintenance cost of the software.

Depending on the nature of the project and process, metrics can be chosen/ configured and used. Measures & metrics in Software have certain issues vis-a-'vis the way it happens in the conventional world of Science. These are to be understood and taken care.