Metrics for an Agile project

Q. How are we doing on delivering agreed scope of the current release?
A. Burn-up chart by iteration for the release. Below is a burn-up chart Manju created for reporting status on one of our large programs.



Among other things, this graph shows:
1. Scope changes (demonstrated by fluctuations on the "Total Scope" line
2. The gaps between succeeding status lines reflects in-process / wait stories. Larger than normally accepted gaps indicate bottlenecks. E.g. Dev is a bottleneck due to the huge gap between Analysis Complete and Dev Complete
3. Inventory of stories that are ready to go live (demonstrated by the "Showcase Passed" line
4. Actual completion status (demonstrated by the "Showcase passed" status line)

Q. How are we doing on throughput? How much value are we delivering? What is the trend - running faster, slowing down?
A. Velocity graph by iteration for the project. Only "Done" stories considered for velocity calculations. Below is a velocity graph Manju created for tracking velocity on one of our large programs. The 3 iteration average was first brought to my notice by Santosh, who was using it in one of his projects. I find this extremely valuable, as it balances the ups and downs into something like a trend line.

Why iterative development?

"Until you have seen some of the rest, you can't make sense of any part" - Marvin Minsky.

Minsky says this in the context of describing complex systems. This applies as much to software systems as to intelligence. How can we help users describe a complex system? Wouldn't building some of the rest help them in making sense of the parts.

Bottleneck - Cont.

Below is some data from my previous project:

Wait stages:
Ready for Dev 78
Ready for BA Acceptance 25
Ready for QA 14
Ready for Showcase 12
Ready for SAT 50

In-process stages:
In Analysis 71
In Dev 92
In QA 10

Its clear that Development is the bottleneck. Development takes the longest time among all the stages. Things just don't move as fast here. So, we push more work into this stage. That is the reason for the high in-process work. And more work means multi-tasking for the developers and consequently, diluted focus. That further adds to the time stories take to move out of this stage. And of course, you just cant push enough through the Development stage, so the inventory piles up. This could be the situation in most software development projects. The symptoms of this bottleneck sometimes showed a high inventory in other stages. But they could be traced back to the Development stage in most cases.

I wonder why we didn't look into the Development stage itself and saw what was happening WITHIN the stage. That could have helped us understand how to speed up the Development process.

Bottlenecks

My last project had a huge bottleneck at system acceptance test (SAT) stage - the SAT team was not able to sign-off stories at the same pace as the dev teams. Though I don't readily have the data, I am certain that the in-process time at the SAT stage was not high. Between the dev team completing the story and the SAT team picking it up were 2 steps - deployment into SAT Servers and showcasing these stories to SMEs and SAT team. Though we were deploying into SAT on a weekly basis, the showcases were done only once in 2 weeks.

There are 2 questions that bother me:
1. Is SAT the bottleneck? Is bottleneck identified purely by the inventory before that stage?
2. How can we ensure we manage the SAT process better?

My thoughts:
1. SAT is not the bottleneck here. If we define bottleneck as the number of stories that can be "processed" at a particular stage, given their ability to work full-time, SAT was not the bottleneck - as their ability to sign-off stories was high (demonstrated during the later iterations of the release). Even if we consider inventory as a measure, given that showcase is a mandatory step before SAT and showcase is done only once in 2 weeks, showcase could be the bottleneck. SAT is like the final assembly. What consumes time is not the assembly itself, but the wait for all parts to come through before they start.
2. I reckon one thing that can be done is: reduce the batch size. Do more showcases. Once a week maybe.

Prioritizing stories based on relative affordability

We use value delivered by the story as a primary measure of priority. Would it be more meaningful to consider Relative Value / Cost (relative affordability) as a formula for determining the priority of stories? Cost can be a relative measure of the complexity or effort for implementing and testing the story, determined by the development team (BAs and QAs) - in other words relative size of the story.

This relative affordability of a story gives us a measure of the worthiness of investing in a story. Judging purely based on value is inappropriate because the cost may be prohibitive.

Estimating value of technical stories

How can we determine the value of a "technical story"?

As discussed below in the case for relative value: The value of a story derives from making the system more attractive for end customers. And that could either be developing a new feature or making the system scale better.

For technical stories that fall under non functional requirements (requirements being the key word here), it should be possible to establish their value based on the above definition. An extreme position would be to de-prioritize any NFR / story that cannot establish how it would make the system more attractive for end customers. There is a possibility that inadequate articulation of the value may lower the perceived value of a tech story. But hey, I would ask the tech guys to articulate the value in a different and better way rather than let it get in the back door.

Note: Value need not necessarily be limited to returns in the short term. Long term benefits should also be considered to be of value, in some cases, more valuable than short term gains.

The concept of Relative Value

One of the interesting things that Chaman brought to my notice was details around TOC and that Throughput should be measured as sales (Rs / dollars).

Now, most people see Throughput in terms of story points that the dev teams estimate for stories. This is natural but inconsistent with the spirit of TOC. Story points, no matter that they are nebulous and a relative measure, are estimated by the people doing the job (devs, qas, etc.). And that firmly land them on the cost side. While it is meaningful to get this estimate, for the purpose of measuring throughput in the context of TOC, we may find "value" estimate by business more appropriate.

Now, given that the stories are not independent units but assimilate into a larger system which is then sold to end customers or used by end customers, the value of a story can't be defined with precision. The value of a story derives from making the system more attractive for end customers. As this is an abstraction of "value", its estimation becomes subjective to interpretation and nailing it down with accuracy difficult. Hence, relative value. My suggestion would be to use the standard sizing / estimation concepts for this. With some changes. Get the business to answer 2 questions:

1. By implementing this story, how much more direct (and indirect) revenue will the system generate? If this cannot be determined quantitatively, then ask the second question:
2. By implementing this story, how qualitatively useful is it making the system for end users / customers?

Use triangulation to ensure the relative accuracy of these numbers. This should enable us to measure relative value being delivered every iteration and hence throughput.