At Ivar Jacobson International (IJI) we have been involved in many agile adoptions ranging in size from single teams to entire IT development organizations. The single biggest problem we see is organizations not understanding why they are changing the way they work – they don’t visualize the goal, set any targets, measure the improvements, nor demonstrate the benefits generated.

This isn’t a problem where an empowered team is looking to improve their own way-of-working. However, it is often a showstopper when attempting a broader adoption, such as an organizational transformation or where the team is not yet empowered.

The key here is to establish a set of actionable measures to drive the change and inspire the teams. These should explicitly support the principles and values being promoted and challenge the teams to improve.

In this article we will look at one way to establish a measurement dashboard to support your agile transformation.

What Should You Measure?

To gather empirical evidence of the effect of the transformation you need to have both:

  1. Measures of the improvement – measures that quantify how well the transformation is going, the amount of cultural change and benefits being generated. For the teams to be truly empowered these measures should be kept as process and practice independent as possible.
  2. Measures of the mechanisms that produced the improvement – measures that quantify what is being used where, demonstrate the uplift in capability across the agile teams, and show whether the transformation is on track to achieve self-sufficiency

These measures will complement and, in many cases, reuse the measures that are being used by the teams to drive and control their work.

In this article we will focus on how to measure the improvements. The sad thing is that there is no standard set of improvement measures that can be promoted or prescribed as suitable for all agile transformations. Each agile transformation is different; with different goals to achieve and problems to address. What is needed is a set of measures focused on the improvements that need to be made. This set of measures will need to be inspected and adapted to ensure that it continues to be relevant as the transformation progresses and the initial improvements are built on to generate more and more business benefit.

Measuring the Improvement: Building an Improvement Dashboard

Our preferred approach to establishing an improvement dashboard is to use our Better, Faster, Cheaper, Happier framework. This is summarized in Figure 1 below.

Figure 1 – The Better, Faster, Cheaper, Happier Measurement Framework

Figure 1 – The Better, Faster, Cheaper, Happier Measurement Framework

This is a simple framework that helps organizations produce a balanced set of improvement measures that are aspirational rather than operational. It is important to achieve a balance between the measures, so that you don’t increase delivery speed whilst sacrificing employee or customer satisfaction, or improve quality whilst lowering productivity.

This proven framework helps organizations to identify and implement practical measures that:

  • Focus on the real business drivers, goals and benefits
  • Measure the things that really matter – the effects not the causes
  • Show whether things are really improving – eliminating seasonal fluctuations by looking at performance over a year, not a week or month
  • Consider how people feel – gathering both qualitative and quantitative data
  • Are simple and easy to understand – not complex derived measures and indexes
  • Don’t depend on using particular processes and practices – allowing the old and new worlds to be compared
  • Have relevance to the business – clearly showing where the business would like to see improvements
  • Have targets and obvious trends – the on-going trends are more important than point measures

The result will be a set of measures and targets that everyone can understand and support.

Some Illustrative Measures

There are lots of measures that could be relevant to your transformation. For illustrative purposes Figure 2 shows a compilation of some of the measures that we have seen to be popular within organizations embarking on large-scale agile transformations. For example, when starting out nearly everyone targets on-time delivery before focusing on cycle times – as a wise man once told me less late is still late.

Figure 2 - Change Initiative Improvement Dashboard

Figure 2 – Change Initiative Improvement Dashboard

A couple of things to note about this dashboard are:

  1. This is an illustrative dashboard and has never been used in this form by any specific organization. All the customers have used some of these measures, but they have also all had their own specific measures to address their specific problems.
  2. This probably lists too many measures for use at any one time. Most adopters start with eight or so measures, which is more than adequate to cover the four quadrants. At IJI we have built up a library of more than 30 candidate measures that support common organizational goals. The important thing is to understand the goals and the underlying concerns rather than to just go shopping for measures.

The measures shown in Figure 2 are:

Better:

  • On-Time Delivery – Percentage of releases that are released on-time
  • Escaped Defects – Mass of defects that escaped from 1) development teams to acceptance test and 2) from acceptance test to live in the last year
  • Meets Business Needs – Survey of whether the customers think the software delivered met their business needs
  • Level of Priority 1 Incidents – Number of priority 1 incidents reported in the live environments in the last year

Faster:

  • Time to Production – Number of weeks it takes for a project to get a release into production once the project is started banded by overall project cost
  • Decision Making – The time it takes to commit to progressing a project or producing a release from decision to evaluate the request
  • Timeliness of Delivery – Survey of whether the customers think that the software was delivered in a timely fashion suitable for their business

Cheaper:

  • Improved Value for Money – Survey of whether the customers think that the software represents good value for the money spent
  • Overheads – The cost of non-productive work done by the development teams (such as fixing escaped defects, support, administration etc.)
  • Stabilisation and After Care – The cost of stabilising and supporting releases in the warranty period after they initially go live

Happier:

  • Business Satisfaction (organisational level only) – Survey of whether IT’s business partners are happy with the service provided by IT
  • Team Satisfaction – Survey of whether the teams developing the software are satisfied with the way that they work
  • Customer Satisfaction – Survey of whether the business representatives and users directly involved in the teams’ work are satisfied with the way the teams work

It is important that all the teams, projects and programs within the scope of the transformation, regardless of their level of agility or buy-in, can use the dashboard. For example, KPN in Netherlands was one of the first organizations to use the framework. They applied it to more than 14 programs and 400 projects within their IT Innovation organization. In the spirit of autonomy and empowerment the programs were allowed to opt out of the transformation if they were confident they could meet the targets. Unsurprisingly the teams that decided to stay with a waterfall way of working showed little or no improvement across the board falling well short of the average performance in all cases. For more on the KPN measurement story see http://www.ivarjacobson.com/resources/resources/case_studies/

Getting Started

The dashboard is created through a series of short workshops where the objective is to understand the most important areas to improve, and to identify practical, intuitive measures to evidence the improvements. The people involved should include leaders, executives, customer representatives, senior managers plus other key stakeholders with a specific interest in the improvement goals and results.

The workshop helps participants to understand the business drivers, goals and needs, and set some meaningful improvement targets. It starts by simply brainstorming what the words better, faster, cheaper and happier mean to the participants focusing the conversation onto the goals of the transformation, and proceeds through various rounds of discussion, consolidation and voting to identify key areas for improvement. This helps to quickly establish what measureable benefits are expected from the transformation and makes sure that they are aligned to the business goals.

The end result is an overall dashboard containing a set of intuitive measures that support the organizational goals and present the overall targets for the next stage of the transformation. This enables everyone to see the targets, the current status, and the improvements taking effect. Typically the measures are captured on a monthly basis providing an instant feedback mechanism on the effectiveness of the approach and whether or not the desired improvements are materializing.

Wrap Up

A truly agile organization is a learning organization that is continually refining and improving all its practices. It is an organization that is focused on continuous improvement and delighting its customers.

To reach this promised land you need a laser-like focus on results, whilst not forgetting why you are implementing an agile approach. Too many agile transformations stall when they are unable to demonstrate that the new way of working is any better than the old way. The mistake they make is to think agility is an end in itself rather than an enabler for better business performance. Quantify the results you are looking for and make sure you measure the effect of the transformation.

It’s astounding how many agile transformations fail to track the benefits they are generating, and in many cases seem to leave the organization no better off than they were before. The people involved, particularly the sponsors and other executives, can also be very impatient. Creating an agile organization is not something that happens overnight. Agility is no silver bullet and not every agile team, or project, will be successful. Patience is needed to see the transformation through and reap the full benefits of agility. Measurement is needed to make the benefits visible.

In our experience the biggest benefits are achieved in the second year as agility becomes business as usual and the effects of agile working ripple through the value chain. Sadly if you don’t demonstrate how you are going to measure the improvements and make them relevant to the business you often don’t get a second year.measuring-enterprise-agility

Join the Discussion

    • Curtis

      I really like this approach and have taken copious notes on this for use with my own coaching.
      I am a little perplexed, however, about how to actually implement the Decision Making measurement. It sounds a bit like the KanBan Lead Time measure. But that doesn’t necessarily measure “decision making”. That particular effort would often be identified as part of the “wait time” in an effort such as Six Sigma, but I am not sure how you all go about measuring this.
      Could you clarify, please?

      • admin

        Hi Curtis,

        Glad you enjoyed the blog. Apologies for the delay in replying I’ve been having a proper summer holiday this year 🙂

        The particular situation where customers have used the Decision Making measure is in situations where there are formal funding processes for the commissioning of ‘projects’ and where it is important to separate the time spent making the decision (feasibility studies, estimations, slicing up, approvals, impact analysis, risk analysis, start up etc) from the time spent getting it done once it has been decided it should be done.

        So for the big ticket items – such as new initiatives, new projects, genuine epics, new use cases – it can be worth measuring the time spent traversing the early states as well as the time spent in development once the item is pulled. It’s not just wait time as considerable amounts of touch time can be involved as well.

        The sad thing is for many organisations more time and effort is spent on making the decisions than on actually developing things (it can also be useful to measure the cost of the decision making as this can often be higher than the cost of doing the thing especially for the smaller items).

        This often gets lost when people measure cycle times as they either 1) don’t bother measuring the time spent on the items that get rejected, and 2) often gets left out as people focus on development efficiency ignoring the earlier effort that went into getting well-formed items into the backlogs

        Hope this helps.

        Cheers
        Ian

    − 7 = 2