The most important things cannot be measured. The issues that are most important, long term, cannot be measured in advance.

—W. Edwards Deming

Working software is the primary measure of progress.

—Agile Manifesto

Metrics

Metrics are agreed-upon measures used to evaluate how well the organization is progressing toward the portfolio, large solution, program, and team’s business and technical objectives.

Thanks to its work physics, timeboxes, and fast feedback, Agile is inherently more measurable than its proxy-based predecessor, the waterfall process. Moreover, with Agile, the “system always runs.” So, the best measure comes directly from the objective evaluation of the working system. Continuous delivery and DevOps practices provide even more things to measure. All other measures—even the extensive set of Lean-Agile metrics outlined below—are secondary to the overriding goal of focusing on rapid delivery of quality, working Solutions.

But metrics are indeed important in the enterprise context. To that end, SAFe provides guidance for various metrics that can be applied for each level of the Framework. The links below navigate to the entries on this page.

Portfolio Metrics

Lean Portfolio Metrics

The set of Lean portfolio metrics provided here is an example of a comprehensive but Lean group of measures that can be used to assess internal and external progress for an entire portfolio. In the spirit of “the simplest set of measures that can possibly work,” Figure 1 provides the leanest that a few Lean-Agile portfolios are using effectively to evaluate the overall performance of their transformations.

 

Figure 1. Lean portfolio metrics

Back to Top

Portfolio Kanban Board

The primary purpose of the Portfolio Kanban board is to ensure that Epics are weighed and analyzed prior to reaching a Program Increment (PI) boundary. This way, they can be prioritized appropriately and have established acceptance criteria to guide a high-fidelity implementation. Further, the business and enabler epics can then be tracked to understand which are being worked on and which have been completed.

Back to Top

Epic Burn-Up Chart

The epic burn-up chart tracks progress toward an epic’s completion. There are three measures:

  • Initial epic estimate line (blue) – Estimated Story points from the lean business case
  • Work completed line (red) – Actual story points rolled up from the epic’s child Features and stories
  • Cumulative work completed line (green) – Cumulative story points completed and rolled up from the epic’s child features and stories

These are illustrated in Figure 2.

Figure 2. Epic burn-up chart

Back to Top

Epic Progress Measure

The epic progress measure provides an at-a-glance view of the status of all epics in a portfolio.

  • Epic X – Represents the name of the epic; business epics are blue, and enabler epics are red
  • Bar length – Represents the total current estimated story points for an epic’s child features/stories; the dark green shaded area represents the actual story points completed; the light green shaded area depicts the total story points that are in progress
  • Vertical red line – Represents the initial epic estimate, in story points, from the Lean business case
  • 0000 / 0000 – The first number represents the current estimated story points, rolled up from the epic’s child features/stories; the second number represents the initial epic estimate (same as the red line)

These measures are depicted in Figure 3.

Figure 3. Epic progress measure

Back to Top

Enterprise Balanced Scorecard

The enterprise balanced scorecard provides four perspectives to measure performance for each portfolio—although the popularity of this approach has been declining over time in favor of Lean Portfolio Management (LPM), as shown in Figure 1. These measures are:

  • Efficiency
  • Value delivery
  • Quality
  • Agility

These results are then mapped into an executive dashboard, as illustrated in Figures 4 and 5.

Figure 4. A balanced scorecard approach, dividing measures into four areas of interest

 

Figure 5. Converting the above metrics to an alphabetical rating and aggregating them can provide a look at the bigger picture for the enterprise

For more on this approach, see chapter 22 of Scaling Software Agility: Best Practices for Large Enterprises [2].

Back to Top

Lean Portfolio Management Self-Assessment

The Lean Portfolio Management team continuously assesses and improves their methods. This is often done through a structured, periodic self-assessment process. When the LPM team completes the spreadsheet, which you can download below, it will automatically produce a radar chart like the one shown in Figure 6, highlighting relative strengths and weaknesses.

Figure 6. Portfolio self-assessment radar chart

Download Portfolio Self-Assessment

Back to Top

Large Solution Metrics

Solution Kanban Board

The primary purpose of the Solution Kanban board is to ensure that Capabilities are reviewed and analyzed prior to reaching a PI boundary. This way, they can be prioritized appropriately, using established acceptance criteria to guide a high-fidelity implementation. Further, the features can then be tracked to understand which are being worked on and which have been completed.

Back to Top

Solution Train Predictability Measure

To assess the predictability of the Solution Train, individual measures for an Agile Release Train (ART) can be aggregated to create a Solution Train predictability measure, as illustrated in Figure 7.

Figure 7. Solution Train predictability measure

Back to Top

Solution Train Performance Metrics

To assess the performance of the Solution Train, individual performance measures for an ART can be aggregated to create a set of Solution Train performance metrics, as illustrated in Figure 8.

Figure 8. Solution Train performance metrics

Back to Top

Program Metrics

Feature Progress Report

The feature progress report tracks the status of features and enablers during PI execution. It indicates which features are on track or behind at any point in time. The chart has two bars:

  • Plan – Represents the total number of stories planned for a feature.
  • Actual – Represents the number of stories completed for a feature. The bar is shaded red or green, depending on whether or not the feature is on track.

Figure 9 gives an example of a feature progress report.

Figure 9. Feature progress report, highlighting the status of each Feature compared to the Program Increment plan

Back to Top

Program Kanban Board

The primary purpose of the Program Kanban board is to ensure that features are reviewed and analyzed prior to reaching a PI boundary. This way they can be prioritized appropriately, with established acceptance criteria to guide a high-fidelity implementation. Further, the features can be tracked to understand which are being worked on and which have been completed, as well as to track the performance of the Continuous Delivery Pipeline.

Back to Top

Program Predictability Measure

To assess the overall predictability of the release train, the team PI performance report aggregates all teams on the train to calculate the program predictability measure, as illustrated in Figure 10.

Figure 10. Program predictability measure, showing two of the teams on the train and program (cumulative)

The report compares actual business value achieved to planned business value (see Figure 22).

For more on this approach, see chapter 15 of Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise [1].

Back to Top

Program Performance Metrics

The end of each PI is a natural and significant measuring point. Figure 11 shows a set of performance metrics for a program.

Figure 11. One train’s chart of performance metrics

Back to Top

PI Burn-Down Chart

The PI burn-down chart shows the progress being made toward the PI timebox. It’s used to track the work planned for a PI against the work that has been accepted.

  • The horizontal axis of the PI burn-down chart shows the Iterations within the PI
  • The vertical axis shows the amount of work, in story points, remaining at the start of each iteration

Figure 12 exemplifies a train’s burn-down measure.

Figure 12. One train’s PI burn-down chart

Although the PI burn-down shows the progress being made toward the PI timebox, it does not reveal which features may or may not be delivered during the PI. The feature progress report provides that information (refer to Figure 9).

Back to Top

Cumulative Flow Diagram

The Cumulative Flow Diagram (CFD) is made up of a series of lines or areas that show the amount of work in different steps progressively in a Value Stream. For example, the typical steps of the program Kanban are:

  • Funnel
  • Analyzing
  • Backlog
  • Implementing
  • Validating on staging
  • Deploying to production
  • Releasing
  • Done

In the cumulative flow diagram in Figure 13, the number of features in each step is plotted for each day in the chart.

Figure 13. Sample Program Increment cumulative flow diagram

Back to Top

Agile Release Train Self-Assessment

As program execution is a core value of SAFe, the Agile release train continuously works to improve its performance. The self-assessment form, which you can download below, can be used for this purpose at PI boundaries or any time the team wants to pause and assess their organization and practices. Trending this data over time is a key performance indicator. Figure 14 gives an example of the results on a self-assessment radar chart.

Figure 14. Agile Release Train self-assessment radar chart

Download Agile Release Train Assessment

Back to Top

Continuous Delivery Pipeline Efficiency

This metric weighs the different steps of the continuous delivery pipeline coming from the program or solution Kanban. It evaluates the efficiency of each by measuring the relationship between touch time and wait time. It can become the foundation for value stream mapping. Some of the information can be taken from tools, especially around Continuous Integration and Continuous Deployment, while some of the other data might need to be logged manually in a spreadsheet. In the case of manual data, provide an estimate of the average touch and wait times.

 

Figure 15. Continuous Delivery Pipeline efficiency

Back to Top

Deployments and Releases per Timebox

This metric is meant to demonstrate whether the program is making progress toward deploying and releasing more frequently. It can be viewed on a PI basis, as shown in Figure 16.

Figure 16. Deployments and releases per Program Increment

Or we can zoom in to see how releases are handled in mid-PI, as shown in Figure 17.

 

Figure 17. Deployments and releases per Iteration

Back to Top

Recovery over Time

This measures the number of rollbacks that occurred either physically or by turning off feature toggles. It overlays this data with a point in time where we deployed to production, or released into production, to see how these relate to needs for rollback.

Figure 18. Recovery over time

Back to Top

Innovation Accounting and Leading Indicators

One of the major goals of the continuous delivery pipeline is to enable the organization to run experiments quickly to allow end Customers to validate hypotheses. As a result, both Minimal Marketable Features (MMFs) and Minimal Viable Products (MVPs) must have a definition of the leading indicators for the business outcomes of those hypotheses (see the Epic article for more details). This permits measurement using real innovation accounting instead of vanity metrics.

In most enterprises, such leading indicators will reoccur. So it’s important to monitor them continuously, as well as overlay the results with the timing of releases of the different features.

Figure 19 shows some metrics that were gathered from the SAFe website to demonstrate leading indicators for our development efforts.

Figure 19. Leading indicators for SAFe website innovation accounting

Back to Top

Hypotheses Tested over Time

A major goal of hypothesis-driven development is to create small experiments that are validated as soon as possible by customers or their proxies. The metric in Figure 20 shows how many hypotheses were validated in a PI and how many of them failed.

Figure 20. Hypotheses tested over time

In an environment of quick testing (see [3] for more information), a high failure rate is actually a good thing, as it allows the team to rapidly identify, validate, and focus on the good ideas.

Back to Top

Team Metrics

Iteration Metrics

The end of each iteration is the time for each Agile Team to collect the iteration metrics they’ve agreed upon. This occurs in the quantitative part of the team retrospective. One team’s metrics set is illustrated in Figure 21.

Figure 20. One team’s Iteration Metrics
Figure 21. One team’s chart of Iteration metrics

Back to Top

Team Kanban Board

A Team Kanban process evolution is iterative. The team’s bottlenecks should surface after defining the initial process steps (ex., define, analyze, review, build, integrate, test) and Work in Process (WIP) limits, and after executing for a while. If not, the team refines the states or further reduces the WIP until it becomes obvious which state is ‘starving’ or is too full, helping the team adjust toward a more optimum flow.

Back to Top

Team PI Performance Report

During the PI System Demo, the Business Owners, customers, Agile teams, and other key stakeholders rate the actual business value (BV) achieved for each team’s PI Objectives as shown in Figure 22.

 

Figure 21. Team PI Performance Report
Figure 22. Team PI performance report

Reliable trains should generally operate in the 80 – 100 percent range; this allows the business and its outside stakeholders to plan effectively. Below are some important notes about how the report works:

  • The Planned total BV does not include stretch objectives to help the reliability of the train
  • The actual total BV does include stretch objectives
  • The achievement percentage is calculated by dividing the actual BV total by the planned BV total (actual BV ÷ planned BV)
  • A team can achieve greater than 100 percent (as a result of stretch objectives achieved)
  • The effort required for stretch objectives is included in the iteration plan’s load (that is, it is not extra work the team does on weekends)

Individual team totals are rolled up into the program predictability measure (see Figure 10).

Back to Top

SAFe Team Self-Assessment

Agile teams continuously assess and improve their methodology. This is often done through a structured, periodic self-assessment process. This gives the team time to reflect on and discuss the key practices that help yield results. One such tool is a simple SAFe team practices assessment, as appears in this spreadsheet: When the team completes the spreadsheet, it will automatically produce a radar chart like the one shown in Figure 23, which highlights relative strengths and weaknesses.

Figure 22. Team Self-Assessment radar chart
Figure 23. SAFe Team self-assessment radar chart

Download Agile Team self-assessment

Back to Top


Learn More

[1] Leffingwell, Dean. Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise. Addison-Wesley, 2011.

[2] Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises. Addison-Wesley, 2007.

[3] https://www.youtube.com/watch?v=VOjPpeBh40s

Last update: 19 October, 2017