On The Nature Of Portfolios - Epics - Feedback Loop

On The Nature Of Portfolios - Epic Feedback Loop

“Always show your working out!” was the mantra of my maths teacher in senior school. This series of blog posts “On the Nature of Lean Portfolios” is an exploration of Lean Portfolios. It is the thought processes running through my mind, exploring the possibilities so that I understand why things are happening rather than just doing those things blindly. It is not intended to be a fait-accompli presentation of the solutions within Lean Portfolios but an exploration of the Problems to understand whether the solutions make sense. There are no guarantees that these discussions are correct, but I am hopeful that the journey of exploration itself will prove educational as things are learnt on the way.

Epic Feedback Loop

Previous postings have explored how SAFe’s recommended prioritisation mechanic, Weighted Shortest Job First, can be challenging at the Portfolio level because the Shortest part of the algorithm is dominant and the smaller, shorter items tend to get prioritised first. The previous post explored slicing an Epic into smaller Epics, but found that there are scenarios where the overall value could be lost when sliced; so examined different ways of keeping track of the overall value.

The other end of the spectrum is to retain a single large Epic that maintains a focus on the overall value, but this introduces problems of it’s own; how to track progress. The Epic Feedback loop is key.

Epic Feedback Loop

  • The Epic negotiates Features into the backlogs of Agile Release Trains.
  • The Agile Release Trains release a version of their Solution that contains the Features.
  • The Solution is used by the Customer
  • The Leading Indicators or Business Outcomes change due to the Customer’s use of the Solution
  • The Leading Indicators and Business Outcomes inform the Epic’s strategy and influence the next set of Features going into backlogs

This cycle repeats until the metrics show that either the Epic has achieved success or that it’s not working and should be cancelled. An Epic should be producing information every Program Increment to justify its continued existence.

A common mistake with Portfolio’s is to treat the Portfolio as Work Management; getting teams to do work, checking that they are working, but checking that work has been done doesn’t answer the question “is the work valuable?”

Lean Portfolio Management should be about Risk Mitigation, mitigating the risk that this idea, this Epic, will waste the investment that has been made in the Development Value Streams and the Trains and Teams within. Regular feedback, every Program Increment, can be used to demonstrate that the investment that has been consumed has generated value.

Leading Indicators are early Risk Mitigation measures to demonstrate that the overall premise of the idea is valid and that the Business Outcomes should follow. The Business Outcomes are measurable metrics from the business that demonstrate that value is being achieved. If either of these aren’t tracking in an expected fashion then that is an indicator that either corrective action needs to be taken or that work on the idea should be stopped; they form the inputs to help run the Lean Start-up cycle.

Cancellation & Approval

The phrasing is deliberate; the process is “Cancellation and Approvals” and it’s always in that order.

Why such pedantry on the wording?

The Portfolio Kanban is a flow based system that utilises pull mechanics. Items should only be pulled forward through the Kanban if there is space. To create that space items need to either be completed or cancelled. Therefore Cancellation is deliberately the first step in the process; create space before approving more work to fill that space. There is no point approving new work if the capacity to deliver it isn’t there.

How do we know what to cancel?

This is where the metrics of the Epic come into play; The Business Outcomes and the Leading Indicators. Every PI the Epic should be providing an update to those metrics to convince the Portfolio that the Epic is making progress and it should be allowed to continue consuming capacity from the Trains. Leading Indicators become vitally important because Business Outcomes are lagging indicators and some proof is needed, even in the first PI after the Epic has been approved, that the Epic is making progress.

Good Leading Indicators are honouring the Lean Start-up mentality, or incremental delivery.

  • Lean Start-up mentality is running experiments to provide proof that we should continue.
  • Incremental delivery can demonstrate progress towards a bigger goal, and is measured on released or releasable functionality, i.e. Value delivery, rather than work being done.

Psychology of cancelling things

This isn’t intended to be a detailed treatise on psychology, I’m not qualified, merely a couple of observations to bear in mind .

  • Big things are harder to cancel. As much as everyone talks about “Ignore Sunk Costs” it’s very hard to ignore them!

    Big things also pose a challenge because the Leading Indicators themselves might lag; it takes time for a big thing to gain momentum and for the results to come through. The challenge is in forming good Leading Indicators, the experiments, the early steps, that show that progress is being made from Day 1.

  • Small things are easier to cancel. The sunk costs haven’t built up, it is easier to ignore them.

    However, too small and there might not be enough time to make decisions. Epics that are only a single PI in length don’t give the opportunity to decide to cancel; they are completed before the next decision point. Therefore anything that’s approved always consumes the effort regardless of whether it was valuable or not.

Epics need to be sizeable enough so that they have the opportunity to have decisions made around them; otherwise the opportunity to decide if this is valuable occurs only at the point where the Epic is approved to consume investment. The challenge, as we’ve seen in earlier posts, becomes that larger Epics struggle in the WSJF, the smaller ones will win, which can become a race to the bottom, everyone is slicing Epics up into smaller Epics to get them prioritised, but get too small and the feedback loop is lost, construction of the Lean Business Cases becomes the dominant factor.

Conversely cancellations could occur more regularly than once per-PI but this has it’s own problems. If an Epic is cancelled mid-PI, what happens to the planned work from the Epic? The PI Plans for the trains will need to be revised1, which will cause disruption to the teams and challenges as work from the higher backlogs needs to be pulled forwards to fill the space created but this is occurring outside of PI planning events where teams are present to negotiate the collaborations between them that are required to get the value delivered.

Psychology of things being cancelled

This is the other side of the cancellation; not the person making the cancellation decision, but the people receiving the cancellation decision. The Epic Owner whose Epic has just been cancelled, the Teams who have been doing work or had planned to do work for the Epic that has just been cancelled. Treated wrongly the cancellation can look, to the receivers of the cancellation, like all their hard work and effort was for nothing.

Explain the business decision behind the cancellation. The aim of Portfolio Management is to minimise the losses, minimise the investment that has been consumed that hasn’t resulted in a gain in value. Cancelling something, particularly cancelling it early, should be seen as a success because it allows the business to redirect the investment towards a piece of work that will result in more value.

Think Scientific Method. In the Scientific Method a No answer, a failure of the experiment, is as important as a Yes answer, a success of the experiment. It tells you what doesn’t work and that information can inform the decisions of what to do next. Turn the apparent failure into a success and explain how it is going to move things forward. Thank the teams for their efforts to generate the information that has allowed the business to make a sensible decision.

What To Look For In The Metrics

The metrics that an Epic is reporting provide insight that will influence decisions on Cancellation or Declaration of Success2. The graphs below illustrate patterns in the business outcomes and how they can be interpreted. Key:

  • Green – Target business outcomes, with target date indicated by the diamond
  • Red – Actual business outcome measured each timebox
  • Orange – Predicted business outcome based upon change over the last measured timebox
  • Pink – Target leading indicator, with target date indicated by the diamond
  • Purple – Actual leading indicated measured each timebox

  • Vertical axis is value
  • Horizontal axis is time, each division representing a Program Increment

For the purposes of illustration only one or two metrics are being shown per-graph, real life Epics will be tracking several Leading Indicators and Business Outcomes.

All targets need both a target value and a target date.

Epic Metric Scenarios 1 - 4

Scenario 1 – Default

Scenario 1 represents the default or standard development scenario. The rise in Business Outcomes starts off slowly, gathers pace as people start adopting, then tails off; a classic S curve.

In this particular scenario the Business Outcome has flat-lined at 90% of the target value and has a single PI to close the gap before the target date.

If the Business Outcome was something more aspirational, say a sales target, if it’s not met exactly is that a problem? The sales target will have been an estimate, what was the error margin on that estimate? If it’s going to cost more to close the gap than the value that will be gained in return; is it worth doing? Perfection might not always be achievable nor economical sensible.

If the Business Outcome was a compliance issue, such as the GDPR example from the previous post, then nothing short of 100% is acceptable and the Epic Owner will need to determine why has progress stalled and what needs to be done to close the gap.

From the above two descriptions it’s obvious that context is king, for this Scenario it’s a sales target, the value described by the Business Outcome was an aspirational target and that the Epic has got within 90% of the target is good enough.


Scenario 2 – Building Success

Scenario 2 represents an earlier point in the standard development of Scenario 1. The Epic is a few Program Increments in and the Business Value is starting to rise. A simple linear interpolation (shown in Orange) based upon the last timebox measured. The interpolation shows that if it continues on the current trend then it will deliver the target value ahead of the target date. In reality the curve will probably follow an S shape like Scenario 1; it will slow down.


Scenario 3 – Too Slow

In this scenario the Epic has been running for 5 PIs and the Business Outcomes are building, but slowly. A simple linear interpolation based off of the last timebox shows that it will not reach it’s target outcomes before the target date.

The sharp rise of the S-curve hasn’t happened and the expectation is that it should be visible in the data after this amount of time. Should this Epic be continued? If the engineering work has been done and this is the financials building up, then maybe it makes sense to let it continue. I would be doing calculations around how much it’s going to cost in terms of support and operations, and now that they are needed for a longer period is the cost-benefit ratio still in favour of the benefits?

If this was a piece of compliance work that must be done before the target date then this is a signal that some form of action needs to be taken. Why isn’t progress being made more rapidly? Is it not a high enough priority? Are there too many other Epics in-play that are diluting the effort of the organisation? Is the work just harder than initially thought?


Scenario 4 – Too Soon to Tell

In this scenario the Epic has been running for 3 PIs and the Business Outcomes are building. A simple linear interpolation based off of the last timebox shows that it will not reach it’s target outcomes before the target date.

The sharp rise of the S-curve hasn’t happened but it might be too early in the life of the Epic to be able to see the rise starting. Should this Epic be continued?

I would give it the benefit of the doubt, but warn the Epic Owner that if the Epic isn’t showing signs of hitting the steep rise of the S-curve then it might be stopped as it is moving into the territory of Scenario 3.


Epic Metric Scenarios 5 - 7

Scenario 5 – Perfection

In this scenario the Epic has been running for 3 PIs and the Business Outcomes show that it is exactly on track. It will deliver it’s target outcomes at exactly the target date.

Any alarm bells ringing?

Nature doesn’t do straight lines, and engineering and development, problem solving, is a naturalistic system that moves in fits and starts. If it looks too good to be true, then it probably is. Are they really measuring an output a result of the changes that have been made or are they measurement something that has ended up being a proxy for time?

The Epic is tracking the wrong metrics, or people are calculating the metrics based upon interpolation rather than real-world measurements.


Scenario 6 – Nothing

In this scenario the Epic has been running for 4 PIS and the Business Outcomes haven’t moved; they remain where they were on the day the Epic started.

A real world example of this scenario would be a system replacement endeavour. The Business Outcomes are never going to move until the new system replaces the old system and that will only occur once all the replacement work has been done. The Business Outcomes will only move once all the investment has been made.

Taken at face value the correct course of action should be to cancel the Epic; it’s not generating value. However, the business case is still valid, it makes sense to replace the system therefore the work should be done. The real challenge is the lack of early feedback, there is no way to know whether the work that is consuming the investment in the teams is generating value.


Scenario 7 – It’s All Happening Behind The Scenes

In this scenario the Epic has been running for 4 PIS and the Business Outcomes haven’t moved; they remain where they were on the day the Epic started. This would be a copy of Scenario 6, but this Epic has Leading Indicators showing that progress in being made in the background.

To hold true to the Agile principles4 those Leading Indicators should be demonstrating actual working software, functionality replicated and validated on the new system. If the Leading Indicator is a measure of “work being done” then the Epic is in Scenario 5 and the numbers are just a proxy for time passing.

The Leading Indicators should be giving the Portfolio Management confidence that progress is being made and therefore confidence to continue the Epic.

Leading Indicators should have their own targets outcomes and target dates, shown on the graph as the dark pink line. This provides direction to the trains and teams below, but also if the Epic isn’t meeting those targets it has to ask why and do something about it as describe in Scenarios 3 & 4.


Next Steps

This post has looked at the Epic feedback loop and how the metrics around an Epic drive decision making. In the next few post I want to explore whether Epic Flow really matters?

#1 Agile Principle #2 : Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

#2 Interpreting the metrics was also discussed as part of the presentation I did on “The Dynamics Of Decision Making within the Portfolio Kanban” for the SAFe 2020 World Summit, video is available here.

#3 An alternative approach would be to utilise the Strangler Fig Pattern proposed by Martin Fowler which would make the Business Outcomes immediately visible compared to a cut-over approach.

#4 Agile Principle #7 : Working software is the primary measure of progress.

Contact Us