Project Concerns vs. Customer Concerns

The traditional “iron triangle” of project management often crowds out our thinking about what we should really be focusing on in product delivery.

Although things like scope (count of functionalities, features), cost and time seem easiest to measure — and therefore become the things we measure and elevate — they really have little to do with product success. Rather, customer concerns like return on investment, fitness for purpose and sentiment are more important things to measure if we are interested in building great products — and they’re actually not as difficult to measure as we might have previously thought (if you’re not convinced, see Douglas Hubbard’s How to Measure Anything).

Image credit: Yoan Thirion

The Eight Stances of a Transformational Leader

Inspired by Barry Overeem’s 8 Stances of a Scrum Master, I have been talking about (and will again at the upcoming Agile Kanban Istanbul and UnlimitedAgility conferences) the eight stances of a transformational leader. I’ll be publishing my own white paper with a fuller explanation, but for now here’s a snapshot.

First, I use the term stance in the sense of “a mental or emotional position adopted with respect to something.” So it’s not a title or a role, but a way of being in a particular context. By transformational leader, I mean simply anyone working in a VUCA environment who, as Amy Edmondson says, plays a role in creating and nurturing the culture we all need to do our best work.

  • Organizational Refactorer: Like refactoring in programming, refactoring is a technique for restructuring an existing work environment by altering its internal structure without changing its external behavior. It’s the act of making small-J-curve, kaizen changes to reduce friction (caused by accidental complexity) and make it easier for people to do their jobs. As the goal of refactoring code is clean code, the goal is a “clean” organization.
  • Strategy Deployer: This is the act of setting direction and providing clarity of mission to foster organizational improvement in which solutions emerge from the people closest to the problem. The goal is aligned autonomy and a leader-leader culture.
  • Anzeneer: This portmanteau coined by Josh Kerievsky means “safety engineering.” That is, we need to protect people by establishing psychological and physical safety in everything from relationships to workspaces, codebases to processes, products to services.
  • Coach: Coaching is teaching others to be leaders and building an organization that can sustain its success. As Jeffrey Liker says, leaders are responsible for creating an environment in which future leaders can blossom.
  • Environmentalist: An environmentalist holistically (re)creates and stewards the environment in which people grow, with particular awareness of context or organizational “terroir.” Just like a winemaker will fail if he or she tries to grow a certain grape varietal in a place that is very different from another, so too will an organizational leader fail if he or she attempts cookie-cutter solutions or to install frameworks or initiatives irrespective of culture or context.
  • Experience Designer: Similar to the experience design of products, this is means consciously creating meaningful interactions centered on the employee, with particular focus on intrinsic motivation (mastery, autonomy, purpose); “alleviating people’s problems and bringing them joy.”
  • Experiment Curator: An experiment curator models and celebrates behaviors and creates the environment for employees to learn and share learning through experimentation. It’s basically creating a place for others to learn in.
  • Flow Manager: Taken from the Kanban Method, flow management means to optimize the end-to-end flow of value in a system. Leaders need to make visible and look after wait flow as much as work flow, actively reducing dependencies and managing the system for smooth, fast flow, rather than utilization.
StanceIn three words
1. Organizational RefactorerReduce accidental complexity
2. Strategy DeployerLeader-Leader culture
3. AnzeneerMake safety prerequisite
4. CoachEnable over do
5. EnvironmentalistPassion for “terroir”
6. Experience DesignerDesign for engagement
7. Experiment CuratorFoster learning culture
8. Flow ManagerOptimize the whole

Project-to-Product Principles

I’ve increasingly been helping organizations looking to go from “projects to product,” so I’ve curated/co-created/stolen a few principles for anyone who wants a tl;dr version of how to go about it. I’m indebted to the work of Jez Humble, Marty Cagan, Matt Lane and John Cutler.

In moving toward product-orientation, we prefer:

  • Outcomes over outputs
  • Solving problems over building solutions
  • Options over requirements
  • Experiments over backlogs
  • Hypotheses over features
  • Customer-validated learning over PO assumptions
  • Measure value over measure cost
  • Flow over utilization
  • Optionality over linearity
  • Product vision, strategy, personas and principles over product roadmaps
  • Small-batch delivery over big-batch delivery
  • Optimizing for assumptions being wrong over optimizing for assumptions being right
  • Engineers solving problems over POs dictating requirements
  • Teams of missionaries over teams of mercenaries
  • Business-driven over IT- or PMO-driven

Sticking to the Plan is Not the Solution

[Note: To commemorate the agile manifesto‘s 20th anniversary, this is the first of 12 posts in no particular order on the manifesto’s principles.]

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

Manifesto for Agile Software Development

Changes to the plan mid-sprint. Constantly-changing priorities. Those words strike fear into the hearts of many team members. One popular agile-health assessment even downgrades as immature those teams whose “plan changes frequently.”

The agile manifesto addresses the idea of change, but it’s not to fear it or to view it as a sign of immaturity — rather the opposite: We are to welcome change and value responding to it over following a plan. So what’s going on?

I think this is a case of confusing the symptom with the problem.

It’s true that constantly changing priorities are often a smell of disorganized product management or unclear strategy. But those dysfunctions need not create additional ones. And it’s quite possible that those changing requirements actually, you know, create competitive advantage. Far too many teams and organizations blame “changing priorities” for their own inability to deliver quality product. I often see diagnoses of teams having difficulty delivering with prescriptions like “Stick to the sprint plan.” Changing requirements is not the problem, so sticking to the plan is not the solution.

The problem is not that the plan changes but that the team or organization lacks the capability to work in such a way as to accommodate change easily.

If changes to the  plan result in any of the following things, you’re doing it wrong — and sticking to the plan isn’t going to help you improve:

  • Taking technical shortcuts
  • Working overtime
  • Getting stressed out
  • More defects (leading to the anti-pattern of defect sprints)

I understand why teams and consultants instinctively reach for the “stable plan” panacea: Teams commit to a slate of work, whether it’s a sprint, iteration or program increment, which they’ve often spent time estimating. Stakeholders (including well-meaning scrum masters) exert some amount of pressure on them to deliver what they’ve committed to, heedless of the nature of knowledge work (i.e., complex). Even when external pressure is low, the very design of a sprint or PI plan implies that success correlates to following the plan; I have encountered many teams who suffer low morale simply because they delivered something different from what they planned, whether it was because they weren’t able to estimate complex work or because a stakeholder changed midway through the plan.

And of course, teams are often held accountable for others’ — product management’s or HiPPOs’ — decisions. Sometimes, it’s because stakeholders behave badly: They are aware that their demand creates stress and pain on the team and refuse to acknowledge and respect a team’s finite capacity — they pile on tasks but refuse to remove any. But often, stakeholders do it out of benign ignorance, lacking a feedback mechanism to inform their choices (or, as Don Reinertsen calls it, “an economic framework”) and assuming that the team will figure out how to handle it. Therefore, it is incumbent upon the team to work in ways so as to accommodate change more comfortably.

However, the solution often put forward is to “stick to the sprint plan.” This flies in the face of the manifesto value to prefer Responding to Change to Following a Plan. Clearly, when a stakeholder makes a mid-plan reprioritization decision, it’s because of a real or perceived business need (a.k.a. “customer’s competitive advantage”). The question of whether it actually does provide a competitive advantage is secondary here. The team has an obligation to “welcome this changing requirement” whenever it happens. In what universe it is a good business decision to follow through on building a feature that we determine customers don’t want simply because we need to follow a plan?

People don’t really have a problem with changing requirements; we have a problem with being prevented from doing quality work. So the problem isn’t that we don’t want new information but rather how we accommodate it.

What makes it difficult to do that? Many things contribute, but I’ve seen two in particular:

  • Too much work in progress
  • Not enough investment in technical practices

The good news is that both of these things are typically within the team’s control.

Too much work in progress

Regardless of whether you work in a timebox (sprint, iteration, PI), your team — as well as you as an individual — should have control over how many things you work on at once. Just because you’ve “committed” to a plan of delivering 20 work items doesn’t mean you need to start all of them at once. That means stopping the inane practice of having individual team members sign up for work items (or worse, having someone assign them) before they start. Limit the number of in-progress work items to something less than the total number of people in the team. (Reinertsen again: “We will always conclude that operating a product development process near full utilization is an economic disaster.”)

This pattern replicates itself at the enterprise level. Organizations are unable to respond to big or urgent demands without significant disruption because they are saturated with “projects.” Most executives that I work with don’t even know how much organizational WIP they have, so for them, making it visible is the first step. But the reality is that when plans become promises and success is measured by conformance to them, we effectively lock in the largest possible batch size, which only exacerbates the problem of dealing with change (and of course is one of the worst things an organization can do).

Not enough investment in technical practices

Talking about XP practices seems so … 2000s. And yet I think that our industry has gotten worse, if anything, with respect to building quality in through things like TDD, pairing, loose coupling, emergent design and continuous integration. Here’s the thing about engineering practices and those stakeholders who are making those changing requirements: They probably don’t have a clue about the practices, and that’s okay. Once you pick up a story, take the time to do it right. The team’s definition of done should include things like proper testing, integration (maybe even deployment!). If it means that the story takes more than two weeks, that’s okay, too. Teams that don’t do these things are reasonably afraid to change the plan because they can’t safely undo (see long-lived branches) or share work (see “not my code”). But here’s the reality: If you avoid doing these things, delivery will never get easier. And, of course, the investment in technical practices will only happen if you limit work in progress.

Other options for better accommodating change:

  • Stop planning in artificial time boxes (whether at the sprint or quarterly level) and instead plan just-in-time using pull (e.g., replenish the Next column based on capacity signals).
  • Stop punishing teams and individuals for not completing their sprint or release “commitments,” including the tracking of the ridiculous sprint-completion rate and other “success” metrics based on conformance to a plan.
  • Spend less time estimating work, as it represents a sunk cost that inhibits change accommodation (and likely has very little information value anyway).
  • Show the cost of interruption. If we’re concerned about undisciplined product owners making careless decisions, make the cost of the decision visible. We can do this with simple flow metrics and post-commit discard rates. If we’re tracking delivery times, we can easily see the “flow debt” that we incur by pausing or dropping work to tend to expedite requests. (Typically this is apparent on a delivery-time frequency chart, manifesting as a bimodal distribution.)
  • Use and make highly visible policies that show how you treat work, such as the selection criteria the team uses to pull work items. For instance, some executives, if they knew that their requests were being treated as “drop everything and work on this now,” would think twice about the requests (I’ve met a few).

The bonus is that if we solve the problem rather than the symptom, we’ll actually save time by avoiding daft things like trying to get better at estimating, debating “carryover” stories and coming up with complicated prioritization schemes (just FIFO it). If we’re able to accommodate change gracefully, those remnants of the agile industrial complex go away.

I would even go so far as to aver that being able to accommodate change mid-sprint is actually a fantastic “agile health” metric. After all, if you can’t do that, you won’t be able to do anything very well, whether it’s scaling, product pivoting or increasing throughput, quality or speed. Don’t hide the problem by requiring the team to “stick to the plan.” Fix the underlying issues so that the team can support the organization in what it’s trying to do, which is to exploit variability for economic benefit. Isn’t that the whole idea of business agility?

Flight Levels and Aligned Autonomy

Flight Levels is a thinking model for organizational improvement. As Klaus Leopold says, it “helps you find out where in an organization you have to do what in order to achieve the results that you want.” Flight Levels is effective at that because it stresses the idea of leverage and coherence across the multiple strata and teams of an organization.

In doing so, it provides a way to model and marry strategy development and strategy deployment, effectively fostering a way for leadership at every level to take root and flourish. I’ve spoken about this connection previously, but, until now, I hadn’t fully connected the dots between how Flight Levels can be combined with the “aligned autonomy” matrix to provide a useful way to visualize where an organization is and how Flight Levels can help create a path toward aligned autonomy.

Aligned autonomy is the idea that, rather than the conventional view that alignment and autonomy are opposites, they are actually separate concerns. The insight is credited to the 19th-century Prussian military leader Helmuth Von Moltke and popularized in modern times by Stephen Bungay, who wrote in The Art of Action that

… there is no choice to make (between alignment and autonomy). Far from it, [Von Moltke] demands high autonomy and high alignment at one and the same time. He breaks the compromise. He realizes quite simply that the more alignment you have, the more autonomy you can grant. The one enables the other. Instead of seeing of them as the end-points of a single line, he thinks about them as defining two dimensions.

More recently, Henrik Kniberg, in his inimitably accessible style, expanded on the concept by describing the organizational culture types of each quadrant:

In the organizations with which I have worked, the elements of the Flight Levels tend to manifest themselves in particular ways that align to these quadrants:

Let’s take each one in turn.

Authoritarian-Conformist

This quadrant is where highest-level leaders make little to no distinction between “what and why” and “how.” This is the realm of top-down decision-making, going too far into detail about implementation and often handcuffing the people doing the work from making better decisions about execution because of a misguided centralized authority. Meanwhile, because of this obsession with the “how,” these leaders are often derelict in their duty to work “up a level” in strategy, where their contributions are most needed. As a result, these are leader-follower organizations, in which people are trained not to take action and have to ask permission for any decision of importance.

We might depict their Flight Levels as overlapping or compressed at one level; that is, people who should be developing and evolving strategy are too concerned with the day-to-day operations of teams. To use the flight metaphor, these are people who who be flying at the airplane level but can’t remove themselves from the butterfly-level details. Or, as Bungay explains: “Far from overcoming it, a mass of instructions actually creates more friction in the form of noise, and confuses subordinates because the situation may demand one thing and the instructions say another… trying to get results by directly taking charge of things at lower levels in the organizational hierarchy is dysfunctional.”

Von Moltke saw the same behavior in the military:

In any case, a leader who believes that he can make a positive difference through continual personal interventions is usually deluding himself. He thereby takes over things other people are supposed to be doing, effectively dispensing with their efforts, and multiplies his own tasks to such an extent that he can no longer carry them all out.

The demands made on a senior commander are severe enough as it is. It is far more important that the person at the top retains a clear picture of the overall situation than whether some particular thing is done this way or that.

These organizations find it difficult to scale effectively, because their leadership’s inattention to strategy and intrusive concern with implementation details creates a passive leader-follower culture.

The challenge for these organizations then is to use Flight Levels to encourage higher-level leaders to begin to distinguish between the “what and why” and “how,” and focus on setting “directionally correct” strategy while trusting teams and Level 2 Coordination to execute.

Micromanaging-Indifferent

In this quadrant, the concerns of operations, coordination and strategy are variously overlapping, disconnected and/or non-existent. Here we observe:

  • Rampant and invisible WIP
  • Low employee engagement
  • No clear org vision/strategy
  • Siloed, undiscoverable tools
  • Tribal, network-based knowledge
  • Busy but unproductive people
  • Redundant, unshared work

Work in these organizations is perhaps best described by Barry O’Reilly when he says that “When people lack clarity they will optimize for what is in their control, output that is attainable to them but not necessarily the outcomes you want to produce.” To the extent that any measurements exist, activity-based metrics reign here.

The challenge for these organizations is perhaps to simply acknowledge the possible existence of Flight Levels and their relationship to each other. The simple but daunting task of making work visible is a necessary first step.

Entrepreneurial-Chaotic

This is the realm of disconnected teams. They have broad autonomy but little awareness of their relationship to strategy and often of their relationship to each other and the wider end-to-end value stream. In some cases, they do have their own Level 3 Strategy, but they are not unified to a common organizational strategy; they function more as warring fiefdoms under a single name. Sometimes, this organizational culture is the outcome of growth, as we might see in the progress of a startup to a scale-up, in which leadership hasn’t commensurately matured with the new needs of the organization. But it can also occur in the context of a bloated Authoritarian-Conformist organization, whose strictures are too unwieldy to control and where leaders with some authority attempt to break free, making their own plans because it’s the only way they can get work done (e.g,. grey market of tools). In both cases, the work is disconnected from strategy. The organization lacks an ability to see itself from the 30,000-foot view.

People in these organizations generally make lots of decisions on their own, until the decision is somehow related to understanding strategy. Since leadership either keeps strategy closely held or, as is more often the case, doesn’t really have a strategy, this can cause tension, frustration and disengagement, as connection with higher-level purpose is missing. This often extends into career development, as well.

The challenge for organizations in this quadrant is to instantiate Level 2 Coordination and Level 3 Strategy. Starting points can be to identify desired organizational outcomes (Level 3), shift attention to end-to-end metrics (Level 2), make work visible and to use yokoten (lateral deployment) to create awareness.

Innovative-Collaborative

This is the Leader-Leader ideal, which is fostered by clear delineation of concerns at the operational, coordination and strategy levels of Flight Levels. The lightweight but comprehensive modeling of these concerns in Flight Levels provides enough separation of “what and why” from the “how” for people to act autonomously but aligned toward the organization’s desired outcomes:

  • Intent is expressed in Level 3 Strategy in terms of what to achieve and why.
  • Autonomy in Level 1 Operational gives freedom of the actions taken in order to realize the intent; in other words, about what to do and how.

The “just-enough” strategy ensures that empowered teams and individuals are working on the right things, not merely working on things the right way. As one neutral observer of Von Moltke’s manifestation of the Innovative-Collaboration organization noted:

Every German subordinate commander felt himself to be part of a unified whole; in taking action, each one of them therefore had the interests of the whole at the forefront of his mind; none hesitated in deciding what to do, not a man waited to be told or even reminded.

— Art of Action

Afterward

It’s important to note that many organizations aren’t monolithically characterizable in one single quadrant, nor do they always manifest in one quadrant over time. That is, certain groups in an organization may be Entrepreneurial-Chaotic, while others are Authoritarian-Conformist. Or while they may generally be Authoritarian-Conformist, they have moments when they exhibit Entrepreneurial-Chaotic.

As a result, it’s helpful to pay attention to those specific behaviors and move in the general direction toward aligned autonomy using Flight Levels, realizing the the organization may change in fits and starts.

How to Forecast Before You Even Start

One question that people who are friendly to the probabilistic-forecasting mindset often ask is “I understand how to forecast with delivery data once the project is underway, but how do I forecast before I even start?” Assuming that you absolutely need to know that answer at that point — heed Douglas Hubbard’s advice* — a simple probabilistic way to do it is through reference-class forecasting. Conceptually.org has as good a definition of it as anyone:

Reference class forecasting is a method of predicting the future by looking at similar past situations and their outcomes. Kahneman and Tversky found that human judgment is generally optimistic due to overconfidence and biases. People tend to underestimate the costs, completion times, and risks of their actions, whereas they tend to overestimate the benefits. Such errors are caused by actors taking an “inside view”, assessing the information readily available to them, and neglecting other considerations. Taking the “outside view” — looking at events similar to ours, that have occurred in the past — helps us reduce bias and make more accurate predictions.

An easy metaphor for reference-class forecasting is home sales. We’re trying to forecast something that’s complex — lots of market dynamics involved in something that’s essentially never been done before (how much someone will pay for this particular house at this particular point in time). We use a variety of economic and housing data — zip code, square footage, construction, features — to create a reference class of “comparables.” (If you really want to geek out, see Zillow’s Forecast Methodology.)

Most organizations have delivered some number of projects — maybe not this exact project in this exact tech stack with this exact team, but with attributes that are comparable to it.

An Example

Here’s an example. A company was considering a new initiative. They needed to know approximately how long it would take (time being a proxy for cost but also market opportunity cost). They took the traditional inside-out approach — attempting to predict how long something will take by adding up all known constituent tasks — and estimated it at about a year. This inside-out approach being subject to the Planning Fallacy, we decided to also try a reference-class forecast.

  1. We took a list of the 50 most recent projects, going back a few years. We needed only a pair of dates for each one: When the business officially committed to the project (confirming this commitment during the “fuzzy front end” is often the most tricky bit) and when it went to production.
  2. We then categorized each project by meaningful traits, like project type (legacy or greenfield), team size (small, medium, large) and dependencies (many or few).
  3. We viewed the data on a scatterplot chart.

Unfiltered Forecast

You’ll always have a tension between needing enough data (small sample sizes can be distortive) and relevant data. The good thing about reference-class forecasting is that it’s inexpensive (and better) to run multiple views. First, we ran an unfiltered forecast — all 50 projects.

Unfiltered reference-class forecast

This yielded a high-level view onto how long projects take overall. Half of the time (50th percentile), projects finish in 383 days, or a bit more than a year. But that leaves a lot of projects — the other half! — that take longer. How much longer depends on the level of confidence we seek:

  • 50% of the time: in 383 days (a little more than a year)
  • 70% of the time: in 509 days (1.4 years)
  • 85% of the time: In 698 days (nearly 2 years)

Filtered Forecasts

Of course, this new project will be different (as always!), so not all of those projects are really relevant. So we filter based on characteristics similar to the project we’re forecasting: It’s a legacy project, with a small team and many dependencies. We had such 12 projects in that reference class. Its confidence intervals are indeed different from those of the entire set:

a filtered reference-class forecast
  • 50%ile: 607 days (1.7 years)
  • 70%ile: 698 days (1.9 years)
  • 85%ile: 776 days (a little more than 2 years)

Those numbers were larger than the whole set, so that was disappointing. Maybe we can look at the problem a bit differently. What about legacy projects with many dependencies that were staffed by medium-sized (rather than small) teams. (Perhaps what Troy Magennis said about reducing the effect of dependencies with slightly larger teams was right!) We had 11 such projects:

Wow! That is quite a different story. I guess Troy was right!

  • 50%ile: 303 (less than a year)
  • 70%ile: 400 (a little more than a year)
  • 85%ile: 509 (1.4 years)

We now have three different reference-class forecasts to use. They at least give us some options to inform our thinking (especially as regards team-sizing decisions). Knowing which reference class to use is more art than science, so I like to consider a few options rather than locking into one (especially the one that paints the rosiest picture!).

Once we do get started with the project, we will of course want to do probabilistic forecasting with actual “indigenous” delivery data. But before we even start, we have informed ourselves from the outside-in — averting the Planning Fallacy — on when we might expect this particular new initiative to be done.

* Hubbard says essentially “Of course you need to estimate development costs when making a decision about IT investments. However, you don’t usually need to reduce the uncertainty about those costs to make an informed decision. Reducing the uncertainty about the utilization rate and the likelihood of cancellation of a new system is much more important when deciding how to spend your money.”

Strangler Pattern… for Estimating?

Martin Fowler long ago popularized the metaphorical “Strangler Pattern” (since updated to “Strangler Fig Pattern”) as a more graceful and less risky way to rewrite an existing system. He wrote of the Australian strangler fig plant:

They seed in the upper branches of a tree and gradually work their way down the tree until they root in the soil. Over many years they grow into fantastic and beautiful shapes, meanwhile strangling and killing the tree that was their host. This metaphor struck me as a way of describing a way of doing a rewrite of an important system… An alternative route [to all-or-nothing big-batch replacement] is to gradually create a new system around the edges of the old, letting it grow slowly over several years until the old system is strangled.

When introducing organizations to probabilistic forecasting — which I simply describe as answering the question “When will it be done?” with less effort and more accuracy — the move from traditional estimating can often seem like a similar problem to that of Fowler’s legacy application swap: It’s a big change, fraught with risk, affects a lot of people, and we’re not entirely comfortable with or sure about how it works.

For these reasons, and because most sustainable change is best effected via gradual, safe steps, I guide teams to apply what is essentially the strangler pattern: Keep what you have in place, and simply add some lightweight apparatus around the edges. That is, continue to do your legacy estimating process — estimation meetings, story points, fibonacci numbers, SWAG and multiply by Pi, whatever — and alongside that, let’s start to track a few data points, like commit and delivery dates.

Kanban Method encourages us to “start with what you do now,” and one of the benefits of this approach (besides being humane and not causing unnecessary emotional resistance) is that it helps us understand current processes. It’s quite possible that a team’s current estimating practice “works” — that is, it yields the results that they’re seeking from it. If that goal is to provide a reliable sense of when something will be done, doing the simple correlation of upfront estimate to actual elapsed delivery time will answer that question (spoiler: Most teams see little to no correlation). That in itself can help people see whether they need to change: It’s the Awareness step of the ADKAR model. Continuing existing practice while observing it also helps us decouple and filter out the stuff that is valuable, such as conversation that helps us understand the problem and possible solution. NoEstimates, after all, doesn’t mean stopping all of the high-bandwidth communication that happens in better estimating meetings.

Meanwhile, we’re collecting meaningful data — the no-kidding, actual dates that show the reality of our delivery system. These are the facts of our system, as opposed to how we humans feel about our work, and as one observer famously noted, “Facts don’t care about your feelings.” But facts and feelings can “peacefully” live alongside each other for a time, just as the strangler fig and host tree (before the fig kills the host, of course). You can then start running Monte Carlo-generated probabilistic forecasts in the background, which allows you to compare the two approaches. If the probabilistic forecast yields better results, keep using it and gradually increasing its exposure. If for some reason, the legacy practice yields better results, you may choose to “strangle the strangler.” Most groups I work with end up appreciating the “less effort and more accuracy” of probabilistic forecasts, and after a time start asking “Why are we still doing our legacy estimating practice?” At that point, the strangler fig has killed the host, and all that remains is to safely discard the dead husk.

So to summarize the strangler pattern for estimating:

  1. Keep doing your legacy estimating practice.
  2. As you do, track delivery dates (commit point through delivery point).
  3. Run a correlation between the two sets of numbers (upfront estimates and actual delivery times).
  4. Continuously run probabilistic forecasts alongside the legacy estimates.
  5. Check the results and either keep the new approach or revert to the legacy.

As with most knowledge work in a VUCA world, whether it’s coding a new system or introducing new ways of working, reducing batch size — of which the Strangler Pattern is a type — offers more flexibility and reduced risk. If you’re interested in a better way of answering the question “When will it be done?” but need to do so incrementally and safely, the strangler pattern for estimating may be an idea to plant. (Sorry, couldn’t resist.)

Agile-Consulting Radar Update

[In the spirit of the ThoughtWorks Tech Radar, I publish my own personal realtime Agile-Consulting Radar. In addition to expanding the blips on the radar, you can find more detail on each in my editorialized glossary.]

This update has a plethora of new tools — and a few old practices — to help organizations and teams work in our current remote-first — or is it remote-only? — and ongoing VUCA world. And it contains some holds — and even avoids! — for long-held agile sacred cows, like user stories and feature-based road maps, sure to provoke some raised eyebrows.

Ubiquitous Remote Work

For the last five months, organizations have been forced into remote work whether prepared or not, and the foreseeable future will require facility with ways of working that are fit for this purpose. This has predictably led to a burst of new tools into the market, many of which are worth a try, and perhaps unpredictably to a rediscovery of venerable practices, like the Core Protocols, Team Agreements, Personal Kanban, and Pomodoro Technique, that enable remote and asynchronous work. Our remote world also requires that we sense and respond to how our teams and colleagues are doing, so competencies like anzeneering and facilitation, along with metrics like Total Motivation and Engagement, are key.

Rethinking Conventional Agile

From velocity to user stories, backlogs to points-based estimation, no agile cow is too sacred to be slapped with an avoid label. It may be time to refactor your agile work processes with the original intent of the Agile Manifesto in mind. That includes an orientation toward value, which is why traditional feature-based roadmaps are out and outcome-based roadmaps are in. The agile community has perhaps finally come to grips with our environment, namely that we typically are working in complex rather than complicated domains, which is why competencies like Experiment Design and Systems Thinking and Sensemaking are blips.

Business Agility

As Klaus Leopold writes, “Agility of an organization is not about having many teams,” but “agile interactions between the teams.” To that end, the radar includes a few blips to support organizational or business agility. These are practices, like Operations Reviews, Leadership at Every Level and Flight Levels — and tools like X Matrix — that create aligned autonomy and connect action to strategy throughout all level of the organization. Speaking of all levels, the radar also contains a couple of items related to scaling, including guidance for unscaling or descaling.

It’s About Flow

Quite simply: Focus on practices, competencies, metrics and tools that enable flow: Flow management, Throughput/Delivery Time/WIP, iterations as checkpoints (rather than planning boxes) and blocker clustering. Avoid practices and metrics that inhibit flow, such as multitasking, high utilization and unlimited WIP.

user story card

Stop writing stories, start validating working software

Barry O’Reilly exhorts today’s leaders to “break the cycle of behaviors that were effective in the past but are no longer relevant in the current business climate, and now limit or may even stand in the way of your success.” After more than 15 years of writing, “refining,” “grooming,” estimating, documenting and coaching people about user stories, I believe it’s time to unlearn them.

I realize that’s a big statement and likely to be not a little controversial. But in a vast and sad irony,  user stories have become the heavyweight documentation and process that they were meant to replace.  Indeed, as Kent Beck shared, “Somebody came up to me and said, ‘We want to do software development but we just can’t stand all this ceremony and this Agile stuff. We just want to write some programs.’ Tears came into my eyes… How can it be that we’re right back where we were twenty years ago?”

Not unlike points estimation, user stories have taken on a life that I’m sure the manifesto signers never intended, and, as Beck alludes, was precisely what they were hoping to escape. As Martin Fowler has remarked, “a lot of what is being pushed is being pushed in a way that, as I said, goes really against a lot of our precepts.” My problem isn’t with user stories per se; it’s with how we use — push — them in practice. If we are dealing with an Agile Industrial Complex, then user stories are the weapons that convince us that we can’t be safe without them, that we simply need more of, and that themselves become the gauge of success rather than the things they’re meant to help us achieve.

But it’s amazing what happens when you strip away the accrued behaviors and get back to the heart of agile: I’ve been working on a side project building a web app with a friend, and I’ve been able to experience how refreshing and dynamic software creation can be when we free ourselves from the methodological fetters that we inherit and reinforce, of which stories are one (and one that I’ve been guilty of perpetuating).

In a vast and sad irony,  user stories have become the heavyweight documentation and process that they were meant to replace.

Curiously, user stories are nowhere to be found in the agile manifesto or its principles:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Let’s focus on a few of the principles in particular and how I see the current practice of user stories as actually inhibiting agility and transgressing the spirit of the principles.

Working software is the primary measure of progress.

Quick: What metrics does your software delivery team and/or organization track? I’m guessing that nine out of 10 of you will say some kind of output-related metric whose origins are based in user stories: story points, velocity or even count of stories.

In my side-project experience, my teammate (who, by the way, is remote and in another continent) and I never even countenanced the idea that any of those story-based concerns would matter, so much that they don’t exist for us. We care chiefly about building something that people will use and enjoy using.

How might we focus on working software? To start, replace the daily standup — a Mos Eisley spaceport of anti-patterns rooted in stories* — with micro-demos (e.g., every day or every-other day). Then constantly validate that you’re building the right thing by letting people use it.

If you don’t think this is a problem for teams today, consider: How much time (meetings), effort and conversation do you and your team spend in the business of stories as compared to the business of validating whether you’re solving business problems and realizing outcomes?

Simplicity–the art of maximizing the amount of work not done–is essential.

Speaking of time, how much time in dehumanizing meetings,  Jira jockeying and estimation meetings have you endured under the guise of agile? Many organizations and teams have unnecessarily complicated their delivery systems with over-engineered story-tracking tools, incessant refinement meetings and elaborate processes. Try working without stories, and you might find you have a lot fewer meetings — and more time to do value-added work.

Business people and developers must work together daily throughout the project.

User stories, far from being the linking point between business and IT, have instead become another way to pass the buck onto IT without any business responsibility or involvement, creating a collective wedge of miniature contracts. IT needs involvement from the business? We’ll give you a product owner, who will in turn spoon-feed you requirements, er, stories. I think this crossing the business-IT divide is really the hardest part of agile delivery, but one that will be exposed when you remove stories. It will be a forcing function to align business and IT.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

I, like many agile coaches, have for years taught the “Three Cs” of user stories: Card, Confirmation, Conversation. The card is a promise for a conversation! The only problem is that in practice, the card replaces the conversation. When stories take on a life of their own (e.g., in ticket systems), they become an impersonal interface not unlike the tomes of spec docs that we used once upon a time.

Conversation in real life is spontaneous, organic and natural. It flows from trustful relationship with and proximity (even if remote) to other humans. It’s possible to do this in delivery teams, but it’s inhibited when conversations are reduced to scheduled sessions and transactional instructions. And these “conversations” betray a mindset that says that the product owner knows best, and that developers need to seek their permission, rather than product people being comfortable stating the problem and what outcomes we need and letting smart people creatively solve problems. (John Doerr captures the essence of this in his awesome expression missionaries not mercenaries, which Marty Cagan elaborates on.)

The first value of the manifesto is about Individuals and Interactions over Processes and Tools. User stories and all of their trappings have become the epitome of process and tools. Let’s free ourselves to interact as people.

If not Stories, then what?

It’s actually easier than you might expect, though when you think about it, it shouldn’t surprise you. Software makers (and I use that term broadly to mean programmers, designers, quality advocated — anyone who enjoys creating software) are usually creative problem solvers. Give them the problem, and they’ll not only discover the solution, they’ll enjoy the experience of freedom, knowing the goal they’re aiming for. And they’ll create even better solutions that what we might’ve originally conceived (and codified on a user story).

Rather than define my point of view in the negative as only something you should not do (I’m not starting the #NoStories hashtag), I’ll propose what I think teams should do. All you need are some enabling constraints:

  • clear product vision and goals
  • trust and psychological safety
  • low level of work in progress
  • tight feedback loops

In the case of my side project, all three of those were in place:

  • Vision: Vision isn’t a one-time slide in a deck. My teammate and I talked when we started about what we wanted to accomplish and have reiterated and progressively elaborated on that vision. By living the product vision each day, you really can live without user stories, but it does take clarity from the product person and enrollment from the team.
  • Trust and safety: My friend (actually he was nearly a stranger when we started, but we have become friends) and I had an initial conversation to understand each other’s personal goals and interests. Ever since, we’ve been in regular contact and have reinforced the mutual trust and psychological safety — no punishments, no incentive plans, no minimum “average velocity” — nothing other than helping each other realize our goals.
  • Low level of work in progress: Without my permission, my teammate sets his plan of attack for the day, knowing that whatever he works on, he’ll focus on building someone that I can see. This is part of the creative freedom that developers need: Without the central planning of a product owner that is reduced to too-discrete pieces that often don’t sequentially make sense from a development standpoint, my friend is able to follow his creative instincts and engage in flow.
  • Feedback loops: Since my teammate is constantly building and releasing — and because he knows that I’m not expecting perfection on the first try — we casually and frequently exchange feedback, not only validating that we’re on the right track but also allowing what has been built to generate new ideas and learn from “happy accidents.” What we are creating is better than any user stories could’ve guided us toward.

I concede that it’s possible for teams to succeed with user stories (I’ve been on some of those teams), in which case this call to unlearn isn’t for you. But taking a clear-eyed view of the state of software delivery today, I’m concluding that, in general, user stories are inhibiting agility rather than enabling it.

I propose breaking out of the single-loop learning cycle that attempts to improve how we do user stories and, rather, taking a double-loop learning approach to questioning whether we should use stories in the first place. Free people to do their best work through humanizing patterns like high-bandwidth conversation, psychological safety and flow. This is how creative software building works.  I’m becoming convinced that people would enjoy their work more if we let them work in this way, and that ultimately organizations will be able to create the kinds of software products that are even better than they envision and realize the outcomes that they’re hoping to achieve in the first place.

* Story-based anti-patterns:

  • This story isn’t estimated.
  • There’s no way that’s a five.
  • Let’s schedule a meeting to refine those stories.
  • These acceptance criteria are in the wrong format.
  • The PO is out today; you’ll have to wait.
  • Can you update your cards, please?
  • Tell me what to type in the description.
  • Can you make a story for that before I do it?
  • Is this a bug or a feature?
  • This story won’t fit in the sprint.
  • I’m not sure we can commit to that.
  • I think we need a spike before we do that.
  • That’s a task, not a story.
  • We can’t work on that until it’s been refined.
  • Can you please add the details?
  • What am I going to accomplish today? Story #654 and maybe start on Story #679.
  • I can’t estimate that if I don’t know the solution.