How to Forecast Before You Even Start

One question that people who are friendly to the probabilistic-forecasting mindset often ask is “I understand how to forecast with delivery data once the project is underway, but how do I forecast before I even start?” Assuming that you absolutely need to know that answer at that point — heed Douglas Hubbard’s advice* — a simple probabilistic way to do it is through reference-class forecasting. Conceptually.org has as good a definition of it as anyone:

Reference class forecasting is a method of predicting the future by looking at similar past situations and their outcomes. Kahneman and Tversky found that human judgment is generally optimistic due to overconfidence and biases. People tend to underestimate the costs, completion times, and risks of their actions, whereas they tend to overestimate the benefits. Such errors are caused by actors taking an “inside view”, assessing the information readily available to them, and neglecting other considerations. Taking the “outside view” — looking at events similar to ours, that have occurred in the past — helps us reduce bias and make more accurate predictions.

An easy metaphor for reference-class forecasting is home sales. We’re trying to forecast something that’s complex — lots of market dynamics involved in something that’s essentially never been done before (how much someone will pay for this particular house at this particular point in time). We use a variety of economic and housing data — zip code, square footage, construction, features — to create a reference class of “comparables.” (If you really want to geek out, see Zillow’s Forecast Methodology.)

Most organizations have delivered some number of projects — maybe not this exact project in this exact tech stack with this exact team, but with attributes that are comparable to it.

An Example

Here’s an example. A company was considering a new initiative. They needed to know approximately how long it would take (time being a proxy for cost but also market opportunity cost). They took the traditional inside-out approach — attempting to predict how long something will take by adding up all known constituent tasks — and estimated it at about a year. This inside-out approach being subject to the Planning Fallacy, we decided to also try a reference-class forecast.

  1. We took a list of the 50 most recent projects, going back a few years. We needed only a pair of dates for each one: When the business officially committed to the project (confirming this commitment during the “fuzzy front end” is often the most tricky bit) and when it went to production.
  2. We then categorized each project by meaningful traits, like project type (legacy or greenfield), team size (small, medium, large) and dependencies (many or few).
  3. We viewed the data on a scatterplot chart.

Unfiltered Forecast

You’ll always have a tension between needing enough data (small sample sizes can be distortive) and relevant data. The good thing about reference-class forecasting is that it’s inexpensive (and better) to run multiple views. First, we ran an unfiltered forecast — all 50 projects.

Unfiltered reference-class forecast

This yielded a high-level view onto how long projects take overall. Half of the time (50th percentile), projects finish in 383 days, or a bit more than a year. But that leaves a lot of projects — the other half! — that take longer. How much longer depends on the level of confidence we seek:

  • 50% of the time: in 383 days (a little more than a year)
  • 70% of the time: in 509 days (1.4 years)
  • 85% of the time: In 698 days (nearly 2 years)

Filtered Forecasts

Of course, this new project will be different (as always!), so not all of those projects are really relevant. So we filter based on characteristics similar to the project we’re forecasting: It’s a legacy project, with a small team and many dependencies. We had such 12 projects in that reference class. Its confidence intervals are indeed different from those of the entire set:

a filtered reference-class forecast
  • 50%ile: 607 days (1.7 years)
  • 70%ile: 698 days (1.9 years)
  • 85%ile: 776 days (a little more than 2 years)

Those numbers were larger than the whole set, so that was disappointing. Maybe we can look at the problem a bit differently. What about legacy projects with many dependencies that were staffed by medium-sized (rather than small) teams. (Perhaps what Troy Magennis said about reducing the effect of dependencies with slightly larger teams was right!) We had 11 such projects:

Wow! That is quite a different story. I guess Troy was right!

  • 50%ile: 303 (less than a year)
  • 70%ile: 400 (a little more than a year)
  • 85%ile: 509 (1.4 years)

We now have three different reference-class forecasts to use. They at least give us some options to inform our thinking (especially as regards team-sizing decisions). Knowing which reference class to use is more art than science, so I like to consider a few options rather than locking into one (especially the one that paints the rosiest picture!).

Once we do get started with the project, we will of course want to do probabilistic forecasting with actual “indigenous” delivery data. But before we even start, we have informed ourselves from the outside-in — averting the Planning Fallacy — on when we might expect this particular new initiative to be done.

* Hubbard says essentially “Of course you need to estimate development costs when making a decision about IT investments. However, you don’t usually need to reduce the uncertainty about those costs to make an informed decision. Reducing the uncertainty about the utilization rate and the likelihood of cancellation of a new system is much more important when deciding how to spend your money.”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s