Triangle Illusion

The Date-Driven Project: Cracking the Agile Paradox – Part 1 of 4

In this article, we’ll take a closer look at schedule-driven projects, which I touched on in an earlier post, entitled “Feature Driven or Date Driven?

All date-driven IT projects have competing goals, derived from the vertices of the so-called “Iron Triangle:” budget, scope, and schedule, where quality is an orphan (more on that some other time). Anyone confronted with this three-cornered cudgel knows that while budget and time are scalar, measurable values, scope is fuzzy.

Functionality can vary based on factors such as the number of edge/corner cases covered or omitted. Technical scope is susceptible, for instance, to the iceberg effect, whereby a feature may work precisely as requested, but with limitations hidden below the water line, such as scalability, extensibility, and security. Scope variability has the power to deform the iron triangle.

It’s no secret that software projects often fail when budgets and schedules are dictated without regard to scope, leaving stakeholders furious that their doomed expectations have gone unfulfilled. To close the gap between what’s been completed and what’s actually needed they’re forced to inject funding and accept delays.

For a project leader looking to predict milestones and measure progress, the fixed constraints of time and money favor structured project management, the so-called “Waterfall” model, but inherent uncertainty begs for Scrum.

Scrum is predicated on the belief that the only rational way to proceed is to write code and see where it leads. After all, why pretend we know how long a project will run, or consequentially, how much it will cost, when most of the time those guesses are substantially wrong? A Scrum team chips away at a prioritized list of requirements until either something useful emerges, or time or budget run out. You could reasonably argue that this will be the likely outcome anyway, with or without an end-to-end schedule.

However, this leaves stakeholders disadvantaged. If you’re a developer put yourself in the shoes of a business manager who has an automation problem to solve and a fixed budget. Before committing to a contract she must know cost. And not only does she have a budget, she needs to evaluate ROI. She’s not going to spend $300,000 on a software solution if she can hire a temp to do the same work for a total lifetime cost of $200,000. I shouldn’t have to say that businesses have limited resources to throw at IT teams. Even if a new business initiative has potential to generate multiples of revenue, the upfront investment is a risk because the predicted opportunity may never pan out. The quantified risk is the ratio of potential return vs required investment. Like Las Vegas odds-makers, effective managers cap their risk on speculative initiatives such that over the long run the winners will outnumber the losers. What they don’t do is approve projects based on ambiguous deliverables and variable costs.

This leads to two competing assertions:

  1. Software engineers often can’t predict the precise cost of a software development project because that project may include too many unknowns.
  2. Business stakeholders often can’t commit to an open-ended software development project without predictable feature set (scope) or cost.

A solution does exist.

The Non-paradox

Despite the dogma, waterfall and Scrum are not mutually exclusive. However, to blend them we need to modify the ground rules of each.

Traditionalists think of project activities in monolithic chunks. That’s the legacy of Waterfall, in the truest sense of that word. The term Waterfall traditionally refers not to the project schedule, but to the decrepit Software Development Lifecycle (SDLC), in which products are (1) planned, (2) designed, (3) implemented, (4) tested, and (5) released, in that order. Over the past several decades, many software development projects have succeeded with this methodology. Unfortunately, even more have failed.

While the old SDLC works well for industrial projects, like building bridges and large machines, when it comes to software development, it’s riddled with problems. Agile techniques have taken a three step journey away from Waterfall:

  1. It makes little sense to draft lengthy, complicated specifications for a system that changes organically over time intervals shorter than the expected project duration. Furthermore, stakeholders often (not always) don’t understand exactly what they need until they begin to see part of a system in action.
  2. If we don’t design an entire system before implementation begins, we can’t rationally estimate the total effort required to complete that system, assuming we even understand what it means for it to be “complete.”
  3. Without estimates, it’s not possible to predict or accept a completion deadline.

This wholesale rejection of scheduling stems from the argument that software development is inherently unpredictable, and therefore schedules are useless. Worse still, schedules imply an un-keepable promise that the engineering team will deliver like clockwork on that schedule, which will only lead to resentment and tears.

The only way Agile methods address the feared consequences of scheduling is by requiring stakeholders to disavow any expectations. It’s a one-sided argument for a practice that introduces substantial business risk, which once again leads to resentment and tears.

It looks like an intractable problem.

It isn’t.

Just as we use roadmaps and turn-by-turn driving directions together to plan and complete cross country road trips, we can reduce uncertainty in complex projects by combining the big picture qualities of traditional project plans with rubber-to-the-road Agile execution.

In Part 2, we’ll break down the scalable process of organizing and scheduling a date-driven software engineering project.