Rebooting the Agile Manifesto – Part 1

Over the next three articles I’m going to revisit a sacred cow. I’m not advocating we overturn the venerable Agile Manifesto. However, I do wish to recover its original wisdom. In Parts 1 & 2, I’ll examine two of the AM’s four postulates and then in Part 3, I’ll explain why I think it sometimes seems the revolution has let us down and what we can do about it.

“Individuals & Interactions Over Processes and Tools”

Projects are the work of people, and when talented people click, magic happens. Process needs to get out of the way.

I’m not saying process can be entirely transparent. That would be great, but unrealistic. I do mean we should unburden project teams from as much process as possible. And that’s where disagreements set in.

If process can’t be transparent, at least I can.

My rants typical and recommendations share a common theme—planning. Planning is often stamped “DANGER. WATERFALL.” That reaction expresses fears of a resurgence of pre-Agile methods, which organized projects into distinct phases to be completed in sequence. Back in the dark times, the metaphorical term “gates” described distinct transitions from one project phase to another, where the gatekeepers were officials within the organization who had to sign off on each stage before the team proceeded to the next.

The backlash to Waterfall process was a complete rejection of staged development in favor of try-test-accept. It’s an example of poor interpretation of the Agile Manifesto, as if it declared, “Individuals & Interactions INSTEAD OF Processes and Tools.” Waterfall naturally emerged from the need for predictability and order, especially in complex physical construction projects, such as bridges, ships, aircraft, tanks, etc. If Boeing could blueprint an aircraft, order the materials, and rivet it all together, in that order, then why couldn’t IBM do the same thing with software?

It didn’t always work so well, but not because project planning is intrinsically incompatible. Even for Boeing, GE, and hundreds of other industrial companies waterfall doesn’t always work as expected, but that’s because we usually hear about the high profile failures — like the long delayed 777 airliner — and not the successes. Take the supreme example of putting “a man on the moon by the end of the decade.” The moonshot required extensive project plans and demonstrated that project management works. It also happened to prove the value of trial and error within a managed project when the massive thrusters that launched the Saturn V rocket were designed and refined iteratively in a series of expensive, thunderous, and spectacularly catastrophic tests. Software developers claim project management doesn’t work because software development is inherently unpredictable. But if NASA could put a man on the moon, on schedule, in just over eight years, with almost no existing technology and no prior experience, then we should be able to design, plan, implement, and ship a mobile banking app on time and within budget.

Success depends on a combination of expectations and execution.

Most of the anxiety seems to come not from planning, but from scheduling. Let’s not confuse the two. A schedule is a plan with target dates, a superset of a plan. The plan is just a list of stuff to do, preferably stack-ranked according to importance (priority) and dependencies. Dependency management is nothing more than a way to prevent team members from getting stuck in holding patterns. Interacting individuals, to paraphrase the Agile Manifesto, can sometimes work out their dependencies without tools, but for very large projects, with tens or hundreds of participants, it can be pretty tricky to keep track of all those relationships. Unless each Scrum team of 4-8 members can work in total isolation, someone needs to make sure things are getting done in the most efficient order.

In traditional project plans, often and incorrectly called “waterfall” plans, the project manager details out every task required to complete the project, complete with assignments, durations, and dependencies, with the ultimate goal of translating all those metrics into a schedule. This typically means setting a start date, adding up all the work, factoring in the sequencing, and letting a project management tool calculate a completion date.

The complaint I hear most from software engineers is that two factors make such plans unreliable:

  1. The requirements keep changing and therefore the pre-calculated completion date is not realistic, or worse, a complete fantasy.
  2. Estimates are guesses, often made before designs are complete, and therefore inaccurate.

Agile methods were supposed to address the first issue by throwing out the plan so no one would get twisted up when the requirements do inevitably change. In Scrum, the most common Agile approach, we plan one or a few iterations (Sprints) at a time, and accept changes to the requirements before the start of each Sprint.

As for the second objection, Scrum replaces estimates of effort with complexity assessments. in a Scrum exercise called Planning Poker, engineers collectively assign each unit of functionality, or “User Story” a value in Story Points, typically a Fibonacci sequence number between 1 and 21 (1, 2, 3, 5, 8, 13, 21), representing complexity, with the hope that these relatively crude measurements will converge on a predictable Velocity, a measurement of average number of Points completed by the Scrum team in one Sprint. Hence, complexity is abstracted from time. All that matters is that some quantity of complexity is delivered in each Sprint. Points are useless for preparing schedules, which is a safe guard against meddling, schedule-happy project managers.

These two tactics are supposed to replace “processes and tools.” In fact, the phrase “Agile process” should be an oxymoron. But Scrum is not process-free. Scrum requires a few essential artifacts and rituals.

Running the Agile Gauntlet

We begin a Scrum project with a Product Backlog, which is the list of all the known Requirements, without regard for timetables. This presupposes that some Product Owner or Owners, an official Scrum role, have gathered those requirements and expressed them as User Stories, populated with Acceptance Criteria (aka Success Criteria). Next, we keep a Sprint Backlog, which lists all the things we plan to deliver during a single Sprint. From the Sprint Backlog, we graph a Burndown Chart, a visual representation of how much of the planned Sprint work we’ve completed from day to day. Before each Sprint, we convene a Sprint Planning Meeting to determine how much of the work from the Product Backlog we think we can accomplish in the next Sprint. Then we hold daily Standup meetings to share what we’ve completed, what we’re working on, and where we’re blocked. After Standups, we conduct Parking Lot meetings to unblock those blockages. At the end of each Sprint, we hold a Product Demo, where we exhibit the completed work and obtain sign-off from all our stakeholders, followed by a Retrospective, in which the team reviews what went well and what they could do better in the future. All that, plus the many milestones we set within each Sprint, such as Code Freeze or Feature Complete milestones, regression tests, and releases.

Papa’s tired.

No wonder so many Scrum teams hire Scrum/Agile coaches. Every team I’ve worked on has had to devise their own tactics to deal with the day to day realities of supporting active customers while expanding their applications to remain competitive. Bugs throw off our Sprints, not to mention production emergencies, illnesses, vacations, meetings, build and branching processes, customer demos that require stabile deployments to dedicated environments, test automation, and especially the lack of it, and so on. I don’t know — seems like a whole lot of process and tools.

It’s not clear that traditional project management methods are any more burdensome than Scrum. They both require practices and tools. They both have advantages and disadvantages. Project teams should adopt techniques from either end of the spectrum that help them achieve their goals, which includes making promises they can keep.

While this post may be about the need for process, my mission is to examine which processes are most effective, in which circumstances. Although I’ve hinted here the pros and cons of Agile vs traditional methodologies, in other posts I have and will continue to dig more deeply, and I’ll clarify why I’m skeptical of one-size-fits-all solutions.