Rail yard in action.

The Date-Driven Project: Cracking the Agile Paradox – Part 3 of 4

In the previous installment of this series, we talked about scaling application designs and breaking those designs into tasks and estimates in preparation for scheduling. Now let’s take a look at how to go about building that schedule.

The Scrum metric for capacity or work completed over time is Velocity. Over several Sprints, Scrum teams calibrate themselves by assigning complexity Points to User Stories, and then calculating how many Story Points they complete in each fixed-length Sprint. For teams who’ve developed consistent Velocity, this works really well. You can walk through the entire Product Backlog, a list of planned Stories, and assign Points to each Story. To get a target release date, you just divide the total Points in the Backlog by your team’s Velocity, yielding a number of Sprints. Since Sprints are fixed in length, they translate nearly directly into calendar time, with allowances for holidays and other planned or unplanned absences. Points benefit from the Law of Large Numbers, in that the greater the number completed each Sprint, and the greater the number of total Stories estimated in the backlog, the more likely the estimates will converge on a consistent average.

For estimates given in Points, the actual duration is self-correcting because it’s based on continually adjusted Velocity, which varies over time according to factors such as team stability and application maturity. Points introduce a degree of indirection, which reduces the need for full re-estimates as team productivity changes. It’s kind of a circular argument, however. If we don’t expect Velocity to remain constant, then we can’t use Velocity to predict the schedule at the outset of the project. Points may save us from perpetually revisiting our estimates, but they don’t necessarily yield much certainty.

Velocity is a little idealistic, or at least optimistic. Despite best efforts, Scrum is often susceptible to the same constraints as traditional project management methods, most notably, functional dependencies and specialized skills.

In an ideal Scrum team every member would have excellent “full stack” skills, which means anyone on the team could complete any development task, whether it’s a service end point written in Java and designed to scale to millions of users, a user interface written in JavaScript, an automated testing framework, or one of many other specialty skills. Such super-coders are rare. So, given a team of specialists, what would it mean to assign a Story a complexity value of five Points? Is it primarily services development, UI, SQL, NoSQL, something else? Unless the Backlog contains a list of Stories that perfectly match the skill composition of the team, Sprints will tend to tip out of balance.

Velocity is supposed to average out the effects of functional and skill dependencies, but that places even more burden on the abstraction. The further we distance ourselves from the nitty-gritty details, the less accuracy we can expect, again caveated by the law of large numbers.

From Backlog to Schedule

If you’re working toward a deadline, you need a schedule. That doesn’t mean you need to sacrifice Scrum. It just means you need to plan ahead.

It would be a mistake to give up Sprints as your execution methodology. Unlike a traditional milestone, a Sprint incorporates additional constraints designed to lead to refined functionality and high quality. Each Sprint ends not with a simple inventory of completed tasks, but a set of tested features, establishing a new, solid product baseline. To me, that’s the overwhelming value of Scrum. Despite Agile’s promised benefits, your projects may not always adjust well to shifting requirements, but there’s no question that exiting each Sprint with clean, stable code will get you to the finish line faster than the waterfall method of postponing testing and debugging until after a coding marathon.

Sprint planning for a date-driven schedule begins with a groomed backlog, fully populated with sub-tasks and estimates. Use your scaled designs to develop a plan with appropriate precision for the required deadline. But that’s not the final planning step. The backlog is almost always stacked in priority order, but priority alone won’t determine your schedule. You need to consider all types of dependencies, which include external, as well as basic functional sequencing, along with resource capacity.

You’ll build your schedule by mapping out your backlog on a timeline. Well-crafted Stories will usually contain sub-tasks, estimated in hours or days, and differentiated by discipline, such as Service Development, UI Development, and Testing. The time-based tasks, not their parent Stories, are the best building blocks of a predictable schedule. Choose your time resolution — hours, days, or weeks — according to the precision of your specifications. There’s no sense in estimating in hours if you’ve only sketched out your design. The precision of your estimates should reflect your degree of confidence.

To sequence tasks on the timeline, evaluate not only how tasks align with each other, but also with available staff resources. This is a critical difference between open-ended and date-driven Scrum. In Scrum, we don’t usually pre-assign work to individuals or skill pools, but the best way to pace out a timeline is to at least guess at possible assignments. We can always reshuffle in-flight.

With pre-assigned and estimated tasks tiled out on a timeline, a backlog begins to look an awful lot like a traditional project plan. However, unlike traditional planning, you’ll want to slice the schedule into a series of pre-planned, equal-duration Sprints. As Scrum demands, you’ll still start each Sprint by validating your assumptions and recalibrating for unexpected changes in scope or complexity, but with the added guidance of your end-to-end roadmap.

Some project management applications are more friendly to Scrum/Agile than others. LiquidPlanner.com, for example, incorporates various Scrum features, such as Burndown Charts and Board views. Of course, there are other criteria for selecting project management products, none of which we’ll explore here.  

You don’t absolutely need a Project Management tool to build your schedule, but I’d strongly recommend it because it makes the plan more useful for calculating the impact of inevitable changes. With a fully modeled schedule you can insert and remove tasks, or change estimates and let the tool recalculate your dates. Another advantage of a project plan view is that it can show you gaps, where dependencies have left unused capacity. For example, your Services/API developers may get ahead of UI. You’ll use the schedule to staff your team or to rearrange work to make best use of existing members.

Schedules are time-based. We accept that we’ll need to make changes (see Plan for Change), and the project plan becomes the tool with which we can show stakeholders how those changes will impact our dates. Stakeholders ask for predictability; engineers ask for flexibility. The fair and beneficial solution is to balance all concerns. More on that subject in the fourth and final part of this series.