the practical program manager

a contrarian's rough guide to getting stuff done

Straw-Man Agile Project Sizing: Part 3

by

In Parts 1 and 2, I suggested a systematic way to tackle rough software project sizing for budgeting and strategic business planning. You may not agree with my approach. It’s just a suggestion, a straw-man for straw-man estimating. The point is that software projects are sizable even before they’re fully defined, as long as we qualify our estimates.

Identify the Risks

“The devil is in the details,” as they say. Innocent-looking requirements can blow up into novel and complex problems with intricate solutions. Some typical features that befuddle engineering teams include data synchronization, scheduling algorithms, workflows, and integrations with external devices and services. When you deliver a high level project estimate, you absolutely must call out these schedule breakers. Tease apart what’s known from what’s unknown.

Calibration

For a work in progress, calibration is easy. You should have many epics to compare against. However, I’d still recommend strongly that you represent your estimates in T-shirt sizes, with stated comparisons to these other features, rather than representing your estimates with inappropriate precision. Remember, when you estimate in person-hours, the implied precision is also to the nearest hour, even if you say +/- 30%. If you state your estimate in person-months, the implied precision is much lower. Like it or not, to most recipients of numerical information precision implies accuracy.

For a green field project, you’ll want to compare complexity to previous initiatives. After three or four iterations you’ll find that you have a pretty clear understanding of how your own scoring system relates to actual effort. If you have the time and patience for it, it wouldn’t hurt to calibrate by retroactively scoring a completed project.

From Complexity to Cost

The final step is to translate complexity into useful, if imperfect costs. It’s worth stating that cost is not synonymous with schedule. Cost is total effort. A timeline is what you get when you match resources with expected effort and dependencies to predict dates.

Time to Complete != Capacity/Estimate

Timelines require planning, not just estimates. However, I do use a rule of thumb to expand total effort into approximate calendar duration. For me, it means Capacity/Estimate + 30%, which is usually (though not always) sufficient to allow for dependencies, holidays, vacations, and unplanned absences.

To Pad or Not to Pad

Be careful about mentioning padding. By which I mean, don’t. If you tell an executive that an initiative will require three months, but that you padded it to four, she’ll hear, “we think it could be done in three months, but worst case is four.” Better to say, “Based on what we know today, it looks like roughly four months of work for 3 software engineers and one test engineer.” If you can get away with it try, “about a third of a year,” or “within two calendar quarters.” Don’t accidentally make promises you can’t keep.

Trained Intuition

As your team or product stabilizes, complexity scoring will calibrate your business planning estimates. Over time your scoring system may become unnecessary. Most engineers can rapidly assess the complexity implied by a straw man data model or interaction model. The real reason for grinding through scoring exercises is to train yourselves as a team to make better, educated guesses with decreasing effort. As long as you’re vigilant and keep your eyes open for land mines hidden in the implied requirements, you’ll rapidly tune this skill. Be patient. I’ve observed that cohorts are better and faster than individuals at breaking bad habits and developing good habits and skills.

As I said at the end of Part 1, one of the benefits of this top down approach to sizing is that it’s not throwaway work. The insights you glean will lead directly into the detailed analysis required as you populate your Product and Sprint backlogs. Whether you invest a few hours or a few days, you’re sure to end up with a useful technical roadmap, along with a list of clarifying questions for your Product Owner(s).

Straw-Man Agile Project Sizing: Part 2

by

In Part 1 we talked about the challenge of furnishing cost estimates for loosely defined software development projects. These are typically initiatives still in the proposal stages, for which business stakeholders need cost before they can determine the potential return on investment and budget.

In this second of three parts I’ll suggest specific examples of ways to sketch in enough detail to arrive at a low fidelity estimate sufficient for business planning.

Tools

I like to begin an exploratory design with a visual “straw man” of the business process I’m modeling. A straw man is a rough proposal, a stand-in for a final design, simple enough to hack away at until we land on a workable approach. If everyone on your team is skilled with OO thinking, then you may choose to jump to Sequence Diagrams for your straw man designs. I prefer old-fashioned flow charts because they tend not to prematurely suggest implementation details, and it’s relatively easy for all stakeholders, both technical and non-technical, to follow the arrows connecting boxes representing actions and diamonds representing decisions. Sketch out as many of these as you need. In one design exercise involving twenty stakeholders from four countries I projected flowcharts on a whiteboard where the team could draw their revisions directly on the diagrams. We then snapped photos to capture the changes. It was a makeshift “smart board.” If you have more advanced technology in your meeting rooms, so much the better.

Along with the process flows, you should invest some time in business rules. Flowcharts may show how things will work under normal circumstances, the “happy path.” Business rules fill in all the minutiae that won’t fit neatly into your diagrams. Don’t get lost in implementation details, such as data input validations until after the project is approved and funded. The business rules you’re interested in here are usually common sense business constraints such as “Each payment may be applied to only one invoice,” or “Accounts may be deactivated, but not deleted until after seven years following last activity.”

The third step, which may be flipped with the first, is to inventory the data required to support the feature. Personally, I like to start with a data model. Incidentally, I used to believe this was the “right” way to design systems, but since I’ve declared myself anti-dogmatic I’m compelled to acknowledge that we each have our own way of breaking down a problem. Yet…

Lay of the Land

While sketching out an approximate data model you can skip the utility attributes, such as event timestamps and update counters, but you should be able to rapidly list all the relevant entities and attributes required to represent the static and transactional units acted upon by the anticipated processes. Capture the essential elements, such as Last Name, First Name, Address, Credit Limit, Terms (30, 60, 90 days to pay), Transaction Amount, Transaction Date, Transaction Description, etc. The number of attributes and the number of documents or records into which they are organized imply a degree of complexity.

For example, you may find that your new feature requires three new data entities — tables in a SQL database or documents in a no-SQL database — comprising a total of forty-five new attributes. The complexity factor isn’t so much how many attributes you need to add, but instead, how many of those attributes are computed, their relationship constraints, and how many will be transformed in some way.

To score the complexity, you’ll need to establish some standard measures. For example, any data entity usually implies CRUD interfaces, which contributes to the weight of the change. So a new table, regardless of the number of attributes, could, for example, represent an arbitrary baseline score of 20 points. That gives you a starting point. It’s a value you’ll establish, against which you’ll compare all other changes. Like card-counting in Blackjack, you can then score up or down. A non-computational field, such as Street Address, would have a complexity value of zero. Each foreign key relationship may add +1. Fields that derive their values from processes applied to other entities or attributes, such as account balances summed from transaction amounts, may count as +2 or more.

Once you’ve captured business processes and a data model inventory, stop and determine whether you need any further detail to arrive at a complexity estimate. You may not. It depends on the constraints such as those I listed in Part 1, such as regulatory compliance, security, scalability, and other non-functional requirements that usually don’t appear in the business processes because they represent plumbing, and aren’t especially interesting to business owners. Sorry managers, but it’s true. That’s why you hire IT managers who do think about all that stuff that just needs to get done. It’s also why things take longer than you expect. Security concerns alone can multiply the complexity and cost of a project.

From here, if necessary, you can increase the accuracy of your guess by diving a little deeper. The next logical steps would be to sketch out the UX and identify backend service endpoints (API functions) that would likely be needed to support the business processes and business rules. Not detailed interfaces, just the most obvious use cases, such as the CRUD functions and essential business logic.

Just like an Agile Spike, time-box this sizing exercise to control your depth of detail. After two or three rounds on separate features, you’ll know how much time it gets to your desired precision. Spend a few hours or a couple of days, not weeks. That would not be a SWAG. And if you do have the luxury of completing a detailed design before providing sizing, take advantage of it because it’s guaranteed to save pain later.

In Part 3 we’ll wrap up, proceeding from complexity to approximate cost and throw in a couple of caveats.

Straw-Man Agile Project Sizing: Part 1

by

Annual budgeting is in progress and the leadership team want to know how much it will cost to deliver on the planned objectives for the coming year. The problem is, the engineering team hasn’t spent more than ten minutes discussing those products or features, and therefore have only a superficial understanding of functionality, or in some cases, no understanding at all.

 

You’ve probably seen the Dilbert comic(s) of this scenario.

 

Yet, whether we like it or not, we are responsible for making educated guesses to support decision makers. We work for organizations that burn money in pursuit of specific goals and objectives: product launches, revenue growth, cost reduction, scientific discovery, landing a Mars rover, etc. Which slams us straight into the engineer’s dilemma: How can we know how much work is needed until we know exactly what we’re building?

 

If the goal is to lock down cost, we can’t. It’s simply unreasonable and risky to expect a spot-on estimate based on fuzzy, simplified, or incomplete requirements. However, we can provide guidance sufficient to inform business planning.

 

The least helpful thing we can do is make completely wild guesses, which are almost always wrong, and lead either to complete failure, or to cost and schedule blowouts and burned out staff. So-called t-shirt sizing, in sizes small, medium, and large, isn’t much better. What we need is a systematic method for sizing a nascent project, less detailed but analogous to the detailed design and estimating we’ll do once the project is chartered.

 

Complexity Scoring

In Scrum we assign complexity with Story Points. Stories, however, are meant to represent specific actions to be performed by the user (technically the ‘actor’). The value of Story Points for course-grained estimating depends on the degree to which the requested feature or application has been decomposed. However, we typically don’t begin defining Stories until the Product Owner(s) have spelled out relatively detailed requirements.

 

One way to size a software development project without a detailed drill-down is to apply a non-story-based complexity scoring system. This is something you can invent within your team. Like Story Points, your scores don’t need to be universally calibrated to some arbitrary — and noexistent — global standard. They only need to be measured against projects previously accomplished by your own team.

 

If you can’t begin with Stories, you’ll need to decompose the proposed application into some other elements to which you can assign complexity. Any particular application comprises some combination of familiar components:

 

  • Data Inputs .
  • Data elements/data model.
  • Data quantity/velocity.
  • Breadth and depth of state changes.
    • Transformations required.
  • Outputs
    • Including reporting, document printing.
  • User Interfaces.
  • Interfaces to external systems (API).
  • Security/Access Control.

 

Other complexity factors include:

 

  • Novelty: Is this a new application or an extension to existing one?
  • Architecture: Web/SAS, Client/Server, Desktop, Mobile App.
  • Usage scale.
  • Regulatory compliance.

 

It’s a pretty basic list. No big revelations here. However, no matter what solution we may be building, we can use these landmarks to pace out the trails and boundaries of a proposed application. Notice that I’m hiding architectural analysis behind the mask of pre-design. That’s why I consider early costing not only necessary for business planning, but also a useful step toward implementation. Is this ‘waterfall?’ Only if you lock down the design before implementation, which is not only a bad idea, it’s also futile. Things change — hence Agile.

 

In Part 2 we’ll discuss some specific tactics we can use to identify an application’s defining characteristics, which we’ll then use to assign complexity metrics suitable for costing.

Dread-free Software Estimating

by

You may believe that time estimates are useless. You may be correct. Estimates fulfill a specific purpose, predictability, which may not apply to your project or initiative. Any schedule-driven project, Agile or otherwise, hinges on the accuracy of estimates. If you don’t need predictable delivery, you can skip this headache altogether. If you’re lucky enough to work on a product that has no timetable, don’t waste your time on extensive planning. Make yourself a list of things to do, a Product Backlog, and knock them off at whatever pace your team can manage.

You may also prefer Story Points over time estimates. For some teams the combination of Story Points and average Velocity yield predictable timelines, especially for stable, experienced teams who’ve worked together through many Sprints. Points differ enough from time estimates to merit a separate article, so I won’t address that technique here.

I’ve explained why and when time-based estimates are useful or essential in my previous four-part series of posts, “The Date-driven Project: Cracking the Agile Paradox.” In this article, I’ll move on to addressing estimating’s specific challenges and how to tackle them.

Accuracy

Stating the obvious, what makes an estimate useful is its accuracy. I wish I could say there was a prescription for accurate estimates. There isn’t. Estimating is a soft science whose accuracy depends on several factors, especially experience. Consequently, it’s unreasonable to expect inexperienced engineers to estimate accurately. Therefore, the greater the cumulative experience among members of the team, the greater predictability you can expect.

To complicate matters, the raw career experience of the people responsible for estimating isn’t the only experience factor. Their tenure with the team, product (aka program), project, and subject matter also affect accuracy, often by wide margins.

Software estimating truly is uniquely difficult. Software development is not entirely repetitive. Even the most mundane enterprise applications can be riddled with novel problems. By definition, it’s all but impossible to accurately predict the effort required to solve a problem never before seen.

It seems like an intractable problem. However, there are ways to calculate predictable schedules from fuzzy estimates. First, let’s define fuzziness, or more precisely, precision.

Precision

Accuracy describes a measurement; precision describes the variability of that measurement. You could say, for example, that I’ve estimated 5 days for a task, and that would have an implied precision of plus or minus one day. If you were to express the same estimate as 40 hours, the implied precision might be plus or minus one hour, but certainly not plus or minus one minute. You could also qualify either of these with explicit precision, e.g. +/- 2 days, or +/- 16 hours. When you’re estimating tasks, you want to choose a degree of precision — hours, days, weeks, or months — that appropriately expresses your certainty — or lack thereof — at the time of estimation.

Imprecise estimates are useful for roughly sizing projects, but not for nailing down ship dates. Precise estimates, when they’re accurate, lead to predictable schedules, at a cost. In general, both accuracy and precision correlate with the degree of detail furnished in design documents, such as functional requirements or technical specifications. Design takes time of its own.

Generally, the more granular your tasks, the better your estimates. It’s not that estimates on smaller tasks are intrinsically more accurate. The quality of your estimates comes from the work you do to break down the problem. Decomposition teases out the relevant questions and answers needed to stake out measurable units of work. You arrive at better estimates by increasing knowledge and decreasing ambiguity.

Measuring Uncertainty

Now let’s talk about the dark side of qualifying estimates with uncertainty. Say you have a hundred or more estimated tasks. The next step is to drop them into a schedule and calculate a completion date. But what are you supposed to do with the uncertainty? Depending on how well the implementation problems and solutions are understood in advance, you’ll likely end up with precision that varies from task to task. Usually, this is expressed not with precision, but as certainty, e.g. “We’ll complete the credit card address validation API in 20 hours, with a certainty of 80%.”

How does 80% certainty translate to days and hours on a calendar? Does it mean the task could take 20% more or less than 20 hours, i.e. 16 to 24 hours? Certainty isn’t meaningful without a significant statistical sample. When a meteorologist says there’s an 80% chance of rain, we assume it’s likely to rain. What she’s really saying is that out of 100 such predictions, it will rain on 80 of those days. On any particular day it will either rain or it won’t. And the prediction says nothing about how much it will rain. Could be a quick shower or a twenty-four-hour deluge. Another way to state this type of certainty is, “There exists an 80% chance I will complete this task within the time estimated,” which only says there is a 20% chance of failure, and then nothing at all about how much additional time might be required, i.e. yes or no, but not how much longer.

Knowing that 80 out of a hundred tasks will finish within the estimated time isn’t helpful. Let’s say you lined them all up sequentially and calculated a project duration of 175 days. Great. How will you use the value of 80% certainty to determine best and worst case? You might assume you could use the broken math from above and calculate precision by applying 20% in either direction, for a range of 140 to 210 days (+/- 35 days). If that’s close enough for your business stakeholders, run with it. The low number is the sum of all best cases, and the high number, the sum of all worst cases. The actual duration will land somewhere within that 70-day gap, an example of low, and mathematically dubious precision.

Hone In On the Range

Given that high precision estimates are unrealistic, it may seem I’m conceding the entire exercise. I’m not. There is an estimating technique that factors in uncertainty in a way that actually, if counter-intuitively leads to predictable schedules.

It’s not as simple as adding up the lows and the highs. That will give you an enormous range of uncertainty. Instead, the summation is based on the statistical method of summing the variance. Think of it this way: If you state a range of uncertainty for each task, some will end up requiring the worst case, some the best, and many in between. Over a large number of tasks, we want to find the average effect of all that uncertainty, not the cumulative best and worst cases.

In his book, Software Estimation: Demystifying the Black Art, Steve McConnell explains the mathematical details well, and it’s simpler than it sounds, requiring only basic arithmetic. If you’re familiar with a few concepts, you may already understand the steps:

  1. Calculate the Standard Deviation (SD) by dividing the difference of each high/low pair by 2.6 (Task_SD = (high – low)/2.6).
  2. Square each SD to obtain the Variance (Task_Variance = Task_SD^2).
  3. Sum the task Variances.
  4. Take the square root of the summed Variance to get the aggregate SD.
  5. Sum the means (averages) of each high/low pair.

Apply the aggregate SD to the overall mean (average) to yield the overall low and high estimates (Overall_Low = Average – Aggregate_SD/2, Overall_High = Average + Aggregate_SD/2).

The result is an unusually strong predictor, based on evidence from a sizable industry sample. The magic number, 2.6, represents the 80% band within the normal distribution. In other words, it assumes that your team will correctly identify the minimum and maximum effort 80% of the time, which is about the best you can hope for, unless you allow them to give extremely broad ranges (e.g. 1 to 1000 hours), and therefore forfeit so much precision that the entire exercise becomes meaningless. The risk in your schedule is in the outliers, so I actually do encourage my teams to toss out a broad spread for any task they find especially tricky to predict, but those are the exceptions.

The arithmetic of range-based estimates is not intuitive, yet not terribly complex. It’s a lightweight statistical method that yields an aggregate range, which is much narrower and more refined than the sum of the lows and highs. The basic calculation is straightforward when you’re calculating total effort for a project as a costing exercise.

Note: No matter what estimating method you use, be careful how you translate estimates into schedules. You can’t simply divide the total expected effort by team capacity per week, month, or Sprint to calculate calendar time. Functional and resource dependencies always stretch timelines.

The range-based method can be incorporated into a schedule, although the math becomes trickier. Estimating total effort for a list of tasks is not the same as estimating completion dates for daisy-chained tasks, with cumulative uncertainty. If you were completing each task in succession, one at a time, the cumulative low and high would predict a range of completion dates. But that’s not how a project works. Some work occurs in parallel, and some in sequence. The milestones and final deadline depend on the critical path. It’s not easy to construct a timeline from best and worst cases, which, as I explained above, are not simple sums, and depend on the path the team takes through the scheduled work. Add to that the gradual decrease in uncertainty as tasks are completed. For a project of any size large enough to benefit from this technique, it’s too painful to calculate the timeline by hand and then revise it continually as the project progresses. You could try a spreadsheet, but spreadsheets are not great tools for modeling complex schedules.

The best option is to use a project management tool that calculates rolled up durations, such as LiquidPlanner.com, the only project management application I’m aware of that runs on range-based estimates. I’m not affiliated with LiquidPlanner.com in any way, but I do recommend them because of this feature. It’s possible that Smartsheet’s (Smartsheet.com) hybrid spreadsheet and project management features, with some finagling, could work as well. If you don’t use a project management tool with these capabilities, you can still benefit from range-based estimates to pre-calculate total effort, and then plan your schedule using the high/low averages (means) for each task.

Range estimates work best over a largish number of tasks, obeying the so-called “law of large numbers.” Your uncertainty on individual tasks can vary widely. Yet over a hundred, or several hundred tasks, it averages out into a surprisingly accurate prediction. At least, that’s been my experience so far. For more thorough coverage of this and other methods, purchase Steve McConnell’s, Software Estimation, mentioned above.

Story Slicing

by

One challenge of Agile development is the process of breaking down business requirements from Scenarios into a set of clear and distinct User Stories. Evolving software architectures have forced us to adjust some definitions.

Story Craft

User Stories begin with a declaration:

As a <user/actor>, I want to <do something> , in order to <get some result>.

Each Story should describe a small, yet complete unit of functionality sized such that a software development team could code, test, and potentially release that feature within a single Sprint.

By breaking down and delivering an application or new feature in a string of User Stories, usually several each Sprint, the engineering team enables business stakeholders to validate the application in small steps, potentially catching design or implementation defects before they germinate into tenacious weeds (crabgrass). Sounds reasonable. And it is. Except sometimes it’s not so easy to find the perforations.

Strictly speaking, a User Story represents the work required to complete an atomic user action. Stories belong to Epics, larger collections of User Stories that add up to a complete feature. “Managing User Accounts” is an Epic. Saving the user name, password, and related information of a single user is a Story within that Epic, as are “Change Password,” “Delete User,” etc.

The Plot Thickens

User Stories with feature scope work well for stand-alone client software, such as mobile apps. However, in systems of federated components, this traditional definition of a User Story as a self-contained unit of work may become unwieldy.

Many modern applications, especially enterprise systems, are built on multi-tier, service-oriented architectures (SOA). In such systems, one user click will typically trigger a cascade of messages among several specialized services to collect data, process it, and store it in a data repository. An apparently simple unit of functionality, as experienced by a user, can mask layers of clockwork. Because some functions are composed of multiple components, they may be loaded with dependencies that cause the work to stretch into multiple Sprints, delaying sign-off by the business stakeholders.

My friend and colleague, George Dawoud, has redefined Stories and their parent containers. To him “Create a User Account” is not a Story. It’s an Agile Theme. Saving the user name and password is an Epic. Within that Epic are multiple technical Stories, one for each component, such as:

  • A Web user interface.
  • A service to accept and store data from the Web client.
  • Test automation to validate each component and the integrated Epic.

Each unified and independently testable unit of implementation is a Story, without regard for the fully integrated function exposed to a system User. This decomposition makes it more practical to complete Stories within single Sprints, and clears the integration log jams that often occur toward the end of each Sprint. For an application built on an SOA, it also aligns process with structure.

To make it possible to complete a feature in a series of successive Sprints, design each feature from end to end. Technically decomposed Epics require more detailed specifications because there is less opportunity to negotiate interfaces and protocols during development. If you have clear enough requirements to design features up front, which always leads to better results anyway, consider dispensing with conventional definitions of User Stories and build your application in Technical Stories.

Beware: It’s all too easy to slip into Waterfall, with front-loaded design and back-loaded testing. Don’t succumb to the temptation to postpone all testing to the final, integrated Epic, as if it were a User Story. Test the UI, to the greatest extent possible, and then test each affiliated service as it rolls off the production line. You want to minimize the defects that surface during final integration.

Twists & Turns

The contract in Scrum is that business stakeholders may introduce changes between Sprints. In an ideal Agile project, business stakeholders will review completed features and either ask for revisions or authorize continuation to the next most useful function. However, when we implement the service tier of a feature in one Sprint, and then the PO changes the requirements before we implement the UI in the following Sprint, it could invalidate the work already accomplished. Is this any worse than completing the entire feature and then changing it in the next Sprint? It is if you have shared dependencies.

One backend service will likely be consumed by several user-facing functions. Dependencies compound over time, such that a small change may trigger a chain reaction through multiple components. The longer the delay between the start of work and delivery, known as the Cycle Time, the greater the risk of high impact changes. This is a weak argument against technical slicing, however, because the PO could just as easily introduce a new feature in a later Sprint that broadly affects work already accomplished, with similarly dire consequences.

One way to limit late-breaking changes is to build the UI first and obtain stakeholder sign-off before investing in the underlying services. Because the UI is the face of most features, a hands-on UI demo is most likely to prompt modifications or more substantial rethinking by business stakeholders. Nailing the UX/UI implementation first will limit more substantial, costly changes in the services tiers. This is another positive benefit of cleaving a feature along technical boundaries. Build the UI first, with either mock services (costly) or hard-coded test data (cheap).

Epilogue

Superficially, the two definitions of Story look like alternatives. But it’s not really a choice. It’s progressive. We typically slice Features into Functions to create conventional User Stories. Breaking down those User Stories into Technical Stories is just the next step. No need to sweat the terminology change, promoting User Stories to Epics, and Epics to Themes.

It’s a pretty simple trade-off:

  • If you’re working from relatively stable requirements and have the opportunity to design features carefully before you start coding, you’ll save scheduling headaches by breaking User Stories into Technical Stories.
  • If you’re iterating through an evolving set of requirements and solutions, then it’s more efficient to deliver work in increments of fully formed functions.

The Date-Driven Project: Cracking the Agile Paradox – Part 4 of 4

by

Of course we need tools and techniques to successfully manage a project, but we also need agreement among all the stakeholders — agreement on budget and schedule, agreement on functionality, and agreement on rules of engagement.

In this series, I’ve discussed some of the advantages and disadvantages of Agile methods, Scrum in particular. Pristine Agile methods are ideal for projects in which requirements are not well understood at the outset and require some iterative development to achieve clarity. We’ve also shown, however, that open-ended Scrum is incompatible with fixed deadlines, because with little or no functional definition in advance, the schedule is unknowable. Period.

One final common misconception about Scrum is that new requirements can slipstream in at each new Sprint, building incrementally on what’s already been accomplished. That may be true for small adjustments, but the impact of an innocent-looking change won’t always confine itself to future work. Often — very often — with or without a deadline, scope changes can derail a project by breaking or invalidating completed work. When business stakeholders demand a schedule, they must accept and acknowledge that Agile methods don’t magically bend time and space to shoehorn scope changes into the same continuum. That’s their end of the deal.

Software engineers are also on the hook. The only way to predict a completion date is by spelling out and estimating the work from start to finish. I know it’s difficult to convince road weary developers that estimation is a worthwhile exercise, but it’s unavoidable when you’re facing a deadline. One reason estimating causes engineers to experience anxiety is that estimates are too often seized upon by business stakeholders as a blood pact. But a project plan is a measurement, not a promise.

Engineers need to stop rejecting project plans as cudgels, and start recognizing their value as communication tools. A plan is a data-rich model. As quantitative practitioners, engineers should recognize the value of data in discussions of project scope and schedule, especially when that data is supported by proven experience and results. When business stakeholders inevitably need to insert new requirements, the project team will use the project plan to evaluate mathematically how scope changes will impact the schedule.

Given the added complexity, business stakeholders must choose whether a project actually must be date-driven. No question that many business initiatives are tied to the calendar by seasonal demands, competitive pressure, or as a consequence of budgetary constraints. However, business stakeholders need to weigh the tradeoffs between hard schedules and open-ended Agile development. Scheduled projects require more up-front effort and limit opportunity for revision during development, while purely iterative projects allow more flexibility to explore and shape a business automation solution at the cost of predictability. To be sure, these are two endpoints of a spectrum. We can tune predictability by choosing how much to plan vs. how much to work out in a “design build,” a euphemism for “trial-and-error.” What’s important is knowing that you’re making an explicit choice and why you’re making it, and then sticking to the accountability agreement.


Rules of Engagement – An Agile Covenant

Before we tackle a date-driven project of any significant scale, the stakeholders, both business and technical, must agree to the specific principles and practices used to plan and support the schedule. I propose we augment the Agile Manifesto with a balanced agreement among stakeholders:

Covenant for Agile Software Development

Agile methods elevate communication among stakeholders and increase product quality.

Agile methods are compatible with date-driven development, insofar as we plan for success.

Therefore:

A specific date demands specific requirements. The deadline for a project can be determined only when either the deadline supersedes the requirements, or the project scope is known at the outset.

Time is not elastic. When stakeholders ask a team to deliver by a specific date, they must accept that scope changes, however necessary they may be, will almost always impact the deadline.

Planning decreases risk. Initial planning is the best protection against scope and schedule changes.

Project plans are tunable. All stakeholders in a project, including the engineering team, must negotiate the schedule risk, scaling specifications and pre-planning according to the firmness of the deadline.

A predictable schedule begins with accurate estimates. Engineering teams must make best efforts to furnish accurate estimates, with uncertainty stated as plus/minus variance or in time ranges, and in time units (e.g. hours, days, or weeks) appropriate for that uncertainty.

Expect the unexpected. No matter how much we plan, we’ll always discover unanticipated requirements and obstacles. Leave room in the schedule to absorb change.


A successful project begins with common expectations and clear responsibilities, and ends with fulfilled needs and satisfied participants.