As a software engineering leader for hire, my assignments require raising velocity and visibility while reducing the risk of missing milestones and budgets. For me, Agile is the substrate, the base on which software is developed.
I also believe in working hard to distil problems down to their simplest form, making them trivial to understand and adopt. When I enter a new situation, I look for the most painful, urgent problems that must be solved. Using Agile principles, I try to synthesize customized, perfectly appropriate solutions that can be adopted with minimum difficulty.
Errant estimation is a problem
In my recent work with two companies, I found that estimation was a big problem. At multiple stages of projects, teams made casual and unrealistic estimates. As the work progressed, frequently these estimates proved wrong, breaking both schedules and budgets.
In both organizations, I crafted initiatives to improve estimation. I started by researching established estimation methods including Work Breakdown Structure (WBS) and User Story Mapping (USM). Combined, these two approaches recommend successively subdividing work into smaller, simpler blocks. Both methods also emphasize finding relationships and interdependencies between blocks of work.
I worked to understand what most often makes estimates wrong. I reviewed estimates that failed to accurately predict work. Frequently such estimates only considered core coding work. Testing, test development, user feedback, DevOps integration, security, performance or business resiliency were all allocated little or no time. An engineer could write the code in the allotted time. Making the code truly shippable would cause an overburn of effort and elapsed time.
Estimation guiding principles
Resulting from this effort, I made a set of guiding principles for better estimation:
- Build a complete diagram of the required software components and dependencies. Continually update the diagram as new components are discovered.
- Develop a checklist for validating estimates. This will ensure the estimates are sufficient for completing truly shippable code.
- Work collaboratively. Allocate estimation work to a diverse team of engineers and encourage reviews for each estimate.
- As the component reviews are completed, review the diagram for interdependencies and allow for related delays.
All Agile practices should be organic, empirical and iterative. This means that they build on existing knowledge, are based on evidence, not theory and are constantly improving. Measuring estimation quality and accuracy should part of every sprint review and retrospective. The estimation process should be subject to continuous review and improvement.
I started developing software in my early 20’s. Back then, I would have predicted that software development in the 21st century would be predictable with budget and schedule overburns a rarity. It hasn’t worked out that way. Too often, software is late, over budget and weakened by poor design and bugs. Poor estimation is one root cause and better estimation is a key solution.
Nurturing future estimators
My son Harry graduated high school and started his first year of Computer Science at the University of Bristol during this same time. Harry is the best combination of nerdy and cool, a musician and slightly taller than me. No surprise, his friends are also nerdy, cool musicians and aspiring computer scientists.
Not infrequently, Harry’s projects expanded beyond the allotted time. I decided to give Harry an assignment to improve his estimation skills. Later, we also offered it to several of his friends.
Each student made a spreadsheet like the illustration below. They added a spreadsheet row every time they started a coding task. They recorded the date, a description of the assignment and an estimate of time to complete.
The students recorded the actual time to complete at the end of every task. If the estimate and actual differed, they provided the reason or other comments in the worksheet. It was a very simple exercise.
In principle, Harry and his friends will learn good estimation by the time they complete their degrees. Would you hire a new university graduate who makes good estimates? I’ll provide updates as Harry and his friends proceed.