Software Estimations - Everybody needs them. Nobody likes them

In the world of software engineering, there exists a notable tension between two groups: the engineers who build the software and the business stakeholders who need to manage expectations and make strategic decisions. Engineers often embrace an “it’s done, when it’s done” mindset. This perspective arises from the complexities and uncertainties inherent in software development, where unforeseen challenges can arise and project scope can change dramatically.

From the engineers’ viewpoint, providing estimates for how long a project will take can feel like an insurmountable risk. They fear setting commitments that might lead to overwork, stress, and disappointment when estimates fall short of reality. On the other hand, businesses thrive on estimations; they require these figures to plan resources, schedule projects, and communicate timelines to clients. This deadlock presents a challenge that, if unresolved, can hinder productivity, morale, and the overall success of software projects.

What Solutions Do Exist Right Now?

Several methodologies and practices have emerged to address this issue:

1. Agile Estimation Techniques: Many teams have adopted Agile methodologies, which incorporate estimation practices like Planning Poker, story points, and t-shirt sizing. These techniques encourage team collaboration and consensus-building in arriving at estimates.

2. Confidence Intervals: Presenting estimates alongside confidence intervals allows teams to express the range of uncertainty associated with their estimates, promoting a more transparent dialogue with stakeholders.

3. Timeboxing and Iterative Development: By breaking projects into smaller, manageable iterations, teams can provide periodic estimates for each segment, reducing the pressure of predicting the entire project timeline upfront.

What is Good and Bad About These Solutions?

Each solution has its strengths and weaknesses:

• Agile Estimation Techniques:

  • Good: Foster collaboration and collective ownership of the estimate, hence increasing buy-in from the team.
  • Bad: Can still be heavily influenced by the team’s psychological biases, leading to over-optimism or pessimism.

• Confidence Intervals:

  • Good: Offer a realistic portrayal of uncertainty, setting clearer expectations for stakeholders.
  • Bad: Might lead some stakeholders to feel ambivalent or confused if they’re not sufficiently educated on how to interpret these intervals.

• Timeboxing:

  • Good: Limits overcommitment by allowing adaptation to new insights, creating flexibility in project management.
  • Bad: Can lead to the risk of focusing too much on short-term goals at the expense of long-term vision and architectural integrity.

How Can We Improve the Solutions to Get to a Better Place?

To better bridge the gap between engineers and business needs, we can consider the following improvements:

1. Enhance Communication: Establish better communication channels between developers and stakeholders. Workshops that educate stakeholders on the complexities of software development can create empathy and reduce frustration.

2. Regular Review and Retrospective: Implement regular reviews of estimates and actual outcomes. This iterative feedback loop can improve future estimates and builds trust with stakeholders as they see responsiveness and accountability.

3. Establish a Culture of Estimation: Encourage a culture where estimation is viewed as an integral part of the process rather than a chore. By framing the estimate as a learning opportunity, engineers may feel more comfortable engaging in the process.

4. Use Historical Data: Leverage past project data to inform future estimations. Historical context can help mitigate biases and provide more accurate benchmarks.

5. Incorporate Buffer Time: Allow for buffer periods in project timelines to accommodate unforeseen issues, thereby reducing pressure on engineers to meet arbitrary deadlines.

Resources and References

Here’s a list of relevant resources that can provide further insights into the topic:

1. “Agile Estimating and Planning” by Mike Cohn - A comprehensive resource on Agile estimation practices.
2. “The Lean Startup” by Eric Ries - Discusses iterative development and how to measure progress.
3. “Scrum: The Art of Doing Twice the Work in Half the Time” by Jeff Sutherland - Offers insights into time-driven methodologies.
4. “The Software Engineering Manager’s Handbook” - Contains sections dedicated to estimation and project management.
5. Agile Alliance Website - A repository of articles and resources on Agile methodologies.
6. “User Story Mapping” by Jeff Patton - Discusses techniques for managing expectations in software development.
7. “Managing Software Debt” by Chris Sterling - Explores the importance of balancing between estimation and technical debt.
8. The Atlassian Team Playbook - Various plays to improve team collaboration around estimation.
9. “How to Measure Anything” by Douglas W. Hubbard - Offers principles for measuring things that seem immeasurable, like software development.
10. Project Management Institute (PMI) Guidelines - A foundational resource for project management principles, including estimation.

By embracing improved practices and fostering a culture that values estimation as a necessary and useful exercise, both engineers and business stakeholders can reach a more productive equilibrium, resulting in smoother project execution and greater overall satisfaction.