Product 101 - Don't Go Chasing Waterfalls

If you've spent any amount of time in Product, you have heard the inevitable...

"Oh no, don't worry, we are not pure Agile.... we are taking a hybrid approach..."

This covers all manner of sins but 99% of the time, it is a nod to the fact that people within an organisation generally want a defined project plan with clear deliverables and an end date. Non-product people hate ambiguity and love Waterfall as a result so we say things like 'hybrid' so that everyones gets the warm and fuzzies...

Navigating an organisation's love of Waterfall can be tricky!

Beware!

While the Waterfall methodology was once the standard for software development, it has become increasingly scrutinised for its limitations and potential pitfalls. Today we examine the dangers of relying on Waterfall in software projects, highlighting how it can lead to inefficiencies and challenges in delivering high-quality software with speed to market.

Understanding the Basics: What is the Waterfall Methodology?

Despite its origins within software development, the Waterfall methodology is a cornerstone in the world of project management (note that we say project NOT product), Its structured and systematic approach makes it suited to managing projects with fixed scope and an end date (something that products NEVER have).

Originating from a time when manufacturing and construction industries dominated, Waterfall's methodology laid the groundwork for many of the project management practices we see today, as it provides a framework for managing projects in a linear and sequential manner.

Tracing the Waterfall to its source…

Waterfall can be traced back to the manufacturing and construction industries of the late 1950s and early 1960s. These sectors required a clear, structured approach to project execution, where end products are reproduced in large quantities. In this world, a longer linear process of discovery, design & documentation prior to build is essential: Ultimately, to produce non-digital products quickly once in market, factories and processes etc need to be firmly set.

Once in market, physical products also do not need to change quickly, meaning that extra time finding market fit upfront reduces costs overall.

Inception and Formalisation

The formal introduction of the Waterfall model is credited to Dr. Winston W. Royce, a computer scientist and software engineer, who published a paper titled "Managing the Development of Large Software Systems" in 1970. Royce's background in aerospace engineering and his work with software projects for NASA provided him with the insights necessary to articulate the need for a more structured approach to software development.

In his paper, Royce described a sequential design process, which outlined a series of steps that begin with conception and continue through maintenance. Although his original paper highlighted the potential flaws and limitations of this linear approach, the model was quickly adopted.

Adoption and Mainstream Acceptance

Early Adoption

Waterfall gained traction during the 1970s and 1980s, primarily because it aligned well with the needs of industries where requirements were stable and predictable. Its straightforward, phase-driven approach appealed to project managers who needed a clear framework to manage complex projects.

Becoming Mainstream

The adoption of Waterfall into mainstream project management was further solidified through its inclusion in military standards. The Department of Defence and NASA began using the Waterfall approach for their software projects due to its rigorous documentation requirements and structured processes. This endorsement by major governmental agencies played a significant role in legitimising the methodology.

Additionally, the Waterfall model became a staple in the curriculum of project management education, cementing its place as a foundational teaching tool for aspiring managers. Textbooks and academic courses propagated its principles, contributing to its widespread acceptance across various sectors.

Evolution

Despite its popularity, Waterfall faced criticism as industries evolved and became more dynamic. The rigidity of its linear phases proved to be a disadvantage in environments where requirements could change rapidly. This led to the development of alternative methodologies, such as Agile, which emphasise flexibility and iterative progress.

Key Milestones and Influences

• 1970s: Royce's paper sets the stage for the Waterfall model, leading to its adoption in software and engineering projects.

• 1980s: The model sees widespread use in government and military projects, reinforcing its reputation for reliability.

• 1990s: As technology industries evolve, the limitations of Waterfall become apparent, prompting the exploration of more flexible methodologies.

• 2000s and Beyond: The rise of Agile and other iterative approaches highlights the need for adaptability, though Waterfall remains prevalent in industries where projects are well-defined at the outset.

Waterfall phases are an uphill climb!

Waterfall 101 – The phases

The Linear, Sequential Approach

Waterfall is distinguished by its linear, sequential process, where progress flows in one direction, like a waterfall, through distinct phases. This clarity and predictability make it an attractive option for projects with well-defined requirements and objectives.

Phase 1: Requirements

The first phase is requirements gathering. This foundational step sets the stage for the entire project. In this phase, project managers and teams engage in detailed discussions with stakeholders to gather and document all necessary requirements. The goal is to capture a comprehensive understanding of what is to be achieved by the project.

During this phase, the focus is on creating a detailed requirements document that will guide the rest of the project. This document serves as a blueprint, outlining everything from system specifications to deliverable features. It's essential that this document is thorough and accurate, as any missed requirements can lead to significant challenges later on.

Example:

Consider a project aimed at developing a new customer relationship management (CRM) software. During the requirements phase, the project team would engage with sales teams, customer service representatives, and IT staff to understand the features and functionalities needed in the CRM system. This might include requirements like integration with existing tools, user authentication features, and reporting capabilities.

Phase 2: Design

Once the requirements are clearly defined, the project moves into the design phase. Here, the focus shifts to how the project's requirements will be executed. This involves creating system architecture, design documents, interface designs, and data models.

The design phase is typically divided into two sub-phases: high-level design and detailed design. The high-level design provides an overview of the system architecture and design, while the detailed design delves into the specifics of how each feature will function.

Example:

Continuing with the CRM software example, the design phase would involve creating wireframes for the user interface, deciding on the database architecture, and detailing how different modules of the CRM will interact with one another.

Phase 3: Implementation

With the design documents as a guide, the project team moves into the implementation phase, where the actual development work is carried out. In this phase, developers write code, build features, and integrate systems as per the design specifications.

The implementation phase is often the most resource-intensive, requiring a keen focus on adhering to design documents and requirements. It's crucial that the development team works closely with the design and requirements documentation to ensure that the final product is aligned with stakeholder expectations.

Example:

For the CRM project, implementation would involve developers coding the user interface, setting up the database, and integrating third-party APIs. This phase continues until all planned functionalities are developed.

Phase 4: Verification

Verification, sometimes referred to as testing, is the phase where the developed product is rigorously tested to ensure it meets the initial requirements and design specifications. This phase is critical in identifying any defects, bugs, or deviations from the expected functionality.

Testing can be broken down into various types, such as unit testing, integration testing, system testing, and user acceptance testing (UAT). Each type of testing plays a role in ensuring the product is robust, functional, and ready for deployment.

Example:

In the CRM example, verification might include testing the software's ability to handle large amounts of user data, ensuring integrations with other systems work seamlessly, and confirming that the user interface is intuitive and user-friendly.

Phase 5: Maintenance

The final phase in the Waterfall methodology is maintenance. After the product has been tested, verified, and deployed, it enters the maintenance phase. This phase involves ongoing support, bug fixes, and updates to keep the system running smoothly.

Maintenance is an ongoing process that ensures the product remains functional and relevant over time. It can also involve the addition of new features or enhancements based on user feedback and changing market conditions.

Example:

For the CRM software, maintenance might include releasing patches for any minor bugs, updating the system to accommodate new business processes, or adding new functionalities requested by users.

For some, Waterfall seems comforting, like a security blanket.

The Illusion of Certainty

When it comes to software products, Waterfall only gives an illusion of certainty…

Senior Stakeholders' Perspective

Waterfall can be particularly appealing to senior stakeholders and governance bodies due to its promise of certainty. The model's linear progression and detailed documentation create an illusion that the project is under control, with clear timelines and deliverables. However, this perception often overlooks the inherent uncertainties and complexities of software development, leading stakeholders to underestimate the risks involved.

The Rigidity Trap

Software development happens and evolves at pace. Requirements are bound to evolve. Market dynamics, technological advancements, and customer feedback necessitate continuous adaptation.

Waterfall however, is built on a foundation of rigid sequential phases, where each stage must be completed before the next begins. This rigidity means that any initial plan is likely to become outdated as the project progresses.

Real-Life Example

Consider the case of the U.S. Department of Defence and its Future Combat Systems (FCS) initiative. Initially structured using a Waterfall approach, this project faced significant setbacks as requirements changed over time. The inability to adapt swiftly to evolving technological needs and battlefield realities resulted in the ultimate cancellation of the project after billions of dollars had been invested.

 

The Misconception of Efficiency

Document-Driven Delays

Stakeholders and leadership often perceive detailed requirements documents and rigid project plans as tools to enhance efficiency and reduce costs. However, these artefacts can have the opposite effect. The heavy emphasis on documentation increases the time to market, delaying the realisation of value. Teams become bogged down in creating and maintaining extensive documentation rather than focusing on iterative development and delivery.

The Cost of Over-Documentation

A study by the Standish Group found that large-scale Waterfall projects typically deliver only 42% of the initially planned features. The extensive documentation and upfront planning do not necessarily translate to better outcomes. In contrast, Agile methodologies emphasise working software over comprehensive documentation, allowing teams to pivot quickly as requirements change.

Documentation, documentation and more documentation... that's the waterfall way!

Inhibition of Feedback Loops

Stakeholder and Customer Engagement

One of the significant drawbacks of Waterfall is its limited capacity for integrating feedback. The methodology's rigid phases restrict opportunities for stakeholder and customer input, which are crucial for refining requirements and ensuring the product remains aligned with user needs. This lack of feedback loops can result in a final product that misses the mark, requiring costly revisions post-launch.

One-Sided Stakeholder and Customer Engagement

The Communication Gap

In Waterfall projects, stakeholder and customer engagement often becomes a one-directional communication of a predetermined project plan. This approach emphasises adhering to a set sequence of phases, where requirements are gathered upfront, and subsequent phases focus on design, implementation, and testing. Consequently, there is little room for ongoing dialogue and iterative feedback throughout the project lifecycle.

Example: Healthcare.gov

The rollout of Healthcare.gov (USA) is a well-known example where stakeholder engagement was primarily confined to initial planning stages. The lack of ongoing communication and feedback loops during development led to numerous issues at launch, resulting in a website that was not fully functional and required significant post-launch revisions.

Limited Capacity for Integrating Feedback

Rigid Phases and Feedback Integration

Waterfall's rigid phase structure limits its ability to incorporate feedback. Once a phase is completed, revisiting it is often discouraged due to time and budget constraints. This can result in projects that fail to adapt to new information or evolving stakeholder needs.

Example: Denver Airport Baggage System

The Denver International Airport's automated baggage handling system exemplifies the pitfalls of limited feedback integration. Designed using Waterfall, the project encountered numerous technical issues due to inadequate stakeholder feedback during development. This led to significant delays and cost overruns, eventually requiring a complete overhaul of the system.

Consequences of Inhibited Feedback Loops

Products Missing the Mark

Without continuous feedback, final products may not align with user needs or market expectations. This misalignment can necessitate costly revisions or lead to delays that erode stakeholder trust and project viability.

Example: Nokia's Symbian OS

Nokia's reliance on a Waterfall-like approach for its Symbian OS development contributed to its inability to adapt to the rapidly changing smartphone market. The lack of flexible feedback integration resulted in a product that lagged behind competitors in terms of features and user experience, ultimately impacting Nokia's market position.

Lengthened Delivery Timelines

Excessive Documentation and Governance

Waterfall's emphasis on detailed documentation and governance processes can significantly extend delivery timelines. While thorough documentation is valuable, the model often leads to excessive paperwork that slows down progress. This bureaucratic approach can delay the development process, hindering the ability to respond quickly to changes or new insights.

Delayed Realisation of Value

The Cost of Lengthy Requirement Phases

In product development, the realisation of value is achieved by taking a product to market and meeting customer needs. However, Waterfall's extensive requirement gathering phases result in significant delays before any value is realised. The protracted upfront planning can lead to wasted resources and missed market opportunities.

Example: Blackberry's Decline

Blackberry, once a leader in the smartphone market, suffered significant losses partly due to prolonged development cycles. Their commitment to long requirement phases delayed product launches, allowing competitors like Apple and Android to capture market share with more agile and iterative approaches.

Market Fit and Iterative Design

Loss of Market Fit

Market dynamics are continually evolving, and a product developed under Waterfall's rigid timelines may lose market fit by the time it is released. The static nature of Waterfall does not accommodate changes in customer preferences or competitive landscapes.

Case Study Comparison: Waterfall vs. Design Thinking

IDEO's design thinking approach contrasts sharply with Waterfall. Design thinking involves iterative, customer-focused methodologies that prioritise prototyping and feedback, ensuring products meet real user needs. In contrast, Waterfall's requirement gathering is detached from user interaction, risking a mismatch with market demands.

Fictional Example: The "NextGen" Tablet

Imagine a fictional product, the "NextGen" Tablet, developed using Waterfall. By the time it launches, it lacks features that competitors have already introduced, such as enhanced AI capabilities and seamless integration with smart home systems. Had the product followed an iterative, customer-focused approach, it might have aligned with market trends and user expectations, ensuring better market fit.

The Burden of Excessive Governance

Adding Time Through Governance

Waterfall's rigid governance structures necessitate extensive documentation and adherence to predefined processes, which add time to the development cycle. These layers of bureaucracy can hinder swift decision-making and adaptability.

Distrust and Delays

Project teams working under Waterfall may feel distrusted by their organisations due to the rigidity and constant oversight. This perception leads to longer task completion times as teams meticulously document every detail to avoid blame for any discrepancies in timelines.

Buffering Due to Accountability Pressures

The over-reliance on documentation creates a culture of caution, where teams add additional buffers to their timelines to protect against potential overruns. This unnecessary padding further extends delivery times and delays market entry.

Products Managed as Projects

Lifecycle Consideration

In a Waterfall-driven environment, products are often managed like projects, with a focus on completing a set series of tasks rather than considering the entire product lifecycle. This project-centric mindset can lead to inadequate products that fail to evolve with market demands or user expectations. Without ongoing iteration and improvement, the software may quickly become obsolete or fail to deliver long-term value.

The Product Lifecycle vs. Finite Projects

Understanding the Distinction

A product lifecycle comprises multiple phases, including conception, development, launch, growth, maturity, and eventual decline or renewal. Unlike projects, which are designed to achieve specific objectives within a set timeframe, products require ongoing iteration and improvement to remain relevant and competitive. Treating a product like a project fails to account for the dynamic nature of market demands and customer needs.

The Waterfall Approach and Its Implications

The Waterfall methodology is inherently suited for project management due to its linear, sequential process. It emphasises upfront requirement gathering, followed by design, implementation, testing, and maintenance. While this approach can provide clarity and predictability in project management, it is ill-suited for managing products that require flexibility and adaptation.

The Stop-Start Nature of Waterfall in Product Management

Lengthy Requirement Definition and Ramp-Up

The Waterfall approach often involves extended periods of requirement definition and project ramp-up. This not only delays the initial launch but also results in a stop-start dynamic throughout the product lifecycle. By the time a product reaches the market, initial assumptions may already be outdated, necessitating significant rework and adjustment.

Example: Kodak's Digital Camera

Kodak's journey with digital cameras highlights the pitfalls of treating products as projects. Initially approaching digital camera development with a Waterfall mindset, Kodak failed to adapt quickly to technological advancements and market shifts. This stop-start approach, characterised by lengthy planning and execution phases, ultimately led to missed opportunities and a loss of market leadership.

The Need for Continuous Adaptation

Importance of Ongoing Iteration

For products to succeed, they must continuously evolve in response to market trends, technological advancements, and user feedback. This requires an adaptive approach that prioritises iteration, customer engagement, and rapid prototyping. Unlike Waterfall, methodologies like Agile or Lean enable teams to make incremental improvements and pivot based on real-time insights.

Risk of Failure Without Adaptation

Without continuous adaptation, products risk becoming obsolete or irrelevant. In fast-paced industries, a lack of responsiveness can lead to declining user engagement, reduced market share, and eventual failure. Organisations that fail to recognise the necessity of treating products as living entities rather than finite projects are likely to struggle in maintaining long-term success.

 

Increased Development Costs

Vendor Relations and Change Requests

Working with vendors under the Waterfall model can lead to escalating development costs, particularly through the mechanism of 'change requests.' As requirements evolve or new needs are identified, requesting changes can be cumbersome and expensive. Vendors may capitalise on these changes, increasing project costs and causing budget overruns.

The Cost of Change

In the Waterfall model, requirements are typically defined at the project's outset. However, as projects progress and new needs emerge, changes to these requirements become inevitable. Each change request under Waterfall can trigger a formal process, often involving extensive review, approval, and implementation stages. This cumbersome process not only delays timelines but also increases costs, as vendors may charge premiums for making these changes outside the initial scope.

Vendor Capitalisation

Vendors engaged under the Waterfall model may capitalise on change requests to increase their revenue. Since many contracts are structured around initial requirements, any modifications can result in additional charges. This practice can lead to significant budget overruns, complicating financial planning and resource allocation for the organisation.

Example: Public Sector IT Projects

Public sector IT projects often exemplify the financial strain caused by change requests under Waterfall. These projects, characterised by complex requirements and evolving needs, frequently experience cost escalations as vendors charge for each alteration, leading to significant budgetary challenges.

Overlooked 'Establishment Costs'

Ramp-Up Costs and PMO Establishment

One of the hidden costs often overlooked in Waterfall projects is the expense associated with ramping up a Project Management Office (PMO). Establishing a PMO involves significant investment in personnel, infrastructure, and processes to manage and oversee the project. These costs are incurred long before actual development work begins, contributing to the overall budget.

The Burden of Upfront Documentation

The Waterfall methodology places a strong emphasis on comprehensive upfront documentation. This includes detailed requirement specifications, design documents, and project plans. The creation of this documentation requires substantial time and resources, representing a considerable portion of the project's initial budget. Organisations often underestimate these costs, viewing them as necessary for project initiation rather than ongoing expenses.

False Economy of Perceived Cost Efficiency

Organisations adopting Waterfall may perceive it as a cost-efficient methodology due to its linear and predictable nature. However, this perception is challenged by the reality of high establishment costs and the financial burden of managing change requests. The focus on upfront planning and documentation can create a false sense of security, masking the true cost implications that emerge during the project's lifecycle.

Operating in Waterfall isn't necessarily any cheaper than other methodologies

The Decline of Waterfall in Favour of Agile

Embracing Agile Methodologies

The limitations of Waterfall have contributed to its decline in favour of Agile methodologies, which prioritise flexibility, collaboration, and continuous improvement. Agile's iterative approach allows for regular feedback, rapid adaptation, and a focus on delivering value throughout the product lifecycle. This shift reflects a broader recognition among software development teams and product managers that Agile better addresses the dynamic nature of software projects.

While the Waterfall methodology offers a structured approach that can be initially attractive, its drawbacks pose significant risks to successful software development. By understanding these dangers and embracing more adaptive methodologies like Agile, organisations can improve their ability to deliver high-quality software that meets stakeholder and customer needs.

Why Organisations Cling to Waterfall Project Management Despite Its Pitfalls

In the realm of project management, the Waterfall methodology has long been a staple—praised for its structure and predictability. However, as the pace of business accelerates, many organizations struggle to adapt to the dynamic needs of modern markets using this linear approach. Despite its limitations, many companies remain steadfast in their commitment to Waterfall, viewing it as a beacon of certainty and security amidst the chaos of change.

The Allure of Certainty

At its core, the Waterfall methodology offers a linear, step-by-step process that promises predictability. For many organisations, this translates to a sense of security. With clearly defined phases and milestones, leaders can feel assured that projects will proceed according to plan. This sense of control is particularly appealing to stakeholders who are uncomfortable with the ambiguity inherent in more flexible approaches, such as Agile.

The Agile-Waterfall Conundrum

In today’s fast-paced environment, Agile development has become synonymous with innovation and adaptability. Agile teams thrive on flexibility, rapid iteration, and continuous feedback—factors that enable them to respond swiftly to market changes and customer needs. However, when these teams operate within organisations firmly rooted in Waterfall, they often face a unique challenge: the need to overlay a "waterfall" framework on their Agile processes.

This hybrid approach aims to bridge the gap between the rigidity favoured by management and the adaptability required by development teams. By incorporating Waterfall elements into their plans, Agile teams strive to reassure leadership and stakeholders by providing the structure and predictability they desire.

 

Challenges of the Hybrid Approach

1. Dilution of Agile Principles: Adding a Waterfall layer can dilute the core principles of Agile, potentially stifling innovation and flexibility. Agile’s strength lies in its ability to pivot based on real-time feedback, a capability that can be compromised when constrained by rigid Waterfall phases.

2. Increased Complexity: Managing a dual-layer approach can lead to increased complexity and confusion. Development teams may find themselves juggling conflicting priorities, trying to meet the demands of both Agile and Waterfall methodologies.

3. Resource Strain: The hybrid model can strain resources, as teams are required to produce additional documentation and reports to satisfy Waterfall requirements, detracting from the time and energy needed for actual development work.

   
   

Bridging the Gap

To successfully navigate the tension between Waterfall and Agile, organisations can adopt a few strategic measures:

• Educate and Communicate: Foster an organisational culture that values education and open communication. By helping stakeholders understand the benefits and limitations of both methodologies, organisations can promote more informed decision-making.

• Tailor the Approach: Customise the project management strategy to fit the specific needs of the project and the organisation. This could mean using Agile for certain projects while applying a more Waterfall-like structure to others, depending on the context.

• Incremental Change: Introduce Agile principles gradually, allowing stakeholders to become comfortable with the increased flexibility. Demonstrating success through small wins can help build trust in Agile methodologies.

While the lure of certainty keeps many organisations tethered to the Waterfall methodology, the evolving business landscape demands a more nuanced approach. By understanding the strengths and weaknesses of both Waterfall and Agile, and adopting a tailored strategy that incorporates elements of each, organisations can better satisfy the diverse needs of their leadership and development teams. This balanced approach not only enhances project success but also positions companies to thrive in an ever-changing market.

Well done if you made it this far!

Yes.... This article on Waterfall was extra long to prove a point.... At some point, the over-reliance on documentation with Waterfall inhibits one's ability to understand and stay focussed!