From Agile Pathologies to Agile Health
In my agile-lean coaching, training, and consulting engagements, I often come across software development teams and projects that claim to follow an agile development methodology, usually Scrum. When I dig deeper, I often find that many of those team are suffering from agile pathologies – incorrect or unhealthy ways of practicing agile methods. Four pathologies, illustrated in Figures 1 through 4, are quite common. After presenting these pathologies, I will diagnose three root causes for them, and offer the right way for agile development.
Pathology 1: Agile development “sprints” assigned to software development lifecycle phases
In this pathology shown in Figure 1, Sprints 1 through 4 are dedicated to typical waterfall life cycle phases: Design and Code development (Sprint 1), QA testing (Sprint 2), Defect fixing and Regression testing (Sprint 3), and Packaging, Delivery and Deployment (Sprint 4). Design and code development for all features is scheduled for Sprint 1 timebox; QA test cases are developed and run for all features in Sprint 2 timebox; all defect fixing and regression testing is done in Sprint 3 time box. By the end of the Sprint 4 timebox, working software may be delivered and deployed if everything goes well.
Why is this pathology and not real agile development? After all, the team is working in fixed time-box iterations, which it proudly even calls “sprints”! This is pathology for the simple reason that in each sprint or time-box, the team is not producing potentially shippable software. The team may deliver working software only after 4 time boxes (if everything goes well as planned). Note that the cycle time for each working feature is 4 sprint timeboxes long! Just by splitting a release cycle into time-boxed lifecycle phases does not make it agile development. It will not deliver many of the benefits of agile development, such as working software after each sprint, rapid customer feedback after each sprint that guides future development, short cycle times for features (at most a single sprint timebox), predictability, etc.
There are variations on the basic pathology shown in Figure 1. For example, the defect fixing effort can be combined with the QA testing effort in Sprint 2. This can happen while regression testing is carried out in Sprint 3 or in additional sprints as regression testing is sadly often not automated. The time it takes for regression testing to complete continually grows as more and more prior functionality needs to be regression tested manually. Deployment in Sprint 4 may be assigned an additional sprint of its own, etc. This variation on Figure 1 is still pathology as working software may be delivered at best the end of Sprint 4 timebox (not at the end of each sprint timebox). Any other variation on the pathology of Figure 1 would still be pathology.
Pathology 2: Agile development “sprints” assigned to technology tiers
In this pathology shown in Figure 2, Sprints 1 through 4 are dedicated to technology tiers: User Interface development (Sprint 1), Business Logic development (Sprint 2), Data Layer development (Sprint 3), and Integration, packaging, delivery and deployment (Sprint 4). User Interface analysis, design, development and testing for all features is scheduled for Sprint 1 timebox; Business Logic analysis, design, development and testing for all features in Sprint 2 timebox; Data Layer analysis, design, development and testing for all features in Sprint 3 time box. By the end of Sprint 4 timebox, working software may be delivered and deployed if everything goes well.
Why is this pathology and not real agile development? This is pathology for the simple reason that in each sprint or time-box, the project team is not producing potentially shippable software. The project delivers working software only after 4 time boxes (if everything goes well as planned). Just by splitting a release cycle into time-boxed technology tiers does not make it agile development. It will not deliver many of the benefits of agile development as explained in Pathology 1.
There are variations on the basic pathology shown in Figure 2. For example, Integration, Packaging, Delivery and Deployment effort (Sprint 4) may be separated into two or more sprints. Any variation on the pathology of Figure 2 would still be pathology as there is no working software at the end of each sprint.
Pathology 3: Mini-Waterfall inside sprints
In this pathology shown in Figure 3, a team has 4-week sprints (as an example) where weeks 1 through 4 are dedicated to analysis (week 1), design (week 2), code development (week 3), and testing, defect fixing, packaging and delivery (week 4). In effect, analysis, design, code development, and testing, defect fixing, packaging and delivery “phases” happen sequentially inside a so-called “sprint” creating a mini-waterfall. Analysis work for all features scheduled for the sprint is scheduled for week 1, followed by design work for all features in week 2, followed by code development work for all features in week 3, followed by testing, defect fixing, packaging and delivery of all features in week 4. First, it is unrealistic to expect that these efforts can all work out exactly on nice weekly boundaries.
More importantly, agile development methodology requires and encourages you to have a cross-functional, self-organized team to work on a rank-ordered backlog of features for a sprint, to have much tighter integration among design, code development, testing and defect fixing for each feature; and lean development further requires you to focus on minimizing the cycle time for features.
In this pathology, as all these guidelines are ignored, working features will be delivered at best at the end of week 4 if everything goes well. There is a high risk that a number of features will be in testing or defect fixing state even by end of week 4. The earliest the team would know if it is implementing the feature incorrectly might be 3 weeks delayed, while in proper agile-lean development this discovery will happen much sooner reducing the waste.
Pathology 4: Water-Scrum-Fall
In this pathology shown in Figure 4, a team indeed follows agile development method during the software development release cycle with four sprints. In each sprint (sprint 1 through 4) working software is developed, with concurrent design, code development and testing activities going in tight loops. However, prior to Sprint 1, there is a rather large phase spanning several months for developing and analyzing requirements and their sign-off. Similarly, at the end of Sprint 4, there is a rather large phase spanning several months for packaging, delivery and deployment of software. In other words, the Scrum-based software development release cycle (Sprints 1 through 4) is preceded by a Requirements Analysis phase and followed by a Packaging, Delivery and Deployment phase, hence the apt name for it Water-Scrum-Fall.
This is pathology because all requirements for the entire release cycle are drafted, documented and signed-off right upfront losing the benefits of sprint-by-sprint feedback. Although every sprint results in “working” software, it works only in development or QA environment. It is not delivered until end of Sprint 4, which will very likely cause unpleasant surprises and deployment issues related to the code that was developed as early as Sprint 1. Any variation on this pathology would still be pathology.
Root causes for agile pathologies
In my experience, there are three major root causes for agile pathologies:
- Lack of real understanding of agile development: Although teams may not have a total lack of understanding of agile development, they often suffer from superficial understanding, partial understanding, or misunderstanding. For example, a team or its manager may genuinely believe that it must try one of the 4 approaches illustrated in Figures 1 through 4 in the blog at least as an interim step in its transition from the current waterfall or ad hoc method to agile nirvana. The team or its manager may not realize that such an interim step is nothing but pathology. It will do only harm. This root cause (lack of real understanding) can be remedied with proper training, consulting and coaching.
- Implementation difficulties: A team may have good understanding of the right agile development process (discussed shortly in this blog), but may find it difficult to practice it. This may be due to organizational impediments, cultural issues, or change management issues related to changing from the current waterfall or ad hoc development process to a proper and healthy agile development process. Often team members are split across organizational silos (such as Analysts vs. Development vs. QA silos, or Technology tier silos) without much of a cross-functional, self-organized agile team behavior. This situation is often exacerbated with a team split between developers in one location (often the US or headquarter) and testers in a lower-cost off-shore center or a contractor vendor. Managers of these organizational silos may drag their feet and may not fully support proper agile development process. Or the managers may genuinely want to transition to proper agile development, but struggle with the goal. Good on-site coaching will help to address this root cause.
- Lack of systems thinking: Different stakeholders (managers, ScrumMasters, Product Owners, software developers, testers, technical writers, analysts, etc.) may be trying to locally optimize from their narrow perspectives and interests, but missing the holistic whole systems view that requires them to focus on delivering highest value features in short sprints to customers to get their feedback for iterative, incremental delivery of working software. This root cause can be remedied also with proper training, consulting and coaching.
What is the right way for agile development?
As shown in Figure 5, the right way for agile development requires a team to deliver potentially shippable software at the end of each sprint of a release cycle, with analysis, design, and code developing and testing activities all going on concurrently inside each sprint (these activities are not executed as a mini waterfall). Lean development further requires maximizing the value flow, minimizing the waste, and reducing or minimizing the cycle time for each feature.
This way for agile-lean development creates its own challenges, both at change management level as well as operational level. And at each level, the challenges can be either technical or cultural or both. I will explain the challenges and potential solutions in a series of future blogs (how to transition from agile pathologies to agile health). Stay tuned.
Have seen or experienced any of the agile pathologies (or any variations thereof) discussed in this blog? I would love to hear from you either here or by e-mail (Satish.Thatte@VersionOne.com) or hit me on twitter (@smthatte).