The Bad and the Ugly of Agile

Deprecation of Upfront Tasks
The prize undisputedly goes to the deprecation of "upfront" activities, in particular upfront requirements and upfront design.

Agile criticism of "Big Upfront Anything" includes some perceptive comments. It is true that one cannot fully comprehend requirements before the development of the system; that requirements will change; that the architecture will have to be improved as implementation proceeds. Those observations express some of the fundamental difficulties of software engineering, and the futility of trying to define everything at the beginning.

There is, however, no argument for shunning the normal engineering practice — the practice, in fact, of any rational endeavor — of studying a problem before attempting to solve it, and of defining the architecture of the solution before embarking on the details. The alternative proposed by agile methods is an ad hoc approach: identify some functionality, build it, assess and correct the result, repeat. It is no substitute for serious requirements and design.

Iterative development is great. Trying out ideas on a small scale before you make final decisions is great. Treating requirements as a living, changeable product is great. Reassessing design decisions on the basis of results is great. Insisting on regular deliveries (once the basic structure is in place) is great. Refactoring is listed at the end of this chapter as one of the significant contributions of agile methods. None of these ideas justifies forsaking the initial tasks of analysis and design.

In other cases we can see the pros and cons of agile ideas. Here there is no place for equivocating: neglecting these upfront steps, as agile authors advocate, is guaranteed to harm your development.


User Stories as A Basis for Requirements
User stories play a useful role as ways to check the completeness of requirements, but to use them as the basic form of requirements means forsaking abstraction. In addition, they ignore the critical Jack-son-Zave distinction between the machine being built and the domain that constrains it.

The resulting systems are narrowly geared to the specific user stories that have been identified; they often do not apply to other uses; and they are hard to adapt to more general requirements.

User stories are no substitute for a system requirements effort aimed at defining the key abstractions and the associated operations (the domain model) and clearly separating machine and domain properties.


Feature-Based Development and Ignorance of Dependencies
A core idea of agile methods is that you can treat software development as a sequence of implementations of individual features, selected at each step on the basis of their business value. It would be great if such an approach were applicable, but it exists only in a land of make-believe. Difficult projects do not lend themselves to this scheme: they require foundational work (building core architectural elements, such as a communication or persistence layer) which extend across features; and the features interact through dependencies, causing complexity of the "multiplicative" kind.


Rejection of Dependency Tracking Tools
The potential complexity of feature interactions requires a careful analysis of task dependencies; projects can skip this analysis only at their own risk. The advice to stay away from Gantt charts and dependency-management tools is not only naïve but detrimental. Such tools are not a panacea for project management but have proved their value over several decades of use. They can help agile projects just as well; dogmatic rejection of useful tools is a self-inflicted wound.


Rejection of Traditional Manager Tasks
The self -organizing teams promoted by agile methods, with no manager having the traditional duty of assigning tasks, are the best solution for a few teams, and are inappropriate for many others. The picture of the manager as an incompetent control freak is a caricature. Many software projects have been brought to completion, and many projects on the brink of failure have been rescued, through the talents of a strong manager. Imposing a single management scheme on everyone is arrogant.


Rejection of Upfront Generalization
Agilist rightly note that the primary responsibility of a project is to deliver working software to its customers, and that too much early concern for extendibility (ease of change) and reusability (applicability to future projects) can hinder that goal, especially since it is not always clear initially in what direction the software will be extended and which parts will need reuse. But these observations are not a reason to reject the concept of generalization altogether. We have seen that such an attitude directly contradicts the professed agile principle of "welcoming change". Good software developers do not wait for change to happen: they plan for it by designing flexible architectures and solving more than the problem of the moment.


Embedded Customer
The XP idea of a customer representative embedded in the development team does not work well in practice, for reasons explained in an earlier discussion. The Scrum notion of a product owner, however, figures below in the list of excellent ideas.


Coach as a Separate Role
The Scrum idea of a dedicated Scrum Master is good for Scrum, but not appropriate for most projects. Good development requires not just talkers but doers.


Test-Driven Development
Test-first development, and the requirement of associating a test with every piece of functionality, appear in the lists of good and excellent ideas below. So does refactoring.
Test-driven development is another matter. A software process defined as the repeated execution of the basic steps of TDD — write a test, fix the code to pass the test, refactor if needed — cannot be taken seriously. With such an approach one is limited to tunnel vision, focused on the latest test. An effective process requires a high-level perspective, considering the entire system.

While test-driven development is extensively discussed in the literature, industry has made its choice: it is not broadly practicing this technique. (On the other hand, many companies have adopted user stories. One may only hope that they will realize that replacing requirements by user stories is the same as replacing specifications by tests.)


Deprecation of Documents
Agile criticism of document-heavy processes that produce little real customer benefit is right on target for some segments of the industry — although in some cases, such as missioncritical systems, little can be done about the situation since the documents are legally required by certifying agencies. (And not just out of bureaucratic inertia. Even the most enthusiastic agilist might feel, when flying to the next agile conference, that it was not such a bad idea after all — not total "waste"— to assess the plane's software against a whole pile of certification standards.)

Outside of specific industries with high regulatory requirements, a strong case exists for lightening up the document infrastructure. It is true, as agilists emphasize, that "design" in software is not as remote from production (implementation) in other engineering fields. Modern programming languages help, because they make it possible to include some of the traditional design in the code itself. (Some of my own work has addressed this issue.) None of these observations, however, can justify the deprecation of upfront plans and documents. Software engineering is engineering, or should be, and sorely needs the benefits of a careful predictive approach, as well as the supporting documents.