PRE-AGILE ERA

First a timeline is presented based on the computer technology in use and the nature of the software development during it. The early age of computers were the time when the sequential computing was the dominant way of processing data. That period then turned to what is here called the Evo-IID period.

The second view is that of the process and engineering cultures. The word culture is used here in the meaning of "the set of shared attitudes, values, goals, and practices that characterizes an institution, organization, or group" (Wikipedia, 2012). The process culture refers to people in software organizations whose main interest is in the work steps and the work results, and managing those steps and results. The engineering culture refers to those whose main inters are in the software engineering methodologies and the ways of doing tasks, and using methods, techniques and tools for doing them. Iivari, Hirschheim and Klein (1998) specifies methodology as "an organized collection of concepts, methods, beliefs, values and normative principles supported by material resources", and "a technique or method, in this context, consists of a well-defined sequence of elementary operations which permits the achievement of certain outcomes if executed correctly". Further, making distinction between the method and the technique is based on the definition in Oxford dictionaries (2012), where the method is defined as "a particular procedure for accomplishing or approaching something, especially a systematic or established one: the quality of being well organized and systematic in thought or action". The technique is specified as "a way of carrying out a particular task, especially the execution or performance of … a scientific procedure". Particularly, the definition of the method contains an element of though allowing several interpretations of a topic, whereas the technique is more execution of a procedure in one and correct manner. In the following text the process culture contains management methodologies, methods, techniques and tools, and the engineering culture contains engineering methodologies, methods, techniques and tools.


The Early Age of Computers

The early age of computers, when viewed from the software development point of view can be characterized as technology being unreliable, software of limited size, processing in sequential batch mode, and software engineering as writing programs for straightforward processing of data. The focus of the software development was in the engineering. During 1970s' a large number of methodologies were published. The structured programming was based on the principle of top-down design (Wirth, 1971; Mills, 1971; Dahl, Dijkstra and Hoare, 1972; Parnas, 1972; Stevens, Myers and Constantine, 1974; Myers 1975; Yordon & Constantine, 1975). These were largely program-focused methodologies, but similar system design methodologies were made available as well, which included means to design user tasks. ISDOS was developed in the university of Michigan, and SADT in the company called Softech. Also the ISDS/HOS methodology (Hamilton & Zeldin, 1976) was published. They all were based on the principle of reducing large problems into smaller problems, solving the small, and then build them up as the solution of the large problem. In addition to this Warnier (1974) and Jackson (1975) presented their different approaches to the program design. Logical construction of programs (LCP) and Jackson Structured Programming (JSP) were based on the analysis of the input and output data and nearly algorithmic construction of programs.

Overall, during the 1970's the engineering methodologies became very popular, if not a hype of that time. However, the human factors in the software development got attention as well. IBM was applying the chief programmer team approach (Baker, 1972). Brooks (1975) wrote the book "Mythical Man Month" which became so popular that 1995 an anniversary print of the book was taken. Further, in a research by Ramsey, Atwood and Campbell (1979) "several major human factors deficiencies and problems were identified. Formal software design methods differ in terms of: Applicability to problems of different types, size or complexity; susceptibility to design errors; and constraints and limitations imposed on the software designer. Various methods limit the designer's ability to select an appropriate problem representation, prevent the designer from utilizing relevant knowledge and experience, or impose potentially significant information loads on the designer." One of their recommendations was to consider different type of methods for different type of problems.

The process culture focused on the application of the project management to the software development. Many of the tools for the project management were already available, like the PERT charts and the critical path method, the Cantt charts, and the work break down structures. So the large projects started to apply them in practice. In 1970 Royce (1970) wrote about management of the software development, which is often interpreted as the basis of the waterfall model. The other influential document was the military standard DOD-STD-2167 published on the 4th June 1985, and its more referred update DOD-STD-2167A (DOD 1988) from 29th of February 1988. They were designed to guide the development of the US defense area projects, but were adopted by many other companies as well. When the more advanced real time systems were emerging the projects got more complex, too. The trailblazers were the military organizations that were several years ahead of the business organizations. The software development was closely integrated with the development of the hardware that was on the latest edge of innovation. There were significant controversy between the very settled project approach described in these standards and projects that progressed in interactive and incremental manner via experimentation (Larman & Basili, 2003). Contractually projects followed the standard process, but several successful cases of IID were also reported.

These successes of the IID did not, however, lead to an avalanche of it in the software development. On the contrary, many military area projects and most of the civil business area projects ended up using the waterfall model. That is, in the strict order (1) specify the scope, (2) approve the scope, (3) sign the specification project, (4) define the detailed content, (5) define the detailed project plan with resource, cost and time, (6) sign the project contract, (7) execute the project contract, (8) approve the project results, and (9) conclude the contract. This description of steps deviates the more commonly presented steps: define, design, implement, test, commission by brining in the decision and contract management points. It is not possible to firmly indicate that the ease of the business decisions and the accustoms to the traditional project management practices would have overridden the technical experience but clearly the waterfall model has been more appealing to the business management than the software development staff, from very early on. The waterfall model became the dominant approach for developing software.

In summary, the 1970's could be called a period of ordered work by disciplined people. The ordered work means doing the defined engineering tasks using the defined engineering methods, techniques, and tools. The disciplined people means executing the work in the planned and controlled manner. This is not the whole picture though. In several occasions and forums the evolutionary and incremental and iterative principles and practices were presented (Larman & Basili, 2003). It was the battle of which one is right, not of which is applicable to what.


The Evo-IID Period

The Evo-IID period was characterized with the growth of size and complexity of the systems to be built. Particularly the US defense industry was facing the challenge. The path taken to solve this problem led to ever-increasing process and management discipline that culminated in the process maturity culture. Particularly the 1990's were heavily process centric. The process maturity and models like CMM (Paulk, Curtis, Chrissis and Weber, 1993a 1993b) and Spice or ISO15504 (ISO/IEC, 1997) were overly popular. All software organizations felt the pressure to the maturity driven process improvement (Haley, 1996; Clark, 1999; Li, Chen & Lee 2002). Models for smaller companies were developed as well (Coleman & Verbruggen 1998). The second corner stone was the project management models, the most notable reference being the Project Management Body of Knowledge (PMBOK) by Project Management Institute (PMI 1994, 1996, 1999, 2004, 2008).

The third set of processes can be found in the product line/family management (SEI/SPL, 2012; ESI/SFE, 2012) models. They extend the two previously mentioned models to cover the management of the whole product development organization. The fourth cornerstone of the process centricity is the IT infrastructure Library (ITIL, 2012) by the Office of Government Commerce (OGC), UK which has widely spread outside UK, and influenced on other standards as well. It is similar to the product line models but covers the management of the organizations internal IT infrastructure investments. Continuous assessment and improvement of processes are a part of all of these models.

Some doubts have been presented if the desire to have the maturity of processes was exceeding the capability of organizations to actually achieve them (OConnell & Saiedian, 2000). Having all processes defined was becoming more important than having them fully operational and beneficial. It is often said that the agile avalanche was the counter reaction to this stiffening process culture that the agile approach was born to cure the process paralysis (Biske, 2006; Whitten, 2008).

However, the Evo-IIP period contributed to the agile approach much more than just providing the process culture as a target to shoot at. Many of the core principles of the agile development were promoted already during the 1970's and 1980's, and even more intensively during the 1990s when the process culture was on its peak. The pre agile age had indicated the significance of uncertainty and the need for innovation during the development, not just before it. The two hallmark models from that time were the spiral model by Boehm (1986) and the evolutionary development approach defined by Gilb (1989). On top of these two, the overall list of prominent names contributing to the incremental and iterative development is long (Larman & Basili, 2003). In addition, many successes are pointed out in their article, some in very large projects.

The next step paving the agile path was the boom of Rapid Application Development (RAD), described by many, but most notably in the book by James Martin (Martin, 1991). One list of the characteristics which distinguishes a RAD project from traditional systems development projects at that time, can be found in (Hull, 1995):

- Significant business user involvement.

- The use of 'time boxes' to manage the overall project.

- A rigorously applied methodology.

- The use of automated design and build tools (CASE) to create prototypes.

- A limited number of system developers relative to the number of business users involved in the project, and significantly out of proportion to the scale of the project in business impact terms.

- An almost evangelical zeal, which you can feel within the organization as you walk through the door regarding the use of these techniques. This results in a much higher degree of user involvement than in a traditional development and consequently a RAD project approaches a PC systems development in terms of user involvement.

The RAD approach itself was developed rapidly as well. "As a generalization, we might propose that RAD as a development approach currently lacks a firm underpinning of systematically verified development principles. A substantial amount of practitioner material has been produced on RAD, but little systematic theoretical and empirical assessment of RAD approaches has been undertaken" (Beynon-Davies & Holmes, 1998).

The engineering culture saw the growth of the formal engineering methods, for the requirements specification, for the architecture design, and for the software and program design. Object orientation in the program design brought a major change to the previous procedural software design and coding. They were supported with tools. Modeling and CASE-tools (Computer Aided Software Engineering) were covering the development work from the conception to the code generation and testing. The integrated CASE (ICASE) environments, the integrated project support environments (IPSE) and the like were introduced. RAD and the jointed application development (JAD) (Carmel, Whitaker and George, 1993) were challenging the engineering method development as well, as too heavy. Discussion of the lightweight development led to reduction of the design effort and the use of design methods, and increased use of prototyping (Gottendiener, 1995). The special room as defined for JAD (Carmel et al 1993) was relying more on customers and developers working together than a strong design process.

Source of Information : Agile and Lean Service-Oriented Development