Agile and agile methodologies

I have been working with an agile team, using Scrumban, for the last year, building APIs and microservices. It occurred to me that most of the guys around me have not read anything about agile, beyond some basic articles, and not even the agile manifesto or the Scrum guide. I therefore decided to level up on some training, to match the experience I had so far. The intent was to see if what I had picked up so far was true to the spirit of agile, and to see what other methodologies offered.

Training courses and certifications

I recently attend a 3 day course called the Scrumstudy Agile Master (SAMC), and writing the Agile Expert Certified (AEC) exam, which was very interesting. It covered a whole range of agile methodologies including Scrum, Kanban, Lean, XP, TDD, DSDM, Crystal and FDD.

While its not on the list of the most well-recognized agile certifications, which are those developed by the Scrum Alliance, Scrum.org, and PMI. Scrum Alliance was founded in 2002 by Ken Schwaber, who left in 2009 to start Scrum.org. Newer entries include those offered by ICAgile and Scaled Agile. Here are the top 5 Agile certifications which are the best for Agile professionals who want to build their career with Agile methodology.

1. PMI-ACP (Agile Certified Professional)
2. Scrum Alliance (Certified Scrum Master/Certified Scrum Product Owner/Certified Scrum Developer)
3. Scrum Alliance (Certified Scrum Professional)
4. Scrum.org (Professional Scrum Master/Professional Scrum Product Owner/Professional Scrum Developer -1)
5. SAFe Scaled Agilists

But keep in mind the value of the certification is in the learning, not the cert itself.

How did agile start?

Adaptive Management is a structured, iterative process of robust decision making in the face of uncertainty, with an aim to reducing uncertainty over time via system monitoring. In this way, decision making simultaneously meets one or more resource management objectives and, either passively or actively, accrues information needed to improve future management. Adaptive management is a tool which should be used not only to change a system, but also to learn about the system. Because adaptive management is based on a learning process, it improves long-run management outcomes

The origin of adaptive management can be traced back to ideas pioneered by Frederick Taylor in the early 90s:

The use of adaptive management techniques can be traced back to peoples from ancient civilisations. For example, the Yap people of Micronesia have been using adaptive management techniques to sustain high population densities in the face of resource scarcity for thousands of years. In using these techniques, the Yap people have altered their environment creating, for example, coastal mangrove depressions and seagrass meadows to support fishing and termite resistant wood.

The ideas behind feature-driven (incremental) approaches started at least in the 90s.

If you haven't read Weinberg's Quality Software Management series, or McConnell's Rapid Development, never mind Smith and Reinertsen's Developing Products in Half the Time, you might not realize how early the agile approaches started. I read Fast Cycle Time when I program-managed for a client.

By now most of us know that a group of experienced guys in the software industry got together in 2001 to discuss the state of delivering software, and to come up with a better way to do this.

Agile software development comprises various approaches to software development under which requirements and solutions evolve through the collaborative effort of self-organizing and cross-functional teams and their customer(s)/end user(s). It advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages rapid and flexible response to change.

The term agile (sometimes written Agile) was popularized, in this context, by the Manifesto for Agile Software Development. The values and principles espoused in this manifesto were derived from and underpin a broad range of software development frameworks, including Scrum and Kanban.

The result was the Manifesto for Agile Software Development, with its 4 values and 12 principles. But what I have realised is that this "agile movement" really started before then.

The authors of the Agile Manifesto wanted to help the software community succeed. They had seen—in their work—that lightweight approaches, based on finishing value and getting feedback worked much better than any process-heavy approach.

Think about that for a minute. These people were practitioners. They saw the death marches in projects. They saw that the lack of team-based work made everything more difficult. They saw that without continuous learning, we would repeat the same problems again and again.  And, the thing that surprised me the most when I read the manifesto was the integration of the customer into the team.

Of the 17 original signatories of the Agile Manifesto, most of them, before 2001, had already come up with better ways to deliver software, over using the traditional waterfall approach:

• Kent Beck: developed XP in 1999. He developed extreme programming during his work on the Chrysler Comprehensive Compensation System (C3) payroll project. Beck became the C3 project leader in March 1996. He began to refine the development methodology used in the project and wrote a book on the methodology (Extreme Programming Explained, published in October 1999)
• Alistair Cockburn: developed Crystal in the 1990s. In the early 1990s, Alistair Cockburn worked at IBM and was tasked with building a methodology for object-oriented development. He interviewed successful teams and found that, although they weren’t following a formal methodology, they shared some common ways of working. From this research, he constructed a family of methodologies, along with principles for tuning them. He called this family of methodologies Crystal.
• Ken Schwaber and Jeff Sutherland: developed Scrum in 1995: Ken worked with Jeff Sutherland to formulate the initial versions of the Scrum framework and to present Scrum as a formal process at OOPSLA'95
• Lean can be traced back to 1936 with Toyotas method of reducing waste

So the meeting of the authors of the manifesto was perhaps simply about combining the best values and principles of their learnings, leaving aside the practices.

Iterative and incremental development methods can be traced back as early as 1957, with evolutionary project management and adaptive software development emerging in the early 1970s.

During the 1990s, a number of lightweight software development methods evolved in reaction to the prevailing heavyweight methods that critics described as overly regulated, planned, and micro-managed. These included: rapid application development (RAD), from 1991; the unified process (UP) and dynamic systems development method (DSDM), both from 1994; Scrum, from 1995; Crystal Clear and extreme programming (XP), both from 1996; and feature-driven development, from 1997. Although these all originated before the publication of the Agile Manifesto, they are now collectively referred to as agile software development methods. At the same time, similar changes were underway in manufacturing and aerospace.

Scrum itself, originated from an HBR article in 1986 entitled The New New Product Development Game by Hirotaka Takeuchi and Ikujiro Nonaka:

The authors described a new approach to commercial product development that would increase speed and flexibility, based on case studies from manufacturing firms in the automotive, photocopier and printer industries.They called this the holistic or rugby approach, as the whole process is performed by one cross-functional team across multiple overlapping phases, in which the team "tries to go the distance as a unit, passing the ball back and forth".

The Scrum framework was based on research by Schwaber with Tunde Babatunde at DuPont Research Station and University of Delaware. Tunde advised that attempts to develop complex product, such as software, that weren’t based in empiricism were doomed to higher risks and rates of failure as the initial conditions and assumptions change. Empiricism, using frequent inspection and adaptation, with flexibility and transparency is the most suitable approach.

In the early 1990s, Ken Schwaber used what would become Scrum at his company, Advanced Development Methods; while Jeff Sutherland, John Scumniotales and Jeff McKenna developed a similar approach at Easel Corporation, referring to it using the single word scrum.

Ken and Jeff worked together to integrate their ideas into a single framework, Scrum. They tested Scrum and continually improved it, leading to their 1995 paper, contributions to the Agile Manifesto in 2001, and the worldwide spread and use of Scrum since 2002.

In 2011, there was a celebration of the 10 year anniversary of the agile manifesto. Even then, they mentioned that it was really coining a term for existing processes:

The other problem with “Agile at 10” is that agile processes are not really ten years old. When we seventeen middle-aged white guys met in Snowbird in February, 2001, we didn’t invent agility. All we did was give a name to a set of values that encompassed the way a whole bunch of people (including ourselves) were already producing software. I dare say much of the work at Xerox PARC 25 years before our meeting would be considered agile. I know that much of the software development that was being done in the 90s was. So, in a way, all we did was to provide the seed crystal of a name—the subsequent growth came from all those thousands of teams that had already been doing things in an agile way. Our seed gave them a name and a set of four values on which to hang their existing practices.

There were other follow up manifestos written, like the PM Declaration of interdependence and Software Craftsmanship Manifesto, which are not that well know.

The hallmark of agile

An iterative and incremental development approach that embraces change (rather than fighting change like waterfall), together with involving the customer and continuous learning and feedback, are the core values of all/most of the agile methodologies. One of the biggest advantages of any agile methodology is the short feedback loop. This, coupled with the understanding that the methodology is the not goal, rather continuous learning is, can help organisations learn, unlearn and relearn as the need arises. They can be modified, integrated and adopted in the way that best suits the organisation.

They have been arranged below in terms of weight - we start discussing the most lightweight approaches first: those that have have the least amount of practices, and roles:

• Kanban emphasises just-in-time delivery, and focusses on visualising the current process in order to identify constraints, and limiting WIP to minimise task switching, disruptions and delays. Kanban does not suggest any practices or roles, but simply to visualise and understand your current process, which will allow you to optimise flow. Kanban makes use of lead time as the metric for process improvement
• Lean focuses on reducing waste and optimising the whole process, and kaizen: nothing is perfect, there is always room for improvement. Techniques include value stream mapping and queuing theory
• Scrum is all about self-organising teams, working in short, time-boxed periods, collaborating with the customer on a value-driven list of priorities, delivering frequently and iteratively. Scrum makes use of velocity as the metric for process improvement. Scrum suggests various practices, and is very light: it only has roles for a Scrum Master, Product Owner and the development team. This works naturally for small organisations, but large enterprises with heavier org structures may struggle to fit with Scrum's limited roles and meetings
• XP focusses a lot on the actual software development processes, by including coding practices like TDD, CI and pair-programming, minimising technical debt, while also promoting a flat management structure, and stressing on face-to-face communication, working with the customer and getting feedback, and simplicity: building only what you need today and not predicting what is needed tomorrow. XP mentions the bullpen  - a large room with ample space for movement, and many whiteboards, designed for developers to work and communicate. The bullpen facilitates pair-programming, and allows the customer to be present. XP is not without its detractors, as a whole book has been written against it
• DSDM embraces customer involvement, and fixes time, cost, and quality, while varying scope/features to keep the project on track,  in order to deliver on time. DSDM is heavier, in that it defines much more roles, that perhaps fits better into a large enterprises org structure: Project Manager, Business Analyst, Team Leader, Technical Coordinator, Business Advisor, etc.
• Crystal is family of adaptive, "stretch-to-fit" software development methodologies. Four vital factors determine the weight, and thus which level of Crystal to be applied. Crystal Clear is the lightest, suitable for non-critical projects, while Sapphire and Diamond are employed for larger projects based on bigger teams and higher criticality. The different levels define additional roles required. Crystal defines practices such as Early Victory and Walking Skeleton, which are MVP-type parts that you build early on and quickly. It defines side-by-side programming: similiar to XPs pair programming, but less intensive.
• FDD: Feature Driven Development operates on the principles of completing a project by breaking it into small, client valued functions that can be delivered in two weeks or less. FDD defines six mandatory roles, plus additional supporting roles. FDD has some direct links to DevOps, by defining roles and processes for Build and tooling engineers

Adopting and Adapting agile frameworks

Thus far we have discussed the origins of agile, and the different methodologies, in order for us to understand which one or combinations we can apply, based on our needs.

The most important thing is to understand that agile is a mindset and culture, while the methodologies are just practices that can enable that mindset. Agile Does Not Equal Scrum: Know the Difference. Agile is great, Scrum is OK-ish:

I have a problem with the way many people talk about Scrum. They say “agile/Scrum,” as though they’re the same thing.No, no, no. Scrum is just one way to approach agile. It’s a terrific project management framework, especially for cross-functional teams that work on one project at a time.Agile is broader than just Scrum. Scrum, for some teams, is a straightjacket that doesn’t fit. Agile, assuming you can see past the Scrum rituals, might fit quite well.

Based on my recent learnings and experiences implementing large scale agile and DevOps in a large enterprise, I am surprised that Scrum is the most common. I would think that large bureaucratic enterprises with bloated org structures might be better off starting with FDD, DSDM or Crystal, as it maps better to the orgs structure with multiple roles.  

Agile is all about continuous improvement, and supports adaptation. Sometimes agile methodologies are tailored to suit the project. Radical changes and strict adherence to methodologies might seem like extreme approaches, because each project brings its won requirements. Different agile practices are interrelated, and it's important to understand before altering a practice. However, be wary of common pitfalls:

• methodologies that are made heavy by adding practices may not improve the success rate
• methodologies that get embellished over time
• untried, or used once only, do not ensure success.

I like the hybrids of Scrum/XP, or Scrumban: Scrum and Kanban hybrid, which uses the practices and team structure of Scrum, with the visualisation, pull system and WIP limits of Kanban.

A perfect combination in my mind, perhaps idealisticly, (too heavy maybe?) would be:

• Scrum, for its practices of Sprints, daily standups and retro
• Kanban for WIP limits and developers pulling tasks when they have capacity
• XP for pair-programming, TDD and the bullpen
• FDD, for some of its additional roles that maps better to an traditional orgs structure, and domain object model, and planning, designing and building by features
• Crystal for its Early Victory (MVP-like) and Walking Skeleton

Beware of Fake Agile

Dave Thomas, one of the original signatories, says that its time to kill agile:

I don’t think anyone could object to a ban on the word when it is used as a noun. That’s just plain wrong. “Do Agile Right” and “Agile for Dummies” are just two of the innumerable attacks on the English language featuring the word. They are meaningless. Agile is not a noun, it’s an adjective, and it must qualify something else. “Do Agile Right” is like saying “Do Orange Right.”

But, beyond the grammar problem, there’s a bigger issue. Once the Manifesto became popular, the word agile became a magnet for anyone with points to espouse, hours to bill, or products to sell. It became a marketing term, coopted to improve sales in the same way that words such as eco and natural are. A word that is abused in this way becomes useless—it stops having meaning as it transitions into a brand.

And thats what Fake Agile is: agile in name only. Scaling agile seems to be yet another pitfall:

When I hear about scaling frameworks, I wonder who the customers are. I'm pretty sure the frameworks are the answer to managers who want the results of an agile approach. However, these managers often don't want to change themselves and change the culture for an agile approach. Agile “scaling” is rarely the answer to the organization's problems. More often, it's descaling. Helping managers visualize the cycle time might be a first step.

More agile reading

Steve Denning has a host of extremely well written articles on agile, as well as his book "The Age of Agile".