The Age of Agile Hasn't Even Begun Yet
Posted on
The Age of Agile Hasn't Even Begun Yet
Back when I worked at Rackspace, I once used to work with someone who, over time, I came to be good friends with. We shared many similar thoughts on software development practices, and frequently had engaging conversation about project management and evolution. Recently, he posted a link to an article which I have some disagreement with, and I'd like to address these disagreements here. The article is titled, "The Age of Agile Must End", posted February 12, by Michael Burnett.
I don't think it's any secret that Michael Burnett doesn't like agile development methods. He writes,
> Agile development has been the #1 operational principle of tech for > over 20 years, unchecked, unquestioned.
Of course, Michael must not have bothered to search on the Internet for any of this, because if he did, he would have found the HUGE religious wars that occurred between advocates of heavyweight, formal development methods, such as those put forth by the SEI, and the extreme programmers and early Scrum advocates like Kent Beck, Ward Cunningham, and Ken Schwaber. But, I am not Google, so I won't bother to list those debates here. Suffice it to say that, eventually, even the mighty SEI had to come around and see the light, and even accounted for agile methods when they released their CMMI framework.
Unchecked and unquestioned, indeed.
Still, in an attempt to provide evidence for his thesis, Michael strongly insinuates, in a not-so-covert way, that waterfall and lean approaches to software development are essentially equivalent, only further demonstrating his willful ignorance of the history of agile and so-called non-agile methods.
After discussing the introduction of lean production systems in America, Michael then writes:
> The Waterfall Scapegoat > > The burgeoning software industry largely followed these processes; > applied to digital products, they became known as Waterfall > methodology.
No.
They didn't.
In fact, software engineers here in the USA weren't even aware of the lean production system until at least the late 70s/early 80s (and I'm being deliberately generous here, as I wasn't in the software engineering field at this time; it's more likely to be around 1988 based on texts I've read since), while we've been basically following the Waterfall process in one form or another since at least the mid-50s.
In fact, Waterfall didn't even acquire its name until 1970 when Winston W. Royce wrote "Managing the Development of Large Software Systems" for IEEE WESCON (Wikipedia, accessed 2023-Feb-19) as an example of what not to do in software engineering. Funny, really, considering Royce didn't even use the name Waterfall in his paper. That was adopted organically by the greater development community. You're welcome.
In fact, the whole idea of iterating on software wasn't even remotely popular at the time; it wasn't until Boehm's paper, "A Sprial Model of Software Development and Enhancement", that the idea of iteration became common vernacular. That was in 1986, more or less contemporary with Scrum and Lean, in case you were wondering. But, we'll get to Scrum later.
For the record, lean software development, as a thing people would actually start talking about, didn't start to get a foothold until after the introduction of Kent Beck's Extreme Programming made alternative models of software development popular enough to talk about without a high probability of actually getting fired. That's a whole decade (more or less) after thinking on lean software engineering started, and two and half decades after Waterfall was coined.
Yup, that's right -- agility was hotly debated as far back as 1970. Possibly even earlier. But, we won't let facts stand in the way here. Michael continues,
> The term Waterfall has become a catch-all to describe a linear > development process that culminates in a fully realized product release.
Yes and no. It really depends on what it is we're linearizing here.
The author is correct that lean depends on a linearized set of
production steps. The existence of swim lanes
implies this, and even a
cursory study into lean's history with the Toyota Way will demonstrate it.
However, these production steps are arranged in one or more pipelines (each
given its own swim lane), just like a real factory would be, so that
multiple components of a project can be worked on concurrently, and
integrated to form larger components. This could be one big integration
like this:
Task A ---------.
|
Task B -------. `--|
`----|
Task C ------------|----> output
.----|
Task D -------' .--|
|
Task E ---------'
Time ----------------------------->
which I emphatically do not recommend; or, it could be more gentle, like this:
Task A ---.
|
Task B ---+--------|
|
Task C ------------|----> output
|
Task D -------+----|
|
Task E -------'
Time ----------------------------->
Thanks to lean's pull-based workflow, each task can proceed at its own natural rate of progress. Contrast against Waterfall, where each task's scope and progress are rigidly defined a priori by requirements analysts and project managers (resp.) first. If any of these needs to change, the whole process starts over from scratch (more or less), which is demonstrably extremely expensive. It's right there in Royce's paper. So, it's natural that companies tend not to do that, which is why usually companies compromise on delivery dates, quality, or both.
Even someone incompetent in the art can tell you that these approaches
differ strongly. With Waterfall, teams work on exactly ONE component
through a distinct succession of top-down refinement steps, only the last
couple of which are related to actual coding effort. Granted, some
allowance for prototyping is made; but, nothing like what you would find in
any typical agile
team.
Moreover, with lean development, any blockage in the flow of development of a component is intended to be addressed by on-demand, proactive offers of assistance from other team members to help unblock the stages involved. If you're just sitting around, twiddling your thumbs, because your supplier for a component (be it another company or the person in the cube across from you) is not delivering on time, then it is in your best interest to reach out and offer to help. This is strongly similar to Extreme Programming's rule of Collective Code Ownership. This can happen along a single swim lane or across swim lanes, as necessary, thanks to its pull-based nature.
Note that there's no discussion about blaming your coworkers here. One should not feel guilty for being slow on the team, as long as you can learn from the experience and grow.
The Waterfall method, in contrast, is extremely push based and highly
regimented. If there's a blockage, production usually doesn't stop while
the blockage is cleared. Instead, resources might (if you're lucky)
be re-allocated to different projects in an effort to unblock, and failing
that, the slower team mates are frequently just terminated as
ineffectual and/or a drag on team progress. In the worst cases, such as on
government contracts, you might even find yourself getting paid for
literally doing nothing due to the strong compartmentalization involved. If
you've ever heard the phrase critical path
or bottleneck
(typically used in phrases like don't become the bottleneck,
or,
we don't want to be on the critical path
) while working on a
commercial project, that's a good indicator you're actually using some
aspect of Waterfall methodology somewhere in your project planning.
This kind of misunderstanding appears frequently in many critiques of agile processes (note: lowercase-A), and frankly, it rather pisses me off. We in the agile community have been trying to communicate these differences for decades now (30+ years in my case), but we seem to be summarily ignored for the convenience of thought-leadership articles like the one linked above. Eventually, we just stopped caring, and now we're stuck with the spread of misinformation as authors play the game of Telephone, first having gotten their information via that same game of Telephone.
Unfortunately, this game has real consequences on us as software engineers.
Please note: this blog article is not intended to denigrate Waterfall and to push agile methods as an unconditional truth above all else. While it is no secret that I strongly prefer more agility in software development than less, there are successful counter-examples on my resume. My current employer, for example. But, one of my most fondly remembered experiences as an engineer was while working at a company called Plum District. When I was there, we incorporated the V Model (traditionally associated with heavy-weight methodologies like Waterfall) successfully with Scrum. For our team, V worked fantastically, allowing us to deliver new feature requests and bug fixes on-time and within budget in almost all cases.
In fact, of all the places I worked which had truly successful, happy teams working on products, none of them wore a badge of honor just for using agile or non-agile processes. None. All of them are happy to admit, "We take a little of this, and a little of that, and a little from those," approach. They are quick to adapt and react to problems that the team as a whole is having. It is the team that defines the processes used, not some consultant, and definitely not some manager who hasn't touched a line of code in thirty years. The paper certification they have is ultimately worthless, be it from the International Scrum or the Software Engineering Institutes.
Reading further into Michael's essay shows the gradient of misinformation into disinformation. But, before he does, he writes the following, which I believe unconditionally:
> The Agile vs. Waterfall argument is a false dichotomy: a company can have > a long-term strategy and carry it out while still releasing the product > incrementally, iteratively.
Michael's entire essay could have been replaced with the above paragraph, and all that needed to be said would have been.
But, Michael goes on about Scrum, again getting some very significant details wrong. Knowing these details is important to know when to, and when not to, apply Scrum in a project.
> As product owners and engineers attempted to apply lean manufacturing > principles to software development, they found ways to mimic an assembly line.
Unfortunately, that's not true. Companies have been treating programmers as an assembly line since the 1950s, as I discussed above. Scrum evolved contemporaneously with lean. You might even say that Scrum and lean are outright competitors in the agile market.
Scrum was intended to address the needs of companies that needed to pivot fast. The idea was to gather a cross-functional team into a war-room, and to bang out a product that met some new requirements before a competitor could move into their market of interest first.
The fact that Scrum has escaped the war-room and become de rigueur is the fault of both product management and those consultants I spoke of above. See, Scrum was exceedingly successful at what it did, so naturally, it was applied at wider scales of production: the on-going day to day maintenance of a project, for example. Here, Scrum languishes like a fish out of water. It is, literally, wholesale, unequivocally, and indisputably the wrong tool for that sort of job. Lean is head and shoulders superior at this task, precisely because on-going maintenance is so much more like running a production line.
Michael then goes on to list all the bullet points making Scrum, well, Scrum. (Not really; these largely derive from Extreme Programming, but I digress.) But, one item sticks out as wholly false to me:
> There's a Daily Standup to have on-the-fly "operational improvements."
No.
Just, no.
I mean, yes, but not in the way that Michael is scare-quoting operational improvements. So, in the end, just, no.
That's not the purpose of this meeting. It is intended to be a status report by the developers to report what they've done, what things are blocking them, and what they intend on working on next to their fellow developers. This is literally text-book Extreme Programming here, and has as much to do with organization-wide "operational improvements" as literally any other programming methodology on the planet. The fact that Michael quoted this phrase tells me Michael has little grasp on the utility of a methodology at all. Even Waterfall is intended to improve operations. Otherwise, what's the point?
But, he goes on:
> (Only developers are allowed to speak and discussion is not allowed > because that leaves the machinery idle.)
Wha??!!
No.
Again, just, no.
The reason why developers are allowed to speak is, because, it's a status
meeting. Discussions happen all the damn time. That's how one
engineer can respond to another engineer's report of a blockage. That's how
junior engineers can gain insight from senior engineers. That's what
operational improvements
is all about. If discussions turn out to be
lengthy, then you schedule future meetings on the topic. This is just
normal, every-day, good meeting etiquette. Michael seems to be upset
that good meeting etiquette was codified. Based on this reaction, I
strognly suspect Michael would be equally enraged by a project Code of
Conduct. If so, Michael is expressly not the kind of project manager you
want on your team, agile or otherwise.
The reason only engineers can discuss things is because stake holders have their own meetings in which they can voice requirements changes. If a stake holder were to have a voice during a stand-up, then what's the point of planning sessions with a customer? Of requirements analysis? This is something that both the SEI and every agilist alike actually agrees on: change requests that come too frequently will kill a project dead.
Dead.
Period.
This is not open for debate. This is documented fact. As I write this, we'll have, what, 67 years of research on this topic somewhere in SEI's library? I encourage you to read some of the reports SEI has written about large projects, both successful and failure alike. It is a literal Hollywood trope to have management come in and just randomly change a requirement, forcing engineers to rip out and re-work software on an ever-running treadmill of change requests. Hasn't anyone seen Office Space recently? That's not just a comedy; it's also a documentary.
Finally, there's this gem.
> The Backlog isn’t core to the Scrum practice, it’s an artifact of the > collateral damage and detritus of everything jettisoned during the sprint.
I ... just don't know how to respond to this except to say that I've stopped reading the article critically right here. Literally, there is no further point in continuing. Michael has made plain for the world to see he has no understanding what-so-ever of development methods and the tools they employ to manage the process.
At the end of the six bullet points defining Scrum, Michael writes,
> You can see how Scrum is fairly illogical already, but then it gets so > much worse.
Actually, if you study literally every other software methodology on the planet (including Waterfall!), you'll find that they also have:
-
Planning sessions where stake-holders are allowed to periodically review progress on the product. They're the ones spending the money to fund the development, after all.
-
Planning and status sessions where teams interact to discuss feature development. In agile terms, this would be your daily stand-up and sprint planning sessions.
-
Change and configuration management are strongly encouraged. Granted, this came a bit later in Waterfall's evolution, but by the time V and Spiral models came out, this was just a given. In practical terms, this amounts to the use of things like "backlogs" (which Michael says is an afterthought in Scrum), revision control systems, ticketing systems, etc.
-
Retrospectives are highly encouraged by SEI for all of their heavy-weight processes and light-weight processes alike. Even Watts Humphrey himself, in his books on the Personal Software Process, encourages self-reflection on a periodic basis to re-evaluate what is and isn't working for you, the developer. And he admits openly in his books, if you can't get PSP, don't bother with TSP.
With this knowledge, one starts to see how Scrum is not only extremely logical, but also how it fits into the larger ecosystem of development methods one can choose from for a project.
To further illustrate how far removed from reality Michael is, he writes down below:
> The combination of Agile principles and Scrum practices is disastrous > for startups.
Really? I openly invite Michael to try and convince the boards of directors of the venture firms backing Meta (nee Facebook), Alphabet (nee Google), Amazon, and Space-X that their clients' respective methods of software development is disastrous. Even Microsoft, which once invested heavily in PSP and TSP back in the Windows 2000 era, has since switched towards more agility, at least for some projects. Please! I really want to see what happens.
I do agree with what Michael wrote above in his essay, which I'll repeat here:
> The Agile vs. Waterfall argument is a false dichotomy: a company can have > a long-term strategy and carry it out while still releasing the product > incrementally, iteratively.
I cannot emphasize how much I agree with this single statement. But, that's only because a broken clock can still be right twice a day. So, Michael, I emplore you to follow your own advice on this topic:
> Just stop.
You're hurting more than you're helping.