Why Agile Happened
Podcast · Why Agile Happened

Why Waterfall Was Actually a Warning

by ZumiLabs · 45m
0:00 45m

Transcript

Made automatically from the audio, so it may contain mistakes. Click a line to play from there.

Host 1

So back in 1970, there was this computer scientist named Winston Royce, right? And he published a paper that had this highly detailed diagram in it.

Host 2

Right. The famous diagram.

Host 1

Yeah, exactly. It showed software being built in these very neat downward sequential steps. You know, like requirements, the design, coding, testing, and finally operations.

Host 2

And that diagram basically birthed the whole waterfall method.

Host 1

It did. I mean, it completely dominated the global tech industry for decades. But there is a massive historical twist here.

Host 2

Oh, it's one of the biggest misunderstandings in tech history.

Host 1

Seriously. Royce actually drew that diagram as a warning.

Host 2

Yeah, a literal warning.

Host 1

He published it in his paper as a prime example of a really flawed, risky model, one that like desperately needed iterative feedback just to survive contact with reality.

Host 2

But the entire tech industry just completely misunderstood it.

Host 1

They adopted it as their unquestioned gold standard. A system fundamentally designed to fail.

Host 2

It is arguably one of the most expensive reading comprehension failures in modern business history.

Host 1

That is such a good way to put it.

Host 2

Because the industry was just so desperate for you know, the illusion of predictability. So they clung to the straw man. They started freezing software requirements into these massive fixed scope contracts.

Host 1

Which locked teams into what were basically multi-year death marches, right?

Host 2

Exactly. Locking them in for years before a single user ever saw the product.

Host 1

And that inevitably resulted in those massive multi million dollar launch day disasters that kind of haunted software engineering for a whole generation.

Host 2

Yeah, it really did.

Host 1

So welcome to our deep dive. This is actually the first part of our ways of working from agile to AI native series.

Host 2

Super excited to get into this one.

Host 1

Same here. And in our previous discussion, we laid out the entire map for this series with one strict rule. And that rule is every major shift in how we build software, whether that's from waterfall to AI native workflows is ultimately just an attempt to shorten the feedback loop to real users.

Host 2

That is the core of everything we're talking about, shortening that loop.

Host 1

Right. So if you are a developer listening to this right now, or you know, an engineering lead or a product manager who wants to know how software teams actually operate behind all the corporate buzzwords, this analysis is custom-built for you.

Host 2

Yeah, our mission today is really to deconstruct the actual mechanics of survival in software engineering.

Host 1

We want to look at why agile actually happened.

Host 2

Right. Separating the actual empirical evidence from the the industry folklore.

Host 1

And frankly, we just want to strip away a lot of the nostalgia and the methodology worship to find out what actually works on the ground.

Host 2

Because there's a lot of methodology worship out there.

Host 1

Oh, a ton. So to do this, we are tearing into a pretty massive stack of source material today. We've got Martin Fowler's Foundational Essays, the history of the original 2001 Agile Manifesto.

Host 2

Which is a fascinating story on its own.

Host 1

It really is. We've also got modern critiques from original founders like Ron Jeffries and Dave Thomas, the U.S. Digital Services playbook, and this incredibly rigorous academic takedown of the famous Standish Chaos report by researchers Evelyn's and Verhoof.

Host 2

That academic paper is just it's brilliant.

Host 1

We will definitely get into that. But I want to start by kind of dissecting the trap itself. Because on paper, waterfall actually looks incredibly logical.

Host 2

It does.

Host 1

It makes total sense if you're coming from a traditional mindset.

Host 2

Right.

Host 1

So why did this highly predictive sequential engineering mindset fail so spectacularly when people applied it to code?

Host 2

Well, the root of the failure really lies in a concept that Martin Fowler calls activity-based decomposition.

Host 1

Okay. Activity-based decomposition.

Host 2

Yeah. Okay. So when you manage a project this way, you are basically breaking the massive endeavor down by the type of activity being performed.

Host 1

Like the phases.

Host 2

Exactly. You mandate that your team will spend, say, three months doing nothing but requirement analysis.

Host 1

Just writing documents.

Host 2

Right. Then the next four months are strictly for architectural design. And then the following six months are purely coding.

Host 1

And finally you have like three months of testing at the very end.

Host 2

Exactly. That's the activity-based breakdown.

Host 1

I mean, I see the appeal for management, honestly. It borrows so heavily from traditional civil engineering.

Host 2

Oh, absolutely.

Host 1

But you finalize the blueprints for a suspension bridge. The structural engineers calculate the exact load-bearing capacity of every single steel cable. And then the welders simply follow the schematic.

Host 2

Right. Because you definitely don't start pouring concrete while you're still debating the length of the span.

Host 1

Exactly. That'd be insane. For a bridge.

Host 2

But here's the thing. Software is not concrete. And it is not steel.

Host 1

Right.

Host 2

Treating software like traditional engineering relies entirely on this idea of predictive planning. And predictive planning assumes that the underlying requirements, the market conditions, and all the technical constraints will remain completely stable over like a two-year project lifespan.

Host 1

Which I mean in the tech sector is virtually impossible.

Host 2

It's laughable, really.

Host 1

Yeah. I mean, think about it. Two years ago, AI wasn't even fundamentally integrated into most consumer products.

Host 2

Right.

Host 1

And now it's literally the baseline.

Host 2

Exactly. The domain itself is constantly in motion. But there is a deeper, even more intrinsic problem here.

Host 1

Okay, what's that?

Host 2

In software development, domain knowledge actually emerges during the building process itself.

Host 1

Oh, I see.

Host 2

Yeah. You cannot fully understand the problem you are solving until you begin trying to solve it in code.

Host 1

So the coding isn't just following instructions.

Host 2

Exactly. Writing code is not the manufacturing phase of software. Writing code is the design phase.

Host 1

Oh, that's a great distinction.

Host 2

The compiler is the manufacturing plant. The developer is doing design work every time they type.

Host 1

I want to push on this bridge analogy a bit more because it's just so pervasive in corporate management.

Host 2

It really is everywhere.

Host 1

If building software isn't like building a skyscraper or bridge, where you absolutely need those blueprints finished before pouring the foundation, what is a better mechanical analogy of what developers are actually doing out there?

Host 2

Think of it less like building a bridge and much more like exploring a completely new continent. Or maybe even like performing complex surgery on a living organism.

Host 1

Surgery, really?

Host 2

Yeah. I mean, you can look at all the MRI scans you want beforehand, right? You can create a highly detailed predictive surgical plan based on those scans.

Host 1

Okay, sure.

Host 2

But the very moment you make the first incision, you might find that a major artery is, you know, slightly to the left of where the scan indicated, or maybe there's some unexpected scar tissue.

Host 1

Right. The reality doesn't perfectly match the map.

Host 2

Exactly. And in that situation, you don't put the scalpel down, wake the patient up and spend three months redrawing the surgical plan in a boardroom.

Host 1

No, obviously not. No. You lose the patient.

Host 2

You can't do that. You have to adapt your technique immediately based on the reality that is right in front of you.

Host 1

That makes total sense.

Host 2

So a rigid set of turn by turn-by-turn directions is not just useless in that surgical scenario. It is actively dangerous.

Host 1

Because it would force you to cut through the artery just because that's what the plan said to do.

Host 2

Precisely. What you actually need in surgery and in software is a deep understanding of anatomy, a clear goal, and the agility to pivot instantly.

Host 1

Wow. That surgical adaptation really highlights the most fatal flaw in waterfall's activity-based decomposition, doesn't it?

Host 2

Oh, completely.

Host 1

Because by pushing the most unpredictable, volatile elements of software development, which are always testing and integration, to the very end of the cycle, you are basically operating blind the whole time.

Host 2

That is the compounding risk of waterfall. I mean, imagine waiting 18 months to finally integrate the database team's code with the user interface teams code.

Host 1

That sounds terrifying.

Host 2

It is. Because if you discover a foundational flaw in your initial assumptions at month 18, you don't just experience a minor delay.

Host 1

No, it's huge.

Host 2

You experience a catastrophic cascading failure. You have to backtrack through a year and a half of built-up assumptions, redesigns, and tightly coupled code.

Host 1

Which is why those projects always went so wildly over budget.

Host 2

Exactly.

Host 1

So the alternative to activity-based decomposition is what Fowler calls feature-based decomposition, which is driven by adaptive planning.

Host 2

Yes. Instead of doing all the design for everything, and then all the coding for everything, you take one tiny vertical slice of functionality. Like what? Say just a user logging into their account. That's the slice. You analyze, design, code, test, and integrate that single feature within a matter of days or maybe weeks.

Host 1

Okay, so you complete all the activities, but just for a microscopic slice of the product.

Host 2

Right. And doing that allows you to update your mental model and your overall project trajectory based on what you actually encountered during the build.

Host 1

Which brings us to a very specific kind of famous moment in time. February 2001 at the lodge at Snowbird Ski Resort in the Wasatch Mountains of Utah.

Host 2

Yes. The Snowbird meeting.

Host 1

This is where the rebellion against that predictive engineering mindset finally crystallized. And honestly reading the story of this meeting from Jim Highsmith's account, it is highly entertaining.

Host 2

It really was a gathering of outliers. I mean you had 17 software professionals there representing all these various lightweight methodologies that had been brewing in the late nineties.

Host 1

Things like extreme programming, scrum, adaptive software development crystal.

Host 2

Right. And Highsmith affectionately referred to this group as organizational anarchists.

Host 1

Which is so fitting because looking at the sources, they couldn't even easily agree on the location for this meeting.

Host 2

No, they argued about everything.

Host 1

Yeah. Chicago in the winter was deemed far too cold and boring. Anguilla in the Caribbean was warm but logistically too difficult to reach for a short weekend.

Host 2

So Snowbird ultimately went out basically just because of the skiing.

Host 1

Exactly. Though according to the lore attendees like Martin Fowler spent the first day mostly trying to ski on their heads.

Host 2

Which is such a funny image. But that anecdote perfectly highlights the personality of the room. These were deeply independent thinkers.

Host 1

And fierce competitors too, right?

Host 2

Oh yeah. In many cases they were direct competitors in the consulting and software space. Alistair Coburn, who created the crystal methodology, explicitly noted that he didn't expect this specific group of agilites to ever actually agree on anything substantive.

Host 1

But against all odds, they really did. I mean, they produced the manifesto for agile software development.

Host 2

They did.

Host 1

And structurally, it's actually brilliant. Because it's not just a list of rigid rules, it's a list of value trade-offs.

Host 2

Right. The four values.

Host 1

Yeah. They stated that they valued individuals and interactions over processes and tools. They valued working software over comprehensive documentation.

Host 2

And customer collaboration over contract negotiation.

Host 1

And finally, responding to change over following a plan.

Host 2

Those four values completely shifted the entire paradigm of the industry over the next decade. But there is a massive caveat printed right at the bottom of the manifesto that is routinely just totally ignored today.

Host 1

I am so glad you brought this up because this is where the industry gets it completely backwards so often. The caveat explicitly states that is while there is value in the items on the right, we value the items on the left more.

Host 2

It is a critical distinction. They were not anarchists demanding the abolition of all documentation everywhere.

Host 1

Right. They weren't saying don't write anything down.

Host 2

Exactly. They were not saying that planning is inherently bad or that tools are useless or that you shouldn't have business contracts.

Host 1

That would be crazy.

Host 2

Right. They were simply stating a prioritization framework for when push comes to shove. When you are forced to make a hard choice on a project, delivering working software is simply more important than updating a comprehensive specification document.

Host 1

And they backed up those four values with twelve principles.

Host 2

Yes.

Host 1

Just synthesizing them from the sources. The absolute highest priority was the early and continuous delivery of valuable software.

Host 2

Early and continuous.

Host 1

Right. And they explicitly welcomed changing requirements, even super late in development, seeing them as a competitive advantage rather than a scope creep disaster.

Host 2

Because if the market changes, you want to be able to change with it.

Host 1

Exactly. They also believed that the best architectures didn't come from some ivory tower architect handing down tablets, but emerged from self-organizing teams.

Host 2

It's a really profound philosophy. But you know, as one attendee joked at the time, it also sounds a bit mushy. Mushy. Yeah. Like philosophy is great for a ski resort, but software is a brutal deterministic discipline. I mean, you have to type specific characters into a machine to make it work. Good vibes don't compile.

Host 1

Right, exactly. So how did these seventeen organizational anarchists actually translate this somewhat mushy philosophy into daily mechanical engineering work?

Host 2

Well, that's where the actual practices come in.

Host 1

Let's dig into those actual mechanics from our sources. And I really want to start with a concept from the C2 wiki that sounds almost offensively simple when you first hear it.

Host 2

Oh, I know what throne you're talking about.

Host 1

Yeah. Do the simplest thing that could possibly work or DTSTTCPW.

Host 2

That's quite the acronym.

Host 1

Seriously. But when I first read this, my immediate reaction was, Well, this just sounds like a mandate to write terrible lazy code.

Host 2

A lot of people think that.

Host 1

Right. It sounds like permission to just hack something together and completely ignore the long term consequences.

Host 2

That is the most common misinterpretation for sure. But it couldn't be further from the truth. Doing the simplest thing actually requires immense discipline.

Host 1

How so?

Host 2

The crucial part of the phrase is that could possibly work. If you need to implement, say a new search filter on a website, you write the most straightforward minimal code imaginable to achieve that exact result.

Host 1

Okay. So no extra bells and whistles.

Host 2

Exactly. It might just be a super basic conditional if statement, but you get it running, you put it on the screen, and you prove that it solves the user's immediate problem today.

Host 1

There is a fantastic quote in the sources from Mikhail Kalashnikov, the famous weapon designer that perfectly encapsulates this vibe. He said, All that is complex is not useful. All that is useful is simple.

Host 2

That's a great quote.

Host 1

It really is. Doing the simplest thing forces the developer to get out of that hypothetical whiteboard phase and get a real feedback loop running instantly.

Host 2

Yes, exactly. But getting something on screen quickly creates a new risk, right? Spaghetti code.

Host 1

Oh yeah.

Host 2

If developers are just hacking things together with simple if statements to solve today's problem over and over, the code base will rapidly turn into an unmaintainable nightmare.

Host 1

Right. So how did the early agile pioneers prevent that from happening?

Host 2

Well, that is the crucial second step that everyone always forgets. Refactoring. Refactoring. Precisely. The rule isn't just do the simplest thing and walk away. The rule is do the simplest thing that works and then immediately refactor the system.

Host 1

So you clean up after yourself.

Host 2

You have to. You step back and say, okay, the software now has this new capability. How do I redesign the underlying code so that it is elegant, clean, and seamlessly integrates this new feature without adding technical debt.

Host 1

But refactoring safely requires a safety net, right? I mean, you you can't just go in and rip off the foundational code every time you add a simple feature, or the whole system will just collapse.

Host 2

Right. And the underlying mechanism that makes doing the simplest thing possible is a comprehensive suite of automated tests.

Host 1

Automated testing.

Host 2

This is the how behind the agility. You simply cannot be agile if you are terrified of breaking your own software.

Host 1

That makes total sense.

Host 2

If you have a robust automated test suite, you can write the simple solution, see the test pass, and then ruthlessly refactor the code to make it clean.

Host 1

And if you mess up.

Host 2

If you make a mistake and break a previously working feature, the automated test flashes red instantly. That immediate sub-second feedback loop is what allows developers to embrace simplicity without fear.

Host 1

Wow. That immediate feedback just fundamentally changes the psychology of the developer, doesn't it?

Host 2

Completely.

Host 1

Which leads us nicely into the next major mechanic of the sources. And it's closely related to this YAGNI.

Host 2

Yes. You aren't gonna need it.

Host 1

Right. I want to spend some time on the history here because the origin story from the C3 project is just fascinating.

Host 2

The C3 project, the Chrysler Comprehensive Compensation System, is legendary in software engineering history.

Host 1

Yeah, so in the late 90s, Chrysler was trying to build this unified payroll system for tens of thousands of unionized auto workers.

Host 2

And payroll sounds simple to outsiders, but it is horrifyingly complex when you factor in union rules, overtime, shift differentials, and taxes across multiple states.

Host 1

The project was basically drowning in complexity. It was essentially failing.

Host 2

It was.

Host 1

And then they brought in Kentbeck and Ron Jeffries to rescue it. And one of the primary tools they used to rescue this massive failing project was Yagni.

Host 2

Right. As Martin Fowler recounts it, there was a developer named Chet Hendrickson who would frequently come to Kentbeck with a list of capabilities he thought the system would eventually need.

Host 1

The hypothetical features.

Host 2

Exactly. He would say things like, Kent, we are building the tax calculation module right now. While we are in here, shouldn't we build in the abstractions for handling Canadian tax law? Chrysler might expand it to Canada next year.

Host 1

Which sounds so responsible, right?

Host 2

It does. But to each of those speculative requirements, Kent Beck would simply reply, you aren't gonna need it. Do not build it.

Host 1

Wow. Just a flat no.

Host 2

Is the ultimate defense against speculative over engineering.

Host 1

But Feller breaks down the actual economics of Yagney in a really compelling way in the sources. It's not just about like saving a developer's time today, it is a strict mathematical calculation of risk.

Host 2

It really is.

Host 1

Let's run through Fowler's economic breakdown, because I thought this was brilliant. He uses an example of building software for an insurance company.

Host 2

Right, the storm risk example.

Host 1

Exactly. So you were building a module to calculate storm risk. And suddenly you think, in six months, the business is probably going to want to calculate piracy risk for our global shipping clients. I should build the complex abstractions for piracy risk into the architecture right now, while my head is already in the code.

Host 2

Which is exactly how most developers think.

Host 1

Right. Yagny says absolutely do not do that. But if I'm a developer, it really feels cheaper to do it now while I'm already modifying the system. Why is that an economic fallacy?

Host 2

Well, Fowler outlines three specific costs that totally destroy that logic. First is the pure cost of delay.

Host 1

Okay. Cost of delay.

Host 2

Yeah. If you spend two weeks building piracy risk abstractions right now, you are delaying the delivery of the storm risk feature by two weeks.

Host 1

And the storm risk is what they need now.

Host 2

Exactly. The storm risk feature is what the business actually needs to generate revenue today. You are actively withholding value from the user based on a pure guess about the future.

Host 1

That's a great point. And the second cost is the cost of being wrong.

Host 2

And in software, let's be honest, you are usually wrong about the future. Always. Maybe the business strategy pivots entirely. Maybe global shipping routes change and piracy isn't a concern anymore. If you never actually need that feature, you just burned two weeks of highly paid engineering time on literally nothing.

Host 1

Okay, but let's play devil's AfKIF for a second. What if you are right?

Host 2

Okay.

Host 1

What if in six months the business comes down and says, we need piracy risk calculations immediately? Haven't you just saved the company a ton of money by anticipating it early?

Host 2

You think so, but even if you guessed right about the general need, Fowler points out that you will almost certainly guess wrong about the specifics.

Host 1

Oh, because you don't have the context yet.

Host 2

Precisely. When you build a feature six months before it is needed, you are building it without the full context of what the users actually require.

Host 1

You're just guessing.

Host 2

Right. You might build an architecture for tracking pirates in the Atlantic Ocean, only to find out six months later that the user specifically needed data on the Pacific.

Host 1

Oh wow. So your code is useless anyway.

Host 2

Worse than useless. Now you have accumulated technical debt. You have to pay what Fowler calls the cost of repair to rip out all the presumptive code you wrote too early just so you can build what they actually want today.

Host 1

Because every unneeded abstraction, every speculative class or interface you add today just makes the code base harder to read, harder to understand, and harder to modify tomorrow.

Host 2

And exactly. Yagni keeps the code base malleable. But again, just like doing the simplest thing, Yagni only works if you have that automated testing safety net, allowing you to add the features easily later on when you actually do need them.

Host 1

So Yagni requires a code base that welcomes change rather than resisting it.

Host 2

Absolutely.

Host 1

So we have DTST T C PW, and we have Yagni. Both of these are really about minimizing current effort, but the most mind-bending, almost counterintuitive concept Fowler introduces in our sources is sacrificial architecture.

Host 2

Oh, I love this concept.

Host 1

I mean, this completely upended how I thought about software engineering value.

Host 2

It is a profound shift in perspective. Sacrificial architecture basically means accepting right now that in a few years you will likely need to throw away every single line of code you are writing today and rebuild the entire system from scratch.

Host 1

Which sounds like a failure to a lot of people.

Host 2

Right. But crucially, it's realizing that this destruction is a sign of immense business success, not engineering failure.

Host 1

Fowler uses eBay as the prime historical example for this. Because in 1995, eBay wasn't some grand, highly available distributed system. It was literally a set of Perl scripts written by Pierre Omidyar over a single weekend.

Host 2

It was incredibly basic.

Host 1

But those simple Perl scripts proved the business model. By 1997, eBay was exploding in popularity, and the Perl scripts just could not handle the traffic. So what did they do? They tore them down completely and rewrote the entire platform in C.

Host 2

Right. And then by 2002, the C architecture was hitting its limits as eBay became a massive global behemoth. So what did they do? They tore it all down again and rewrote it in Java.

Host 1

Okay. Now look at that through the lens of traditional waterfall engineering.

Host 2

Oh, a waterfall purist would have a heart attack.

Host 1

Exactly. They would look at the 1995 ProScripts and say, but a massive failure of foresight. You should have anticipated the global scale and built the Java architecture from day one.

Host 2

But if OmidR had tried to build a massive distributed Java architecture back in 1995 just to test a weekend auction idea, eBay never would have existed.

Host 1

No, of course not.

Host 2

It would have taken two years to build, it would have been incredibly expensive, and it wouldn't have met the needs of a tiny startup that was just trying to find product market fit.

Host 1

The architecture that supports a thousand users is fundamentally different from the architecture that supports 10 million users.

Host 2

There is a famous rule of thumb from Google engineer Jeff Dean that perfectly captures this idea. He says, design for 10x growth, but plan to completely rewrite before you hit 100x growth.

Host 1

Plan to rewrite.

Host 2

Yeah.

Host 1

Wow. Yeah. Wow.

Host 2

You are not building a monument for the ages. You are just building a disposable vehicle to get your business to the next stage of scale.

Host 1

That completely liberates a development team.

Host 2

Yeah.

Host 1

I mean, if you accept that your architecture is sacrificial, you stop over-engineering for a hypothetical future that is five years away.

Host 2

Exactly. You build what you need to survive today, knowing that if you succeed, you will have the capital and the resources to rebuild it better tomorrow.

Host 1

And we actually see these iterative user-focused mechanics scaling up far beyond just startups in Silicon Valley. If you look at the U.S. Digital Services Playbook, which is this incredible open source document maintained by the federal government, it explicitly pushes massive federal agencies away from monolithic acquisitions.

Host 2

Yeah. The government used to operate on the ultimate waterfall model. It was legendary.

Host 1

Like writing a thousand-page requirement document for a new healthcare portal or something.

Host 2

Exactly. They'd hand it to a massive defense contractor and then just wait three to five years for delivery.

Host 1

And when it finally launched, it was almost universally broken, wildly over budget and completely misaligned with what citizens actually needed.

Host 2

Almost every time. But the US DS Playbook changed that by mandating user research driven phases like discovery, alpha, beta, live.

Host 1

So breaking it down.

Host 2

Yeah, they force agencies to deliver small pieces of value continuously, proving the concept with real users before they write those massive multimillion dollar checks.

Host 1

The empirical evidence in favor of iterative development just became impossible to ignore, even at the highest levels of the federal bureaucracy.

Host 2

It really did.

Host 1

So let's take a breath and kind of synthesize where we are. We have this elegant, developer-centric philosophy that came out of Snowbird. We have these highly disciplined economic mechanics doing the simplest thing, Yagney, sacrificial architecture, all heavily backed by automated testing.

Host 2

Right.

Host 1

It sounds incredibly logical. It sounds liberating, honestly. So how on earth did a movement based on doing the simplest thing mutate into the bloated bureaucratic corporate nightmare that so many enterprise developers suffer through today?

Host 2

To understand the dark mutation of Agile, we really have to follow the money. And we have to look at the data that drove corporate panic back in the 1990s.

Host 1

Oh, the infamous 1994 Chaos Report. If you have been in literally any software management meeting or taken any project management course in the last 20 years, you have seen a PowerPoint slide citing this exact report.

Host 2

Almost guaranteed.

Host 1

It claimed that the software industry was in a state of absolute unmitigated crisis.

Host 2

The Standish group published this report and delivered a genuinely shocking set of statistics. They claimed that across the entire industry, there was only a 16% success rate for software projects.

Host 1

16%.

Host 2

That's abysmal.

Host 1

It is.

Host 2

They also claimed 53% of projects were challenged, meaning they were severely over budget or late, and an astonishing 31% failed outright and were just canceled.

Host 1

And those numbers absolutely terrified the industry. I mean, they were cited in major academic papers, mainstream media reports, and supposedly even in a briefing to the president of the United States.

Host 2

They became the foundational justification for imposing heavy, rigid command and control frameworks on software teams.

Host 1

Right, because management looked at a 60% success rate and essentially said, well, developers are chaotic artists who are totally out of control. We need to lock them down with intense oversight.

Host 2

The narrative was totally set. Software is failing, and we need tighter management to fix it. But there was a massive problem with the Standish Chaos Report.

Host 1

The data was completely flawed.

Host 2

Fundamentally, mathematically flawed.

Host 1

I read the IEEE paper by Evelyn's and Verhoof from Bria Universiteit Amsterdam that is included in our sources, and I have to say, it is one of the most satisfying academic takedowns I have ever read.

Host 2

It's so thorough.

Host 1

It really is. They finally got their hands on a massive empirical data set, over 5,000 forecasts from real world projects totaling hundreds of millions of euros, and they decided to rigorously fact check the Standish methodology. And the conclusions they drew completely dismantle the industry's founding myth.

Host 2

The core issue that Evelyn's and Verhoof exposed is how the Standish group actually defined a successful project.

Host 1

This is the crazy part.

Host 2

Right. According to Standish, a project was only considered successful if it met three binary criteria. It had to be completed exactly on the initial time estimate, exactly on the initial budget estimate, and it had to offer exactly the features and functions that were specified on day one.

Host 1

I really want the listener to let that sink in for a second. Standish defined success solely by adherence to the initial estimate. Nothing else mattered.

Host 2

Exactly. They did not measure user satisfaction at all. They did not measure return on investment. They didn't even measure if the software actually solved the business problem it was supposed to solve.

Host 1

So wait. If you discovered halfway through a six-month project that a highly complex feature was actually totally useless to the end user, and you wisely decided to cut it to save the company $50,000.

Host 2

Standish categorized your project as challenged or a failure.

Host 1

Just because you didn't deliver the original scope.

Host 2

Yes. You were quite literally penalized for saving money and adapting to reality.

Host 1

That is wild.

Host 2

It gets worse.

Host 1

Okay, EQF, what is that?

Host 2

Basically, EQF measures the area under the curve of your estimates over time compared to the final actual outcome. So a high EQF, like a five or a six, means your initial estimates were highly precise and unbiased, meaning you didn't artificially pad your budget just in case.

Host 1

Okay, that brings us to the tale of two organizations from their paper. Let's look at organization Y. This was a highly mature, really competent financial services provider.

Host 2

Right.

Host 1

Evelines and Verhoof analyzed their cost and functionality forecasts, and organization Y was incredibly accurate. They had a median EQF of six point four for cost.

Host 2

Which is excellent. It means they were giving honest, precise estimates to management without hoarding extra budget.

Host 1

They were essentially a best in class engineering organization. But when the researchers took organization wise actual project data and ran it through the strict Standish Chaos Report definitions of success.

Host 2

This highly accurate organization scored a dismal 35% success rate.

Host 1

35%. Because even slight, completely rational adaptations to reality, like coming in 5% over budget because of an unexpected technical hurdle or dropping a minor feature caused them to fail the rigid Standish criteria.

Host 2

Right. But then the researchers analyzed Organization X.

Host 1

Okay, organization X.

Host 2

Organization X was a large multinational corporation. When Evelines and Verhoof looked at their internal data, they found massive systemic overestimations.

Host 1

Like intentional padding.

Host 2

Exactly. The project managers at Organization X knew they were being judged heavily on coming in under budget. So what did they do? They sandbagged. They deliberately padded their estimates. If a project was going to cost 10,000 euros, they asked for a hundred thousand.

Host 1

Wow.

Host 2

Their estimation quality was terrible. They had a median EQF of just 0.43.

Host 1

But because they massively, fraudulently overestimated their budgets and timelines, they easily came in under budget and on time every single time. And how did they score on the Standish scale?

Host 2

Under Standish rules, Organization X achieved a 67% success rate.

Host 1

So just to be absolutely clear here, the industry's defining metric for software failure was actively penalizing competent teams for adapting to user needs and forecasting accurately.

Host 2

Yep.

Host 1

While simultaneously rewarding teams that actively hoarded budget, lied on their estimates, and delivered bloatware.

Host 2

That is exactly what the data shows. The Standish numbers created a perverse incentive structure. Project managers realized that the only way to look successful was to demand massive budgets and long timelines up front, freezing the scope to guarantee they hit the target.

Host 1

And yet this deeply flawed, sensationalized data drove a generation of executives to enforce command and control project management. Just to fix a crisis that was largely a phantom of bad statistics.

Host 2

It's a tragedy, really.

Host 1

Now, to be completely fair and impartial to the sources, because we aren't just here to preach the agile gospel. You mentioned earlier that there is a time and place for rigid high assurance upfront planning. It's not always a mistake.

Host 2

No, it isn't. And Martin Fowler points this out explicitly. Context is everything. Right. If you are the NASA space shuttle software group, predictability is not just possible, it is mandatory. You have a massive team, incredibly stable, well-understood physics requirements, and if you have a bug, astronauts die.

Host 1

The stakes couldn't be higher.

Host 2

Exactly. In that highly specific life-critical context, high ceremony, massive documentation, and heavy upfront design are completely rational.

Host 1

But the problem is an e-commerce website is not a space shuttle. An internal HR tool is not a space shuttle. Business software is highly volatile and unpredictable.

Host 2

Right. But management, terrified by that 16% success rate myth from Standish, tried to treat every single web app like a space shuttle.

Host 1

And when the Agile manifesto came along in 2001 and offered a more empirical, adaptive way, management was deeply threatened. They didn't want to let go of the illusion of control that predictive planning provided.

Host 2

Which leads us directly to the dark mutation of Agile. How did this philosophy actually get co-opted?

Host 1

Well, Martin Fowler describes a linguistic phenomenon in the sources called semantic diffusion.

Host 2

Semantic diffusion.

Host 1

Yeah. This happens when a word is coined with a very specific technical definition. But as it spreads through the wider community, and especially through marketing departments, it loses its original meaning entirely.

Host 2

Right. Agile stopped being a set of values and just became a hollow buzzword.

Host 1

It became a highly lucrative brand, honestly. And Holly Cummins, another software thinker, takes it a step further with a concept called semantic inversion.

Host 2

Semantic inversion is fascinating. That's when a term is used so poorly that it ends up meaning the exact opposite of its original intent.

Host 1

The opposite.

Host 2

Yeah. So Agile, which was explicitly designed to empower developers and prioritize individuals over processes, became a strict process-heavy framework used by management to micromanage developers.

Host 1

Fowler specifically calls out something he terms agile imposition. And when you read his essay on it, it makes perfect sense why this is such a massive red flag in corporate environments.

Host 2

Let's look at the logic of it. The very first value of the Agile manifesto is individuals and interactions over processes and tools.

Host 1

Right.

Host 2

And a core principle is that the best architectures and designs emerge from self-organizing teams.

Host 1

Self-organizing being the key phrase.

Host 2

Therefore, if a CTO or an upper management committee buys a massive branded agile framework like SAFE, for example, and imposes it on a development team from the outside, mandating exactly how they must work and which ceremonies they must hold.

Host 1

They are violating the foundational premise of Agile.

Host 2

Exactly. You cannot mandate self-organization. It's an oxymoron. It's like demanding that someone act spontaneously on a strict schedule.

Host 1

That's a great way to put it.

Host 2

A team must choose its own process to fit their specific technical context and team dynamics. Forcing Agile onto a team is agile imposition, and it universally breeds resentment and failure.

Host 1

And yet a massive corporate machinery sprung up to facilitate exactly this kind of imposition. Fowler wrote a really scathing critique of the certification industry in his essay titled Certification Competence Correlation.

Host 2

He argues a very simple point there. For any certification to be useful to an employer, holding that certification must correlate with actual competence and the underlying skill.

Host 1

Right. Like if I hire a certified structural engineer, I expect them to actually know how to calculate load balances.

Host 2

But in the software world, agile certifications like certified scrum master rarely correlate with the ability to actually write good software or deliver value.

Host 1

I mean, you essentially take a two-day weekend course, pay a thousand dollars, take an open book, multiple choice test, and suddenly you are a master of software delivery methodologies.

Host 2

It's wild. And yet these virtually useless certifications became mandatory entry gates for hiring in the enterprise world.

Host 1

It became this multi-million dollar industry of courses, assessments, and badges that had absolutely nothing to do with technical excellence.

Host 2

You had an army of consultants selling agile for dummies to management. They stripped out all the difficult, highly disciplined engineering practices like test-driven development, continuous integration, continuous refactoring.

Host 1

And they just left behind what?

Host 2

Just a bunch of meetings, story points, and velocity metrics.

Host 1

Which brings us to the heartbreak of the original founders. It is incredibly rare to see the creators of a movement actively try to destroy their own creation.

Host 2

It is.

Host 1

But by 2014, Dave Thomas, one of the original 17 signatories at Snowbird, wrote a blog post literally titled Agile is Dead.

Host 2

Yeah. He begged the industry to abandon the noun agile because it had been completely co-opted by vendors and consultants and returned to using it as a verb or an adjective, agility.

Host 1

Right, lowercase A.

Host 2

Yeah.

Host 1

He distilled the entire concept of agility down to four incredibly simple steps. Find out where you are. Take a small, meaningful step towards your goal, adjust your understanding based on what you learned from that step, and repeat.

Host 2

That is the entire mechanical loop of software development.

Host 1

And Ron Jeffries, another original founder who helped rescue that Chrysler project, went even further in 2018. He wrote a piece bluntly titled Developers Should Abandon Agile.

Host 2

Jeffries coined the term Dark Scrum or Dark Agile.

Host 1

Dark Scrum.

Host 2

He observed that when these frameworks are imposed poorly by management, they do not help the development team. They actually make life exponentially worse.

Host 1

How so?

Host 2

They result in constant interference, endless reporting meetings, less time to do the actual technical work and unrelenting pressure for management to constantly increase quote unquote velocity and go faster.

Host 1

Fowler actually gave a speech about the state of Agile in 2018. And he told this horrifying story that perfectly encapsulates what dark scrum looks like in the real world.

Host 2

Oh, the six developers story?

Host 1

Yes. He visited a supposedly agile enterprise organization. There was a team of six developers, and they were writing software for exactly four internal users. Just four.

Host 2

And yet, those six developers had never spoken to those four users. Never. In fact, they were strictly forbidden by management from talking to them. Instead, they had to communicate exclusively through a designated scrum product owner.

Host 1

And over the course of a single year, that team had rotated through four different product owners.

Host 2

That's unbelievable.

Host 1

Let's visualize this. Six developers, four users, and they aren't allowed to pick up a phone and talk to each other. They are communicating through a revolving door of middle management who don't understand the domain. And the company has the audacity to label this bloated process agile.

Host 2

It is the complete antithesis of what happened at Snowbird. As Fowler pointed out, Kent Beck had originally suggested calling the movement conversational rather than agile.

Host 1

Conversational. Right.

Host 2

I like that. Yeah. Because building software requires an ongoing, direct, high bandwidth conversation between the people writing the code and the people using it. Dark Scrum builds bureaucratic walls, exactly where there should be open conversations.

Host 1

So we have painted a pretty bleak picture here. We have developers trapped in what Ron Jeffries calls the code mines of Ohio, suffering under top-down imposed agile frameworks, drowning in JIRA tickets, and dealing with arbitrary deadlines based on bad metrics.

Host 2

It's a rough reality for a lot of people.

Host 1

So how does an engineering team actually fight back? How do they survive dark scrum and reclaim genuine agility in their day-to-day work?

Host 2

Well, Ron Jeffries offers a very specific, deeply pragmatic survival tactic for developers who find themselves trapped in these toxic environments.

Host 1

Okay, what is it?

Host 2

His core advice is this detach your mental state from whatever branded framework your management thinks they are running, and focus entirely on the foundational principles of engineering. And his number one survival tactic is a mantra. Ship to survive.

Host 1

Ship to survive. Unpack the mechanics of that for you. What does it mean in like a daily stand-up meeting?

Host 2

It means the ultimate defense against micromanagement, arbitrary deadlines, and unreasonable demands is producing running, tested, integrated software on a highly frequent basis.

Host 1

So just constantly having something to show.

Host 2

If your team can produce a fully operational, working version of the software every two weeks or every week or even every day, you seize control of the narrative.

Host 1

Because you take the conversation out of the realm of hypothetical gaunt charts and pull it forcefully into reality.

Host 2

Exactly. When management comes to you with a massive, impossible list of demands or complains that your velocity isn't high enough, you don't argue with them about story point estimates. You don't debate the philosophy of agile.

Host 1

You just show them the code.

Host 2

You open the laptop, show them the working software, and say, here is what is ready to ship today. It works perfectly. Now looking at this reality, what is the single most important thing you want us to do next?

Host 1

I love that. It completely shifts the power dynamic. If you have working software in your hands, the focus inherently moves from management asking, why aren't you doing all this extra stuff? to a collaborative question. What should we do next?

Host 2

Right. You force the leadership to engage with reality and collaborate with you on prioritizing actual business value rather than just standing back and demanding more speed.

Host 1

And if your team is overwhelmed and missing targets, Jeffries advises something very counterintuitive for corporate environments.

Host 2

Yes. Take on significantly less work in your next sprint until you reach a batch size small enough that you can definitively absolutely finish it.

Host 1

Fully packaged, fully tested, and working.

Host 2

Exactly. Take the political heat for not completing management's fantasy list, but fiercely protect the reality of delivering working software. That is the only way to rebuild trust and turn a failing project around.

Host 1

So for the engineers, architects, and product managers listening right now, how can they evaluate their own organizations tomorrow morning?

Host 2

Well, the sources provide what we'll call the three concrete checks to tell if you are actually practicing iterative learning or if you are just doing waterfall dressed up in two week chunks.

Host 1

Let's run through. Check number one. Look at your requirements.

Host 2

Are your requirements completely locked in before your coding phase begins?

Host 1

If they are.

Host 2

If you are handed a massive stack of requirements by a product owner and told you cannot change them as you learn new information during the build, you are not agile, you are doing waterfall, regardless of what your meetings are called.

Host 1

Check number two involves deployment. Can you actually deploy working, valuable software at the end of your sprint?

Host 2

This is a big one.

Host 1

This is a huge one.

Host 2

If you finish your two-week sprint, but your code has to sit on a server, waiting for a massive three-month integration and manual testing phase at the end of the year before a user ever sees it. That is just waterfall with more frequent meetings.

Host 1

And finally, check number three, autonomy. Did your team choose its own process, and do you have the power and authority to change that process to make yourselves more effective?

Host 2

Right. If a process isn't working, can you drop it? If the answer is no, you are suffering from agile imposition.

Host 1

Excellence in making software is really a lifetime activity. The corporate labels, whether your company calls it Scrum or Safe or Kanban or Extreme Programming, matter far less than mastering the core loop that Dave Thomas described.

Host 2

Find out where you are, take a small step, adjust your understanding based on reality, and repeat.

Host 1

Ultimately, the mechanics of software development are about managing risk through rapid feedback.

Host 2

If your current process increases the time it takes to get feedback from a real user or from the compiler, it is a flawed process, completely regardless of what expensive brand name is attached to it.

Host 1

Which sets the stage perfectly for our next deep dive. Because now that we understand the underlying philosophy, the history of the Snowbird Rebellion, the mathematical flaws of the Standish report, and the traps of Dark Scrum, part two of our series is going to get intensely tactical.

Host 2

We are going to leave the history behind and look at the actual day-to-day mechanics. Right.

Host 1

Scrum, Kanban, and XP in practice. We'll dig into work in progress limits, the mathematics of Kanban Flow, pair programming, and how to actually structure these workflows in modern distributed teams.

Host 2

Should be a really good one.

Host 1

Definitely. But before we sign off, I want to leave you with a final provocative thought to mull over, inspired by Martin Fowler's concept of sacrificial architecture.

Host 2

Oh, this is a question that really challenges the core ego of how most developers view their daily work and their legacy.

Host 1

Exactly. If Fowler is right, and the true is sign of a wildly successful code base is that it eventually needs to be entirely thrown away and rewritten to handle new scale. How does that change your emotional attachment to the code you wrote today?

Host 2

Good question.

Host 1

Are you building a monument meant to last forever, polishing every line for posterity? Or are you just building a temporary bridge to get your team to the next version of reality? Think about that at your next stand-up. Thanks for joining us on this deep dive into the source material. We'll see you in part two.

Save to playlist

Manage playlists →

Share