Extreme Humility

I think ExtremeProgramming and several other similar approaches are based on humility about our abilities:

we won't be able to work out all the requirements up front (so allow mid-course corrections).
we won't get the design right first time. So don't try! Just DoTheSimplestThingThatCouldPossiblyWork and then RefactorMercilessly.
we don't know how implementable this design is. So implement it and see.
we'll make mistakes in coding and interpreting designs. So test everything, run the tests every time you make a change.
we don't understand the problem well enough to predict what we'll need later. So YouArentGonnaNeedIt.
we can't reliably write simple, understandable code on our own, so we work in pairs for constant feedback.

DesignInXp makes some of the same points.

In contrast, the (pure) WaterFall method is arrogant in its confidence:

in the requirements phase, we will identify all relevant requirements, and correctly
in the analysis and design phases, we will interpret the requirements correctly, and produce an implementable architecture and design which will not need changes.
in the coding phase, we will interpret the design correctly and make no significant mistakes (a waterfall plan almost by definition doesn't have a large "rework after testing" phase. A plan may, and indeed should, have rework activity during testing and every other phase. The model does not, however, because "rework" is just "work" iterated.).
in the testing phase, we will test correctly, and (sufficiently) exhaustively
"We know how to design and you don't." That's why the designers "throw the design over the wall" to low pay "peon coders" to do the next phase.

(You can have have WaterFallWithFlowback?. You can then go back it becomes hard to know what you're actually doing, because you no longer know if you've finished (for example) design yet).

See also CleanRoomHumility, YouWillMakeMistakes?

Even an XP cynic like myself (sorry, I've seen too many silver bullets whizzing by in the past 40 years to jump on even inviting band wagons) can see the wisdom of what you say. Nice job, Paul!

Does XP claim to be a silver bullet? Interesting question. I've attempted to answer it on XpIsNotaSilverBullet.

"This is amazing!" -- RajWaghray?

Humility about one's abilities need not lead you into doubting your abilities.

The XP principles may arise out of humility but humility is certainly not the feeling I have when I practice these principles. On the contrary, I feel powerful, confident, and fearless. Perhaps even to the point of arrogance.

Humility does not oppose power, confidence, or fearlessness; it is simply a state in which you have an awareness and acceptance of your flaws. I don't think it's very surprising at all that you feel much more secure in your capabilities when you are conscious of where you lack them - and XP, as a methodology that works well in the presence of human frailty, is going to further encourage your confidence.

This is getting philosophical, but then methodologies seem to tend toward philosophy or religion. :)

But Seriously, don't you think the argument could be made that humility is empowering in a Zen like sort of way?

This quote sums up what I am trying to say:

"The more I learn, the more I realize I don't know."

: -- AlbertEinstein

This is actually an indicator that XP is on the right track. Perhaps because it forces us to recognize our limitations, it makes it possible to overcome them, and thus succeed in our endeavors (in this context to finish the project on time, and in spec). The fact that a feeling of empowerment grows from this is very natural.

Re: In contrast, the (pure) waterfall method is arrogant in its confidence...

Could you tell me where you found this "pure" waterfall method defined? A couple of years ago, I read the WinstonRoyce? paper (dating from somewhere in the 1970's I think) that first captured the term "waterfall" to describe this kind of progression in software projects. I was quite surprised to find that it was nothing like this stupid and rigid model we all like to poke fun at, and I saw no arrogance in his writing at all, only humility. Maybe, part of ExtremeHumility should include studying recent history, although that's not going to be "the simplest thing that could possibly work" for us.

[The Winston W. Royce paper referred to may be "Managing the Development of Large Software Systems: Concepts and Techniques", WESCON technical papers, 14., published in 1970. Does anyone know of an online copy?]

The above reference is correct. I just re-read the paper quickly. It seems the term "waterfall" does not even appear there, although a later paper by BarryBoehm (A Spiral Model of Software Development and Enhancement, 1987 IEEE) credits Royce with the "original treatment of the waterfall model" in the same 1970 publication. The "waterfall" we don't like seems not to have an author.

Waterfall methodology - as I was taught it, at least - includes the expectation that there will be regular cycles to the waterfall. Specify, design, implement and test what you currently know you need. Then specify, design, implement and test the changes to that. Rinse and repeat ad nauseum, until you have code which suits the requirements.

...and requirements that suit the sponsor.

I've yet to meet a developer who admits to ever having bought into a pure waterfall methodology. When I was taught the model back in 1987, it was constantly stressed that the so-called 'phases' naturally overlap and that iterating back through the 'phases' was inevitable. The dogma was that when a problem was identified during, say, 'build' or 'test' you should go back to the 'phase' that gave rise to the problem ('analysis' or 'program design', say), fix the problem there and 'fall' (quickly) back through the phases in between until you got back to the 'phase' where the problem was identified. Only then would the problem be 'fixed' (though you might have patched things up in the interim. In practice we often did this but I don't recall whether it was officially ok).

The disaster (as so often in our business) was the ProjectManagement take on it: 'the only way a waterfall flows is down', 'phase n shall complete on date t when phase n+1 shall start', 'ok I'll build in a week for handover', 'why do I still have analysts on my project when I'm in test?'. I thought I was making progress when I secured the project manager's agreement to the principle that phase n could be declared to be 'complete' when the developer(s) for phase n+1 said so. That way, he could have his plan and we could get on with the development. Unfortunately, phase n+1 developers would never declare phase n complete because if they did the phase n developers would disappear without trace! -- AlanAshtonJeanes

I don't think it requires a complete buy in. The key thing is what happens in response to a shortcoming in earlier phases. If the "fix" is always to make the earlier phase more strict/pedantic, you're in the trap. For instance, in my experience, when the requirements turn out not to be what the customer actually wants by the end of the project, if the constant response is "we need to write down the requirements more precisely/we need more requirements analysis", then I think you're in the waterfall trap. XP (and many other methodologies) accept that the quest for precise, unchanging requirements is unachievable, and deal with it -- PaulHudson

Fair enough. But I don't know of anyone who believes in unchanging requirements either. Or immutable design. Or frozen code (...no, scrub that). If the problem is that the requirement was wrong or has changed, either of which is highly likely, you should(according to the model) go back and see what difference that makes to everything that since has flown and fanned out from it. If you don't, you will still, eventually, have to find and fix the problems, by which time more work will have flown and fanned out from the unfixed problem. That's what I was taught, and I can still see the logic to it.

As has been said elsewhere, it's mostly a question of scale. Like XP, WaterFall works on small scales, with short elapsed times. Today, most people's approach to big development is to avoid it. If you can't avoid it, chunk it until it's manageable. How you proceed within any given chunk is, of course, with ExtremeHumility. -- AlanAshtonJeanes

Agreed, within one project. What I meant was what happens on the next project, or at the post-project review. If the complaint is "Oh no, a requirement changed, we didn't try hard enough to nail the customer down at the beginning" or "the contract shouldn't allow the client to change the requirements", then you're trapped. But I think we largely agree on the overall point. Agreed, that would be the start of a very vicious circle.

Theories:

XP Humility Embraces Many Little Mistakes to Avoid Catastrophic Failures
To Embrace Many Little Mistakes, You Must be a Confident Person
Arrogance is Actually Fear of All Failure, Large or Small

XP does not merely embrace change, it embraces systematic (and collective) mistake-making and lesson-learning. By embracing lots of LittleMistakes, XP reduces the risk of BigPublicFailure. Lots of little mistakes are like lots of naturally-occurring brushfires or controlled burns in a large forest. They constantly manage the risk of big forest fires by controlling the amount of fuel that such catastrophes require.

Confident people can easily choose to embrace lots of little mistakes because it does not feel like failure to them, it feels like incremental success. XP provides a collective and controlled means of keeping the mistakes small, and keeping them flowing at a constant rate.

But you can only be comfortable with a constant onslaught of little mistakes if your sense of self does not depend on continual mistake-free performance. If you know yourself to be fundamentally worthwhile, unplagued by fears of public failure, public humiliation, job loss, etc., then you have the luxury of gung-ho experimentation, mistake-making, and course-correction. If somebody points out an error in your code, and your reaction is positive and thankful ("Oh, cool, thanks. Good catch. Lemme fix that."), you are probably confident enough for the rough-and-tumble of XP.

If your reaction to someone pointing out an error your code is to deny, defend, and rationalize ("Oh, um, I did that for a reason. And I was gonna rewrite all of that soon anyway"), then you may prefer the illusion of mistake-free phases that waterfall provides.

To rephrase the Arrogance of waterfall list from the top of the page:

we will have a single closed-end requirements phase at the beginning, because we are afraid of getting them wrong and having to keep doing them forever.
we will have only one analysis and design phase, because we are afraid of getting it wrong and having to throw away badly-conceived design
we will have only one coding phase, because we are afraid of getting it wrong and having to throw-away badly-written code.
we will have a single big testing phase, because we are afraid of bugs, and we wish to put off as long as possible the miserable experience of slogging through bug fixes.
"We are afraid you might have good design ideas, which might make us look dumb or unnecessary. So we won't let you suggest any."

And the list of XP humble assertions, rephrased in emotional terms:

We don't mind that we won't be able to work out all the requirements up front. Making mid-course corrections will work out fine. In fact, it will be a gas.
It's fine that we won't get the design right first time. The sooner we get it slightly wrong, the sooner we'll get it entirely right.
We may not even be able to build this thing! Cool!

... etc

So it seems to me that XP's likeliest skeptics are those who are afraid of being "found out." Afraid that it will be discovered that they have occasionally designed, coded, and documented badly. And I mean actual fear: the kind that subverts your life by secretly affecting your motivations, your actions, and your ability to enjoy life.

And I think that the arrogance of waterfall, and of traditional software "engineering" in general, is born of varieties of that fear. -- PatWelsh

Making another link here. XP (although by no means perfect) aligns the software development process mechanism with the way that the human brain works. XP seems to me an evolution of TomGilb theories, and seems analogous to the TEFCAS theory of how the human brain learns described by TonyBuzan in HeadStrong.

It is time that we realized that we are flawed beings. If we were *really* smart, we would be able to take situations and immediately spit out the necessary binary code to instruct our machines. All the tools and techniques we have developed are to overcome human limitations.

Well, yes. That is what this page is all about, you know

If you wanted another comparison, you could say that one underlying characteristic of an AgileMethodology is that it tries to think of software development itself as a highly dynamic system - that is, more amenable to bottom-up, emergent order than to top-down order. This is how nature works, and although such systems aren't necessarily perfect or precise, it is usually quite resilient to outside shocks.

YouKnowYouAreaRealProgrammerWhen ...

The following views were expressed with ExtremeHumility sometime ago :)

I know this might seem inflammatory to some - but please consider that it's the opinion of many programmers who undeniably "Get it", myself included:

Real programmers, who seem to just be able to code from birth, are too rare to build a society on. Sure there's quite a few out there, but not the numbers that industry wants to be able to complete all of the tasks it needs done. These real programmers never needed formal computer science, never needed software development lifecycle models. They just Do It, and It Works. Sure they may have dabbled in such things at times, hoping to better explain things to others, but they never needed them. A whole lot of the greatest programmers who ever lived probably never cared about how to develop software by any model.

So, we're left with this vacuum - society wants more of these "programmers", because they can do nifty things, but there just aren't enough to go around. Salaries rise, demand rises. So a bunch of people who aren't really the "real programmer", but might be a passable substitute if you down a quick shot of bourbon and squint your eyes, and a whole lot of other people who couldn't code their way out of a paper bag, step in to fill the vacuum. They consistently fail to output the kind of results people are looking for, because they aren't the right tool for the job, and it's just really hard to "teach" someone who isn't a natural how to do this stuff.

So lots of theories emerge, and books are written on the software development proccess, etc, etc. Mostly these things offer some insight into the nature of the problem, but mostly these ideas are ultimately wrong. Finally, at some point in the evolution of this whole system of subpar code jockeys trying to find their way in the dark, someone starts making this Extreme/Agile Programming idea popular, because it sure seems to make a whole lot of sense.

As it turns out, Darwin wins again, and through stumbling in the dark and falling on their faces with ideas, this process eventually "evolved" a reasonably correct model. It turns out that the basic tenets, ideas, and feelings behind the general area of Extreme/Agile-type software development methodology, are pretty much what the "real programmers" were doing in the beginning. It's almost as if someone built a database of real programmers and spied on them for a few years, and studied them and documented their methods as best they could. This accidental evolutionary system actually found what it was looking for in many senses. Deflate the hype and over-analysis, just mull over the key points of the extreme/agile way of doing things generally, and try to be rational and think like a "real programmer", and you're so much further along the enlightened path than anyone following any previous software development life-cycle hogwash. It's not perfect. It can't encapsulate the genius of the real programmers. However, it's about as close a facsimile as you're going to get, so embrace it. Just don't go being too academic about it and over-thinking things.

Response from another, also in the distant past

Dear sweet merciful delicious Jesus-crackers.

Please tell me that out of the hundreds of skilled professionals that read that entry, I am not the only one who read it and realized that it was a subtle jab at ExtremeProgramming?

From the 4th word it starts with "might seem inflammatory", it talks about "RealProgrammers" versus idiot coders, and how "Extreme/Agile-type software development methodology" can help make up for the use of the latter instead of the former, etc. You call that subtle? How could it be more explicit; by throwing in obscenities?

Ironic to find it on a page called ExtremeHumility, no?

I seem to remember an article by EwDijkstra called TheHumbleProgrammer that addressed some of these ideas in 1972!

Yes, that's been noted before. See ExtremeHumilityAndHowWeAchievedIt

Contributors: PaulHudson, JeffGrigg, BradAppleton, AnonymousDonor, GlennVanderburg, BenAveling, BillCaputo, JennVesperman ...

CategoryExtremeProgramming