Why It Is So Hard To Sell Extreme Programming

In Y2K, I read two books that radically changed my thinking about software development and how it relates to business goals. The first book was ExtremeProgrammingExplained, the second was CrossingTheChasm. XPE told me HOW to EmbraceChange, CTC told me WHY. Both of these books describe a flexible, but highly disciplined process for product development. XPE from the software development point of view and CTC from the marketing point of view.

CrossingTheChasm describes the TechnologyAdoptionLifeCycle. Once I understood the PsychoGraphics underlying the TALC, I could see why so few developers have adopted XP. The Innovators who would be interested in XP are probably already involved. I'm sure that there are a few Visionaries involved (you know who you are). I know that I was attracted to XP because I was looking for an order of magnitude improvement in quality, speed, skill building, teamwork, fun, etc. Are there other Visionaries out there who haven't heard of XP yet? Or maybe they don't see how it would fit in their projects.

I really want to see XP become the mainstream software development method. Why? For purely selfish reasons. I believe it is a better way to develop software and I want to work that way. My worry is that like SA/SD and OOA/OOD, it will take an awfully long time before there is general acceptance. I think that XP is a quantum leap forward just like structured programming and object-oriented programming when they were introduced. A lot of shops paid lip service to those methods, long before they were really using them.

An even bigger concern, is how to cross the chasm to the Pragmatists. The majority of MIS managers are Pragmatists. They are not going to buy into XP en-mass until a lot of their peers are having success with XP. Yes, that is a Catch-22. No offense to anyone intended, but would a Pragmatist call C3 a success? Motorola appears to have completed a multi-national, multi-site, partial-XP project. Would a Pragmatist take a reference from them? I don't know, but I'm not confident. The fact that Pragmatists reference other Pragmatists in their MarketNiche before making buying decisions can not be over emphasized.

CrossingTheChasm says that you need a BeachHead. If you can provide a WholeProduct to that MarketNiche, you can meet their CompellingReasonToBuy?. Then, you can begin to generate the ReferenceBase? needed to convince other Pragmatists to buy. Then you go after another niche, and another, and another, until the momentum begins to feed on itself.

Why do I view XP as a high-tech product? Even though the Extremos say that all you need is a stack of index cards, that isn't going to work with Pragmatists. They want a WholeProduct. History shows that whole industries grew up around SA/SD and OOA/OOD. Dozens of applications were developed. Hundreds of books were written. Thousands of consultants appeared. Tens of thousands of people went to conferences and they all bought the tee shirt. All of this to support a simple idea. It will happen to XP as well, if it becomes a mainstream development method.

-- GaryBrown?

It would be great to have an answer to this question. Speaking as someone who has tried and failed with about ten out of ten coworkers, I'll give my tentative opinion.

Of course, it's going to be a different set of reasons for different people, but so far I've seen the following:

Maybe if you could overcome the communication barrier, you show them how much better things could be, thus overcoming the complacency. Once the complacency is dislodged, the desire to do better can overcome the fears. That's tough, though.

The communication barrier that you have encountered is the chasm. The folks that you are working with are Pragmatists. I'm guessing that you are an Innovator or Visionary with respect to XP. You have a different set of goals. See PsychoGraphics. They need other Pragmatists to tell them about their success with XP before they will be willing to try it. -- GaryBrown?

XP adoption requires significantly more involvement from the clients than traditional methodologies. Adopting it requires climbing two TechnologyAdoptionLifeCycles at the same time, as you are replacing both software creation and RequirementsAnalysis at the same time. For many business clients, a better software development process at the cost of their being frequently distracted from their other duties, which their bosses value more highly, is not a good trade off. Also, they end up more responsible for the results and avoiding responsibility for things you don't control is part of Management 101. -- MarkSwanson

I think that the customers of a software development project need to be a lot more involved. They are responsible for identifying the requirements that bring business value to a development project. Those requirements are not static in most cases, especially in new and emerging markets. To me, the genius of XP is that it places the responsibility for defining business value on the customer. It also give the customer the power and flexibility to achieve that value by steering the project.

I agree, that adopting XP requires significant change throughout an organization structure. I believe that this is a GoodThing. Traditional methods cause communication barriers to form between developers and their customers. XP asks the customer to be responsible for and control the quality, cost and scope of a project. What could a customer be doing that has more value to their boss/organization? -- GaryBrown?

Is it really hard to sell ExtremeProgramming? I had an awfully easy time out of it. Maybe because we were a startup, and our management hired me explicitly to bring in "real software engineering experience" (cough cough - I didn't tell them that software and engineering is an oxymoron).

Anyway, when I said we'll do PairProgramming, I expected the standard "no way you'll have two resources when one will do" bit, but instead I got a "why?". I told them we improve quality, couple it with CollectiveCodeOwnership and change pairs frequently to distribute knowledge around. This minimizes business risk and increases the BusNumber. They bought into it immediately.

I told them we'd do the PlanningGame, and engineering was going to be a service function towards business. They loved it.

True, most developers were hired based on their willingness to go along, but one was pre-existing, and though he hated the idea, he now thinks that it's great. (He's a smart, honest guy with an open mind.)

Anybody had similar (easy) experience in selling XP? -- AlainPicard

I had the same experience in a start up, except that I was hiring all of the developers. I gave each of them a copy of XPE after their initial interview. I only hired the ones that really wanted to work in an XP environment.

Start ups are more likely to be populated with Innovators and Visionaries. See TechnologyAdoptionLifeCycle and PsychoGraphics. It is not necessary to sell them. The Pragmatists that run most IT/MIS shops do need to be sold. And don't forget that they need to reference other Pragmatists who have had success using XP. -- GaryBrown?

I work for a small consulting/custom engineering firm, and I think they will accept XP. The problem we face is that we consult for Fortune 1000 companies. It is clear that they (well, at least the one I'm working at right now) need help in the development process. If they'd brought us in to consult on process, maybe they'd buy it, but I that's not why we're there. But as things stand, I think there are too many political groups (with great turf-protection experience) to convince, IMO. And then there is the delicate nature of us telling the customer that there current process sucks (though you'd think they know that).

If it weren't that XP started at Chrysler, I'd think it was a totally hopeless cause. -- DavidCorbin

Large corporations have many ways to absorb risk and thwart change, as they should, I suppose. XP worked at Chrysler only because all of these mechanisms had failed them at once. Once out of the Y2K bind they reverted to usual corporate behavior, only then as an even larger company. -- WardCunningham

Call me anti-social but I can't face PairProgramming. It's my biggest stumbling block.

Are you speaking from fear, or experience? If experience, tell us why you found it so painful. If fear, BeBraveLittlePiglet.

Also the fact that refactoring is not trivial in C++, VB and many other mainstream languages.

Doctor Doctor, it hurts when I do that! So don't do that.

Are you saying that one can't, or shouldn't do refactoring in C++, VB, and many other mainstream languages?

If you ask me, I think XP is still firmly in the Innovators stage, not the Visionaries stage. The graph at the top of this page shows a 3-5 year time frame, but I believe that time frame is for individual technologies such as the Walkman and the cell phone. XP is a culture, and it takes a lot longer to adopt a culture than it does to adopt a technology. The success with start-ups (above) is where XP will gain its acceptance and cross the chasm, especially start-ups where the person in charge of setting the process is an XP enthusiast. I predict (off the top of my head; and I'm no expert) it will be 2 more years +-6 months before XP moves into the Visionaries stage. From there it should move quickly through to Pragmatists (say another 3-5 years to reach the peak).

I have to disagree. The fact that there are several XP projects under way tells me that the Visionaries are in the game. The Innovators for XP are the folks who say that they are trying SoloXP. Look at pp 57 - 59 in CrossingTheChasm to get a feel for the difficulty in getting the Pragmatist IT/MIS managers to buy XP. -- GaryBrown?

Those who EmbraceChange will thrive in the MarketPlace?. Those who do not will not thrive. The MarketWillDecide? when the peak is achieved.

Part of the problem may be in the name. "Extreme Programming" may conjure up images of CowboyCoding among a less-adventuresome ItManager. See AnExtremeConversation for an example.

Just seeing ExtremeProgramming and CrossingTheChasm in the same sentence made it click for me. I think Gary is right on. But what is a good BeachHead? Startups are probably not it, because they tend not to be viewed as Pragmatists. What MarketNiche does XP fit best into? Well, clearly one in which requirements are likely to change (although my personal view is that the requirements on all projects are likely to change). I need to go home and read the book again.

This is good, and I'm excited, I'm just inarticulate.

-- PaulTevis

I actually think startups are the best way to introduce XP. You get the culture in place right from the start and 'grow' the company on XP (like fertilizer?). Culture is the biggest stumbling block XP has to handle. XP is not management-friendly on the surface (XP requires fewer managers and gives them less control). XP is a ParadigmShift, and ParadigmShifts are brought about by 'the next generation'. If you view corporations as entities (which legally they are, but what I mean is that the culture of a company is what makes it a cohesive 'individual'), then the next generation are the startups. I'm starting one up right now! :-) -- RobHarwood

Thinking about this more, two possibilities come to mind. One is along the lines we have been discussing, i.e. the mainstream acceptance of a discontinuous innovation through chasm crossing, as Gary originally proposed. The other is along the lines of the scenario outlined in The InnovatorsDilemma. In this view, XP is a "disruptive technology" which means that addresses a different set of consumer needs (the author's term is ValueChain) than does the technology it is trying to replace. As a result, most companies reject it because it does not address the needs of their market (because they want design documents, deployment plans, and other things that a heavier-weight methodology provides). Those companies that do adopt it have different needs because they address different markets (rapid deployment, responsiveness to changing requirements, etc). If XP is successful, then it migrates "up-market" as those companies who did not adopt it gradually have their markets eaten away by those who did (because as it evolves, it proves to address the needs of other markets as well). -- PaulTevis

I believe the latter will happen as you suggest. In fact, I'm banking on it. In my field of computing (multimedia), there is a huge need for any process. I believe XP will be my competitive advantage. My job now is to educate my clients. See FirstCreateTheMailbox. -- RobHarwood

Discovered over on VcapsProject
"What you get is a very good environment to apply ExtremeProgramming. Basically we are bean counting.The Problem is that the way we count the beans is very important and is dependent on exactly how the analysts are counting them...this time. Each go around will have a new set of rules and government regulations."

Perhaps financial systems are the right MarketNiche for the chasm crossing. Both VcapsProject and ChryslerComprehensiveCompensation demonstrated the effectiveness of ExtremeProgramming in this market. -- PaulTevis

Discussion about the issues and practicalities relating to OnSiteCustomers? moved to ProblemsWithOnSiteCustomer.

I have to admit that the PairProgramming is the biggest stumbling block for me too. I've tried it, and it was fun, but when I got done I felt like I'd been wasting company time because I wasn't coding myself. To me, it seems that you would have to see at least a doubling of productivity to make it worth it to have two programmers working in a pair. I have a hard time believing you get that much of a boost the majority of the time. -- EddieDeyo

You won't get a boost right away. It takes time to become accustomed to the different roles. When I started out, it was like there were two individual programmers sitting at the same machine. Later, when I realized that each programmer plays a very different part, it became more like PilotCoPilot. That's when my partner and I really saw the benefits. It helps to switch roles fairly often as well, because then you will more quickly see that each person is doing (and supposed to be doing) different things.

Geoffrey Moore's chasm crossing model asserts that selling technology initially involves romancing the visionaries and early adopters, but ultimately the mass market only cares about price and warranties.

XP is an excellent means to this end, but not an end in itself. XP could be the service that creates the product that is crossing the chasm.

The "craft" quality of XP is very effective in the early stages of product development where only 10% of projects survive to CrossTheChasm?. When a product or technology becomes fairly commoditized, then you start to see more repeatable, but less agile, software "engineering" practices start to supersede the software "craftsmanship" practices; perhaps because accountability to shareholders starts to supersede accountability to customers.

-MichaelLeach

Many projects fail due to a lack of sufficient planning and understanding of the goals. To people who keep getting hit with "We don't do enough planning.", a suggestion to do the opposite of BigDesignUpFront (instead of more big design) may come off as ridiculous and not worth further consideration.

Many projects suffer from ScopeCreep, drifting away from the spec into uncertain territory, and insufficient change control. To someone who firmly believes that "We must adhere to the spec, and stop letting the client change things, or we'll never get these projects done!", a suggestion that the problem could be solved by doing away with the spec and letting the client change things more often and more easily just sounds like blasphemy.

When a project suffers from PrematureRelease?, with insufficient QualityControl or QA (usually due to insufficient time), a suggestion to release earlier and more often seems impossible. How can you say release earlier, when we just released a month late and now have to eat the cost of all these bugs?

When processes are failing, and a team really needs a new methodology, they usually have high overhead and insufficient productivity. Pair programming, implementing test frameworks, and spending half your time writing test code just don't sound like things that would improve the bottom line. Sounds like the opposite.

So, given that the major tenets of XP sound more like extreme versions of the problems that caused someone to look for a new methodology, rather than the solutions, how do you sell the synergy? Or how do you get them to listen long enough to sell it?

Would be fair to characterize it as a HighDisciplineMethodology?

A few other people have said something along these lines but I think it bears repeating. As a programmer who has worked with other programmers, I can say that we are, on the whole, smelly and unpleasant people. Additionally, having to be given a buddy, like some kind of kindergarten student on a field trip, is the single most degrading and humiliating thing that I have ever heard asked of a skilled professional. Pair programming is such an unpleasant concept to me that I immediately lose interest in every other aspect of XP and refuse to even consider the idea. I would not only quit a job if forced to pair program, but I would quit programming altogether if it came right down to it.

Re: "I can say that we are, on the whole, smelly and unpleasant people."

Ya don't say? Time for VirtualPairProgramming? Smellavision has not been invented yet and if your partner gets too ranty, pull the plug or run his/her voice thru the Pleasant-A-Tron filter.

Research shows that great ideas and productivity results from working uninterrupted in a quiet place all by yourself. What about brainstorming, you say? Very little comes from meetings. The great ideas are thought up AFTER the meeting and discussed informally. Groups rarely come up with a collectively great idea.

Programming in pairs has become the 2014 buzzword and I'm getting really tired of hearing it. Perhaps it has risen its ugly head because in Europe, a bachelor degree in computer science is a 3-year study with very little practical application. The really great engineers come from the polytechnic universities with 4 -5 years of solid studies under their belts. In America, a bachelor of science is a 4-year degree and the computer science cirriculum is heavy duty programming along with some theory and academic teaching. But the guts of a any good computer science program is producing software engineers with some clue about how to code well.

It's true that novice programmers get flummoxed during debug sessions. If the runtime error says, "file not found", a novice programmer may assume it's true and start checking around for the existence of the file or verify that the code has the file named correctly. If everything looks good, the novice programmer is stuck. He or she knows something is wrong, but can't figure out what it is. The best action at this point is to wander over to the desk of an experienced engineer and ask for some help. The experienced engineer has seen it all and done it all. He or she knows the "file not found" error can be one of many possibilities, NONE of which has to do with the file. Yes, that's right. It could be a server problem or the DBA has temporarily locked the file on the test platform and forgotten to tell anyone about it, or the file has become corrupt or maybe... something else in your code has pointed to a different file and that subroutine is now active, so when you try to access the file you want, you simply can't. The pitfalls of novice spaghetti code and misuse of GLOBAL (PUBLIC) variables wave at you!

Hurray that extreme programming in pairs solves these problems on the fly. But....Huh? Really? Put two novice software engineers together and you have inexperience multiplied by two. It's costing your company twice as much to not figure out the error because programming in pairs means you solve it yourselves. There is no wandering over to the senior software engineer for quick advice. And even if you could, he/she is chained to another programmer, so if you interrupt them with a question, you now have FOUR people involved in the task. How can THAT possibly be cost-effective?

The solution, you say, is to pair an experienced software engineer with the new kid on the block. Wow, do you REALLY want your highest paid software developer sitting idle while some newbie types in code? That is a sure-fire way to de-motivate your top engineer and risk losing his or her expertise to a firm willing to pay more and provide a plushy private office. You know the really great software engineers by the cars they drive: Jaguar and Porsche. Ok, maybe a Lexus for the not-so-sporty types.

A good programmer with experience will out perform two novice programmer paired up. Why? Because after a few hours sitting with a co-worker, the top notch programmer will start to go a bit insane. You have ZERO privacy, which means you can't talk 5 minutes in private when your spouse calls to let you know the conference is running late and you're on your own for dinner or your kid skinned his knee at the school playground and is going to the ER as the school nurse doesn't put in stitches. You can't check your email to confirm your visit to the client site tomorrow because the computer is currently in use by the other half of your pair.

Ladies, don't comb your hair, check your nails, and please try to limit the number of times you run to the bathroom during your period. Honestly, we are programming in pairs and all this female stuff is distracting. Yeah? You say, "I'm a female and job laws against gender discrimination are on my side. You can't even complain because it's sexual harrassment."

Oh boy, what a nightmare!

But the BIGGEST CHALLENGE to paired programming is not just the inconvenience and unnatural setting of being married to a co-worker for 8 hours every day, 5 days a week. It's the constant interruption to your mindshare. You can't concentrate on difficult tasks because you have no solitude to really think. The brilliant ideas you get when you are in the midst of writing code flee away the minute your partner says, "you mistyped that SQL statement". Wow, thank you for interrupting me. I like to fix my typos when I'm done coding in my brainstorm.

And for those of us with 20 years of experience under our hats, those of us who were on the front lines when microcomputers were still considered "toys" by the mainframe crowd, before Microsoft invented certification and you had to design LANS with no clear roadmap, we have forgotten more than you, the programmer with 5 years experience, will learn in the next five years. We will teach you, but programming in pairs is not going to teach you any faster.

You see, the senior software engineer of proven quality needs to continue pushing the frontiers and working on continuous improvement. We invented Agile methodology, we cobbled up hardware to make what we needed when vendor offerings were scarce, we crawled under desks to wire up LANS, we wrote in assembler to bolster early OS and overlaid our high-level object-orientated application software. We experimented, we succeeded, and we learned from our failures. Case in point: A typical software system my team of three spent 2 months programming resulted in a $500,000 ROI (return on investment) after the first 6 months of deployment. And none of that was the result of paired programming. Today, we are moving beyond Agile to hybrid methodologies which break the traditional waterfall CDR into IDR's for clients who fear iterative development with no roadmap. We adjust, we invent, and we will gladly hand you the reins when we retire so can step into the saddle and take off running. We will teach you, guide you, promote you, and celebrate your genius.

But I will not be your programming pair and share my desk with you. I don't even sit next to my husband when I play on Internet during my Sunday afternoon leisure time. Why not? Because the peace which comes with a bit a privacy refreshes the mind. Some of us, who are social but not "relationship needy" thrive on a robust amount of solitude and that is NOT to be misconstrued as being asocial or geeky nerdy introversion.

Clearly paired programming is one of the biggest controversies to land on the software developer's desk. Back in the 80s and 90s, we tossed around the idea, we read every book as they came hot off the press, and those of us who were brilliant designers and coders looked at each other in disbelief because saw straight through the fallacy of being chained to a co-worker. To mentor the new kid on the block is worthwhile. But to pair up your best minds and most capable software engineers without any time for working on individual tasks is, put bluntly, patently stupid.

Advocates of paired programming tend to be business-oriented managers who have no clue what coding actually entails. They envision that two heads are better than one. I won't be surprised if they dream up the "keyboard for two". Their paired programming hype is bolstered by other advocates who say that loner types are egotistical monsters who want to "own" the code and can't work with other people. To them, I say, "Rubbish". I am a professional, I excel at my tasks, and to handcuff me to another person - equally bright or not - is imprisoning my innovation and destroying my ability to concentrate. I'm tired of "consultants", who fumbled their way for 5 years as mediocre software engineers, and now preach paired programming far and wide because they never exceled at coding. Tell me something... do you envision SELLING IN PAIRS? How would that work for you? Maybe cut your commission in half... or would you work twice as hard to double your sales, which would be split with your partner, so you would make exactly what you make now...??? Hm...

For "R-type" personalities, (relationship-oriented), programming in pairs might be the ultimate "fun", resulting in some real productivity. But that is only one personality type. For those of us who are highly motivated, capable, and experienced, with a track record of producing excellent code in a minimal amount of time, you are wasting your money pairing us. We already work as a collective team in our sprints, taking turns as the SCRUM manager for various projects. We code in our solitude, exchange ideas in our meetings, and leave our tracks on the whiteboard for all to see. It works because it is the IDEAL BLEND OF INDIVIDUAL WORK AND TEAM WORK.

Victoria Suominen, May 2014

See also HowToSellGoldenHammers

EditText of this page (last edited May 5, 2014) or FindPage with title or text search