Rons Xp Project

Refactoring suggestion: move some material to XPDoesntScale, and add comments to WhatWorksBetterThanXP. Delete this page from the ExtremeProgrammingProjects list since it's not currently an attempt to apply all 12 principles.

I work at a big company. I won't name it, and if you know me then you know which company this is. I'm going to be purposefully abstract (not vague, see VagueVsAbstract). We're building a reasonably substantial subset of a real product to put in the field to measure certain aspects of such systems. We're a splinter group, in that the people who build these kinds of things for a living aren't really helping us.

Some details:

What is working: What is NOT working: RonCrocker

Hey Ron. Here are some suggestions for fixing things:

As to scale, XP just isn't intended to work with that many people. If you can partition them into autonomous groups, then you might scale it that way, but still at the largest levels of scale you're going to need business planning logistic support that isn't in XP. Try ExtremeUnifiedProcess, TeamFusion, or even just RUP at those scales. -- PeterMerel

You can, too, force people to pair. I know of two such projects. On one, the consultant / team lead came in and said, "You will pair program." The people raised many objections, but in the end he got them to pair. After a couple of weeks, they were even eating lunch to together for a change. -- AlistairCockburn

Okay, let me rephrase that: you can't force some people to pair ... -- pm

The project is over now. I'm going to be speaking at the OOPSLA 2000 panel on XP. Here's what I sent in as my position statement. Kent asked me to post it here...

Ron's position: "XP doesn't scale"

Short statement:

XP is a personal-communication-intense software development process. Pairwise communication grows as the square of the number of people; this number becomes unmanageable at a reasonably small number (<20). Large projects by definition have many people. A project with 40 people is 4 times the communication of a project with 20. One possible solution is to carve the big system into many loosely-coupled smaller projects. XP works in the smaller projects, but the coordination problems and project management among the smaller projects becomes new problems. It seems like an answer might be to have a small "Extreme Architecture" or "Extreme Management" team to coordinate the development teams (the former seems better).

Longer statement:

I was the system architect for a rather large development effort. The system consisted of several like-sized (10-15 engineer) subsystems that needed then to be integrated to make the system. Our project was ideal for an approach like XP - we were dealing with a great deal of unknown-ness, and I've heard XP described as moving from the known to the unknown. Our system level approach was to start system integration early, with a little bit of end-to-end functionality, growing the system capabilities until the whole thing was done. Our schedule had 3 week increments for end-to-end functions, with some shorter increments for pair-wise integration steps. We had a simple system metaphor, easy to remember and meaningful, that was shared among all the teams.

Not only was I the architect of the system, I was also the development manager for 2 of the subsystems. The down side was that I only had 2 of the teams. We did XP (or close to XP - it depends on how literal you want to be), but the other teams were free to do whatever they wanted, as long as they met our incremental integration schedule.

I pledged at the beginning of the project that I wouldn't be making the following statement: Our stuff was never the problem. Well, our stuff never was the problem. We had required functionality in the lab on time, and were flexible enough to fix other subsystem's problems when required. Because we were XP, we were able to add stuff as we needed it and were even able to accommodate temporary changes to support deficiencies of other teams (which, curiously, were categorized as fixes to OUR bugs - if you can fix it, it must be your problem).

Nonetheless, we were still at the mercy of the other teams. Several of the other teams did their process, which could be called iterative or incremental, but the spins were much coarser than our 3 week increments. We had to compensate for their inability to provide their content by building simulators for their parts. This was good, as it allowed us to build confidence in our system working. It was bad because it reinforced our assumptions, but did nothing to determine if we had shared assumptions; often we did not. One particular example that sticks out in my mind is that we used a value of "50" in one specification as the base of some range of shared information. Our implementation treated this as a decimal number, as did our simulators. The real implementation of what we simulated treated this as a hexadecimal number. We didn't find this out until late, because we reinforced our assumption and had to wait until the real implementation showed up.

Managing this project became quite "interesting." Because of our highly integrative approach, we often had stuff that we felt good about. However, because not everyone was playing, we couldn't ship what we had. We also had difficulty telling our management something that they understood. We had a task list for my teams. The task list grew as we went through the planning game. We added tasks as we refined our understanding of what we were doing. We had charts that showed our progress toward achieving our tasks, and we had monotonically increasing number of tasks completed per week, but our "% of tasks completed" number would vacillate based on how many tasks got added each period. We also had data on the integration phases completed. Unfortunately, my management didn't grok this information. They wanted to know if we were done with design, done with coding, done with testing, done with subsystem integration, ready to hand off to system integration, etc. I spent a lot of time trying to convince my management that we were on track, even though it was obvious that we were on track ("Go down in the lab and I'll show it to you") and the other teams might not be ("We're 93% code complete," but does that really mean anything?).

Ok, so it seems like XP doesn't scale. That doesn't mean XP is bad, it just means that it doesn't scale. There are many projects that don't need to scale to 100s of engineers. XP seems to work well there - see the other panelists for that information. We found XP worked nicely for our part of the project, but our approach to building the system this way created quite a bit of different problems. To make XP a scalable approach requires solutions to the coordination issues.

-- RonCrocker

I don't understand how the conclusion, "XP doesn't scale" (with which I agree if scale is properly defined) follows from the experiment. None of the teams did XP on their own. The 12 rules were not followed.

What was done wasn't unsuccessful, exactly, and wasn't XP anyway. So ... from this description, I don't follow the logic. -- RonJeffries

Have to agree with Ron. The conclusion I would derive from this story would be that one good team won't save a large multi-team project if the other teams aren't good. (Must be a better way to word that...) The experiment didn't actually test the hypothesis that XP can scale to big systems by carving "...the big system into many loosely-coupled smaller projects" each of which are XP projects. -- JasonYip

Jason, which Ron are you agreeing with? (I'll assume RonJeffries). I wouldn't say that my teams were "better" than the other teams. I can agree that this project didn't test if multiple XP teams working together works. I still see a fundamental problem in the coordination of the teams. RonJeffries claimed in a separate email that XP doesn't define inter-group communication at all. It seems like there is no definition for how to deal with projects that are bigger in scope than what can be handled by a single XP team. Is there? -- RonCrocker

Surely the problems your team experienced would have existed whatever method they were using. It seems to me that the problems were not caused by your team using a (partial) XP methodology, but by the other teams NOT using it. My conclusion from the evidence reported would be that NOT using XP doesn't scale. -- DaveKirby

Some yes and some no. Some were caused(?) because we had stuff working a lot earlier than the teams we wanted to integrate with. -- RonCrocker

I wrote this yesterday but then had problems posting it... note that _your_ refers to RonCrocker's.

I think that Jason was agreeing with RonJeffries. I'd say that your teams were at least successful if they were functionally complete on time, I'd be impressed to hear that all the others were too. If the others weren't, I'd say that makes your teams "better".

As to inter-group communication, I don't know. I'm far from an XP expert... in fact, most of XP has existed in people's minds for longer than I've been alive. Please don't take this as me discouraging the effort. I work ~5 miles south at another Motorola building, and I'd love to be doing XP (it's a hard sell... especially the snowboardy name and the PairProgramming).

It seems to me that you didn't test if XP scaled. You tried parts of XP in 1/6 of a big project, but you certainly weren't doing anything remotely XP at the higher level which I'd say you'd have to be doing to do XP.

In the example of failure you gave about hex vs decimal, it seems the failure was in the BigDesignUpFront, not the XP. The design failed to specify the interface between two modules. I'm no expert on BigDesignUpFront either, but MythicalManMonth teaches that as soon as someone saw that they were going to make an assumption in the requirements, the assumption had to be documented in the requirements. If it were, then whomever didn't follow the requirements fixes their code. Now, I understand that this is virtually impossible on big Motorola projects (let's just say it takes a while for requirements documents to change), but still... You can't blame that on XP being as how XP's code is the documentation of how the system is to work.

As to the % Completed problem.. I don't see it as a problem. Let me explain. I'm working on a project now that gets it's output fed into a much larger project that's completely out of our (or anyone else's :) control. There's a 10-15 person team working on changing the interface in the much larger project in 2 phases. I'll do a little blow-by-blow of how effective their 'code complete' reports were:

So... the moral to the story: The code complete %'s were completely useless. We're now looking at the very real possibility that the schedule may have gone from 5 months long to at least double that time, and there was never any real warning until after the scheduled hand over times.

Of course, most managers don't just accept that their tried and failed methods don't work, so you'll still need to give them the numbers that they want. Given feature tests, you should just be able to hit print screen and your tester and tell them that the team has now completed 8 out of 10 features. You could then pull a % complete number out by extrapolating extra tasks encountered from past iterations. That number should at least be within 50%... which is better than many of the code complete %'s I've seen.

For somewhat related reasons, I don't think XP as-is can scale directly to large projects successfully.

Most importantly, I don't think current computer hardware allows for rapid enough integration, testing, and refactoring on huge projects. I know of a multi-hundred person project that has been going since the early '90s. The time to build the system earlier this year was about a day. They threw a big pile of $$ at the problem for a huge build machine, and I think they've got it down to about 1.5 hours (there's only one machine that can do that build). If I change the interface on one common object, I'll have to rebuild to test. I don't think it's possible to make the testing painless when it takes over an hour to rebuild, and multiple minutes just to get the new files onto a machine where you could run a barrage of unit tests. I suppose this might be work-around-able if you could somehow automatically separate only those modules that ever get data that's ever derived from data from the unit that was changed, but I don't know of such a system right now (seems like that system itself would take eons to run).

Of course, with a >10 person project, you'll pretty much have to split them up because of communication overhead. After that, you need to tell each group something to do. To still call the whole process XP scaled, I think you'd have to do it without the design up front. Without design up front, I don't quite understand how you could get everybody on the floor running at the same time.

Perhaps you could "grow" a big XP project much like you grow an XP developed program. Maybe you could start with a 10 person group that went through to figure out some things the program had to do, and they could DoTheSimplestThingThatCouldPossiblyWork for a bit. You'd have to start dividing the program through interfaces that seemed unlikely to change very often. Then you could add developers working onto divided portions of the project only. After a time, you could have a great number of developers working XP style behind some interfaces that couldn't change. The code would still be the documentation, but you could build and test parts separately.

It doesn't seem very XP though to have interfaces that are so static though. What do you do when the interfaces have to change? Do you have the original members make a design judgement and push the change down that way? Do you re-group (refactor groups?) so the interface is internal to a group so they can change it but other interfaces become static?

It'd be interesting to see a >100 person XP project in action. Unfortunately, I think I'm a few million short to fund it myself. I'll chip in my $1 of weekly donut money though. It's worth that much to me, and I really like donuts.

RonCrocker: if you're trying an XP project again up at Mot Arlington Heights, let me know. I've never seen one in real life. It'd be nice to talk to some people in person and maybe see if I could work on it (you know you're a computer geek when you a) know what the KMP search is, b) have components with more than a decade between them in the same computer, or c) are willing to blindly transfer to a different group, project, and location when you hear they're using a different development methodology).

-- GeorgeGruschow

I wanted to clarify my view in this more permanent location than the egroup. While I don't think RonCrockers experiment speaks directly to XP scaling, it does speak directly to inter-project coordination and thus indirectly to projects made up of short-cycle and long-cycle processes, and thus indirectly to XP in conjunction with non-XP.

I do think that some of the XP practices that were dropped just barely might have helped, but as RonC sees, the biggest problems were in inter-team coordination. Some PlanningGame extensions might help there, but the basic problem is incremental vs big-bang delivery. Of course, big-bang delivery with multiple teams invariably gets you in trouble, so none of us should feel bad about the problems that Ron has seen and kindly reported.

One suggestion that this report brought to mind was to have the rule be that the first team to define and use the interface to an object owns the definition. That is, if your team builds and tests stubs for my team's product, those stubs' interface definitions are the official definition, unless we mutually agree otherwise. This would give slow-cycle teams an incentive to get actively involved with the fast-cycle users of their objects, to protect their turf. But the only way they could protect it would be to actually be helpful, since the rule is that the first definition in use sticks.

I'd be very uncomfortable using this suggestion. Remember, you goal is to build software someone can use, not to beat up non-believers. Work together and find a consensus approach in the time frame that is most appropriate to both groups. Advancing the idea that XP doesn't play will with others will harm its acceptance, not advance it.

RonCrocker has collected some important information here, and I look forward to seeing it presented at OOPSLA. I sincerely think that even with the short time he'll have in his panel, he has a shot at the most significant reported result. -- RonJeffries

In the on-going discussion between "The Two Ronnies," I think I have clarified my thoughts quite a bit. I can only hope that I have half the impact that Ron J thinks I might... -- RonCrocker

Helping XP scale:

If each subsystem team is using XP, can Team A create SharedAssumptions? by running their UnitTests against the stub Team B is using?

OR

If each subsystem team is using XP, does ContinuousIntegration remove the need for Team B's stub? That is, can Team B just use Team A's subsystem directly?

One of the explicit rules we use in our multi-team project is that the "prime" team can't dictate process onto the "sub" teams. They can dictate (i.e., pick) two of these three: what, when, how much. How it gets done (that is, the sub's own process) is a fixed part of the bargain. So, I guess I'm ducking your comment above, but I don't see why I would be required to force my partner teams to use my process. Let's say you run an XP team. Let's further say that you are being asked to build something by someone willing to pay you to build it. Would you allow those people to tell you NOT to do XP? -- RonCrocker

EditText of this page (last edited February 20, 2003) or FindPage with title or text search