Change Your Organization Diary Part Two February

Start with ChangeYourOrganizationDiary. See also ChangeYourOrganizationTactics.

Next and previous parts:

6 February. I spent the day today going around to each of the people who are going to be involved with the project, explaining the basic concepts, and getting their buy-in. I talked to the on-site customer (to be), programmers, and all the management folks I know. Everybody's a little nervous, but excited.

While I was doing this, I made an interesting discovery. I have a pretty good relationship with one of the management folks, so I was asking him about the political situation at the company. Who was resistant to my plans, who was fearful, who might feel their job was threatened, and so on. In the course of doing that, I discovered that the director -- my boss's boss -- hadn't actually given approval for my project! Well, he approved the project, but he didn't know I was going to do a (nearly) full-blown XP process.

Oops.

It turned out okay. The manager I was talking, who is a long-time aquaintance of the director's, broached the subject. He told me later, "he got a little bit hot when he first heard about it, but settled down as we talked further." Then I scheduled a meeting for Monday. We passed later in the hallway and he smiled and gave me a thumbs-up. So looks like everything's good.

My lesson for the day? Don't assume anything. I hadn't talked to the director because I don't have much contact with him, and was a little nervous about explaining this stuff to him. So I talked myself into believing that my direct management wouldn't let me go ahead without his approval. Luckily, it came out.

There's probably other people that I haven't directly talked to that I need to brief. I'm going to write and circulate a document that describes what I'm doing, and why, tomorrow.

7 February. More stakeholder buy-in. I talked to a project manager (one of six), another manager, programmers, the QA guy, and the person who assigns our project accounting codes and tracks intermediate work products. Things are coming along. Monday, I expect I'll be able to start getting us a place to sit.

Let me explain how things are set up here. I've mentioned in the past that this is a MatrixManaged? organization, but I don't think I've gone into how much overhead that entails. Until today, I wasn't really aware of the huge amount of organizational infrastructure we had. A lot of it is looking like it's excessive.

Each project has:

Each of these people, except the "consultant," reports to a different manager. (The "consultant" is a senior UI or DB programmer.) So theres a manager for UI programmers, a manager for project managers, a manager for DB programmers, a manager for DBAs, a manager for system engineers, and a manager for QA.

The projects we work on aren't that big. Certainly not enough to keep all of those people working full time. As a result, people are assigned to multiple projects at once. Four or five isn't uncommon. I've seen cases where QA was assigned to a project "0.3 hours per day." (That's 20 minutes, folks!) This requires people to do a lot of task-switching, which is very inefficient. It also requires very precise resource allocation and prediction. So precise that it goes wrong a lot, resulting in big hills and valleys in people's workloads.

One of our clients is particularly big and has signed up for multiple projects. They're linked, by which I mean they have related requirements and a somewhat common codebase. Fear of those projects going drastically, horribly wrong is what got me the job of running all of them.

But I can't change the whole organization at once. I need to fit into the existing way of doing things. I have control over how the programmers are organized, but I'm going to present the same external appearance as I would if I wasn't changing anything.

So now we have:

With this setup (which is nothing more than a consolidation of 6 existing projects into one team), several people have noticed and commented on the amount of overhead. It's exactly the same structure as before, just a slightly different view.

Commented how?

Comments along the lines of "Wow, that's a lot of project managers."

Are those 6 project managers really full-time assigned to just these 6 projects? What the hell do they do all day? --BillSeitz

Ahh... organizational refactoring. :)

8 February. Even more stakeholder buy-in. This is a big change for this organization. Today I talked to the boss's boss (the director). It went very well. I now have official approval to proceed. I also talked to one of the project managers. He was worried about the transition to the new approach holding up his need to get stuff done. I thinked we assuaged his fears.

All this buy-in stuff is taking a lot of time, but it's critical. We're making a change to projects for the organization's biggest client. This has people pretty skittish. The good news is that all this work is paying off. The project managers' boss is on board and his support has been very helpful.

At this point, you might wonder: How in the world did I manage to make a migration to an agile process on the company's biggest client? And why would I be so foolish as to try such a thing?

The answer: The risk level was astronomical. Something was going to go badly wrong if we didn't change anything. That keeps coming up over and over again when we talk to people. It's been a huge help in convincing people to switch. Otherwise, I think I would get the normal reply: it's a good idea, but not for us.

Now, faced with the high risk of failure, suddenly it is "for us."

This reminds me a lesson from one of my sales books. I think it was "The New Strategic Selling." (Good book. I recommend it, especially since changing your organization is a sales job.) The book said that there's four basic mindsets when it comes to companies:

You can only sell to the last two types. Only when there is a mismatch between desire and perceived reality can a sale be made.

That's exactly what happened to me. As long as the company was in case 2 (failing, but thinking it's on an even keel), no change could be made. As soon as they realized that failure was imminent, suddenly changes started happening.

13 February. Looking back at my previous entries, I can see that I was kind of giddy. Now reality is beginning to set in. This is a tough change to be making. I think it's possible, but if not, I want to find out soon.

There's a couple of gating factors. I'm not going to give the go-ahead until they're resolved:

The last one is a legitimate issue. There's a lot of work to do... tons... but there's bureaucracy. Things have to be signed before work can start. So we're in the ironic position of being behind schedule (already) but not able to start work. Each individual project is teensy, and each one requires a document to be signed.

In other news, we've now tailored XP to the needs of the organization. It's looking pretty good. The requirement was that we not impact the rest of the organization too much. I think we've succeeded. Each project proceeds along its normal lifecycle (requirements doc, statement of work, scope document, implementation specification, construction, QA, user acceptance test, performance test, deployment) independently. Project managers keep track of the whole mess. Meanwhile, the XP team works in two-week iterations. Any work that the project managers want -- from creating an implementation spec to building software -- comes into the XP team as a story. A "project executive" will prioritize the incoming stories and create a release plan. We even have an on-site customer (a proxy, actually).

I've been working with a counterpart from the business side of the organization to plan the transition. So far, we've presented the above ideas to one project manager, who seemed pretty accepting. We refined our lifecycle diagram after talking with him, and tomorrow will be presenting it to the project executive and chief architect. We already have buy-in from both of them, but I'm looking forward to seeing their reaction to the plan.

Sometimes I feel like overwhelmed by the bureaucracy but other times I feel like we're cutting through it. This new plan, I think, cuts through it. The bureaucracy is preserved, satisfying my goal of not upsetting the organization too much, but the XP team is pretty well insulated from it. Project managers continue with the same job as before (managing project bureaucracy) and the XP team takes stories and produces results.

This is the most complex environment I've ever tried to implement XP in. I think the above plan is the way to do it, but I haven't tried it. And there's still those gating factors. It'll be interesting to see how it all turns out.

I'm not sure I understand the above correctly; are you saying you're implementing a waterfall like process where each phase is done as an XP iteration and each task is a card? -- BrianRobinson

No, not exactly. The XP team will work on regular two-week iterations regardless of what the rest of the organization is doing. But project managers continue to work under the waterfall lifecycle they're used to. The difference is that before, they would explicitely schedule programmers to work on each phase. Now, they schedule the XP team via stories. So, they might have a "Create design specification for project Foo" story.

This is more complex than the regular XP approach, and I certainly wouldn't recommend it, but it's a stepping stone for us. It's been very helpful for me to say that XP is a programming change and that project management is mostly unchanged. In addition, we have multiple projects feeding into the XP team simultaneously. This approach addresses that. It seems like a nice, clean way to have the XP style of doing things coexist with the company's preference for waterfall. I'll let you know how nice and clean it turns out to be in practice.

If you're doing up front requirements and design, how is the implementation going to work? Will you later convert your requirements documents into user stories? How much leeway will you allow yourself in deviating from the design created in the design stage? I can see problems with refactoring which will lead to problems with CodingStandard, ContinuousIntegration, and CollectiveCodeOwnership. You should still be able to get the benefits of PairProgramming, AcceptanceTests and most of the benefits of TestDrivenDevelopment, although a fixed design might make you miss designs that emerge from TestDrivenDevelopment. -- BrianRobinson

We're doing up-front requirements and design because the business requires them, not because we want them. I still intend to have TestDrivenDevelopment, SimpleDesign, and refactoring drive the actual design. The design won't be fixed.

This raises the question: Isn't doing the up-front design a waste of time? Well... yes. I'm going to try to mitigate this by keeping the up-front design focused on third-party products, protocols, interfaces, etc., rather than actual design. But I expect that at some point, we'll have TooMuchToDo and the business will complain. That's when I get to point to the "Create design spec" story card and mention that we don't really need it, and would you rather spend that time on something else?

18 February. I've been updating less frequently lately because there isn't that much to tell. The transition is continuing. This change is touching more people than I expected, which explains why there was so much resistance to it earlier. On the other hand, the further along we get, the better things look.

My business-side cohort has identified the basic stories for a few of our projects. Tomorrow and Thursday, we'll be meeting with project managers to confirm those stories. Then we'll get programmer estimates on Thursday and Friday.

We're also working on the final requirements for buy-in with the boss's boss's boss: an internal work order (IWO) to pay for the time we've been spending and a comparative budget showing how things change with the new approach. Next week will be slow because several key people are out of the office, but a week from Friday we'll be getting the IWO approved and doing release planning. If all goes well, we start our first iteration on Monday, March 3rd!

We still have two big unanswered questions:

One last thing. In talking with some people today, I've learned a little bit more about how this change occurred. When I was given the job of leading these projects, I was given a little bit of authority: technical leadership over the projects. I acted as though I was given something a little different: technical and process leadership over the projects. The thing is, process leadership is the responsibility of the project management folks, not the programmer folks. So that surprised a few people, caused a few waves, but not so much that they decided to put the brakes on. As time has gone on, I've emphasized the programmer-specific aspects of my changes, and they've relaxed. (But in fact, this is going to cause project management changes. It raises the visibility of so many issues.)

I knew at the time that I was pushing things a bit, but I decided to pretend I didn't. This is something I've had success with once, maybe twice before. Sometimes there's a need for a leader, and you haven't really been given that role, but no one else has either. If you just act like it's yours, and are clearly qualified, and are sensitive to people's reactions, and do a good job... then it's yours.

But before I get too self-confident: the jury's still out on whether I'll be successful this time.

And by the way, the first time I tried this, it backfired in the worst way imaginable, to the point of costing me my next job. So be careful! Don't forget the clearly qualified and sensitive to people's reactions pre-requisites! I didn't have either of those, that first time, and it hurt.

Responsible leadership? Sleazy office politics? You tell me.

I just found (22 Feb) some quotes by KrisJohnson on TheSecretOfPower that state my position perfectly:

I have found that it is pretty easy to get a fair amount of "power" by simply (a) gaining others' confidence, (b) acting like you are in charge, and (c) accepting responsibility for your decisions and also for the decisions made by those following you. Most people want to avoid responsibility. If you assume responsibility, subordinates will do whatever you ask and superiors will defer to your judgement because they can all blame you for whatever goes wrong. As long as you have a thick-enough skin to accept all the blame and criticism and you aren't afraid of being fired (or assassinated), you can get a lot done. --KrisJohnson

How does one act like one is in charge?

(I'm not sure I understand the question, but...) Tell people what you want them to do, and act like you expect them to do it. If you have no official authority, then this may involve a certain amount of pleading or bluffing, and may fail miserably. However, if people agree with your goals, believe in your competence, and don't want power themselves, you can easily become the de facto leader of a group of people without any official sanction from anyone else.

21 February. We estimated stories for three of four projects today. (The stories for the remaining project will be estimated next Friday.) It went very well and took about three and a half hours. The project managers weren't included, although they normally would be. Since we were just starting, though, I didn't want the tension of project managers second-guessing programmer estimates.

We estimated the stories over the course of two meetings. The first meeting lasted two hours. I started by giving a half-hour overview of the XP process. There was a bit of skepticism and some questions, but no more than normal. During the second meeting, which lasted an hour and a half, we estimated the remaining stories and finished up with more discussion of project management and how XP will resolve some of the problems the company faces. I'm afraid I ranted a bit at that point, but maybe I didn't froth at the mouth as much as I thought I did. The other XP guy here, who was at the meeting, said that he thought it was good that I explained why we were doing this.

At the end of the first meeting, we talked a bit about the release planning meeting. The estimates turned out higher than even I was expecting, which means our deadlines are probably at risk. Most of the estimates came out at two or three ideal weeks, and my best guess of our velocity is about three ideal weeks. (How did I calculate velocity? Five people times two weeks divided by PI. A reasonable approximation in my experience.)

I'm sure that these estimates will be a shock to the project managers. So I talked a bit about how to deal with estimate pressure. First, I emphasized, never reduce your estimate in response to pressure. Instead, explain how you came up with the estimate. Describe the requirements as you understand them and give the PM a chance to change scope. If they do, re-estimate. If not, though, describe the technical requirements and don't take no for an answer. Under continual pressure, go into more detail. You are the one qualified to make the estimate, not the PM, and everyone knows it. No matter what, don't get confrontational, even though the PM might.

At the end of the second meeting, I had a chance to demonstrate an example of that kind of pressure. We had to call one of the PMs to get him to break a story into two pieces. It was the primary chunk of work for his project, and it's due in exactly one month. So we walked through the process, he broke the story down, and we estimated the two stories at two ideal weeks each. Then he asked, with a bit of trepidation, "how does four ideal weeks map to real time?" I said that I wasn't sure. I'd have that information at the release planning meeting. He reminded me that he had a strict deadline: 21 March. I responded, "I understand your concern. At the release planning meeting, you'll have a say in which order things are done in. With a team of five people, there's more resources to work on your project than before. If it still turns out that there's too much to do, at least you'll know about it now, in time to do something about it." He agreed! I thanked him for his time, and we hung up.

All in all, a great set of meetings. The release planning meeting's going to be a doozy, though. That happens Friday, one week from now.

28 February. Today started out terribly. I've been working to get agile development into this organization for the last five months. About a month ago, as you already know, I got the okay to do it. Today was the final, most important milestone before starting development in an XP style.

So what happened? I stressed out, didn't get to sleep until 4 am, and then slept through my alarm. I slept straight through my first meeting... the one in which we were going to estimate the last batch of stories before the release planning meeting. I didn't get to work until 11 am.

You can imagine how I felt. All this work, down the drain.

Well, it didn't work out that way. Yep, I missed the first half of that estimation meeting. Nope, we didn't get everything estimated. But the release planning meeting, which was scheduled for four hours and started at 12:30, went fine. Actually, it went more than fine. It went great.

I started the release planning meeting with a short presentation about what we were doing and why. I did a terrible job with the presentation. I was still flustered from sleeping through my alarm, and I rushed it. I was disjointed and left out several key points. Fortunately the question and answer session afterwards went better. People raised concerns, but it was clear that we were going to go forward. The PM's had some concerns, but were at least willing to try.

Once the concerns were addressed, an hour and a half after we started, we started on the release planning portion. That was just as difficult as I had expected. We have many projects -- five or so -- all competing for the same programmers. It was clear to everyone that there's TooMuchToDo.

The great thing about this is that everybody instantly understood the problem. There wasn't any of the usual browbeating of programmers or negotiating for a higher velocity. (Well, not much, anyway.) This is a problem I've always had problems with. I think it was better this time because I was very clear about how I estimated velocity in the absence of measurements. I told people that the initial velocity was a guess, and they were welcome to change it if they want. But I also emphasized several times -- and I think this was the key point -- "This is all just a planning tool. We can change the numbers as much as we want, but it won't effect how much work we're actually able to do. So be careful about being too optimistic. The goal here is to make a realistic plan so we can identify problems before they occur." That attitude -- that they were in charge, I was there to help, but that we wanted to come up with a good plan -- worked very well. Very well.

We planned two iterations in that meeting. I was hoping for more, but it was a struggle even getting those done. Every PM has his own priorities, and every one of them has an externally-imposed deadline that wasn't based on programmer estimates. In order to make everything fit, they added two more programmers to my team.

Actually, come to think of it, that was probably why there wasn't as much browbeating and negotiating. Most companies I've worked at haven't had the option of adding more people. Resources, time, and scope are all fixed, so the only thing left to change is the estimate. Here, time and scope are fixed, but at least resources can change. For now, anyway. There are budgets.

After the meeting, I had several people congratulate me on a productive meeting. On March 3rd, we start iteration one.

next: ChangeYourOrganizationDiaryPartTwoMarch

EditText of this page (last edited June 1, 2004) or FindPage with title or text search