Change Your Organization Diary Part Two April
Change Your Organization Diary Part Two AprilStart with ChangeYourOrganizationDiary. See also ChangeYourOrganizationTactics.
Next and previous parts:
8 April. What a day. A mostly liberating one, though.
We're almost finished with our third iteration now. The team is stabilizing, the process is becoming second nature, and most importantly... I'm beginning to trust the estimates and our velocity. That's the good news.
The bad news is that between our estimates and velocity don't add up to our schedule. The schedule is less than half as long as it needs to be.
I've been gently signalling this for a while. I've been reluctant to pull the fire alarm, though, because the estimates seemed high. Now that I have some measurements of our accuracy and velocity, I can see that our estimates have been almost dead-on. I can also see that our needed velocity is so much higher than our actual velocity that we probably can't solve the problem by adding people, either.
So today, I broke the glass and pulled the fire alarm. I've raised the issue with one person. Tomorrow, at our regular release planning meeting, I'll raise it even more publicly.
The dates and scope are set by a client's client. We probably can't change them. I don't think there's any way we can deliver this project on this schedule, and I don't know if we can communicate that.
I'm not sure what will happen next. I expect a fair amount of sturm and drang. Other than that, I'm not sure what the conclusion will be. This could potentially be the end of the XP process; more to come tomorrow. I'm going to be careful not to be defensive about it; I'm doing a great job, a different process won't help, and I'll stand by that.
9 April. I pulled the fire alarm today. Our team needs a velocity of 15 ideal weeks every iteration in order to meet our current schedule. Our actual measured velocity is three ideal weeks every iteration. Ouch.
I've suspected that our schedule was out of whack with reality for a long time. So have other people, to be honest. Words like "aggressive" have been bandied about since I first heard about these projects.
Although I've implied that there's a problem from the beginning, I haven't come out and said it straight out. I didn't want to be a Cassandra. I've been waiting for the moment when our measurements of reality clearly invalidate our wishes for the future. Yesterday, I finally got that. I have confirmation of the dates we're planning on delivering to and I have three iterations of velocity. That allowed me to calculate our current velocity (three) and our needed velocity (15).
I had to be very careful about how I presented this information. I've been hinting at it for the last six weeks and judging reactions, so I had a pretty good idea of what I needed to do. I've noticed a problem with CustomersAndVelocity on other projects I've done: customers mistake velocity for productivity. I wanted to avoid that problem here, because the reaction to a velocity of three would be to ask why a team of six can't accomplish more than three weeks of work in two weeks.
So I didn't just jump up in the front of the room and say, "Our current velocity is three, we need 15, we're doomed!" Instead, I started by talking about the real hours we've put in. I talked about how we were about 75% efficient... that is, we spend about 75% of our time working on project-related tasks. (The rest is administrivia.) I said that we were slightly better than the industry average. I stressed that we were working efficiently. I mentioned that our measured work hours corresponded to ten developer-weeks every iteration... a pretty hefty amount.
Then I started talking about "other" forms of velocity. "This is our measured velocity," I said. "There's another form of velocity: our planning velocity. This velocity reflects our ability to plan over the long term."
I wrote our history on the board. It showed the velocity decreasing steadily over the three iterations, ending at three. I continued, "This number shows that we're having difficulty planning. We're encountering a lot of blocking issues and having unplanned work added at the last moment. As a result, our planning velocity is low. That means that our ability to meet long term schedules is compromised."
We talked about it briefly, and I sat down. Then the on-site customer - who is fantastic - showed his project forecast and added it up. It came out to fifteen. He said that he could only accept three weeks worth of stories because that's what our planning velocity was. People nodded, the time was scheduled, and we ended the meeting.
Why the anti-climatic ending? It's probably because a representative from the client was present. The PMs didn't feel comfortable jumping up and announcing that the process was a failure. That's not something I did on purpose; one of our architects invited him to the meeting. I almost didn't go through with my speech, but after discussing it with some people, we decided to go ahead.
But I think the other reason it went well was that I was careful to present information positively. I started by talking about what we were actually doing, emphasizing that we were working efficiently and getting a lot done. I approached the velocity problem as a planning issue, not a "inability to deliver" issue. Then the schedule discrepancy came up later.
The issue is well out in the open now. We'll be having a meeting next week to address it further, once the client is gone.
11 April. If you read the last entry and winced when you saw "we'll address it further once the client is gone", give yourself a gold star. We should be addressing these problems with the client.
12 April. I had a meeting to discuss our scheduling issue today. We looked at the situation and decided that a) the client wouldn't adjust scope or schedule, and b) we couldn't deliver according to the schedule we had. So we handed the project back to the client. They're going to give it to another team. Actually, they think we'll do "the architecture" and the other team will do "the typing." Silly.
The right decision was made here. I'm happy about it. It had become abundantly clear that the schedule we were given was infeasible. Rather than try and fail, we identified a problem and then made a business decision. In this case, the business decision was to not do the project.
It's not all sweetness and light, unfortunately. There's complicated political issues which will probably make my life difficult for the rest of the year. For example, that whole "we'll do the architecture" thing. But at least we have a schedule that is in line with what we can deliver.
For what it's worth, I find the idea that one team can do "the thinking" while another team does "the typing" to be absolutely ridiculous. What are they thinking? That programming is just a bunch of mindless data entry? Obviously, there's still plenty of client communication issues to resolve. But at least now we have a bit of breathing room to resolve them in.
Okay, I have to ask this. Do you now have two teams doing XP, one, or none? The above, while not explicitly stating such, implies negative consequences for your XP efforts. -- JeffPanici
16 April. I'm not happy about the decision any more. I was happy with it for a day or so, at most, but then I realized it was just a cop-out on my part. What the organization's decided to do is to find AnAcceptableWayOfFailing. We know that the project can't possibly succeed, so we're handing it off to a different team. When that team fails, as the transition costs dictate it must, we'll say "it wasn't our fault." I can't accept responsibility for that kind of lack of responsibility. So to speak.
There's only one solution I can see here. We need to change the way we work with the client. All of our problems stem from our client relationship. The fact that we have TooMuchToDo - a result of not working with the client to create the schedule. The fact that we're finding AnAcceptableWayOfFailing - fear of giving the client bad news.
We currently have half a dozen different "projects" all impacting a single codebase. These "projects" should just be collections of stories. They should be prioritized into a single stream and handed to us by the client, and we should deliver results on an iteration-by-iteration basis, with quarterly releases.
What I need to do is to is to talk to the decision maker at the client directly. I need to talk with him and understand his concerns, then present the idea of a single stream of requirements and quarterly releases. I think I can couch it in terms that he'll appreciate. It would allow us to deliver a number of things he wants: a reduced need for QA environments, greater visibility into our work, reduced bureaucracy...
This is a good idea, but it's a radical change from what we're currently doing. I don't know if I can sell it. I'll need to talk with some high-level management folks and see what they think.
It's also a scary idea. I'm not immune to the idea of AnAcceptableWayOfFailing either. Up until now, I've been safe. I knew the changes I was making would result in improvement. Now I'm proposing changing the client relationship, even when the people here who best know the client tell me that they're hard to work with. I don't know that I can improve things. I might make them radically worse. If it went poorly, I couldn't ethically back out. I would have to see it through to the end. Given how unlikely I think success is, I don't know if I want to take that kind of responsibility.
18 April. I haven't done any programming in the last two weeks. I've spending my entire day just managing process issues. It's amazing how much crap I have to deal with. Part of it is my client; part of it is their client. There's a lot of meetings to go to, and then there's email to catch up on, and then there's actually trying to fix the problems that lead to all of these meetings and email.
It's getting better, though. I talked with my boss's boss today about strategy. We talked about a week ago and I said that the projects were going to fail. He asked if I was interested in taking a bigger strategic role. I said that I was too busy on tactical issues. Since then, though, I've realized that our problems are strategic, and if I'm going to see tactical success, I have to solve our strategic problems.
I've been thinking about it ever since then, and a few nights ago I sketched out some ideas on a piece of paper. Today we talked about them. It went over very well. Surprisingly well, considering that I'm proposing some fairly radical changes. I think part of the catalyst was my "these projects can't succeed if they continue in this way" opening line.
We still have to talk about details, but there's provisional agreement to everything I want. Basically, I want to do a full XP model, end to end. Before, we were just doing XP internally. Now we'll include the client as well. We're going to consolidate the six-plus "projects" we currently have into one. We're going to get the client to provide an XP customer who can prioritize requirements and review iteration deliverables. (We'll still have our own "on-site customer," who's a QA lead with tons of experience with this client.) We're going to eliminate all of the waterfall artifacts... goodbye, scope document! Goodbye, design document! We'll eliminate or consolidate the project managers. I'll start communicating directly with the client.
It's good to recognize that the term 'project' is an artificial boundary around some work, and therefore can be redefined. Are these really all working on a single codebase? Should they be that integrated, or should the code be componentized more? In what sense were the originally-defined 6 projects distinct? Is there a risk that mushing them all together will create a sort of cognitive-transition problem where programmers have to change mental context every time they switch to a new story? If in any sense these are separate projects, what would happen if you had everyone work on a single project until it was completed, and then you started on the next one? Would the final completion date be the same? Or what if they stuck to a single project for 1 release (multiple iterations), then jumped to the next? Just playing with different ways of bunching things... -- BillSeitz
I wish I could say that this approach would eliminate our problems. Unfortunately, it will just give the client more visibility and control over them. That's a good thing, but this won't be a slam dunk. There's just no way to deliver the scope they want in the time frame they want with the staff that we have. And no time to ramp up more people. Doing XP will make that blindingly obvious; the question is, how will they react?
As I said in my last entry, this is a scary proposition. But I think I'm ready for it. I have the consolation of knowing that I can't possibly make things worse. And this is a big client. I'll get valuable insight into the way super-big businesses work. Plus, if I succeed, it will give me ammo next time I try to introduce XP. "What? You think XP is too informal? Well, such-and-such gigantic company used it, and they're as conservative as they come!"
So, things are looking better. As always, I'm eager for the change to occur. It will probably be at least two weeks before I see any real change from this, though. I'll spending that time setting it up.
(So, who's leading the XP team in the meantime? Fortunately, there's another person with experience leading XP teams here. He's not as extreme as I would like, but he'll do a good job while I'm pulled away.)
22 April. I spoke with senior management about pushing XP to the client this afternoon. Everybody was excited about the idea: not so much about XP, but about the idea of simplifying project management, reducing costs, giving the client more visibility and control, and reducing our risk.
So we're going to do it. I'm flying to the client's site next week to pitch the idea. I think it will be an easy sell: rather than talking about XP, we'll talk about providing things the client's been wanting for a while.
****Looking back at my entries, I realized that I haven't really mentioned what's happening with the project. Even if we didn't have problems with excessive scope, this project would be dying. Dying. It's incredibly frustrating. Our velocity this iteration looks like it will be about one. Yeah, one. Ideal week. A team of six programmers (including me), working for two weeks, and we only have a velocity of one. We should have a velocity of seven, at least.
The problem here is that our integration environment is in control of the client. For a variety of reasons, none savory, it's unreliable, in the extreme. We haven't had access to it for two weeks. Honestly, it's amazing that we'll be able to complete even one story. The last two weeks have been particularly bad, but environment problems are the norm around here.
What makes this really frustrating is that we can't seem to get anybody to do something about the problem. The people who control the environment aren't the same as the people we work for. There's some ugly political stuff going on as well. Net result: no ability to do integration testing. So no ability to finish stories. So high risk and low velocity.
Why is this the first place this is mentioned? Has this problem been going on the whole time? Are you saying there are 2 groups within the client organization - one is your Customer, and the other is some 3rd party group whose actions are hindering the relationship between you and the Customer? Is that ugly political stuff all internal to the client groups? How can an integration environment be unreliable? Is this a final-test system which fails to match the live system? And, how the heck are you guys getting paid? Straight time and materials? -- BillSeitz
I'm hoping pushing the XP model to the client - that is, using timeboxed releases, having them prioritize stories, and reviewing progress every iteration - will help out with this as well. Nothing is a better catalyst for change than a customer with a stack of stories. Velocity is never high enough for customers... and this time, improving it will be in their control. I'm betting that change will happen pretty quickly.
About your integration environment, I have experienced similar situations, though may not be exactly as yours. Customer wants some software from you, but due to company policy, the software must be accept/installed/maintained by the almighty Infrastructure Team. Unfortunately, the almighty Infrastructure Team has no interest in having your software running at all, that is just more work load for them, they wish you would just go away. The frustration you are going to experience is left as an exercise for the interested readers.
I find myself in charge of an almighty Infrastructure Team. I welcome any advice about how to best serve the interests of the developers.
The frustration really goes both ways. Doing such acceptance/installation/maintenance is really extra work load for your team. Worst, the division buying the software probably doesn't appreciate this extra work you do. I wish you could get extra resource for these extra work, such as charging the client division, even only on paper to show how much value you have provided. Otherwise, given an already tight budget for your team, adding extra software will be impacting your other services so much that not co-operating/delaying is the only option for you, because you just can't spare the people to help.
That's not quite my story. My almighty Infrastructure Team exists only to make sure that software gets deployed in an orderly, repeatable and transparent manner. Our budget isn't tight, but we can't be wasteful.
Good for you. Some other Infrastructure Team I have to deal with are also responsible for the maintaining the system, that includes minor modification. It is impossible to expect one single team be familiar all different possible hardware/platforms/systems/languages out there, but that's exactly what they need to do. Often resulting in cases like a Windows/ASP guy having to install a Linux/Apache/Java system from scratch (maybe because the Linux guy is already swamped), no matter how much documentation you provide him, he will take a week or two to do it, delaying the whole project by a week (if you are lucky).
Can't you donate one of your Linux/Apache/Java experts from the development team to the almighty Infrastructure Team to speed up the process? Cross training can help.
Didn't tried that, mainly due to the fact that our expert is also swamped with other jobs (well, they said they are going to do the installation/maintenance, of course we won't keep our expert on hold just in case), and the potential cost to us. My gut feeling is that it will likely be reject, and rightly so. After all, the purpose of having the almighty Infrastructure Team do the installation is so that they are sure they can do it again if needed, which is a good thing. But you can't achieve that if someone else came in and install it for them.
Coincidentally, there is an interest article about the impact of various "administration" on software development, see http://www.softwarereality.com/lifecycle/role_fragmentation.jsp and the Slashdot comments on it at http://developers.slashdot.org/article.pl?sid=03/12/06/1611214. One notable quote is: "The interesting thing is that this particular administrator knew that nothing bad would happen to him as a result of his gross ineptitude, therefore he didn't give a damn." Of course, this goes both ways, if a developer knew nothing bad would happen to him if his program messed up the corporate network, he wouldn't give a damn either.
EditText of this page
(last edited June 28, 2006)
or FindPage with title or text search