Frozen Development

A code freeze makes sure development is focused on shipping the product. But what happens right after that? Read on for one story ...

I am opening this topic to articulate what I am experiencing on the project I am on. Maybe this will resonate with many of you out there. Please feel free to add thoughts and link to related topics. I am sure that there is some wiki jargon that already covers what I am seeing.

Recently, we went live with our ambitious project for the client. Now that the code is in production, and that has been for over a week now, we are barred from further development unless a bug appears in production. We are effectively frozen, on standby doing time and materials busy work only semi-related to the project.

Rating the technical ambitiousness of the project, I would say it was modestly ambitious. Concurrent with the development of the application code, we had to develop adapter code for the content management server that was purchased for the project. This server is poorly documented, does seemingly no parameter validations on inputs, and required reverse engineering to discover exactly what some of its APIs really did. In our favor development-wise was our accomplished software architect/project manager who came up with a fine up-front design for the adapter code (with my help, wheeee!) and a narrowly defined scope for the application code that held through most of the development time to production. Also in our favor was the intentional exclusion of any presentation layer to the application - the first incarnation was to be run in a batch mode with no user interaction.

Rating the budget ambitiousness of the project, I would say is was dangerously ambitious. We aggressively underbid our larger competitor. Did we need to do this? It is still unclear to me, and I am still a bit jaded about it since I was the lead developer and I had to work the crazy overtime to get the content management adapter written in time. The competitor failed to complete prior incarnations of the project two times and had already blown gadzillions of the client's dollars. If we bid what we knew we should have bid, would it have mattered? I tend to think not, I tend to think we would have won the bid. In terms of resource budgets, the team was split half and half with two remote near-shore cheapo developers and two of our own. These developers were an anchor because they lacked all but the most beginner Java skills and their management was equally clueless. One developer failed to complete all his tasks which we ended up having to take on, and the other's was so poorly engineered that we had to rewrite much of it. Keeping these developers in the loop and out of trouble was a time sink until we restricted interaction with them to a one hour a day meeting. We knew early on they were going to fail, and that we would have to take on their work. But timewise, even with the near-shore anchors, added work, and tiny local team, we rarely got time extensions (and never money extensions.) I believe we got little over a week added to our project in total. The arbitrary end date set for the project seemed set in concrete.

Now that we have proven ourselves, we are being carried past the end of the project on a time and materials budget to support the project! All throughout the project the talk was cheap: "we want junit tests", "we want x% code coverage", "hmmm ... that agile stuff sounds good, maybe we should try it", and "try not to work any overtime, I don't want you to burn out" ... but all that fancy talk went out the window when we actually started implementation, which was a deathmarch, at least at the beginning. Now that we are done and have the free time, we two developers are all ready to performance tune and refactor to get things right(er) with the code. But the code is frozen, we cannot touch it!

I think this is a huge psychological brick wall that needs to be torn down in enterprises. Post-production is where a lot of neat stuff happens in movie-making, why not the same thing in software-making? Yet I hear the same thing time and time again, either from my management or from the clients', "The code made it to production, you are DONE, do NOT touch the code!" Yet, you and I know, defects are always found after going live on a project. Either a test was never written for that bug and it was never caught, or it was known but no user was ever supposed to run into it, etc etc. If we are working on time and materials, it is our responsibility to the client to devote every free moment to code improvements. The standard dev/QA/prod environments still apply, and getting those changes to production will still require all the same rigor and testing that brought code to production initially.

My hands are tied in proactively going to the business and asking to make code changes. Code changes after we have gone to production means we are admitting defeat; that we failed and have bugs in the production system or that the production system is not perfect! It is not a failure to have defects (since everyone has them), but if they ARE there, do not let loose with the accusations. Instead, I would appreciate encouragement to seek out defects and destroy them before they do any damage. Business people need to have an expectation of bugs in production systems so that there is no anger or remorse when post-production activities include resolving defects. I think this could be communicated at the beginning of a project and properly budgeted for as a post-production phase.

Now the ice over our FrozenDevelopment is beginning to thaw, the code is performing too slowly in parts, and the client is beginning to ask for options on how to remedy the situation. We discussed and attempted to add indexes to fields and tuning the database otherwise that yielded some improvements. But, it was not enough and I was asked, "What can we try next?"

This gave me the opportunity to send the ice-breaker letter (a letter that is sent to the business that opens the door to code changes after a successful project has gone to production) today that everyone, including the project stakeholder who gets to turn the big project steering wheel, got to read. In essence, I explained what I thought was going on with the database, that I thought there were roundtrips we could eliminate, what we were seeing in our query logs, my speculation that there were queries we could improve, and content management adapter improvements that we could make. I did not couch my language in fear or in guilt, but rather in optimistic language to prove that we were on top of it and were capable of a fix.

I think that it is sad that I had to wait for a right time to send an icebreaker letter, but I am glad it comes from the business' desire for code improvement rather than from the need to address a major uncaught flaw in the software or design. Had the need never arisen, we could have rolled off before a flaw was discovered, and then an unnecessary friction between us and the enterprise would be created. This has happened to us before with other clients, and it was painful for everyone concerned.

In essence, I advocate that development never stops as long as developers are assigned to a project, because I argue the best work done on the project can be done after that project goes live. The post-production development need not ever be deployed to the production environment. However, by keeping developers free to make changes and improvements, they will in turn communicate issues found and resolved - which gives the business options they would not otherwise have had.

-- AnonymousDonor

IMO, code freeze is really necessary for a production system. You should refactor/fine tune/cleanup/etc when you can during development, but I think the payoff of them do not justify the risk once the system is in production.

You can, however, write more tests for your frozen code. Given your description of the project, you will probably want more test coverage for the system anyway, why not take the time to cover the system with as many tests as possible? It will certain make any changes easier in the future.

-- OliverChung

In this (and many other projects I have worked on) after we went to production making a code change is a "big deal". Even writing better/more unit tests (which I know we need) is not an option, we simply can't do it without raising a lot of eyebrows and getting a lot of signoff. Plus, our time is currently allocated for everything but development, so we couldn't do it right now even if we wanted to. This is why I hope my icebreaker email works. :)

I agree that code freezes are necessary for production systems, but the act of freezing the code is a configuration management issue, not a development one. A build can be stripped off as a branch in your source control system, and development can continue unabated on the main branch.

Your risk assessment of making code changes after going to production is a natural one. But I am advocating courage because I believe that that will yield extra results where stagnation is apparently the norm. -- AnonymousDonor

I feel like I can make another observation about the consulting biz now that we have moved on to another project, and I can see that this one is settling into its production run. Maybe this belongs someplace else, but here it goes.

In a nutshell, if it wasn't for our IT sponsor who tightly managed scope for the business, we would have never succeeded. I think that in my consulting business career, that was the closest I've come to a DoTheSimplestThingThatCouldPossiblyWork project ever! I think an interesting side effect of this was, after we had our low-expectation/tightly-constrained-scope project success, our statement of work got extended by the business to be an open ended time and materials project! That extension allowed us to implement additional features and refactor the code to give the users performance increases.

I think if it is not possible to run a project as an Agile one, then what we saw might be an acceptable degenerate form of Agile. The characteristics that I identified of this project that might somehow map to AgileProcesses are:

Create the simplest upfront design that is acceptable to the client architects that we could also actually use.
Maintain a constrained scope which gives us the breathing room in the fixed schedule allowing us to prototype with untested technologies before trying to produce production code.
Test driven development when we had enough cushiness in the schedule to justify it to management.
After we ship production code, an enlightened client funds follow-on work to refactor and add enhancements.

Obviously, there's a lot to be desired in the way our project was run, but I think the characteristics I identified definitely increased our chances of success.

-- AnonymousDonor

CategoryStory