Small Iterations Smell

There is a lot of advice saying coding iterations should be very small. This to me smells because it's completely artificial and is not based on the problem or code at all. The problem will tell you HowLongAnIterationShouldBe. Listen to the problem, it will tell you.

Does this imply that there are some problem domains that don't lend themselves to DivideAndConquer where some (if not most) divisions provide some incremental value?

Maybe it implies that all problems you're solving aren't uniform; therefore, small iterations don't always reflect what's right. There's no real consistency to the number of characters you type or difficulty in updating or refactoring existing code through all the iterations and increments of a project. For me, early in a project, I type like a madman. Later on (e.g., after the first few increments), I'm doing a lot more thinking and a lot less coding. Even simple refactorings are harder to make happen. And as things change around the work I'm doing (including my understanding of it!) the nature of how I continue the journey solving problems and spinning off iterations along the way seems to fluctuate. --PhilipEskelin

Interesting. I'd kind of expect, intuitively, to go faster as the project goes on, owing to the presence of clean abstractions and lots of code that is nearly what is wanted. I wonder what is going on in the situations you describe. --rj

Maybe I didn't explain my point well. I didn't mean that the number of words per minute decreases over time, but instead that the number of lines added decreases over time. I think EightyTwentyRule explains what I meant. That last 20 percent is harder and takes longer because you need to think about more when making changes.

For example, let's say I've got a WorkQueue with three items in it. I want to finish them in my next iteration, which I estimate to take about two weeks. But when I start the first one, I realize that the first item was harder and would take longer to complete than originally estimated, let's say more than two weeks. In the next project status meeting, I would describe the situation, and push the other two items to my next iteration. If I stuck to my original plan, the iteration would take longer. That's what I meant by fluctuation.

The above example seems more likely to happen toward the end of a project, when several thousand lines of code exist, rather than in the beginning. Ron, you helped invent a methodology, so I'm wondering whether you think I'm deficient and need to learn more about design, or if this is just a reality for us mere mortals? --PhilipEskelin

Well, I'm not here to assess you. I believe that well-structured code has the property that features are generally easier (or at least no harder) to add as time goes on. If the code has clear objects, nicely modularized, then there tend to be convenient things to use or subclass. On the other hand, I've lived through a lot of code that didn't turn out that way. Maybe I'm just flat wrong. It could happen. You'll have to assess that.

I must be looking through different glasses than you are in this discussion. You made it sound so darn easy, that it made me question my own capabilities. For me, building software and deploying it into production never consists only of neatly adding features to a very pure, elegant, and modularized product. There's lots of other problems, bugs, and complicated integrations I've needed to deal with (on top of adding features) that have made iterations fluctuate. So I wouldn't say you are wrong, or that I'm deficient, but I do believe that in discussing SmallIterationsSmell we were addressing different parts of the process. --Phil

Oh, yes, it seems like there's always crap at the end. But if you have acceptance tests from the beginning, you know the important stuff works. If you do the hard stuff first, you have the bad integrations being done every week. If it takes more than one iteration to clean up, IMO we have done something wrong, and I think that ContinuousIntegration really means that the system is ready to go at any time. See no reason why it can't be done that way. The BugList?.size is zero at the beginning. Why can't we keep it that way?

Zero bugs across the lifetime of a project sounds impossible in the real world. In my experience, ContinuousIntegration and acceptance tests are standard practice, but so much chaos can happen in the middle of a project (e.g., companies get acquired or acquire, people come and go, projects your project is dependent on get delayed or changed significantly without proper change control, COTS software doesn't work like you thought it would) that despite using them it's rarely easy. Change is the only constant. People are people. Mistakes happen. You do your best to deal with issues and problems and mitigate risk. There's always a degree of control that you wish you had (which you don't have) and you do your best to deal with it. In my experiences, these things can make small iterations and releases of the project itself a non-trivial exercise. Don't they ever happen in your engagements? How can it be so simple? Either one of us is crazy ;-), or you're onto a major gestalt paradigm shift methodology that will change the world. --PhilipEskelin

My experience has been that iterations are small, not because of the size of the problem, but because of human nature. Teams need a "cadence" where they resynchronize and see the results of their work (including the customer into that team, too). The longer the cadence, the more likelihood of drift. Also, it's hard for anyone to realistically estimate beyond a certain horizon, which constrains the iteration length. A regular cadence helps maintain a rhythm within the team, sets expectations, lends a certain predictability to the process, etc. A small iteration also helps minimize the risks of going off course, since you will get a course correction.

If the problem is too large for a particular iteration, it's a good exercise anyway to see how to break up the problem into bite-size pieces. If it can't be broken up, that might indicate a need to better understand the problem.

-- JeffMantei

Just so. --rj

My experience has been that iterations are small, not because of the size of the problem, but because of human nature.

This is along the lines of some people will abuse drugs so we need a law against using drugs. Personally i don't appreciate software laws or legal laws that reason like this.

Huh? Isn't the argument that shorter iterations give more, quicker feedback, and therefore better control? Human nature comes into it because developers are humans (a fact often overlooked), and humans 1) are often hugely over-optimistic, 2)operate most effectively within a framework.

Of course, it depends on what you mean by small: XP iterations are only smaller than those of some other approaches. They feel about right to me. Week-long iterations would be too small. I'm reminded of those numerical methods that are both iterative and incremental--many of them improve in accuracy as the increments get smaller, but only up to a limit where the results explode all over the place. For the kinds of problem I've seen XP applied to a three or four week iteration length seems close to optimal, for a bunch of technical and social reasons. YMMV-- KeithB

If it can't be broken up, that might indicate a need to better understand the problem.

You are saying breaking it up is required without reference to the actual problem. That doesn't smell to you? If someother methodology said do X no matter what you wouldn't be suspicious?

Isn't it the solution that is being broken up, not the problem?

Tailor the process. If long iterations seem better, compare them to shorter iterations and find out.

That being said, shorter iterations are a good idea for programmers who, like me, have a very difficult time predicting anything more than a few weeks from now. I admire programmers who can plan farther ahead than that, but I have met few of them in the wild (most of them are in cages at the methodologist zoo). -- WayneConrad

Short iterations (two weeks) work well on my project because it gives us something recent to demo at any given point (including the point at which the money runs out).

I'm having trouble thinking of a feature that can't be broken down into three-week steps. By this I mean (a) efficiently (it shouldn't take twice as long to implement in short steps) and (b) intelligibly (the product of each short step should make enough sense to be testable).

Could you provide a real-world example for us to chew on? --AnonymousCoward

''This is along the lines of some people will abuse drugs so we need a law against using drugs. Personally i don't appreciate software laws or legal laws that reason like this.''

I can't help feeling that the OP has gotten to the core of his issue in the above statement.

Shorter iterations always provide better feedback to the customer, to the team, and to the programmer. When they get too short, the biggest negative impact is that there's no time to do anything. That happens at about one day per iteration in my experience, although that works just fine in some cases. I find the relentless pace thud thud kind of grinding at that frequency, however.

Doubtless there could be domains where it's not possible to do something meaningful in two weeks. So far, however, I haven't found one. I, too, would like an example. --RonJeffries

''My impression was XP touts continuous integration so 2 weeks would be a very long period.''

Appropriate length iterations aren't a terrible idea especially if you don't have any reason to have a structured schedule. See WorkQueue for instance. When you are constantly in ship mode--like, the order to ship the latest build could come at 10am and it'd be out the door at 11am--, I've found it best just to make a central task list that every developer and stake holder has access to. Stake holders put all tasks on the list, prioritize it, etc. Developers, when they are free, go to the list and pick off one of the tasks each. When they are done implementing it (which means the system is back in shippable state, not just "testable," whatever that means), they check off the feature and move on to the next one. I've found that (professional) developers have good time management. Sometimes frivolous features go in, but that's mostly because they are cheap and fun to implement. That lets off a little stress before the next high priority, high stress feature is implemented or bug is fixed. -- SunirShah

Small iterations have been common practice in developing low level interfaces in embedded systems. Many of the iterations are in fact useless from a customer perspective, but invaluable for development; they tell you that one small piece is working. You may at first just set an external signal high and view it with an oscilloscope. You might then add a loop to create a recurring pattern. Next, you might add a piece that sends out a canned, but "real" data. Finally, you actually pass in the real data. Depending on the complexity of the hardware and software, the above sequence may take a day, a week, or a month.

Doing iterative development in such a manner is so second nature to many embedded developers, that seeing developers create full modules and then integrating and testing them months later seems bizarre. Nothing is more frustrating than trying to use some piece of software and find out that some very basic characteristics are wrong.

Small iterations are much more the normal operation in development environments, and the large iteration approach is really the anomaly. A large iteration may have the potential to be faster than many small iterations, if everything goes perfectly, but one small mistake more than pays for the time spent doing the small iterations.

-WayneMack