Test Driven Design Phase Shift

A phase shift is a phenomenon that occurs in TestDrivenDevelopment when simple ReFactoring ceases to be adequate for the incremental changes being performed. The premise of TestDrivenDevelopment is to develop one simple test at a time and ReFactor frequently at a small scale; one starts with an intentionally naive design and allows the addition of new tests to evolve the design to greater levels of complexity, while still trying to keep it as simple as possible with each new test. This can help prevent AnalysisParalysis and other such slowdowns from dealing with complexity and produce more readable code by the theory of YouArentGonnaNeedIt.

Then you rewrite a module using DeprecationRefactor, passing each of the old modules' tests with the new module, then converting each client to use the new module, then retiring the old module.

However, sometimes one finds during TestDrivenDevelopment that the underlying design of some component needs to be revisited altogether; the approach being used to support the existing stories does not extend well to the new stories, and a different approach (which may not have been thought of earlier) must be used. In these cases, the design change required may not be analyzable into small refactorings, so a significant amount of code must be rewritten in one step. This phenomenon is termed a "phase shift", due to its analogy to phase shifts in chemicals: at one temperature, a chemical may be a solid, but if heat is injected, at some point the temperature will stop increasing while the solid converts into a liquid. This is similar to the break in the test-code-refactor cycle necessitated by a phase shift in design, and resulting significant transformation of the component.

Examples of design phase shifts include:

Converting a collection of API calls to a LittleLanguage
Developing or using preexisting ExpertSystems to replace direct procedural implementations
Replacing one algorithm with another, e.g., replace DoubleDispatch with a table-driven approach

Phase shifts can occur outside TestDrivenDevelopment as well, but TestDrivenDevelopment can make phase shifts easier to deal with: the principle of YouArentGonnaNeedIt causes phase shifts to not occur unless they are really necessary, thus avoiding unnecessary and costly rewrites and resulting complexity. Also, the more abstract design to which the code is shifted is always one that is actually mandated by the needs of the user rather than by predicted technical concerns which may turn out to be illusory. Finally, TestDrivenDevelopment can mitigate the cost of a TestDrivenDesignPhaseShift for several reasons:

TDD leads to good SeparationOfConcerns; thus while a module may have to be rewritten, the interface to that module may often remain stable.
One will likely be able to reuse much of the existing code, supplanting only implementation details.

Can someone please describe real-life experiences of working through a TestDrivenDesignPhaseShift in the context of planning and estimating UserStories in iterations? In particular, how does the development team realize that the code requires an overhaul and then inform the customer that over an iteration or two, the team's velocity will be lowered because of refactoring efforts that will yield no increase in user facing functionality?

I've encountered a variety of strategies on actual projects. One approach is to include the TestDrivenDesignPhaseShift activities in the estimates for story development. Another approach is to dedicate separate resources in parallel to story development. On my current project (a fairly large one), we've developed a so-called refactoring team whose job is to tackle such larger scale issues which may be too tough to address within a specific story or set of stories. Instead, these problems are assigned their own stories, prioritized, and dealt with by those members of the team who volunteer to work on them. The matter of how to manage velocity and customer expectations is not trivial, but my overall comment is that the customers and developers must learn to cooperate in order to achieve good productivity. Sometimes it is necessary to accept a slowdown in order to pave the way for subsequent work, and a good project manager knows who to deal with such problems. -- VladimirLevin

the design change required may not be analyzable into small refactorings

I would disagree with this statement. Not only can a major design change be broken into small ReFactorings, it is also the most effective way of implementing the design change. Far too many development efforts substantially under-estimate the effort required for a code rewrite.

The approach described above tries to do the major rewrite first and then fix code to get it to pass the tests once again. This leads to an extended period of time where the code is broken, i.e., the current version has less functionality and correctness than the previous version. This increases the pressure to enhance the previous version, thus branching and delaying the newer version even further.

An alternate approach is to stick with a refactoring approach, and all that entails, with the benefit that the code is always ready for delivery. This does require underlying effort to maintain the old design and new design in parallel, keep them synchronized, and slowly migrate from one to the other. Additionally, the pessimistic time and effort projections that come with planning a migratory, refactoring-based approach are usually far more accurate than the optimistic estimates that come out of a rewrite approach. The refactoring approach forces planners to focus on potential difficulties, whereas in the rewrite approach, these are unknown and thus ignored.

Refactor, don't rewrite.

-- WayneMack

I should clarify the way the refactoring team works. The design changes are indeed usually made incrementally and checked in one piece at a time. However, the purpose of a refactoring story is to clean up or reorganize a piece of code so that it works better in the general scheme of things and makes the design more flexible for subsequent user story development. The main point is that if the change that one wishes to make is significant enough, it can become quite difficult to incorporate that change into development for a user story that has actual new functionality associated with it. In the case of a truly fundamental change, even a whole series of refactorings, or refactorings mixed in with rewrites, may not be enough. In that case, at least some aspect of the system may have to be redeveloped entirely. However, if that part of the system lives behind a well-defined set of interfaces, at least the tests for those interfaces will remain useful. -- VladimirLevin

See also DiscontinuitySpike, HowWouldRefactoringGoFromIndentationToParsing, HologramOfJesus?

NovemberZeroFive

CategoryTesting CategoryTestDrivenDevelopment