Refactor Low Hanging Fruit
Refactor Low Hanging FruitI said that in TestDrivenDevelopment the right design emerges only after you have thrown away a lot of code.
It "emerges" as a combination of you discovering it and you already knowing what it is. Kind of like the end of TheWizardOfOz, when the Wikid Witch of the North told Dorothy that she had the answer with her all along, but she had to learn it for herself. You just tap the heels of your RubyLanguage slippers together three times, and hit the OneTestButton?.
Of course, with any approach, you will always encounter situations where you have to rewrite major portions of the code, or, worse, change the entire architecture. This is likely to happen due to requirement changes, or performance issues, but can happen because your initial visualization was faulty.
In real life, what typically happens is, after the twister, your house lands on top of the previous lead architect, and you peer tremulously out the door at a twisted landscape of big balls of crufty mud, written by short programmers, their growth stunted by long working hours and junk food, under the spell of some lesser process than XP.
This leaves you wondering where to start. Then MichaelFeathers, wearing a Wikid Witch of the North costume, appears with a copy of The Joy of LegacyCode, and smacks you across the forehead with it.
DOROTHY But -- after I add tests to all this legacy code, how to I fix it? Do I split it down the middle? Do I look for the big common patterns --- MIKE Just refactor the low hanging fruit. DOROTHY But that sounds too easy! Shouldn't I make a plan too...But MIKE floats away inside a SOAP bubble, leaving you all alone. With the short programmers crowding around you.
DOROTHY My..! People turnover so quickly here! ... Refactor the low hanging fruit? Refactor the the low hanging fruit? DBA Refactor the low hanging fruit. TOOLS GUY Refactor the low hanging fruit! GUI GUY Refactor the low hanging fruit. MATH GUY Refactor the low hanging fruit. ALL Refactor the low hanging fruit. Refactor the low hanging fruit. Refactor, factor, factor, factor, Refactor the low hanging fruit. Refactor the low-hanging refactor the...So, by finding the simplest possible fixes to the lowest level code within that ball of mud, and by isolating the effects of your changes with characterization tests, you can begin to tease apart its design.
... And ... you ... are ...
Testing your way to dee-velopment Developing tests for the cause You'l find it a whiz, so give it a squiz Each test just gives one little pause The tests you test are bestests tests The bests are tests to test the best Because ... because because, because, because, Because of the wonderful code they does. You're testing your way to dee-velopment The wonderful tests for the cause!-- Phlip2004
[NOTE: Refer to the idiotic dialog in RefactorLowHangingFruitOriginal to see what the following discussion is aboout.]
BlackHat: The quote, "But the other architects will laugh at me if they see my code doesn't have a BigPicture in it," smells like whining to me, used primarily to reduce the reader's opinion of the SuperEgo.
YellowHat: Worrying about not having a BigPicture is a common concern among developers who RefactorLowHangingFruit. The quote is merely used to help show developers with this worry about how silly they are being.
BlackHat: I'd like to see some support for that statement. And anyway, what is this ridiculous "discussion" between the non-identities SuperEgo and Id? Are these names supposed to convey something? They only serve as a distraction from the supposed "point" of this page.
WhiteHat: Several different kinds of DramaticIdentities are used throughout Wiki to convey ideas.
RedHat: But endless repetition of the same statement is not a valid cognitive argument. It is simply naysaying.
BlackHat: SuperEgo, as architect, points out that the project is in a state of mess and needs to be resolved into a reasonable architecture. Id, ever the idiot, responds with the only line he has in the whole "conversation."
YellowHat: Id is repeating the same mantra as a device to show how all the complex problems can be iteratively solved with a simple solution. The purposeful use of repetition is akin to a ZenSlap.
RedHat: The pat answer does, however, put me off a little.
BlueHat: It might be valuable to further explore some of SuperEgo's concerns...could we be clearer with a less ZenMasterAndStudent? approach? Maybe a simpler explanation before the dialog, or even just a simple explanation?
BlackHat: Refactoring works great when you have a bunch of tightly constructed thingys that all do their own work efficiently and effectively. It's not so great when these thingys don't work together because nobody knows what the state of the system is or what unit has priority over what else and when those priorities need to shift. Refactoring can't address that at all.
YellowHat: The point is that although there will be a transitory state, possibly as confusing as you fear, constant and cohesive applications of RefactorLowHangingFruit, will avoid such a state in the end.
BlackHat: No, it won't. The architectural issues SuperEgo brings up can not possibly be resolved through the simple minded approach that the Id 'bot is spouting. Some of us have experienced projects that desperately needed an architect to figure out how to put things all together. Refactoring doesn't solve all the architectural problems, in my experience.
WhiteHat: If you envision an advanced design, don't start with the most complex refactor. Try the simplest ones you can do first. Strategically, the design may temporarily get worse. Keep going until the refactor that will install that advanced design is now the simplest refactor available. If you don't envision any advanced refactor, hit the LHF anyway.
BlackHat: Why should I bother with refactoring a bunch of code that may not address the true concerns of the overall system? If I go back to the drawing board and come up with a plan then I can see if any of the objects, procedural functionality, or data constructs look anything like what I envision the system as needing. I'll refactor what I can use and toss what I can't. The use of refactoring needs to be applied judiciously, not as a sledgehammer solution to all scenarios. It is not the Universal Tool.
GreenHat: The RefactorLowHangingFruitOriginal dialog gave me the "AhHa" that refactoring low hanging fruit applies to many domains beside programming, such as RefactoringWikiPages. The dialog & ensuing discussion on this page don't give me the same insight. The changes made to the dialog above are a perfect example of DisagreeByDistorting.
RedHat: Really? This despite the fact that the original version is filled with the stench of ArgumentumAdInfinitum (ArgumentByRepeatedAssertion, etc.) and AvoidingTheQuestion?
GreenHat: My AhHa came not despite the repetition, but because of it. I'm not a programmer, so I took the dialog more as a metaphor. That detachment let me extract the dialog's message and apply it to other areas, particularly writing and editing. Could it be that you're so focused on the dialog's domain (refactoring as a way to understand legacy code) that you've missed the larger lesson (tackling small parts of a problem first makes the bigger parts easier)?
BlackHat: No. The repetition is used as a blunt instrument in an attempt to hammer flat a problem containing curves. This is not to say that refactoring is not a powerful and valuable tool, just that it simply doesn't apply to all conditions of altering a design in its basic structure.
Discussion
[Note that much of this discussion refers to the original Id/Superego discussion on RefactorLowHangingFruitOriginal.]
This smacks of ArgumentumAdInfinitum. However, the idea of chasing down refactorings is valid in its own right, as long as it is not taken as an article of faith that this approach cures all poor design evils.
I'll tell you the truth, folks - if somebody did something like this to me in a real situation I might be tempted to apply some ParkingLotTherapy. Nobody in his right mind can believe that a simplistic approach like this cures every problem you will encounter. Nothing is that universal. Anybody who tries to tell you that is selling snake oil. -- MartySchrader
Could expand on your objections?
First the "dialog" above would need to be "refactored" into a set of problem/solution statements. I simply can not live with the BS as it is currently stated. The drivel about other architects laughing is particularly grating. Eliminate the fluff and I'll be happy to write a rebuttal. Right now there is nothing to rebut other than the useless repetition. Where's the porpoise?
I've been very successful with refactoring towards high level architectural goals. But RefactorLowHangingFruit is still a very good piece of advice:
If you can't bring yourself to fix small problems, how will you ever be successful with the big problems?
You have to develop a habit that when you see a problem you fix it. Otherwise when you hit a big problem you'll (#1) not really know how to fight it and (#2) big refactorings always create lots of little problems anyway. You'll still have to RefactorLowHangingFruit.
-- JeffGrigg
Unfortunately, there have been many, many hours of my time wasted in refactoring low hanging fruit when I was working on something that needed an axe taken to it. Lots of fixes to code and documents that later ended up in the ashcan. Had I been smart enough to take a longer view of the junk I was working on I would have seen that the particular code or doc simply wasn't needed for the solution I was striving for.
This is not a case of the goal posts shifting after the fact nor of clarity of vision due to more information. The information to analyze had been there all along. It was a matter of my wanting to "make it better" before actually making the Whole Thing work. My fault entirely. Still, not something you want to explain to your boss (or client) when it comes time to fill out a weekly report. -- MartySchrader
This pattern makes Refactoring those big balls o' mud into a hill-climbing algorithm that flattens the hill as you go.
JoelSpolsky has an interesting article describing a refactoring exercise he did, which reads to me like a classic piece of refactoring low hanging fruit: http://www.joelonsoftware.com/articles/fog0000000348.html. -- StephenHutchinson
The Id makes it sound simple, but this pattern has two flavors: refactor the fruit that is easiest to find, to engage flow. Or take time to identify which fruit is truly lowest-hanging.
I thought that the term 'LowHangingFruit' meant that it was "patently obvious to the most casual observer" what was easy - it didn't need great study. Like replacing a lot of:
if (x == <somevalue>)
{...}
else if (x == <some other value>)
{...}
code with switch statements.
I think that's what the dialog above is trying to say. In other words, if you have a big ball of messy code, instead of trying to tackle the whole problem, just untangle a little bit. Keep at it, until the whole ball is gone.
Because, as was stated above, "Why should I bother with refactoring a bunch of code that may not address the true concerns of the overall system?...I'll refactor what I can use and toss what I can't." Don't refactor code you will be discarding later on. Suppose that whole if/then/else statement evaluates something I end up not using? It's a waste of time to refactor it. Eh?
True. However, I usually find out what's useless through the process of refactoring.
<sigh> I'm really not trying to belabor the point, but the issue I was raising is that refactoring doesn't always help to define the real needs of a system, particularly when there is a lot of legacy code involved or if the system was never very clearly defined in the first place. Many clients have paid me a big piece of change to work on legacy systems that I cleaned up nicely, only to discover that the code simply wouldn't meet the New! IMPROVED!! needs of the system as the clueless client (a redundancy, I realize) specified. I have examples of this in mind. In almost all of these cases I could have saved the client moolah by analyzing the real system requirements first and then creating new code later. Had I been given the mandate to discover the real needs of the system I could have done that; in most of these cases the client whined about how many man-years of labor had been invested in the code base, so I had to start there.
In a few of these cases I completely ashcanned the original code and started over from scratchola. The new system was able to surpass the requirements in all cases where the underlying hardware had sufficient horsepower. Refactoring the original code forever simply wouldn't have achieved this. Period, end of statement.
Once again the vague definition of the word "architecture" threatens to derail a well-meaning discussion between programmers. When you say "architecture", Marty, do you mean "macro-structure", or "system requirements"? Because I get the feeling that whichever meaning you intend, somebody else here intends the other. -- francis
Also, remember, the purpose of refactoring is to maintain the current functionality and just change how the functionality is expressed. So, it is a true statement to say that refactoring will not cause code to meet new requirements. Refactoring will lead to changes in efficiency and performance, but whether those changes will be significant, improvements, or regressions is going to be almost impossible to predict. I think it is a separate discussion as to whether it is more appropriate to modify current code or rewrite the code from scratch when addressing new requirements. -- WayneMack
Refactoring techniques help when re-featurizing. If you do a BigRefactor? and let behavior drift a little, it's a Big-Refeaturizing. But this applies much more to DeprecationRefactor than LHF. The latter is about design. -- PhlIp
Francis, my comments about architecture are intended to talk about The Big Picture(tm), whether that entails the relationship of features to each other and to the system overall or to the relationships of components in the system or to integration of the system to the outside world. My point is that one needs to know how components will be used in The Big Picture(tm) before determining if they are usable or not. If a component is not usable I am not going to spend any amount of time at all refactoring the code; it simply gets tossed in the dustbin. -- MartySchrader
Refactoring is merely a means to find a home for new or changed functionality. Refactor to create the appropriate place to make the change, make the lowest risk change possible, and then refactor to make the change clean. This provides a viable alternative to throw it out and rewrite from scratch. Some of the biggest project disasters I have seen were "rewrite from scratch" projects. The ugly code hid a lot of details that had to be rediscovered by the development team. The components of a system are rarely unusable (the system is currently using them), though they may be difficult to understand. Refactoring improves the understanding and understandability of the component while preserving the usability. -- WayneMack
Yeah, that's great for those cases where refactoring applies. The point is that refactoring doesn't apply to all cases of making structural changes to an application. Sometimes the required change is so great -- without appearing to be so -- that refactoring the code until Doomsday doesn't get you any closer to a solution. Architecture does.
At this point we may just have to AgreeToDisagree; our experiences seem to be too different. I have not experienced applications that needed structural change. I have experienced applying large structural change to applications in a single step and seen the following struggle to recreate the original operating characteristics; the end result being a lot of time and effort with little clear benefit to the actual users of the application. Perhaps one day I will experience an application in need of major structural change and will be able to understand your argument. Until then, however, I will favor an incremental approach, which I see being formalized through the combination of Refactoring and TestFirstDesign. --WayneMack
Concur. We will agree to disagree here because of the disparity of our experiences, I guess. If anyone is interested I can dredge up numerous projects over the last 25 years or so that needed architecture to fix the problems; refactoring had been attempted to some degree or another and had failed to bring about the necessary improvements to meet requirements, whether those requirements had changed or not. -- MartySchrader
FalseDichotomy: Perhaps this is more a matter of nuance and interpretation than anything else. RefactorLowHangingFruit is not intrinsically opposed to re-architecting. And the original page over-simplifies, but it does so to make a valid point - that the impact of continuous refactoring as part of the development process is often not appreciated sufficiently. Even if you do need to rewrite from scratch - and I have had to rewrite legacy subsystems more than once - you still need to refactor and change your view as you go. I quail when I see a bunch of people come in to sort something out with a Big Picture. There is no Big Picture(tm) in many business organizations. Or, at the least, the Big Picture(tm) doesn't give you a nice, neat up-front set of criteria and a gameplan. Better to start off with some High-Level But Still Meaningful Basic Principles(tm) that don't dictate exactly how they are to be fulfilled in software. You feel your way, building a rapport with the customers. You refactor and change your view of what system is required as you go along. Sometimes you junk a few day's worth of exploratory work. If you don't go this way, you'll just waste your time writing lots of nice shiny new programs that the customer doesn't actually need. After all, you might only need to rebuild half of that system. But without an iterative, refactoring-based approach, you're not going to be able to sort out which half.
Hmm. I suppose you may deal with clients who are so clueless as to not know what their plan is, but these must be firms without investors. Typically a firm doesn't get investment capital until after they pitch a business plan to a rather substantial number of investors, so many eyes get a chance to look the plan over and poke holes in it if they can. Clients are certainly clueless, but they can't afford to be stupid or wishy-washy if they want to use expensive resources like me. You go ahead and refactor their old crapola. I am going to meet their needs as they have specified to me, starting with what are the improvements of the new system over the old one.
You know, I am a basically lazy guy, so if I can use some old kaka out of the previous product then I will polish this turd up and use it. However, there are way too many times when no amount of turd polish will turn the old junk into a sparkling new widget. You gotta know when to hold 'em and when to fold 'em. Analysis gives you this information, and time wasted in refactoring old drivel won't help a whole lot.
-- Marty Schrader
RLHF is one of those rules that's very hard to go wrong. -- the TOOLS GUY
Generally, that's true. Refactoring is a powerful tool in cleaning up legacy code or adding simple feature enhancement to an existing system. The above discussion is all to act as a warning sign that refactoring isn't always the proper tool to use. In the cases of feature enhancement or performance increases where some fundamental change in architecture is required then refactoring SIMPLY DOESN'T APPLY TO THE PROBLEM DOMAIN!!! I really hope we were all listening that time.
Use the tool that fits. Don't pry open the paint can with a screwdriver. Don't hammer the nail with a wrench. Don't tighten the bolt with a wire cutter. DON'T REFACTOR CODE THAT WON'T BE USED IN A NEW DESIGN!! Do we get it now?
What you're saying makes a lot of sense, but I don't think it takes analysis separate from refactoring to find that dead code. When you want to clean up legacy code, you write some characterization tests to figure out what it does. If in doing so you find that the code isn't needed anymore, you just delete it. But if it is needed, you now have the tests in place to do refactor with confidence.
-- PatMaddox?
Yeah, I'm still not convinced that tests are any better than any other means of determining if legacy code is fitting to a particular altered architecture. (Of course, I am not fond of tests in the first place, being a Fagan convert and thoroughly enamored of the way formal inspections work.) In the progression of development architecture comes after specification and before design. Code is the complete tail end of the effort, so if there's code already there then you have to look at the design that stems from the architecture to see if any of the legacy code fits.
I am not interested in reusing existing code just because a client whines that he put so-and-so many man years of development into it. I have examples of products where multiple man years went into something that wouldn't work in the first place, not ever. Refactoring that trash forever gets you exactly zero. Let me see if I can whip up a design that incorporates the existing code into solving the defined architecture, and then I'll look into refactoring the code. Certainly not before that.
To try to extend that: when dealing with a design where the design fundamentally has to be changed at a basic level, one refactoring-based approach is to attempt to refactor by factoring out all the modules which you will be able to reuse. Then when you rip the basic architecture out and replace it, you have those modules ready and functional. I think once you know what your new architecture is, your perspective on low-hanging fruit tends to include only stuff which will still be needed in your new architecture, but that's not the normal meaning of low-hanging fruit, is it?
You have a good grasp of the point I was originally trying to make. -- MartySchrader
DeleteWhenCooked: Regarding " [Um, RealNamesPlease.]", the DramaticIdentity THE TOOLS GUY comes from the narrative segment of this page. It's not an attempt to hide a real name. (And it wasn't me - please leave it alone.) --PhlIp
Darn! You're right. I missed the one reference to this character from the cast of munchkins flogging Dorothy with their, um, "wisdom" above. Still think THE TOOLS GUY is as big an ass now as he was years ago when he first made that argument. -- MartySchrader
See: RefactorLowHangingFruitOriginal
Contrast: DeprecationRefactor
EditText of this page
(last edited December 27, 2010)
or FindPage with title or text search