Daves Real Example Where Thinking Ahead Would Have Helped
Daves Real Example Where Thinking Ahead Would Have HelpedThis is a response to the Extreme Programming idea that it doesn't help to build software for situations that might arise in the future. This is a real-life example.
Most derivative traders recognize something called a Trade. One kind of Trade is a Swap. A Swap is usually described as a set of Legs. Each Leg represents a set of Payments that will be exchanged between the two counterparties. One interesting thing about Legs is that each is unique to the Swap that it belongs to.
Another kind of Trade is a BondTrade?. A BondTrade? represents the exchange of an amount of money for a holding in some Bond, typically issued by a third party. The Bond itself has much similarity to a Leg: it represents a set of payments that will be exchanged between two counterparties (in this case, the issuer of the Bond and whoever holds the Bond when the payment becomes due). Because many millions of identical Bonds will normally be issued at once, there is nothing about a Bond which is unique to a Trade.
Some developers at my company were asked to build a swap system. In the example I have in mind, the developers took the Extreme approach of ignoring the rumours about the possible future onset of bond trading. They created a nice simple model where the Trade class encapsulated a set of Legs. Over time the assumption that the Trade owned its Legs got incorporated into the data storage and retrieval system, the copy semantics, most of the user interface, and so on.
Then someone said "I would like to trade bonds on this system". One design given this requirement would be to create an abstract superclass Instrument with subclasses Leg and Bond, and have the Trade represent an exchange of Positions (a Position representing some quantity of an Instrument: in the Swap case, the quantity would usually be 1). Of itself, this represents a reasonably straightforward refactoring, but the implications for all the other code that relied upon the ownership assumption made this a very expensive change. Had the ownership assumption not been incorporated, the code would have been slightly more complex, but the change would have been straightforward.
What I glean from this is that first, thinking ahead does help sometimes, but not always. Second, only people with previous experience of the domain should be allowed to think ahead. Anyone who'd previously built a major derivatives system would have got this right. Another example: I've built a few object-relational mapping modules in my life, and I think I know the hotspots well enough for upfront investment in flexibility to pay off: but I'm not sure I could have got this right the first time.
-- DaveCleal
It sounds like you entered RefactoringHell.
This is an excellent example of a time when you need to stop modifying the original code and to create a totally DifferentVersionFromScratch. Do not thow away the original code, simply reimplement it from scratch, and whenever you think something is difficult, take a look at the original source to get some ideas. It is ok to use some CopyAndPasteProgramming, since when all the functionality has been migrated, you will just throw away the original code. -- GuillermoSchwarz
Possibly related: FinancialContractExample
I've screwed up too, many times, and wound up needing changes that were hard to do. I can't remember any, however, where I didn't start by thinking that the dumb bastard (usually me) who wrote the existing code didn't have it properly modularized/factored. As I look at what we do now, I don't see anything in those sticky systems but refactoring we should have done but didn't.
I could be fooling myself. OTOH, we have worked for 2 1/2 years on this payroll thing, adding minor details like unions and multiprocessing, and haven't gotten stuck even once. Either we're
Or it could be that you are being more anticipatory than you think. The very act of being modular anticipates change. Am I wrong? -- RobertDiFalco
Back when I worked for Macromedia, I worked on Shockwave for FreeHand (not to be confused with the much better known original Shockwave, which played Macromedia Director movies). This was a Netscape plug-in which displayed FreeHand files. The first version displayed FreeHand 5.0 files. After we released it we started work on a FreeHand 7.0 version which would ship at the same time as FH7 (There was no FreeHand 6.0 for reasons not worth explaining). SWFH5 shipped two thirds of the way through the FH7 product cycle so there was only about 4 months to get the next version done, so we were really on a tight schedule. A month or two months in we discovered that we'd never put in any versioning support.
We needed to use the same file extension, ".fhc" for both compressed FreeHand 5 and FreeHand 7 files. If a user tried to view a web page with an FH7 file embedded in it and they only had SWFH5 installed, the browser would crash. This was terribly unfortunate.
I worked around this problem by concocting a stripped down FH5 file which just contained a broken document icon with a hyperlink to www.macromedia.com to get the latest SWFH plug-in. The FH7 compressor prefixed this document to the beginning of the FH7 document before compressing it. SWFH5 would just blindly display the first file, and FH7 was smart enough to look for the prefix file and skip it if it was there.
SWFH7 had a versioning scheme built in. It had a way to recognize a later version FreeHand file and it would vector the user to the appropriate place on the Macromedia web site to get the proper plug-in for the specific file type.
SWFH5 didn't need a versioning scheme when it was released, but when SWFH7 was released suddently it did. We should have planned for this from the beginning.
Generally I'm an advocate of YouArentGonnaNeedIt, but sometimes you should plan for the future because sometimes you are going to need it. This was definitely one of those times. -- CurtisBartley
This seems qualitatively different from the previous case, and is one where the expectations of XP break down. Specifically, I am putting some software out into the world, I am going to have to support it as is for some significant length of time, it *will* interact with my more recent code, and I will have no opportunity to refactor it. It's the same as freezing some chunk of your code and saying this is absolutely un-modifiable. At that point you can't do some of the things XP wants you to do, so you do want to think ahead, at least to the point of being able to distinguish which version of the code you're talking to. -- AlanKnight
It seems that there are a class of anecdotes that go like this:
This looks like a case of TwoIsAnImpossibleNumber. -- PeterdaSilva
Recall that in XP we have a specific practice that is future-oriented: WorstThingsFirst. If we had a story at the beginning that said "support a versioning scheme", and if we identified it as a risky issue, we'd move that story forward and work on it. The idea of YouArentGonnaNeedIt (and I'll go back and look it over to see whether I can make it more clear) is that we don't implement things that just come to mind ... "hey, we're gonna need some kind of frammis like my widget, I can generalize my widget and then when we need the frammis, it'll be easier." We let our planning process decide what order things need to be done, and explicitly eschew the kind of one-person opportunistic future-orientation that distracts us from our planned order of development. -- RonJeffries
In the swap system described above, the Trade abstraction seems to have been lost, displaced by Swap. In my experience with XP, one does not abandon abstractions to DoTheSimplestThingThatCouldPossiblyWork. Or, if one does not use an abstraction initially, it is factored in within days or weeks.
However, I agree that folks with knowledge need to be making key XP design decisions as they write code. I spent close to a decade programming derivative-product based trading systems and risk-management calculators on Wall Street. I made a lot of mistakes but I also made it a point to learn the business, which meant understanding Swaps, Swaptions, Caps & Floors, and the highly exotic deals that Traders were making. That knowledge helped me define abstractions which made it easier to incorporate new products into the systems. -- JoshuaKerievsky
Yes, Trade was displaced by Swap - that's the whole point. You don't need the two distinct abstractions until you get to bond trading. -- DaveCleal
Says Who? Do you think XP says "thou shalt not have any abstractions that will work for tomorrow's products?" Perhaps Ron can enlighten me if I'm wrong, but based on my own work, and most especially in observing WardCunningham, one does not dispense with abstractions when doing XP. -- JoshuaKerievsky
Well, that's interesting. I wasn't thinking of "dispensing" with the abstraction - I was assuming that it would never get created by XP. I imagined the XP programmer starting with a test harness. This would presumably define the external protocol of a Swap class, which would then get coded. No amount of refactoring is going to reveal the need for the abstract Trade class - until we get to another kind of Trade, when interface sharing and code duplication is going make it painfully obvious.
I can't see any way that I would be motivated to refactor the Swap class into the Swap and Trade classes without thinking about the future. And then I would be in violation of YouArentGonnaNeedIt. Am I missing something here? -- DaveCleal
No, actually - I now think I'm missing something. I did some programming work that I thought was real XP rather than XPish, but after discussing this abstraction issue with someone in the know, I now see that the introduction of the abstraction without stated need, was XPish rather than pure XP. Sorry for the confusion -- JoshuaKerievsky
An Extreme Programmer would not add an abstraction that was unused in the system at the time of release of the abstraction. We won't build an abstract superclass with only one subclass. The motivation for refactoring the Swap class into Swap and Trade has to be that we need them both, and they have commonality. If we need them both and can see how to make the second out of the first with a little abstraction, we'll go straight there. If it isn't obvious, we'll write Trade (maybe with a little rape and paste), then combine them ala OnceAndOnlyOnce.
I honestly believe that doing the abstraction before you actually need it is a waste, pure and simple. The code has no senders - what good is it? -- RonJeffries
Ron, "code has no senders" is not what happens. People "prepare for the future" by designing very general data structures or by making an abstract class with only one subclass. The code is all used, it is just more complicated than it needs to be. -- RalphJohnson
Sounds like a classic hill-climbing problem: The shorter your horizon, the more likely you are to make decisions you'll regret later. If you leave DoTheSimplestThingThatCouldPossiblyWork at just that, without adding to satisfy the set of stories on the table, and one of the stories isn't "allow for a future where ________," then you're are certainly going to find yourself climbing down off of more hills. This is a requirements issue, not a problem with XP. -- DaveSmith
XP really does say to DoTheSimplestThingThatCouldPossiblyWork to satisfy the one story you are currently working on (and then refactor to get the simplest system structure including that one new story). We really do say not to work on stories that may be coming up. We really do say not to try to put in things for the future (unless WorstThingsFirst has moved a story forward). What would have to be true about the process or the problem for this to work without needing to climb down off a lot of hills? -- RonJeffries
Oh. Well, in that case you're hosed. The process would need to incorporate enough lookahead to prevent "Simplest" from being expensive to undo in the forseeable future. I think one approach is to have "constraint" stories, where one constraint may be "don't do Y, even if it's the simplest way to get to X." -- DaveSmith
A light is beginning to dawn. When RonJeffries says DoTheSimplestThingThatCouldPossiblyWork, he means it. As KentBeck said in the XP BoF, you sit on the other stories and pretend they are not there. But,DoTheSimplestThingThatCouldPossiblyWork is not the only practice that is followed. The process then moves on, so you have proved that it works by running the UnitTests, now you have to satisfy OnceAndOnlyOnce. This is the point at which the original problem Over time the assumption that the Trade owned its Legs got incorporated into the data storage and retrieval system would have been stopped by XP.
In order to satisfy OnceAndOnlyOnce, and the LawOfDemeter, and all of the rest of SmalltalkBestPracticePatterns, you end up doing a lot of refactoring. So yes, DoTheSimplestThingThatCouldPossiblyWork might indeed build a few hills, but then the Bulldozer called OnceAndOnlyOnce uses refactoring to flatten all of the hills.
I'm beginning to get the sense of how DeeplyIntertwingled all of this XP stuff is. Starting off with DoTheSimplestThingThatCouldPossiblyWork makes it possible for the team to create a working solution quickly. Then the hard work starts of smelling all the CodeSmells, and doing the OnceAndOnlyOnce act on the code.
I'm interested in hearing from the XP team as to the relative proportion of time spent between getting the tests to run in the first place, and then refactoring the code to match the XP CodingStandards. -- PeteMcBreen
I spent a year programming pretty much full time on a half extreme project (LifeTech- no stories, big mistake). We were certainly big on refactoring, simplicity, and testing. I spent half of my time entering new code and half of my time with the RefactoringBrowser. Working on the JunitFramework? with ErichGamma, we spent 2/3 of the time refactoring and 1/3 entering new code, I would guess. It's little and simple, so I would expect to spend more time refactoring on a big Java project.
Looking at these numbers is pretty depressing- a third or a quarter of the time spent coding new functionality. However, there is some level of productivity of that coding time at which you can still out program someone who spends all of their time coding. The question is whether you can reach that level of productivity. I think I do, and I have taught others to.
Regarding Dave's assertion above. There is a subtle skill in learning what order to take the stories. You also need to analyze the situation keeping in mind the CostOfDesignCarry. -- KentBeck
Help me understand this. The options I see in the situation DaveCleal describes are
''I wish I'd said that - it's a perfect formulation of my issue. I'm eagerly awaiting responses.
One thing that may help is allowing developers to create their own stories. This is something we already do: in each development cycle, we explicitly reserve around 10%-20% of the available development time for whatever the developers want to do. This tends to get used for this kind of stuff (ie. to add to the budget for a task and allow it to be done more generically than is needed in the short term). Other things developers pick are changes that will reduce the amount of time developers need to spend doing boring support work, and performance work, especially around interactive performance. -- DaveCleal''
My sense is that the ReFactoring work needed to support OnceAndOnlyOnce goes so far along towards getting the results that you want from Option 2 that there is no point in the expense of Option 2. Given that the ReFactoring time is so high a proportion, there must be a lot of work required to reduce CodeSmells.
This conversation is however starting to get to the roots of WhenXpIsUnpopular. -- PeteMcBreen
OnceAndOnlyOnce relies on you noticing that you said something twice. Some things are implicit, perhaps even emergent, properties. Only when you make the decisions explicit can refactoring get to work on them.
How would you write a UnitTest for Option 2? I should think "extensibility" would be a hard thing to test for, which is perhaps an argument for not doing it. -- DaveHarris
I've been suspecting something for a while, and I want to run it past you all. Is there a distinct advantage to using a SystemMetaphor over using the actual "concepts" in the domain? I wonder whether by jumping out of the "real-world" modeling, the developers would have a little disconnect which would allow them to end up making fewer domain-related assumptions.. end up thinking more behaviorially rather than representationally? -- MichaelFeathers
IMO, in any complex application, the "concepts" from the domain are a RedHerring. Sure, some of them make sense: dollars, hours, pay rates. But what's really at issue is the approach you plan to take to doing the job (in our case, paying a guy and recording all the detail). That's where the metaphor comes in, like the factory metaphor that C3 uses. I hold that the trick of teaching CRC or UML with the domain classes is, to a large part, cotton candy. -- RonJeffries
NormKerth has a wonderful phrase that I'm sure I'll garble, but at least you'll get the point. The idea when you produce a version of the system is to leave it in a condition such that it assumes no particular next step, but also prohibits none. We have found that putting WorstThingsFirst, so that you experiment with and thus understand key possible problems, and by always building with MercilessRefactoring, you generally wind up in the condition Norm describes. -- RonJeffries
ExtremeNormalForm, perhaps?
As usual, I'm late to the table, but here's my $0.02Cdn. From my experience, it seems obvious that if you have to make a decision that puts some part of the system out of your control (thereby lending it some permanence) such as choosing the data store schema or shipping the GUI, you put in lots of escape hatches and flexibility to allow you to recover from a mistake or grow in the future. Moving the furniture doesn't help you if you've painted yourself in a corner; similarly, changing internal code structure doesn't help you if you've committed to an incorrect interface. I think it is ridiculous to attempt to program from first principles in these cases because only experience will tell you what contexts aren't undoable and how to avoid shooting yourself in the foot. Patterns help here, naturally. Versioning is a pattern, for instance. -- SunirShah
...implications for all the other code that relied upon the ownership assumption made this a very expensive change.
I have a very strong suspicion that something funny is going on in the other code that depends on that assumption. I would guess that there is a violation of OnceAndOnlyOnce lurking in there. I have seen more than a few situtation in which a seemingly simple refactoring became very expensive because other code had hidden duplication. (This happens very often with RDBMS interface code!) "Refactoring the system until a feature is easy to add" often leads you to make changes in surprising places. -- MichaelNygard
I second this motion. It is hard to paint Urself into such a corner (although many of us are very talented at doing just that), but usually in cases like these I find that many small refactorings were ignored along the way. A bad smell here or there that just never was important enough to get out of the way. I wonder if anyone else has found that this is true?
By the way I think this has nothing to do with XP per se but programming in general - when you don't clean up your code you get into a big mess at the end. In non-XP methods the design helps mitigate this problem by forcing everyone upfront not to cross the design lines.
We learn from experience. XP is saying start fresh for every project and only add stuff in when the need arises. To me this is just stupid. After working on 10 or so projects i pretty much know what it takes to make a large project work. I'm going to add these elements in immediately rather than wait. That's the smart thing to do and it works. I don't know what all these other people do to make this approach fail, but I've seen it work over and over again.
One overlooked downside of not creating appropriate frameworks is the training needed to overcome inertia amongst large groups. If 50 people have been doing something using approach X for a while, deciding to do X differently is extremely hard to do. Developers don't change their ways on a dime. It takes a long time for new approaches to be understood and diffuse into an organization.
If you know that you're going to need to do X, then adding X isn't a violation of YouArentGonnaNeedIt or DTSTTCPW. Perhaps you could give some specific examples of things you "pretty much know" you'll need on a large project, and see whether the XP gurus think you're wrong to put them in from the beginning.
Logging for example. I've been told by an XP guru that you don't add logging to a project until you know you need it. Logging is imho invaluable on any complex product. Those who say different i have to honestly doubt their experience in complex projects.
I agree, XP seems poor for architectural work where "constant refactoring" means eternal churn of all objects. It also seems poor for legacy work. Writing a system from scratch is rare. -- RIH
A good example came up today. Because of tight memory limitations I'm mapping status information for multiple objects into a bit array. To get to the correct attribute for an object I give the coordinates like CalcClearBit?(object1, object2). Thinking the number of attributes for each object might grow I made the code general enough that adding more attributes or objects would be easy, even there was no clear use case for this. The general code was slightly more difficult than the very specific code, but would prove a big win if assumptions changed. They did change and it took me about 15 minutes to add in the new attributes and change the UnitTest. The specific code would have been much harder and take much longer to change. -- AnonymousDonor
RobertDiFalco wisely decided to make this concrete by showing some specific code in ExampleLoggingFramework.
I think a lot of good points were brought up in this topic, I'd like to add a quick metaphor, as it helps me when making decisions regarding coding, and has helped me on a number of occasions. I consider programming somewhat like driving a car at times. Clearly I need to keep my eyes on the road and take care of actions now that are needed now. Even so, as a driver, I am aware that it is important to look ahead. If I know I'm going to have to make a left turn in the near future, and I'm on a 3 lane road, then I definitely want to position myself in the left lane before I get there or things are going to be problematic. Maybe I don't need to be in the left lane right now since my current goal is only to get to StreetX, and the lane I'm in will get me there just fine without adding the work of moving two lanes over, but if I know that I'm going to have to make a left on StreetX, then I definitely want to be prepared. If I were not sure that I was going to turn left on StreetX, but knew that I was going to turn one direction or the other when I got there, I might opt to be in the center lane, as that would reduce the amount of work needed as I approach it. These are design considerations I can make now that would make things far simpler later. I'm not sure if this thinking breaks the XP model, but it is definitely important to consider.
If you need to be in the left lane now to get to where you want to be, then you need it now. Yagni does not apply. The slower you can react, the sooner you need to react. That can mean you need to slow down.
When night driving, if your stopping distance is further than your headlights can illuminate, you are already out of control. If something happens, you have constructed a situation where you cannot react in time.
The problem with the Trade/Swap example is that you DID think and code ahead, albeit without realizing it. The act of encoding an assumption into your implementation was thinking ahead - you implicitly wrote "I am never gonna want to think of Trades separately from Swaps" - you just never thought of it that way. Assumptions are the most dangerous aspect of blindly following the writ of XP zealots. XP is great. But you have to keep in mind the context within which the "rules" were formulated. Applying simple abstractions to leave doors open rather than closing them prematurely is not anti-XP. Design patterns are a perfect example of this - applying the lessons learned from the past. XP is in the implementation, and only using the patterns (or even just the pieces of them) that you need right now. In all the hype and evangelism surrounding XP logical concepts got further and further abbreviated into catchy throw away lines that could be dropped into an argument to impress your colleagues, losing the contexts within which those abbreviated statements were valid. I hate the phrase DoTheSimplestThingWhichWillPossiblyWork? - its a powerful hook but lacks a lot in reality - I have suffered for years on a project where such phrases were recited as though holy writ leading ultimately without any design to an unmanageable mess which we have had to completely throw away and reimplement from scratch. I much prefer the concept of "Delay design decisions until the latest responsible time". Daily on our project XP zealots code themselves off into a corner where they get stuck unable to refactor their way out because they start with ZERO plan much like a hiker leaving base camp without knowing his destination let alone a map of how to get there. As I said earlier I actually agree with the concepts of XP as elided by Kent Beck and Ward Cunningham. What annoy me are the legions of disciples who equate regurgitation of catchy one-liners served out of context with a deeper understanding of the pros and cons of various development methodologies.
When employing the "SimplestThing" approach beware of assumptions by which you may be unwittingly making upfront design decisions. I find the "simplest" thing to do in this regard is to very consciously make your implementation as generic and loosely bound as possible. It actually cost exactly the same up front once you are in the habit of developing good abstract data types. -- Dave Whitla
EditHint: This page is very long and could use some consolidation, but the story should remain. It should also be made plain that YAGNI does not forbid thinking ahead, it merely discourages acting ahead.
EditText of this page
(last edited January 21, 2007)
or FindPage with title or text search