Yagni And Frameworks

ExtremeProgramming advocates putting off implementation of a feature until is explicitly requested by the client: It calls this practice YouArentGonnaNeedIt. But does this apply in the case of a framework?

In this case, let's define "framework" to mean a technology whose absence or presence will have design ramifications spread thinly across the source code. The easiest example is a logging infrastructure: You can defer implementation of logging, but then when you find you do need it you might spend lots of time adding one or two lines to all sorts of classes. Or maybe not.

And maybe it would be useful to ask this question to the Yagni adherents: Does any framework exist that touches enough code that you would consider writing some of the code up-front? If, for example, no testing framework like JavaUnit existed, would you evolve that framework along with the rest of your design and suck up the refactoring work you might need to do later? Does the use of frameworks ever justify BigDesignUpFront?

Moved here from YagniAndLogging

Sadly, the page still seems to mostly be about logging. It's also mostly ThreadMode at the moment. At 2:00 am, I am not going to fix that immediately. I would like to aim the discussion in another direction, though. What about frameworks that you already have for free? What about Yagni and Swing? Yagni and MFC? JFC? C++ STL? Clearly it's illogical to avoid using these tools SIMPLY because the client has not yet asked for their GUI to be extensible. (GUI vs console is another issue entirely.) There is a benefit to using a relevant framework when it is a pre-generalized solution for a segment of the problem domain. If the client says GUI, YouAreGonnaNeedIt.

Perhaps this begs the question, and perhaps not. If the question is whether to construct a framework in anticipation of a requirement, I say nay! Frameworks are a major effort best amortized over a sequence of projects. They are best built by repeated (not stepwise) refinement the (well-factored) code from a sequence of related projects. You get them by saying "We've done this before, let me go grab that code again". Eventually you have a general solution to a category of problems. You reap the benefits of reuse absolutely every time you use this framework, including the first. You know when to use it because you know what kinds of problems it solves. I suppose a better question is when to go out and buy a framework vs. when to let one evolve in-house. In any case, if you have an evolved framework in-house then it's probably a better fit to your staff and history than anything you could buy. Another worthwhile question is which frameworks are unequivocally indicated for what types of projects/problems.

-- IanKjos

...writing the above logging framework when the customer hadn't asked for it...

"Your project is going to be late by a few days; but the good news is, I've got a nifty logging infrastructure in place"

I honestly don't mean to be rude, but this stuff seems like rhetoric; methodology-based Simon says that I can't believe is based on your experience delivering products and applications. If you are timing things so exactly that 10% more time spent on making the logging routines (that you are already developing) reusable and extensible throws you off course, then there is something fundamentally wrong. This point of view, which is very popular, seems to show a misunderstanding of reuse. In actuality, it should take more time to rewrite/refactor the logging stuff each time an unanticipated change shows up than spending a little extra time to create a simple, but extensible logging manager extensible by observers. The time required to make something more resistant to change and extensible is not that much more than making it specialized. Finally, if we made it reusable the last time we needed this functionality, we wouldn't even need to write the code at all, allowing us to actually save time. I don't think that refactoring should substitute designing for change.

What can I say? You're probably right - I can't schedule software projects and I don't understand reuse. Just look at the evidence - all the software projects that have been entrusted to me over the past 10 years have taken from 20% to 100% longer to complete than my original estimates.

It's just that... I have read this book by this guy Beck (and some mail from this guy Jeffries, and so on), and a lot of what they have to say about software development makes sense, especially when I read it in the light of my previous failures.

Now, it's probably like people with an incurable disease - you know in your hearts of hearts that the latest voodoo cure probably won't do anything for you, but still you are desperate enough to give it a try. Right?

And just as for those voodoo cures, you'll never know for sure until you have given it a try.

And now comparing against similar projects your record is what?

Thanks for asking. My first impression is that the cure appears to work, but I was more ill than I thought. Half a year's treatment is probably not enough to correct a 10 year condition; give it time yet. And among other things, it seems that the original disease is contagious within a team, so you need to inoculate everybody. This tends to confirm that XP isn't in fact a Voodoo cure. These are well-known for working every time, the first time, and all the time.

My current project suffered several schedule slips, but on each occasion we were able to trace them to external causes. Lack of customer involvement was shown to be a major cause of delays. Bug rate has been way below what I had on previous projects. -- LaurentBossavit

What the experience of people actually doing it seems to be is that, with a well factored system, the cost of adding it later is not more than the cost of doing it now and you incur neither of the above penalties.

That's not possible. Logging has to be *added* it can't just be refactored into existence. And adding logging after the fact is difficult, from experience. Taking 100000 lines of code and adding logging is a lot of difficult work. Doing it while you are developing will more likely log interesting things.

The XP viewpoint is that, with a well factored system, it will not cost any more to add logging later (when you need it) than it does to add it sooner (before you need it) and maintain it as you are going along (until you really do need it).

That may be the XP position but I'd like to know how it's possible? You honestly think that logging added after the fact will be as complete as adding it in months later? I really don't think so, speaking from experience, unless you mean audit or security logging. I'm also sure my idea of logging is more expansive then the usual dump it to std out. I've always tied logging into a "module" notion that identifies software subsystems. This in turn is used to specify dependencies for a system state machine that can bring the system up, down, or from secondary to primary states.

It all depends on the context. If this is your application - you are your own customer - the above scenario could correspond to giving yourself a UserStory saying, "I need logging; I also want to make up my mind as to what to log, and where to log it to, as we go along." But if this was a customer project you had been working on and, say, you'd spent a day or so writing the above logging framework when the customer hadn't asked for it, and you eventually had to go to the customer and tell them "Your project is going to be late by a few days; but the good news is, I've got a nifty logging infrastructure in place", how good would you feel about that (admittedly nifty) framework?

Of course, if the framework helps you make the software better - in the same way that unit tests do, say; perhaps the structure of the software is such that unit tests can't reach into some portions of it, and you need to instrument the code itself with logging code... oh wait, then maybe you'd be better off refactoring the code so you can do with unit tests what you were going to use logging for, no ? Um, anyway - if you really need it, then you really need it.

Or you could use an existing infrastructure, like Apache's log4j, rather than "roll your own", unless I not only really needed it but also had UserStories from the customer indicating that they want the logging done a certain way, i.e. with some particular set of targets in mind, or some particular log formats anticipated, or some anticipated variation in said targets or formats.)

Or you could use an existing infrastructure, like IBM's log4j...

Well, or something. People always say this and it seems like more rhetoric to me. IBM's Jlog is much more infrastructure than the above, maybe more than is needed. In fact, it would take more time to learn and extend that system than writing the VERY simple log manager shown above. It depends on the system. Going for the COTS solution is not always the simplest solution. After all, a COTS component was not developed using your requirements. As a result, they often have much more (or different set of) functionality than you will actually use.

Finally, another problem I have with your logic is that while XP embraces change and refinement, it seems to discourage all the techniques that work so well for anticipating changes in requirements. For example, you assume new things to log don't show up in a project. I don't know what sort of distributed or large scale systems you've worked on before, but this is just simply not the case with at least those kinds of systems. It's not even the case for a single-user commercial project that need to transition several times. Frameworks are a good and useful thing and the fact is that every XP project uses them. Just consider the SmalltalkLanguage or JavaLanguage frameworks we use. What about the xUnit frameworks? There are tons. Just like all artisans, programmers build up libraries of tools through their years of work and experience. Some of these tools are conceptual, like DesignPatterns while others may be actual source-code. To accept the former while rejecting the latter is indeed a very extreme point of view.

-- RobertDiFalco

Surely you mean accepting the latter while rejecting the former?

No, I mean it as it was said. XP does not seem, in my opinion, to encourage software reuse. In fact, most XP practitioners often talk about reuse being a pipe-dream. However, XP has no problem with concept reuse, such as DesignPatterns. Having experienced real-world reuse of frameworks, source code, and components, I have a difficult time with this point of view - to always create specialized solutions with no eye towards reuse. -- RobertDiFalco

Robert mentioned COTS earlier. Does anyone know what the XP position is on large scale reuse of 3rd party software. Integrating 3rd party libraries or frameworks always requires at least some initial investment. I can see how it would be tempting to scream YAGNI! But apply YAGNI strictly and you end up rewriting functionality you could have bought. -- ChrisSteinbach

Why not buy it when you find out YouAreGonnaNeedIt? The risk with TP software in my view is that you spend an inordinate amount of time trying to push the square peg into a slightly non-square hole. Sometimes it is faster to rewrite specifically than tailor generic stuff. -- NeilWilson

Yes. But when do you say YouAreGonnaNeedIt? Once you've written half an ORB and decide you've wasted enough time? Or when you get bored of maintaining your database engine?

It is worth choosing at least your infrastructure at the start of a project. This way you can simplify data migration during upgrade, product licensing for your customers and concentrate on adding value. -- ChrisSteinbach

Consider a nail and a piece of wood: round peg, often no hole at all. This doesn't stop carpenters from forcing the issue.

Sometimes you will have a hammer, and some of the things around you will be nails. Sometimes glue, a screw, a dowel, a rivet, or a nut and bolt will be more appropriate. Sometimes all of these will work just as well, and the ultimate decision will be nothing more than a matter of personal taste. -- ZygoBlaxell

I don't think XP has a problem with the use of an existing framework, any more than it has a problem with the use of an existing library; the only tension is where the code has to be distorted to fit the framework, without immediate benefit. There is one case i can see where XP permits the use of a distorting framework, or (possibly) the construction of a new framework: where it's part of the SystemMetaphor. If the metaphor for a telephony server is about responding to events, it is legitimate to start by building a framework for managing delivering events. Still, it's common sense (and extreme sense) not to over-engineer this. Am i right, or i have i lost the plot? -- TomAnderson

Moved from YagniAndLogging

It seems to me that what this page is saying boils down to an empirically testable "match":

YouAreGonnaNeedIt: if you have ample prior experience coding a particular kind of functionality in a generic, reusable and extensible manner, and a project calls for even an embryonic form of that functionality, then the time you spend up front pays for itself in time saved on making it more generic later.

YouArentGonnaNeedIt: if you systematically avoid making an up front investment in genericity, reusability and extensibility, and a project calls for repeated use of a functionality such that it needs to be made generic, then the total up front savings more than compensates for the later cost of refactoring the functionality to be more generic.

The entire discussion about what the customer should be specifying or not is, IMHO, entirely irrelevant. The customer always wants the cost of implementation to be as low as possible; with or without a UserStory, you should apply whichever of the two principles yields the lowest cost, all other things being equal. XP tries to make sure each of Business and Development have a change to maximize their relevant contributions to keeping costs down, but YAGNI instructs only Development.

Absent rigorous experimental data, it isn't possible to tell which prediction yields results closest to reality. I invite comment on how such experimental data might be gathered.

However, on the principles themselves, it is interesting to note one invariant and one significant difference.

The invariant is that systems designed under either principle tend to be more resistant to unanticipated change. (Although, on a "meta" level, I would expect that YouAreGonnaNeedIt will produce software resistant to unanticipated changes of the kind you had anticipated, whereas YouArentGonnaNeedIt produces software resistant to the kind of changes that actually happen.)

The difference is that you need a talented programmer with a long experience and a deep bag of tricks to make the kind of decisions that YouAreGonnaNeedIt requires; whereas YouArentGonnaNeedIt only assumes a competent programmer. Since you are paying more for the former, factor that into the cost of up front design.

CategoryFramework