Linus On Design

Back in December 2001, LinusTorvalds posted a message on the Linux Kernel mailing list that's been generating some attention (e.g. on JoelSpolsky's site).

The post is archived here:

http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.0/0004.html

You may also want to read this reply:

http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.0/0089.html

An excerpt (on how Linux is developed):

But I am claiming that there is no common goal, and that most development ends up being done for fairly random reasons - one persons particular interest or similar.

It's "directed mutation" on a microscopic level, but there is very little macroscopic direction. There are lots of individuals with some generic feeling about where they want to take the system (and I'm obviously one of them), but in the end we're all a bunch of people with not very good vision.

And that is GOOD.

A strong vision and a sure hand sound like good things on paper. It's just that I have never ever met a technical person (including me) whom I would trust to know what is really the right thing to do in the long run.

Too strong a strong vision can kill you - you'll walk right over the edge, firm in the knowledge of the path in front of you.

I'd much rather have "brownian motion", where a lot of microscopic directed improvements end up pushing the system slowly in a direction that none of the individual developers really had the vision to see on their own.

And I'm a firm believer that in order for this to work well, you have to have a development group that is fairly strange and random.

I'm bothered by these types of messages only because, in reality, Linux already had a high-level design. People read this sort of thing and then try to apply it to projects that are started from scratch with no reference architectures. To say that there was no common design is misleading.

Linus suggests in that same email that the existing high level design from Bell Labs came about in the same way.

TheEvolutionOfTheUnixTimeSharingSystem seems to imply otherwise. Sounds like plenty of diagrams, performance analysis, prior-work, and black-board meetings to me. Sounds like design, smells like design.

And yet they use the word "evolution" in the title (in 1979!)...

In a response, poor Linus got his own earlier words to the contrary thrown back in his face:

: [Linus:] But that's not how UNIX is or should be. There was more than just engineering in UNIX. There was Design with a capital "D". Notions of "process" vs "file", and things that transcend pure engineering. Minimalism.

Given how slippery the word "design" is, what makes you think Linus means the same thing in these two different posts? In any event, it looks like what he's warning against is BIG design (up front, naturally) - he just doesn't have that particular phrase in his vocabulary (yet). For what it's worth, the original Unix guys seem to have had a lot less BigDesignUpFront than the MulticsOs project did. -- GeorgePaci

Another example of the recent trend of mysticism around software I've noticed lately. Sad to hear it from Linus. I think we're currently in a trendy phase where some people beat up on anything that smacks of methodology or formality. Architectures are just supposed to "evolve" from the code. We're supposed to "listen to the code." We're supposed to "walk at random." What a load of crap! The devil IS in the details, and software artists (programmers) think they can manage those details by heroic, brute force, while software engineers apply a variety of tools and techniques to manage change and complexity. Both paths can succeed or fail, of course. But I'll take the latter course as less risky and less dependent on heroism.

Another example of the recent trend of mysticism around software I've noticed lately. Sad to hear it from Linus.

Linus leads a successful software project. He thinks the ideas he expressed in his post are a key part of why it's successful. Where is he mistaken?

I think we're currently in a trendy phase where some people beat up on anything that smacks of methodology or formality.

You should probably ignore those people. Just as we all should have ignored those in the previous trendy phase who worshipped anything that smacked of methodology or formality. -- GeorgePaci

From above: The devil IS in the details, and software artists (programmers) think they can manage those details by heroic, brute force, while software engineers apply a variety of tools and techniques to manage change and complexity.

Or: Talented, skilled Programmers create high quality software through careful application of great skill. Software Engineers and their Managers think they can aim formality, ceremony, and pretty pictures at a problem to emit quality.

Ask yourself this: if uniform, formal processes are so effective, why does the Rose product from Rational (fervent purveyors of big 'P' Process) crash every 25 minutes, and frequently scramble my diagrams, even after at least 6 or 7 major releases?

How did we get from upfront design, to uniform and formal processes? Are we possibly going down a rat-hole here?

Most formal processes mandate upfront design as a distinct step.

The problem with this stance is that software engineers don't exist. Of course, I am aware that there are a great many people who label themselves software engineers, and even a number of misguided university faculties (often chasing funding) who offer degrees in it. But there is not a single individual on the planet who is a software engineer in the same sense that a mechnical/electrical/whatever engineer is an engineer. More precisely, there is nobody performing software engineering in the same sense that may other disciplines perform engineering. This is subtle, because on the surface the activities can look almost identical at times. Unfortunately, while an EE may be standing on a strong foundation (which the EE doesn't need to understand deeply) of physics and mathematics, the software engineer lacks this foundation. Building our understanding to the point where this is not the case is an admirable goal. Pretending there is not a vast gap still is a daydream. The point behind this is that you cannot get successful software from merely applying 'software engineering' techniques. You may succeed if you have a bit of luck and a number of other hard to quantify inputs. On the other hand, you may not. And you won't, in general, be able to analyse the difference. Until such time that a solid engineering basis exists, you also need other approaches...

Most software engineers (myself included) would never claim to be equivalent with other engineering disciplines. Nor would they claim that "software engineering" techniques are sufficient to build good software. Software is hard, and we haven't been doing it nearly long enough to have figured it out by any stretch of the imagination. Right. I added "also" to 'disambiguate my intent in the above. I was certainly not suggesting that there is nothing to learn from engineering processes. Quite the opposite - this is vital. But blindly following such processes won't get you very far, and in some cases may be detrimental. Then we're in agreement, because I don't believe in blindly following any single process or person...

But I think it is folly to point at the many failures of software and say "See, formal methodologies don't work! We need other approaches!" When the users and managers (who are paying for the software, remember?) ask "Okay, what other approaches?", we can't just say "Just let us do it our way. We'll listen to the code, and let the architecture evolve. But we can't give you any hard milestones or design documents; that's just so waterfall! You're just going to have to trust us!!"

In other words, just because there are disadvantages and weaknesses to every tool, technology, and methodology (see many other pissing matches on Wiki for examples: OO is dead, UML sucks, etc.), doesn't mean we should throw up our hands and devolve to hacking. We should continue to use these flawed tools and methodologies, AND look for ways to improve them, AND look for better tools, AND keep comparing notes with each other one what worked and why, and still make sure we deliver value (in the form of working software, on time and within budget) to our customers, or else this is all an academic exercise.

So who's advocating hacking? Certainly not the XP people. I'm advocating craftsmanship. I accept this, to the degree that it represents the more experienced advocates of XP who know better than to hack. But I've read a fair bit on this Wiki that leads me to believe that there are many people who see XP as a loophole to get out of any formal design or documentation, who basically want to hack. And they may be pretty good hackers, and some of the time they'll manage to build good software. But I sure wouldn't put my money or business on that approach... (I personally consider myself a software craftsman; I take pride in what I do, and I always try to do it right, even when forced to do it waterfall style...)

Uh...so craftsmen don't do any upfront design?!?

I don't know what craftsmen do, but XP people do upfront design. They just do it in really small steps, just as it's needed. Then it isn't really "upfront", is it? And I doubt craftsmen do BigDesignUpFront; they probably have a picture of what they need and start building it, then adjust on the fly with their deep experience and talent guiding them and the eventual outcome is high quality, even if they couldn't describe exactly what they did at the start to ensure it. -- PeterHansen

Master craftsmen might do that. Normal craftsmen follow patterns, blueprints, or their master craftsman. Then hire master craftsmen, and don't try to force them into processes more suitable for lesser craftsmen.

Depends on the craft, too. Good film producers/directors most definitely do a great deal of "design" up front; they generally won't start shooting until they've honed the script.

Film directors face a 6-figure daily burn rate spin-up once the stars and huge crews arrive, so up front design is almost unavoidable for them. More importantly, ad-libs and script tweaking aside, acting is also fundamentally distinct from writing. In general, writing doesn't require the skills of actors, gaffers, and production assistants, and script writers can't usually act or wire up lighting. A comparison to filmmaking breaks down quite quickly since only skilled programmers can design software.

Other crafts might be closer to creating software. Take furniture-makers. Although most use factory-style production runs to reproduce a successful design for a wider market, the best original pieces usually emerge through applying great skill and talent to only a rough idea. (But comparison to that craft also fails eventually, since a piece of software is trivially reproducible once complete. Ultimately CreatingSoftwareIsCreatingSoftware?, and comparisons don't do it justice.)

I'd say that four legs, a back, a seat, and so on are rough ideas. Usually, there is a very formal idea that is then improvised upon, with the improvisation always needing to conform to the same requirements. The cases were someone does something completely different that breaks the mold is a fringe case, certainly Linux is NOT one of these fringe cases. -- RobertDiFalco

For software the design itself is a hack. What formalism? The tools we are supposed to design with, the methods, the discipline, the verification process are nothing but hacks and in many cases are just nothing. UML is not even able to provide decent definitions worthy of a scientific work for the basic concepts it works with (like class, instance, type, parameterized type and so on). If UML is a hack, I suspect that "UML driven" designs are automatically hacks. You might want to say something about Z or CSP or other very few examples of formality that are applied in niche markets. But for >95% of the software industry the design is a hack. So do you advocate hacking over hacking? Is design hacking or architectural hacking better than code hacking? -- CostinCozianu

I'm having a hard time buying this. Are you saying that since design tools and techniques are not perfect that they should be abandoned altogether? I'm not to interested in formal design methods as much as I am in favor of making upfront analysis and design a formal (reentrant) step in software development. I could care less about the process, technique, or method used to do that. If we stopped coding because it was imperfect, we would have no software. Basically, it's generally a good idea to understand the problem domain and have an idea of what you are building before you build it. Design is not mutually exclusive with evolution and change. Maybe I don't understand your point. -- RobertDiFalco

Well, to say they're not perfect, it's a funny way to put it :) Nothing's perfect. I was saying they're a hack. Should we abandon them altogether? No, because we can't. The political pressure is there to use them, at least for documentation. And while I'm such an amateurish technical writer, even I can see that they're lousy even for generating technical documentation. We shouldn't abandon things like conceptual modeling, requirements elicitation and validation, but these are not about design, and I was commenting only about design. Other than that I think we're in violent agreement. But the only thing I know for sure about design is DontThinkInUml?. My preference would be to abandon the tools altogether until we get something better. -- CostinCozianu

To summarize, LinusTorvalds is saying that the way to create Linux was to ScratchAnItch?. It seems to work pretty well. I suppose that ConceptualIntegrity becomes an itch, eventually, when the code becomes a mess and you need to fix it. If the code is a mess but you don't need to fix it, it is not an itch because it works. If the code needs to be fixed and it is not a mess, it can be fixed right away, a very slight itch.

Do not ScratchAnItch? in real life, you may get a bad rash.