Essential Complexity

Some things are just inherently complex, but I would like a better explanation of EssentialComplexity (on its own and in contrast to AccidentalComplexity) -- OleAndersen

Off the top of my head, I can identify three sources of EssentialComplexity:

the problem domain is complex, e.g. CognitiveSciences?
the algorithms are hard to understand, e.g. QuantumComputing
simple systems must grow into complex systems, i.e. they exhibit EmergentBehavior
Algorithms designed around capricious customers, managers, marketers, politicians, etc.

Hmm, I guess the first two boil down to the notion that some systems need to be complex because the "simple" way of describing the solutions cannot be found until we have learned enough about the problem domain by "BruteForce", complex solutions.

An example of the third (lifted from "SimplicityIsOverRated"):

The human body is a system that has been "kept simple", as have all living organisms: they all started out as very simple organisms. Then, generation after generation, simple changes were added. Only those changes which made the organisms better adapted to local conditions were kept; as a result, the complexity of the human body is EssentialComplexity, recording a long and varied history of contingent changes in local conditions.

I disagree totally. "Keeping a system simple" requires intelligent understanding of how simple the system is to comprehend. Building a system through incremental small changes is not sufficient to accomplish this, and is in fact the very archetype of overcomplex software. -- DanielKnapp

Maybe you only think you disagree. The two positions are not mutually exclusive: that EssentialComplexity results from accumulation of simple modifications to an initially simple design; and that intelligent design can keep a system simple. Or at least keep it from becoming overcomplex.

I would not necessarily grant that much intelligent design is required to keep a system from becoming overcomplex. I tend more to the Extreme position that too much design yields overcomplex software.

[Would not changes which have no effect on adaptation also be kept? Also, changes which have a harmful effect, but only on non-reproducing individuals, might be kept.] The latter category may even be selected for, as populations with an oversupply of drones will waste resources on supporting them that would be better spent on helping reproductive individuals to reproduce. The former category of changes can be kept (or lost) without apparent effect, but if circumstances change, populations with a larger reservoir of "ineffective" variations have a better chance of containing individuals who are better adapted to the new circumstances (fortune favours the mongrels).

One form of EssentialComplexity applies to living organisms - it is that which records a long and varied history of contingent changes in functional constraints. This kind of complexity is the same kind that, we are told by no less than KentBeck, will creep into the simplest, most aggressively refactored, most assiduously pair programmed XP project; and the end result will be the same. Much as species eventually go extinct, even a successful XP project must eventually grow to complex, be scrapped, and make way for something new.

Simple systems must grow into complex systems because they can't grow into simpler systems. They are already as simple as they can be, therefore the only direction they can go is toward more complexity. (What StephenJayGould calls a LeftWall? in FullHouse?).

Complex systems can be made simple. This happens in evolution as well as in programming. But intuitively, I would expect complexity in software systems to be retained over time, because it corresponds to functionality, and losing functionality (even to eventually get more) is unacceptable to owners of software systems. (Other factors, such as extremely rapid change in the software world, market forces, etc. may influence the evolution of software, and their effects might be more significant. I don't want to jump to any conclusions.)

Simplification of systems never happens in evolution. Given the above about the history of constraints being stuck in even XP projects, it should be evident that the chief way to reduce complexity is to take what you've learned at the end of a project and start from scratch. It should be even more evident that nature never does that.

Are we sure nature doesn't do that? What about mass extinctions... (could we say that nature has started almost from stratch at least 5 times? http://en.wikipedia.org/wiki/Extinction_event) It's hard to say that nature "learned" anything from those events - merely forgot a lot.

Life digests entropy.

{Genes for non-helpful functions do appear to disappear out of disuse (or are used instead for something else). Mammals lost some color vision genes, for example, because they were mostly nocturnal during the reign of the dinos. Primates had to reinvent an inferior replacement in order to analyze fruit. However, it could be argued that it's "good" to keep such genes around in case the need comes back.}

CompareTo? YagNi

Let's see if I can persuade you to buy any of this.... Consider something as simple as a refrigerator. A cooling mechanism around two boxes. No interesting essential complexity. Now add consumer buying. "Yes, sir, would you like 4, 15, 18, 25, or 30 cubic feet? Freezer inside, above, below, or to the side (left? right?)? What color? Water dispenser with that? Ice? Cubed or crushed? Self defrosting? ...etc..."

Now consider something reasonably complicated all on its own, such as a car. Repeat the above exercise.

There is a kind of complexity that comes from trying to satisfy 300,000,000 individuals that the "neat algorithm" approach won't satisfy and that isn't introduced by shoddy designers. -- AlistairCockburn

Frederick P. Brooks, Jr. (Author of the MythicalManMonth) said "Einstein argued that there must be simplified explanations of nature, because God is not capricious or arbitrary. No such faith comforts the software engineer."

Einstein - "Raffiniert ist der Herr Gott, aber boshaft ist er nicht." (God is subtle, but he is not malicious.) Inscribed in Fine Hall, Princeton University

I mentally translated this as 'Refined is Mr God, but bogus he is not' :) -- DarrenHobbs

I am hoping that the components of the car can each be simplified to their own objects. I realize that no inter-dependent system (how many cars do you know of that have portable, self-contained air conditioners in the engine compartment?) can have truly isolated objects, but to the extent possible these distinctions need to be kept distinct. The refrigerator needs to have a design that allows space for the self defroster, but doesn't require that functionality to provide refrigeration.

Why? People don't upgrade their refrigerators, except to replace the rubber bits now and then; they keep them for approximately ten times the life of the typical large software program, then replace them entire. Thus, refrigerator designs can and should be highly optimized - a model which is bundled with a self-defroster must allow space for same; a model without should not. Cars are perhaps closer to computers, but MUCH better documented and supported (and much more documentable and supportable, perhaps evincing lower EssentialComplexity). I'm just pointing out that not everything in the world is analogous to software, and usually I'd rather read "program" than "refrigerator/house/airplane" when programs are the real subject matter. -- DanielKnapp

This page is unnecessarily complex. EssentialComplexity is that complexity I just can't live without. -- WaldenMathews

Essentially KISS. Simple is as (Fractally Holistic) complexity does. -- GeraldLindsly

Ok, I give up, what kind of meds are you on?

Don't good'ole fashioned cures count? -- gl ;)

Make it as simple as it can be to encompass the EssentialComplexity, no more, no less, fini. -- Einsteinian concept GLized.

Essential complexity is the argument you use to justify how your software works.

"Essential complexity is the argument you use to justify how you worked your software over" -- IvanStojic

A thing exhibit "essential complexity" when its level of complexity is more than a single individual can manage -- Gigi

Instead of thinking about "essential complexity" in the abstract, look for essential vs accidental state. For every piece of long-lived state, ask whether it's possible in principle to write a program that doesn't keep that state. You'll find that many programs spend most of their effort managing accidental state. -- EdFaulkner?