Can An Architecture Emerge
Can An Architecture EmergeSome people say that a software system architecture can be an emergent property of an appropriate development process.
The term 'emergent' has a specific meaning with respect to chaos theory. A structure, in the past, was always assumed to be imposed by some direct structuring force. This is known as skeletal or imposed structure. This would be an architecture that started perfect and stayed that way.
In this context, emergent means emerging from the system rather than being imposed on the system. Take the example of a river's course. On the one hand the course of the river structures the flow of water. Yet the flow of water is also responsible for defining the course of the river.
Or, to take another example, Prigogyne described a number of chemical reactions where the structures seen were emergent, and related to macro-patterns (order 1cm) described by non-linear micro-interactions (order molecular). The structure was an emergent property because it emerged out of chaos, it was a structure made of interacting dynamics of varying stability.
Now I like the sound of that. I don't think a single skeleton allows you to evolve and thereby adapt to changing requirements. An emergent architecture though, that sounds like refactoring to me. XpIsAnarchy .
For some of us, the word "architecture" on this page means macro-scale structure, and the question we're asking is whether these structures can be predicted. So the question becomes "Can the resulting architecture of a software product arise without our having been able to predict its form from the start?" And perhaps this is how all new patterns -- including architectural ones -- appear.
You could frame the question in yet another way: "Is it possible to end up with a clear high level design from concentrating on resolving tensions at a detailed level?". Yes, it can, as long as tension resolution is carried on up. When it works, the smaller elements have exerted their collective will over the big picture. That's what we mean by "emergent".
This is not an argument against a reasonable amount of forethought; you always try to plan the obvious. And emergent effects and planned effects can happen comfortably within the same process, and often do.
The question of this page matters because for any planning exercise, there is always an incomplete but reasonable scope. Unless you're God, you can't plan in the scope that includes all scopes, including itself.
Is there an Architect for the Barn in an Amish Barn Raising?
I think the fallacy is to believe that because there is an architecture there must have been one architect. Most things with an architecture have an architect, it just isn't an absolute. -- TomAyerst
I think a more correct statement than CanAnArchitectureEmerge? is how much will the architecture change? Every piece of software has some type of architecture and any piece of software will have its architecture change over its lifetime. The question comes down to how do we best guide the changes in architecture, with some overall prediction of what will be needed or through a series of just in time changes? -- WayneMack
Once you've produced code to solve a problem, the structure of that code is just information. There are only a few sources that information could have come from:
When is it possible to do a good BigDesignUpFront and stick to it? When we know all the information about the problem itself that we need to structure the solution -- i.e., the final code.
Is it possible to do no up-front design and still reach a good solution? We're often asked to start coding without knowing everything about the problem, after all.
We might have something to worry about if there were a trap that your small-scale refactoring moves fell into. But do we know of any refactoring traps?
If the noise in your system -- differences in code formatting, disagreements over duplication, etc. -- are big enough, they could drown out the information from the problem itself. So then you might never reach a working solution, or might progress too slowly to make it worthwhile.
In summary, you could worry about a development process that depends on refactoring getting stuck, or letting the noise take over. But you don't have to worry about anything else.
Do we even have a clear-cut "problem" to solve? What we are doing is automating human processes, and we provide progressively better approximations of those processes. This implies you will have emergent design whether you start out with an up-front design or not.
Also, you're missing a critically important source of information--the solution. Emergent architecture relies on looking at a solution with a poor architecture and making a better architecture. By breaking this cycle into minutes and hours and days instead years and decades and never as in CodeAndFix, you can quickly converge on a good architecture without the risks of speculative design. When you get really good at it, you go so fast that people accuse you of thinking ahead even when you don't. -- KentBeck
Many of the people who believe that an architecture can emerge also happened to be ExtremeProgramming proponents, and they believe XP is a ProgrammingValueSystem that allows this emergence to happen.
The party line is that you have to start with a SystemMetaphor, which constitutes at least a logical architecture, then ModelFirst, then explore concerns and tool choices with one or more SpikeSolutions, and make contact between developers and users via a SpartanUserInterface. All of these are explicitly architectural activities. They can be conducted iteratively, but enough must be done up front to get to an iterative mode in the first place. It's BigDesignUpFront, not AdequateArchitectureUpFront that gets the flick. -- PeterMerel
Good architectures do not usually emerge. This is obvious by looking at legacy systems that are doing most of the work in enterprises today. No matter how much work has been done on some of these systems, they are still virtually unmaintainable, and not extendable. I think a better way to frame it would be that good architectures evolve (through careful attention to points of elegant simplicity in the design space) and that architectures tend to converge as in biological systems, i.e. given the same environmental constraints for organisms evolving through many different pathways, you tend to get a similar result. This evolution and convergent was one of the goals of the early SCRUM process. Extreme Programming has, as a principle, careful attention to points of elegant simplicity in the design space, so XP may make the case that constraints implicit in the XP process tend to evolve good architecture. In this sense, XP architecture is emergent through constrained evolution, a concept that the naive reader will certainly miss on first reading. -- JeffSutherland
But evolution is an emergent phenomenon, isn't it?
I can see how refactoring can decompose a system into smaller and smaller pieces, what I don't understand is what forces are at work to combine these pieces into meaningful groups. That is what I expect an architecture to convey, a set of generalized, higher level constructs. Any suggestions on what I may be missing or misunderstanding?
Macro-scale structures can form through emergence as small code changes and refactorings aggregate related functionalities.
What you may be missing is that although called 'factoring' (i.e., literally breaking down into smaller pieces), the term is commonly used to mean both decomposition and composition back into dense but loosely coupled units. The force that usually prevails in judgments of what's 'dense' and what's 'loosely coupled' is the force of anticipated change (no, ExistingSolutionsDriveArchitecturalEmergence?. So, if you stand back and watch, you will find emergence happening during refactoring, not at some later time. The discovery that this thing and that thing are alike, and likely to always be alike, is emergence in progress. Hope this helps. -- WaldenMathews
I do know little about XP, but I think that the refactoring is similar to a process of SimulatedAnnealing. This works like a recrystallization process that centers around thinking & words & objects & language. If you repeat this process, certain structures of order emerge because they represent states of lower energy. IIRC such phenomena are quite common in thermodynamics. -- HelmutLeitner
Is "architecture" that which remains when the implementation is done? Discussion moved to ArchitectureDefinitions.
I've been working on a small program ( which measures the abstractness and instability of Java packages. see: http://www.objectmentor.com/publications/oodmetrc.pdf ), using TestFirst and RefactorMercilessly, and I am watching an architecture emerge. The program has been passing it's test for a few days now, and I have been refactoring since. Very soon now, I will have spent more time refactoring the program than I did writing it. -- ErikMeade
The idea of an emerging architecture that is not known before hand may work okay on small business-based systems, but it does not scale. It particularly does not scale for frameworks, foundations, and pluggable architectures. This doesn't mean it isn't great for smaller systems, but, just look at the inconsistencies, PatternChaos, and the AmorphousBlobOfHumanInsensitivity that the various Java API's (and their classes) have emerged into. A little planning, design overview, ConceptualIntegrity, and consistency rules (for patterns and naming) would have gone a long way. The larger a system grows, the more dangerous it is to simply think an architecture will emerge all nice, mature, and happy like. -- RobertDiFalco
I'll assert the opposite. The larger the scale, the more you must rely on emergence. The Java APIs are an excellent example of very smart people designing their butts off and achieving crap. Nobody said emergence was the same as the path of least resistance. It takes short cycles, concrete feedback, and social networks (basically what you said). -- KentBeck
Okay, to clarify, I think you are asserting that they didn't emerge but that they were, indeed, designed upfront and quantified where as I understood that they were pretty much just programmed, tested, found to be useful, and so polished up, tested some more and released -- i.e. no overview constraints. You could be right, I wont dispute that but on another point, I wonder if (at least sometimes [is that a compromise!?]) that using only the (definitely great) techniques of refactoring and emergent design can fail for large resusable systems with a wide audience. Especially, where independent systems (parts) can be taken as a whole. The problem is that once one library (i.e. part) is released it can no longer emerge without impacting other libraries (i.e. sibling parts) or their clients. Worse yet, and I'm pretty sure we agree on this, if the various library teams don't augment intra-team bandwidth with 'interteam bandwidth, with no team overlooking the ConceptualIntegrity of the overall vision, we end up with each library taking it's own path. This has the usual result of chaos and inconsistency throughout the overall system no matter how absolutely cool each individual part may be. -- RobertDiFalco
As far as when you "rely" on emergence, it seems to matter more what your sphere of influence is than how big the final product will be. When systems that weren't designed to be integrated with each other suddenly get integrated, the patterns of interaction are probably emergent. And the work to finish integrating them is based on those patterns, and sort of quasi emergent because there is both the unexpected part and our planned response. I think this may really be what this page is about, unexpected juxtapositions that we have to make work, and the architectures that result. -- WaldenMathews
If you're talking about architecture as ConceptualIntegrity, emergence may not be so easy. If the major SystemMetaphor's change, then you either chose an inappropriate metaphor or chose too early.
Most effective development efforts are initiated by a small group (often one person) who has the vision, and the ability to instigate that vision with an architecture. Perhaps at some point the group gets too big, and an architect (or architects) is needed to maintain ConceptualIntegrity. As to where that line might be, perhaps it varies from project to project.
However, even in small teams, the members will not share all work equally. Each will organically fall into roles they perform a little more than there team mates. Someone will organically rise to the task of dealing with ConceptualIntegrity issues more than the others. Another member will rise to the role of dealing with coding standards and project management more than the others. The only difference from a large team is that they may not explicitly recognize, name, or assign these roles on a small team -- the roles happen spontaneously.
Chaos is easy, no question about that. The hard part is maintaining wholeness at larger scales. As you increase the scale, centralized control of ConceptualIntegrity definitely falls apart, and you must either rely on some other method or accept that chaos is the best we can do. OpenSource is a community that creates wholeness at very large scales. XP strives to create wholeness under high stress situations, which is a different kind of scaling.
CollectiveCodeOwnership isn't done just for the sake of integration and refactoring, but as a design strategy as well. For a non-trivial project, no one person can hold all the design decisions in his head. At some point the team will be making decision decisions, like it or not.
Architectures are always emergent unless the system is a toy. Specifically, a designed architecture would need to provide for all possible constraints for it to remain stable throughout the system lifecycle. But we know requirements always change, right?
Consider termites building a termite colony. My guess is that this works because all termites use the same heuristics to decide what to do. Assuming that's true, the implication is that for a decent architecture to emerge, programmers would similarly have to operate to the same algorithm. That makes the XP metaphor, and the OnceAndOnlyOnce rule, and coding stands and other heuristics all that more important.
If the people working on the program have different ideas of what is good, then the resulting program will not evolve to a program with a good structure. For a good structure to show up, the programmers need to be making changes to the program that follow a common base algorithm, or they are following a common theory in development. Then all their changes might contribute to an emerging structural pattern. Conceivably, the algorithm they follow is different in different parts of the program. Even with heavy thinking along the way as programmers make local changes, if their operating principles / theories / algorithms / heuristic are different, perhaps the structure won't show up.
Suppose Pat thinks copy/paste is a great idea, and Chris thinks OAOO is a great idea (I worked with a senior tech lead like Pat). Good structure will not show up. Suppose Pat likes to hide business logic away from GUI classes, but Chris disagrees. Good structure will not show up, whether they refactor each other's code or not. They must share a common heuristic of what "better" means, and probably a lot more than that. -- AlistairCockburn
I don't believe you can posit emergence on a requirement of common thought patterns, especially since conscious intent is at odds with emergence, but to the detached observer watching a pattern emerge from chaos, common behaviors would likely be evident (in addition to common structure in artifacts), although they (the behaviors) might be part of what emerged, not something the system started with. Think "bootstrapping". -- WaldenMathews
Associations between uniformity of workforce and structure of product might be too obvious for their own good, Alistair, meaning that such correlations don't actually exist. I wonder about questions such as:
When structure happens predictably, it's not emergence. When you design for structure, like attacking a problem using only a uniform workforce, and arming that workforce with conscious design tools and models, you are making structure happen, not letting it emerge. By definition, emergence is slippery: defining its causation makes it go away.
Hmmm. Are we talking about working structures? As in the grouping of programmers? Or are we talking about the structures they produce as code? Or both? I think the former is interesting in that the slightest non-uniformity of people results in very heavy patterning. The butterfly effect I guess, but with nasty black-hole attractors when the system gets out of whack. This implies to me that you need a queen-bee to guide the direction and cohesion of effort, effectively damping the feedback to keep the system on track. More extreme intervention may involve imposing a fixed structure, but that sucks for all the reasons that fixed structures suck.
The latter obeys the river simile. What does not vary becomes part of the framework.
Another possible interpretation of the title question: Once a system has become a BigBallOfMud, is there any way for it to become otherwise, or is that another word for scrapping the current system and starting over (with the knowledge gained) to build again, but this time with structure?
Where does the "big boned" structure of a product come from? Can it percolate up from the tensions and resolutions found in detailed experience of building and using the system? Or must experience always yield only mess, requiring an intelligent human to analyze and plan a structured substitute for that mess? How does an idea like pipe and filer come about? Can a system say pipe and filter to its developers?
My take on all of the above: Problems have essential structure. Although there may be a few different and valid views of the problem, allowing somewhat different structures, the number of valid structures for a given problem is not infinite, nor is it a matter of analyst style. Plausible solution structures are functions of problem structures, more or less. Second, there is noise around everything, obscuring clarity and structure. This noise is essential for emergence, because when structure emerges, it emerges from a sea of noise (what else?). The critical questions do not include "can an architecture emerge". An architecture "can" always emerge, given the right conditions. The real questions have to do with the minimal conditions under which structure will emerge from apparent noise. There are also questions about for whom structure will emerge, since structure it largely embedded in the processes of observation and interpretation.
I especially like the tension resolution part here. It is about feedback. You need to feel the tension. Where you feel it, your instinct should be to resolve it. A tree for example can kind of "feel" badly balanced branches and puts more wood in the areas of highest pressure (it is more a mechanical effect, of course). He thus achieves statics that are better than those of most cathedrals and can hardly be rivaled by an engineering task force. A tree has an architecture. It emerges by tension resolution.
The tree also achieves better dynamics than a cathedral. It is the constat motion of the flexible parts that tells the tree where to grow stronger. If the tree grows too high and too quickly, it will break in high winds; this is what tells the tree to grow out and not up. If the tree grows out too far and too quickly, it will snap under the weight of a heavy snowfall; this is what tells the tree to grow up and not out. Between the two it achieves balance, without the least bit of planning.
You need to be willing, here and there, to let a few branches break. It is how we learn.
BTW: How about comparing Software Development not with building houses but with gardening? -- DierkKoenig
Can a system say pipe and filter to its developers?
I like this question, and my answer is "Yes". And a developer who's listening, and who's had some experience with Unix, or Java's I/O Stream Filters, will hear the system loud and clear. But a developer who's never seen pipe and filter probably won't get what the system is saying.
This is now the best explanation I can think of for why patterns are important, even if you're doing emergent (as opposed to up-front) design: you need to see multiple instances of something before you can recognize a new instance of it. To someone who's never seen a hammer, does anything look like a nail?
Books like the GangOfFour's are phrase books that let you translate what your code is telling you. The Forces section is what you hear from your code (in addition to any obvious code duplication) -- Ha bla piepandfiltr?. The pattern itself is what your code is trying to say (or suggest) - Hey, pipe-and-filter might work nicely here. -- GeorgePaci
Few software developers are building 'cathedrals', they are mostly building 'cottages' and 'small shops'. Craftsmanship and skill are important and they are accessable to everyday folk for this level of construction. It may be that a lot of software projects can get by with FolkArchitecture?. Just a thought. -- TomAyerst
What could a company do to try and create a fertile environment for architectures to emerge? Would that be AgileArchitecture?
Be careful not to mix up Evolving Architectures and Emerging Architectures. Architectures absolutely must evolve. They don't particularly need to emmerge (but this depends on your definition of 'emerge'). The larger a system is, the more API's it needs to provide for ISV's, the more it needs to integrate with different enterprise systems, the less possible it is for an architecture to simply emerge. I would say custom apps would probably have a better chance with complete architectural emergence, but it's pretty difficult for a commercial product that needs to support an ISV aftermarket, system integrators, sales channels, technical marketing and so on. If we are using evolution and emergence as synonyms, then I would agree that all architectures emerges. -- RobertDiFalco
Let's say there are 100 workers constructing your building. On room 2097 of your building you discover this new really cool idea. How easy is it to change the other 2096 rooms? How long will it take to retrain the 100 workers?
Difficult, and much time. Luckily software doesn't work like that. Adding a cool idea to the Room abstraction doesn't necessarily make it any harder to instantiate multiple times.
Of course software does work exactly like that. The cool new idea isn't to make the paint color setable. It's more complex than that.
You'll have to demonstrate how writing software is more like building than writing instructions for building. Real buildings are made of material that has weight and (generall) rigid structure. Changing 3000 rooms is non-trivial if they are made of wood and concrete. Changing 300 Room instances can be trivial if they are made of bits. No matter how complex your cool new idea, you don't have to tear down any load bearing beams to implement it (or test it).
Sure, no problem. Let's say you have a commercial software product with an API for integration. You have 1000 ISV's that are using this API and now you decide that you have a much better way to express some integration points and this new way is not going to be backwards compatable. Hmmmm...too bad you didn't spend a little more time in up-front design. You might have caught this before publishing the API. -- RobertDiFalco
Standard procedure is to release a new version of the API and deprecate the current one.
Which is not necessarily a good architectural strategy. In building new things, one has no problem with this, but in maintaining old things or remodeling them, the reliance on stability, endurance and backwards compatibility are critical. Emergent Architecture should not deprecate currently used components on too short a time scale. The time scale should be based on the life-cycle of the dependent software and associated hardware.
The deprecated API lingers until the benefit sinks beneath the cost (usually well below it). That's why the software I'm using right now won't run on an 8088 CPU or MS-DOS 2.0 operating system. No amount of up-front design would have avoided this. -- EricHodges
I found this buried in an economic paper about the dynamics of open source development (http://web.mit.edu/evhippel/www/UserInnovNetworksMgtSci.pdf):
Saw something like this about how the human brain works - "at a critical point on the edge of chaos" - http://phys.org/news156767725.html
Buried and success on the same sentence? Being successful means making yourself irrelevant. I don't like that. You are much better off introducing flaws. -- Dilbert's Pointy Haired Boss
If things are done properly, all architecture is emergent. This is true even for those who follow BigDesignUpFront (BDUF). Where do you think that design comes from? A responsible project manager would never just take an idea from thin air and jump straight into production. Only an idiot would commit to something that has yet to be proved. This is not to say the world isn't full of idiots. The point is, all reliable designs emerge from a process of evaluating ideas, mitigating risks, and answering the unknowns. This process is called research & development (R&D). It's the phase that preceeds production and helps to ward off any big gotchas. Both XP and BDUF have an R&D phase. The only difference is that BDUF draws boundaries and segregates it from production, whereas XP does not.
Case in point, take the prior example of building architecture. Large structures, especially those with novel features or locations, are always put through an R&D process before any significant construction begins. The building is modeled and loads are tested. New truss designs are assessed for reliability. The ground is evaluated to ensure it'll hold the weight. An architect doesn't just whip out a bunch of drawings and people start to work. They try things out first. If they run into a problem, they back up and try something else. Thus, the plan they eventually commit to has emerged from a process of repeated testing and evaluation.
In short, BDUF is only BDUF if you ignore the R&D that happens beforehand.
- Milo Hyson
See: ArchitectureDefinitions, WhatWillThisBecome, XpNeedsNoArchitecture, PerfectArchitecture, ArchitectTheNegativeSpace, CanMathEmerge
Contrast: ArchitectsDontCreateArchitectures, which is possibly the exact same idea approached from the other direction
EditText of this page
(last edited September 24, 2012)
or FindPage with title or text search