Abstraction Development

Push as far as you want into other worlds, death is the only real escape from this one

Contents

Framing the issue
The nature of abstractions
Developing abstractions
Abstraction prejudice
Automated abstraction tools
A Unified Abstraction Map / Pattern Taxonomy
Implications for SoftwareDevelopment
The awful truth

Framing the issue

It is difficult to discuss these concepts when the meaning of each word is also abstracted. The nature of describing and solving a problem resembles the problem itself. Of course MeaningDependsOnContext. The difficulty in software development is design, not implementation. Developing good abstractions is more difficult than implementing them as code.

The author assumes that any defined words rely on definitions using words that are themselves abstract. It is up to the reader's to draw meaning correctly.

The reader is assumed to be a terran human fluent in the English language.

Reality is assumed to mean the physical substrate humans live in. It is where the corporeal bodies of your loved ones exist.

A solid foundation for abstractions

Reality is the fundamental context for all shareable abstractions. Therefore, the nature of any abstractions fully isolated from reality is incommunicable. Meaning cannot be conveyed without pre-existing context.
As the level of abstraction increases, meta-abstractions are needed to convey meaning. A pattern is a meta-abstraction used to convey meaning about a lower level abstraction. Therefore patterns always exist at a higher abstraction level than any abstraction it references. For any loa(n), a loa(n+1) exists that can convey meaning about loa(n)
The correctness of a non real-world abstraction can not be measured in real-world values. Consider it the AbstractionHaltingProblem?.
Info-diversity is the range of possible abstraction, and reality has the highest info-diversity. The computer world is not a separate or parallel world, it is an abstraction tied to the real world.
Reality holds the monopoly on meaning. We are taught from the youngest age that our imaginations, while useful, are irrelevant unless applied to the real world. As children we want our imagination free to roam unhindered, as adults we find that imagination alone is insufficient to satisfy our desires.
The further an abstraction is layered from reality, the more difficult it is to communicate with others about.

The nature of abstractions

All abstractions:

Are meaningless without context
Are flawed.
Resemble their creator
Are tied to real-world models

Good abstractions:

Are appropriate for the context
Minimize noise introduced by real-world politics
Resemble all creators
Can be easily communicated to other humans
Have been used before as a pattern - this is a ChickenAndEggProblem
Can be analyzed objectively
Are within acceptable tolerance for mapping to the problem domain

Bad abstractions:

Are unique to the situation
Evoke personal reactions
Map noisily to the problem
Are outside acceptable tolerance for mapping to the problem domain

Developing abstractions

Practicing Humans usually learn without understanding how learning works. Every time we knock our heads against a problem, we are practicing.
Practices Obviously software does get written, so solutions are being found, the problem is in classifying them so others can discover and use them.

See:

TooMuchAbstraction
AllAbstractionsLie - Abstractions's Law of Thermodynamics
LevelsOfAbstraction - Discusses a useful triangle.

Forces working against us

Information availability Opposing strictly limited information is the open-source movement. There are many licenses, but the GPL, LGPL, and the BSD license are the most archetypal. The need for different licenses arises from the debate over the definition of "free". Some consider free to mean available to all and usable for nearly any purpose(BSD). Others consider free to mean available to all, but protected, so that it remains available to all(GPL). The LGPL might be considered the middle ground. The GPL protects by forcing the code to be open if it is redistributed(for free or profit) when it is statically linked to a program. The LGPL allows some exceptions to this, when the code is dynamically linked. The BSD license allows code to be "stolen" for proprietary projects. Software code is just one kind of information, and the open source movement is by definition about source code. Groups like Project Gutenberg supply public domain books. Many kinds of information are available for free, but there is little organization to all the public domain information gathering and generating projects. Some types of data will remain available, but there is certainly no public domain computer parseable general knowledge-base.
Compounded layers of abstraction Computer science has given an excellent foundation to practical aspects, but is silent in the face of pure abstractions. Object oriented examples often give the animal example. It usually shows a bird class, the live/die/fly/spawn methods. Then it goes on to describe a specific bird, that inherits the characteristics while adding a few new ones. While using real-world examples are great for conveying concepts, true abstractions are often hidden. The problem is context outside of the real-world. Should the bird abstraction be a specific version of something more abstract? How high should the abstraction go? How specific should it get? How do we send a bird to a file? How do we send a bird to a file that is a a program that is on a remote computer that requires logins and is a different operating system? How do we visualize or convey the structure required to do that without getting bogged in yet more complexity by compounding the layers of abstraction?
Hallucination As developers we build up models of our problem by creating entire worlds full of abstractions. When asked to change our new world we find our carefully constructed abstractions were often inadequate. Traditional science can compare a model against real-world results, but software engineering has no such technique. The very nature of the problem - excessive complexity and the flaws inherent in abstracting - prevents any isolation of the variables we want to test. We are all sharing our mass hallucinations while arguing over whose hallucination is more real, with little foundation to compare against.
Language Compounding the problem is language. Consider these words as an abstraction of meaning encoded in a sentence. Readers will vary in what meaning they extract from the abstraction. They may wonder why certain words were picked, or perceive knowledge beyond what was intended by the author. When patterns are described, the most important component is not the name of the pattern, or even the solution, but the context. The context makes the solution make sense. Otherwise the developer might pick a pattern that others have used before, but use it for an inappropriate problem. Language derives it's power from context - from having a good abstraction of the intended meaning. High level language compilers were a SilverBullet by bringing programming closer to the level of English. It was then possible to discuss and think about code more easily. This is due to our training in language from the real world. The further an abstraction is pushed from common language and concepts, the more difficult it is to describe.
Unwanted complexity Unwanted complexity naturally arises from abstractions that are mismatched with context. However if the context is the real world, an abstraction can never be perfectly be matched with what is real. Any hope for amelioration will come from ways to find the least-error prone abstraction for a given problem. The real world is only a subset of the range of problems. We will need techniques for finding the abstraction that best matches two other abstractions - both of which are many levels of abstraction away from concepts we could expect non-developers to know. The hyper-modal nature of the human mind is excellent at finding *a* solution, but we have precious few metrics to determine how adequate a given abstraction is for requirements that could easily change - as happens in the real world.
The problem of selecting good choices The problem of selecting good choices requires context. For humans, context comes from almost anywhere. We base it on our own past knowledge, information gatherable at the time, even instinct or emotion. For a computer, input is as tightly controlled as it is for a prisoner in solitary confinement. For a computer to select good choices for any non-trivial problem requires multiple sources of context information. Software is usually bloated with specific information and starved for general information. It is partly a socio-economic problem. Data is valuable, and must be massaged into proper formats for computers. While the amount of data software could conceivably use is nearly inconceivable, the amount of data available is extremely limited by price. Copyright distribution issues only add to the problem. Complexity is the most common problem in software, but lack of general information may be an unidentified competitor. Most of us carry around trademarked terms, knowledge that may be illegal to reproduce, protected algorithms, even illegal knowledge about taboo subjects. It is impossible that software could make decisions at the level humans do while being limited by what it is allowed legally know.

Possible parallels with MetaDevelopment?

ArtificialIntelligence The parallel to ArtificialIntelligence is in the nature of complexity. AI advances slowly due to ignorance of the intricacies of balancing complex subtle abstractions. Most assume that the next jump in AI will come when machines are capable of building up abstractions into higher logic as humans do. The complexity involved is overwhelming and raises questions of automatic problem solving, the origins of creativity, consciousness, and the ultimate limits of logic.
FreedomVsSecurity? The parallel to FreedomVsSecurity? is in the nature of balancing microscale and macroscale. The needs of each individual are weighed with the needs of the whole. Just as the needs of each individual's cells are balanced with the needs of the whole. What makes society so complex is the subtle interplay between low level individuals, high level groups, the higher level abstractions of groups, and the highest level - everyone. Humans even have the equivalent of facades. What we know as corporations are the high level abstraction of groups, but made to look like an individual unit to the law. Like most code, a perfect mapping is impossible, and sometimes the law has to treat the units differently. What is key is the balance that must be maintained for the entire system to function properly. Like any security approach, excessively draconian controls make the system bloated beyond it's ability to function, and complete freedom leads to disaster. The abstraction of individuals and groups is a UsefulLie. The problem of balancing abstractions lies in the nature of truth. Like any abstraction, the model will be flawed. The system does not have to function perfectly, it just has to maintain the balance between abstractions. That balance is the truth of the system. When we can formally define how to find and balance that for any abstraction we will have solved more than just a computer question.

Forces that help us

Grounding Many developers practice "grounding" - a technique of "getting back to the real world". Many do so without calling it that. Essentially it is the practice of reducing the problem space by enforcing real-world constraints like time, money, features, memory, environment, past knowledge, etc. In essence, grounding is finding the balance between the abstract and the practical. It can also be thought of as mapping between abstract and practical spaces.
Projection In some circles, the idea of astral projection exists. The idea being to step outside of one's body into an essentially more abstract form. Traditionally, the projected form is tied to the real body by a "silver cord". If the silver cord is severed, the projected form is unable to return to the body. When abstracting/projecting, we remain tied to the real world so that no matter how far we push.
Possibilities Time and evolution will bring new knowledge that could act as a catalyst
Real world solutions Solutions found in practice help frame the problem
Real world mapping Although abstractions are created in mental space, then made tangible as code, they still exist in the real world.

It is possible that humans simply cannot reason about abstractions that are not related to the real world. The source of creativity is elusive, but truly original thoughts are generally considered impossible. A thought is always based on previously knowledge. Abstractions created in software are still tied to consensual reality. Attempts to reach beyond that limit are futile, as any implementation or mental thought about the abstraction will always be tied to our consensual reality context. In essence, our physical reality is the FundamentalContext?.

Abstraction Prejudice

Let us consider the following software development metaphor:

Software Development as Shamanistic principles

Astral Projection - into another world
Spiritual Exploration - of that world
Energy Grounding - to avoid drifting, or to return to this world
Returning - to the real world bearing knowledge

Some would reject the metaphor due to their own opinions of shamanism - as they have experienced in the real world. Real-world opinions affect our abstractions that they would easily trump the correctness of the solution.

Software Development as Hardcore Drug use

Unoptimized state of thinking
Addiction to established practices
Constant need for a fix
High price for little progress

In the context of anti-patterns, this is a suitable metaphor. It is societally acceptable as most drug use is considered by the mainstream as counter-productive. The social aspects weigh so heavily they almost obscure the point of the metaphor in the first place - to describe bad software practices.

Software Development as Psychedelic Drug use

Radically different perspectives
Not addictive and effects can vary widely with each usage
Used when novelty is desired
Indeterminate price for indeterminate progress

In the context of design & brainstorming, this is a suitable metaphor. However is runs into social implications immediately. Do I appear to be advocating psychadelics? Are the effects too debatable? the metaphor attempts to describe a software development technique, but the social implications again obscure the issue.

It appears that the real-world counterpart to a development metaphor should not be a hotly debated topic. This is due to technical considerations like clarity. It is also due to social considerations - specifically the desire not to offend, and the desire not to get dragged into endless debate on the wrong side of the metaphor.

Imagine the obscuring issues that would arise from using the following metaphors:

Software Modeling as Racial Discrimination
Software Testing as Early and Late-term Abortion
Software Prototyping as Child Molestation
Software Security as Summary Execution
Data Mining with Agents as Gang Rape

All other factores excluded, unrelated real world constraints affect what can be practically used as a metaphor or abstraction.

A Unified Abstraction Map / Pattern Taxonomy

Many would like to have a hierarchical taxonomy of patterns, but there are problems in the way:

There are PatternLanguages, but they are only human readable, and usually domain specific.
Non-orthogonal aspects
Difficulties in agreeing what is a pattern or anti-pattern
Lack of formal way to declare the context of the pattern.

What are the non-orthogonal aspects to patterns that prevent creating a hierarchical taxonomy?

Formality of pattern(human readable, machine, both)
Problem Context
Anti-patterns
Language-specifics
Related tools

Prototype pattern taxonomy for software development(ignoring the non-orthogonal problem):

Scientific - Quantifiable and testable patterns(i.e. science)
Social - Related to dealing with other humans
Design - Related to architectural design
Construction - Related to implementation
SelfImprovement? - Related to how the software developer can improve
Configuration - Patterns related to how the software is configured
Trouble - Bugs or antipatterns
Reference - External information sources

Possible taxonomy conventions(where is data included):

A hierarchy with hyperlinks
Implicit by naming convention

See:

Implications for SoftwareEngineering

SoftwareEngineering is any subset of the art/craft/science of using any management style to map anything to anything in any possibly way in any language to run on any machine over any network that is usable by any level of user at any level of quality at any level of cost at any level of abstraction, using any number of workers to solve a problem that is constantly changing.

SoftwareEngineering needs a way to resolve choices efficiently. It is no wonder there is DisciplineEnvy when the BenefitsAreSubjective for most choices when even the BestPractices cannot be agreed upon. SoftwarePlatonism seems interesting

Too many choices is no choice at all

SoftwareEnginnering? has no SilverBullet(where improvement is ~10x), and for the average developer, searching is useless. What SoftwareEngineering has is patterns and processes(where improvement is ~10%). An approach that pre-mixes small improvements to achieve major improvement will fail in practice due to the varied unique contexts. A dynamic approach that generates design abstracts will fail due to software's inability to fully grasp complex problem domains. One possibility is a GuidedDesignAbstractGenerator? where the user specifies the relevant aspects of the problem domain. Such an approach is flawed in assuming that the user understands the problem domain. Software is also terrible at solving very abstract problems. Although software is excellent at specific problem solving, general problem solving would be needed, at a level currently only humans have. The current state of the art does not allow for such an approach. Unless new software implements a valuable new paradigm as yet unknown, it will not be a SilverBullet, no matter what marketing departments say.
No current graphing technique has the potential to be a SilverBullet. Whether the GalacticModelingLanguage, UnifiedModelingLanguage, CaseTools, code dependency visualizations, profiling charts, etc - None alone offer major improvement, although many choices are available and mixable than in many aspects of SoftwareEngineering.
No current language has the potential to be a SilverBullet. The difficulties in balancing high-level abstractions with low-level functionality has already created languages at the far extremes. Raising or lowering the abstraction level offers no benefit in general. As always, many choices are available, and the best language is unique to the situation.
Some have proposed formalizing patterns and processes. Although some processes are quite formal, there is no formal vocabulary to discuss and compare them. Again, infinite choices abound.
It is apparent that much of the territory in SoftwareEngineering has been explored. Unfortunately, like the TowerOfBabel, the explorers have different languages. Even worse, the explorers cannot agree on where they went, or even a map of the territory.

Questions that remain

What are the techniques for balancing abstractions?
What parallels between programming and writing exist for finding the least error-prone abstraction/word to convey intended meaning?
Does the existence of a SilverBullet imply finding a foundation for SoftwareEngineering?
How can we formally communicate how to solve problems?

The awful truth

A good approach to MetaDevelopment? is unlikely to come from extrapolation by architects who do not solve practical problems. The experiences of developers is far more likely to bear pearls of knowledge. The techniques used to solve problems in computer science are not applicable in the subjective abstract space of software development. Currently, experience seems to be the primary indicative factor of the correctness of an abstraction, and then only as a difficult-to-teach, informal, possibly illogical approach.

Developers must accept that standards have won for many reasons other than technical superiority. A mediocre solution implemented is still superior to an optimal solution unimplemented. Social considerations are usually more important than the technical. To avoid AnalysisParalysis, compromises must be made. A developer must have the will to take action. Hopefully that will is forged from wisdom in the fires of experience.

In the real world, will power is as powerful as thought.

Shamanism is not measured by the distance traveled, but by what is brought back

From AbstractionDevelopment: As the level of abstraction increases, meta-abstractions are needed to convey meaning. A pattern is a meta-abstraction used to convey meaning about a lower level abstraction. Therefore patterns always exist at a higher abstraction level than any abstraction it references. For any loa(n), a loa(n+1) exists that can convey meaning about loa(n)

Not always so, I think. For example it is possible to design a type system in which the types are themselves types. Compare with awareness of awareness. Want to take this shamanism meets software abstraction discussion into email? JamesCrook (crookjATindigoDOTie)

You have to be careful; if there isn't an infinite hierarchy then, ceteris paribus, you're subject to the Liar Paradox, which means an inconsistent system. There are other ways to avoid this, but it requires great care. -- DougMerritt

Oh...actually another mathematical principle holds here; intuitively it seems as if any space of dimension N requires a space of dimension N+1, causing an infinite sequence of dimensions. However it was proven long ago that such embedding spaces are not inherently required. This proof was very important in differential geometry (an underpinning of General Relativity) in allowing direct treatment of the curvature of space via intrinsic curvature, without reference to any embedding space.

Similarly here; if you treat a LevelOfAbstraction as a space, it does not necessarily require a higher dimensional space to contain it.

I suppose this is less directly helpful than my earlier point about the Liar Paradox, but I think it helps to keep in mind these areas where mathematics has discovered that intuition yields incorrect results.

CategoryDesignIssues CategoryAbstraction