Abstract Vs Concrete Software

[this was set off by something incredibly insightful AlistairCockburn said about "deep vs. shallow knowledge" in the last couple of days, and I can't find it now, curses!] (neither could I, more's the pity, and I'm sure I couldn't remember it - Alistair)

Before we dive into the discussion of AbstractVsConcreteSoftware, let me say what the definition of Abstract and Concrete seems to be below:

Abstract Software: Software that is used by other software, not directly by people. For example: The Linux Kernel.

Concrete Software: Software that satisfies an immediate human need and that possibly most likely will also use abstract software. For example: A payroll system.

In my experience, there's an important divide in the software community: some projects are abstract, and some are concrete.

Abstract projects are driven by an intellectual problem: How do I save things on a disk? What's the most efficient garbage collector? How can I store stuff and get it back in insertion order? Operating systems, many frameworks, and programming tools are often abstract projects. Much of the published computer science research comes from abstract software projects. Non-software companies sometimes build abstract software; however, these companies often prefer to buy abstract software.

People who are working on an abstract project often have only a hazy idea of the project's ultimate uses. Because of this, the developer strives for generality: if I don't know who will be using this, or for what purpose, I'd better make sure that it scales well, can be extended, and handles corner cases and unexpected conditions cleanly. Abstract software is intended to solve more than one problem.

Abstract projects aren't necessarily non-commercial. Many people build general-purpose frameworks to solve their own problems, then realize "Hey! I have a product here!" Some of them are wrong. However, abstract software can sometimes be so abstract that it can be sold only to other software developers. (HotSpot may be the ultimate abstract project.)

Concrete projects, by contrast, are driven by an immediate need: How do we calculate our payroll? How can we automate our contact-tracing system, while protecting the privacy of people's Rolodexes? How can I sell widgets over the Web? Concrete software can be either custom (C3) or mass-market (Quicken).

Concrete projects generally are in close touch with their customer, if only because the customer hired the developers on the project. Many companies outside the software industry undertake concrete projects to serve their business needs. Concrete projects often buy abstract products to solve non-domain problems: object persistence, multithreading, distributed transactions.

The two sorts of projects attract different sorts of people, and impose different process requirements. I wonder whether some of the XP friction has come from collisions between the abstract software and concrete software worldviews.

Contributors: BetsyHanesPerry, MichaelFeathers

I wonder as well. I come from the abstract side in that my missions have usually been operating systems, relational database systems, language compilers, general purpose software intended for making other software.

Such projects, when funded by someone of finite budget, can have great pressure to come to market. It is difficult for "marketing" to really know what is needed, and the "requirements" for such a product tend to become a laundry list of everything that any competitor has. The marketing people usually have the ExtremeRole of Customer.

Developers of such products think they have excellent vision of what the ultimate buyer will need, and are very inclined to put in capabilities or generality because someone will need it someday.

I suspect abstract projects are more likely to be commercial, in that much the huge bulk of internal projects are directed to concrete goals. What's below is built in part on this questionable assumption.

What are some dimensions where abstract/concrete or commercial/inhouse might differ? Here's a start:

Need to Predict: Commercial projects need to get marketing lined up synchronized with release. Will this increase pressure to hit a specific date, or predict one?
Time To Market: Abstract projects may be more likely to be undertaken for profit than concrete, in that there are lots of companies doing internal concrete work, and few doing abstract. Most of these latter ones are in the business of selling their work. Will this increase pressure to get done soon?

But you're forgetting the research community, who often do abstract work, not to mention the enormous open-source community.

Ah, you know my thoughts so well, to see what I'm forgetting. Are we perhaps talking about the deadline vs no-deadline distinction?

Documentation

Certainly commercial levels of documentation differ from inhouse. Abstract software may have more discrete aspects to discuss? Is there a different need for "internal" documentation? Or is it just that the abstract documentation happens to be the same as the internal?

Success Rates?

What else?

I'm not sure that commercial/inhouse is the same split as abstract/concrete. Quicken is commercial software, but I think it's concrete. Perl is definitely abstract software, but is non-commercial. And inhouse teams at large companies often develop quite abstract software, when they can't find a framework to solve their problems.

I'm sure they're not the same. That's why I pointed out that the distinction was questionable. Would you agree that most inhouse software is non-abstract? What are the dimensions of difference you would think most important?

See next rock.

Let's say that I'm working on a syntax-sensitive word processor because the version of Word for Windows CE really bites on a small screen. Is that abstract or concrete software? There's a researchy component, like what is the smallest set of syntax rules that still make editing considerably simpler than Word. It certainly isn't commercial, because no word processor will ever dethrone Word (until one does, of course). It doesn't have a schedule, because I only work on it a little at a time. But it's ready for its first release when I can write a book manuscript on it, so I know exactly when it is "done". --kb

In my book, the word processor is concrete. It's trying to accomplish a single task (editing a document), and you are expecting people to use it only for that task. On the other hand, if you finish the syntax rules, realize that they are generally useful, and start incorporating them into other programs, then the syntax rules become abstract. In my mind, the most important test that abstract software satisfies is that it is used in many different contexts to accomplish many different tasks. Does this make sense to you?

--bhp

Betsy, I think I understand what you are getting at, but it is hard to get hold of. Are you drawing the distinction between development that is very CS intensive as opposed to development that is very problem domain intensive or user centric? Is this Alistair's deep vs shallow knowledge? I'm reminded of sitcoms and movies about writers. They abound because writers know about writers. In the same way, perhaps, abstract projects abound because programmers know programmers? -- MichaelFeathers

Michael, I think your first sentence (CS intensive/domain intensive) is close, but not all abstract projects involve deep magic. You can write abstract software that isn't advancing the state of the art a jot. Let me try this another way. (Once I get a definition that works, I'll move it up top.)

When I was an egg, we made the distinction between systems programming and applications programming. Systems programmers wrote OSes, compilers, and other close-to-the-metal stuff, usually (pre-Unix) in machine language. Applications programmers wrote payroll software, statistics software, and database software, sometimes, but not always, in higher-level languages.

The distinction I am trying to make in AbstractVsConcrete? is derived from that old systems/applications line. Abstract programs don't stand alone; they are tools or frameworks that are used to build other programs. Concrete programs stand alone: somebody runs them to produce a desired result. Most "systems programs" were abstract; most "applications programs" were concrete.

For example, perl is an abstract program. A script written in Perl that serves CGI pages is a concrete program. TOPLink is an abstract program. A customer-service application that uses TOPLink to retrieve customer records is concrete.

The tension I am trying to capture is "Make it general; we don't know what its final use will be" versus "Make it work in the current case; if we have to, we'll make it general later." Abstract programs maximize generality; concrete programs maximize specificity.

One other feature of abstract software that hasn't been mentioned is that its interfaces have to be correct the first time out. Why? You will be embedding this software into others systems. You don't want to make them change their code at every release. This may mean that some refactorings might not be possible. I believe that this provides a bit of a problem to an XP approach, although it's also ameliorated by the XP use of doing no more than necessary -- FrankAdrian

Is this the same distinction that appears between "pure" and "applied" mathematics? They're both mathematics, they strongly overlap (Runge-Kutta methods, prime number theory), and 1+1=2 in both of them (when conventionally interpreted). But one has at its starting point some mathematical structure that is to be investigated solely for its own sake and the logical consequences are selected for follow-up are selected on their own aesthetic merits. The other has as its starting point some state that is intended to represent some aspect of the real world and what's followed up depends on whether the consequences are useful when interpreted as something that refers to the real world.

DesignPatterns are abstract. A Framework is concrete. See PatternsAndFrameworks.

CategoryComparisons