Too Advanced For Its Own Good

Sometimes elegant, sophisticated, and useful tools are rejected in favor of more primitive tools.

Here we list

some examples (Lisp, Smalltalk, ...)
some speculation about *why* each tool was rejected.

Lisp

The second programming language after FortranLanguage was Lisp. Amazingly, it had some features (in the 50s), such as Garbage Collection and first-class functions, that are only now making some ground in "real-world" languages like Java and C++. Why is not everyone using Lisp?

RichardGabriel considers this in some depth in his WorseIsBetter paper. However, I think he is missing something quite subtle. Lisp was invented as a special purpose language (for AI), just as FORTRAN was invented for numerical work. History has shown that specialist languages flourish better than generalist languages. Witness the AlgolLanguage. However, there has to be wide enough applicability for a specialist language to be used outside its domain. That applicability has to be compelling enough to overcome the limitations of that language inherent in its specialist origin.

I believe that Lisp was, in a sense, too advanced for its own good. Because it is, as it were, pre-optimized for AI applications, and most commercial software just doesn't need that kind of sophistication. Maybe, as the industry and our clients mature, this mismatch will go away and Gabriel's NextLisp will come to the fore. I'm very interested to see if the Scandinavians can do for FP with the ErlangLanguage what they've done for Objects. -- KeithBraithwaite

Lisp was not invented for AI. It began as a way to encode (partially) some fundamental ideas about computation, i.e. it was informed by (not derived from) the lambda calculus. It quickly became the *LanguageOfChoice* for AI, because the people in AI were tackling much harder and/or more poorly specified problems than most programmers, and wanted the sort of flexibility and abstraction power only available in lisp. I do think the association with AI hurt lisp in the AiWinter, and in the ensuing years some languages have narrowed the gap in terms of expressive power. None of the mainstream ones are really that close yet, but the decision (to do AI in lisp) is not as clear cut as it was in the 60s, certainly. Lisp is *certainly* not pre-optimized for AI. It is, however, in some sense optimized for hard problems.

Lisp may be a great language for your own projects, but I am afraid of the odd abstractions that *other* developers would or could invent that would take a while to figure out. There may be no way to reign in MentalMasturbation. More primitive languages tend to result in more consistency, even if it is a bit repititous. Like intermittent-problem car repair, the most difficult part is often finding the problem, not changing the code (parts) itself. I've overdone abstractions in the past (not in Lisp) that confused multiple developers. A language that offers more creativity may even result in more of that. When working in teams, sometimes I sacrifice duplication factoring in order to obtain clarity now. I hate to say it, but removing all known duplication is overrated sometimes. This is one reason why Soviet economics didn't work very well. -- top

I've always felt that blaming Lisp's failure on being "too expressive" or some such story has always been a cop-out. Let's be honest here - Lisp is much harder to learn than other languages. Every other language has made attempts to use familiar syntax - either attempting to look like English (COBOL, SQL) or like high-school math (C, basic, etc.). Lisp only looks like Lisp, unless you know LambdaCalculus (which most don't). The only exception is Perl, but Perl looked familiar to Unix geeks that had already invested mentally in Sed, Grep, etc. Regardless of its power, expressiveness, and sensibility for implementing constructs that are not natively in other, more syntactically-complex languages, it has no curb appeal. To the untrained eye, Lisp looks as legible as bytecode. Other languages are at least minimally legible to someone with a highschool education in math. Not to mention its stubborn refusal to join the VowelGeneration.

SmallTalk's lack of popularity, on the other hand, can simply be explained, historically, by price, and currently by bad marketing. Most newbies simply know of SmallTalk as the ideological predecessor to JavaLanguage - they think of it the same way they think of Cobol. Neither of these languages failed for being "too expressive" or "too obfuscated". Perl, C Macros, and scripted Excel Spreadsheets are used more frequently and produce maintenance horrors that completely outdo clever Lisp macros... but they're all conceptually familiar concepts to the users who learn to use them. -- MartinZarate

It is interesting that this particular misconception persists (and in fact may well be one of the reasons for Lisp's lack of popularity). Lisp is not "optimized for AI",

Not even "as it were pre-optimized", just a little bit, and I mean McCarthy's original, not CommonLisp?

and is not a "specialized language".

Are you saying that it wasn't "invented as a special purpose language"?

Quite the contrary, Lisp is far more general than, say, C, which is a "specialized language" - specialized to program a Von Neumann style of program. Lisp is about expressing computations. Expressing them in Church style, surely?

I believe (or is it just a vain hope?) that up to now, most sofware was sufficiently simple that a straightforward, inexpressive language such as C was sufficient to write most programs, thus C (and its related languages) have been wildly successful. I also believe that the critical point is coming where we want so much from software that an entirely more powerful level of abstraction will be required, and such cannot be supported by C et al. Lisp (or a language implementing most of its great ideas) will then move from being TooAdvancedForItsOwnGood to being WhatWeWantedAllAlong but didn't know we had.

But probably not in my programming lifetime... sigh. -- AlainPicard

Which is what my "most commercial software just doesn't need that kind of sophistication. Maybe, as the industry and our clients mature, this mismatch will go away" was trying to express.

Where we differ, I suppose, is that I don't really feel that there is any reason to be concerned that the industry's problems have (mostly, up to now) been soluble with less sophisticated tools. After all, the industry started out collating census returns and automating commercial arithmetic. -- KeithBraithwaite

What went wrong with Lisp?

Garbage collection was awesomely slow. Yes, it would have been slow in any language, and it's no longer slow now, but to start with, it was indeed slow. Very expensive hardware was pushed to sell its implementations, which never worked well on commodity hardware. Common Lisp became a design-by-committee monstrosity with identifiers more bureacratic and verbose than COBOL. And nobody implemented an operating system with the system interface in lisp except companies interested in selling the aforementioned super-expensive hardware.

To this day, you cannot get a decent free implementation of lisp on Windows.

Smalltalk

Another nice language is Smalltalk. It had many of the ideas in Lisp, and added some of its own, such as pervasive object-orientedness and an integrated development environment. Only now are we seeing some of these ideas becoming mainstream with Java, and then only in a quite diluted way.

What went wrong with Smalltalk? Too advanced for its own good?

See JavaBandwagon, LanguagePissingMatch, and IfSmalltalkIsSoGoodWhyDoesNobodyUseIt.

other things too advanced for their own good

???

I remember using a word processor that would "correct" words as soon as I typed them in. This was incredibly annoying when I needed to use words that weren't in its dictionary.
Does BetamaxSituation fit here?
Motorola MPx - see http://www.brighthand.com/article/Motorola_MPx_First_Impressions