Domain Specific Programming

Many folk in the CommonLisp community claim that when you are programming in Lisp you are more designing a DomainSpecificProgramming language rather than writing a program. Doubtless they're right. But of course a Lisp program is just a bunch of procedures (and macros) that define and manipulate data structures. And so is a C++ program.

I've always understood one of the foundations of ObjectOrientedProgramming to be the construction of program entities whose behaviours are in some way analogous to those of the domain entities. So a C++ or Java program (a good one) also embodies a domain specific programming language, to some extent. So the question is, why does the Lisp community make this strong claim? What are the essential features of Lisp (CommonLisp or the other dialects) that result in Lisp programs being more strongly DSPs than those written in other languages? -- KeithBraithwaite

RealMacros are the essential feature that Lisp provides and languages such as C++ and Java are missing. And, since RealMacros can only conveniently be added to HomoiconicLanguages, the existing forms of C++ and Java are missing out. Without RealMacros, the syntax will never be domain specific, and the sublanguage will never feel domain specific.
I take issue with the statement that RealMacros can only be applied to a Homoiconic language. The problem is that most non-homoiconic languages have a BondageAndDiscipline mentality when it comes to macros. Consider that, pre-compilation, all text-based languages are a close-enough approximation of homoiconity (all the code is in the form of data(text), and thus can be edited as data). After all, what does homoiconicity get you? Procedural construction and inspection of the code itself. Well you can do that with any text - Lisp just lets you do it with lists (which, admittedly, is much, much easier since you can use the same constructs for both things). True Homoiconicity is only required for macros that must be evaluated at run-time - a very, very small set of problems. And in that case, it is still possible for a language to include an intermediate homoiconic level (for example, an oop-representation of all the statements, control-structures, etc. in a block) as in the "Weakly Homoiconic" languages. -- MartinZarate
I never said they can only be applied to HomoiconicLanguages. I said that they can only conveniently be added to HomoiconicLanguages. There is a difference. And Lisp is hardly the only HomoiconicLanguage, though it and its dialects (like scheme) are the only ones I know of that use a list-syntax for everything. I would say, however, that lacking some sort of procedural construction of the code, you don't have RealMacros -- you're missing some of that fundamental power -- e.g. the ability to investigate the insides of code prior to deciding what to do with it. However, it is possible to write other macro systems that will do that job even on text; it's just... messy? ugly? a PITA? those things readily come to mind. It's bad enough that nobody has made any heroic attempts to make them work elegantly in C++ and Java. Anyhow, Lisp doesn't evaluate macros at runtime; it evaluates them at compile-time. Without RealMacros capability, you can't really call the language good for DomainSpecificProgramming because what you end up having to do is write a whole different language, then /compile/ it with a different language processor to produce the stuff you can compile down to working code -- you end up with two separate languages, rather than one integrated language that happens to be good for writing DomainSpecificLanguages. And, yes, you're correct that there can be an 'intermediate homoiconic level'. If macros are allowed to introduce syntax (e.g. operators like '+'), then that would even be the 'natural' thing. However, such things don't exist in the standards of many languages (e.g. C++ and Java) and we don't have the power to make them exist.

In LISP you can add new statements to the language. Yes, strictly speaking they are just function definitions, but since the LISP language notation is functional, they are just like built-in statements.

In C++ you can add a function and call it. For C++ to be like LISP you'd have to be able to add new statements like for(;;) or if(). Since C++ only has a couple of statements, people hardly notice them. But a C++ program is a list of statements, many of which are function calls. A LISP program is a function call, consisting of other function calls, some of which define new functions, and others of which use them.

Basically there is no syntax in LISP. Just LotsofIrritatingSillyParentheses. --RonJeffries

So, taking the C/C++/Java style "for", it's the '{' '}' getting in the way. "for" looks almost like a function call, but not quite, and couldn't really be made into one because the block of statements following is just a block of statements, and not a closure. It seems like this couldn't be fixed without turning C into Lisp. -- KB

A similar, but different feeling arises in Smalltalk. There, everything is an object. Like LISP, there is essentially no syntax to the language (though there is a bit more than in LISP). Everything is a message send, including defining a class or method.

Then again, we could treat "for" as a method, if we took the thing between the '{' and '}' to be an object, but that would turn C in Smalltalk. -- KB

There's something odd that happens in your mind when your language is made up of all the same things, elephants all the way down. Jagged edges seem to go away and things become smoother. I'd explain it better ... but I can't. --rj

It's very much like the difference between Chess and Go. --

Well, I've heard similar claims made about ForthLanguage - writing a Forth program consists of defining words that you combine to solve your problem. RebolLanguage seems to make similar claims, but I have even less experience with that.

I'm not sure what makes these languages more amenable to this idea - it may just be that language and culture strongly favour composing programs from blocks. Of course, in the case of Lisp at least, the environment encourages the reuse of parts of previous programs. After a while, you'd expect a DSP to arise almost naturally out of the basic parts of different applications in the domain. -- BurkhardKloss

ElephantsAllTheWayDown?... that's good. Part of it is that in C or similar, part of your text is talking to the compiler, part is talking to the execution engine, part is fixed syntax that again talks to the compiler, the function calls are invented within the program. Syntactically, function calls are not linguistically interesting or natural.

In LISP and Smalltalk and Forth, the text only talks to the execution engine. Nothing talks to the compiler (well, the parens in LISP do, as do the [:;.] in Smalltalk. So the text all and only says what to Do. That helps a lot. Inventing new units of expression looks and feels just like using previously packaged units of expression. --AlistairCockburn

Right. The concerns of execution environment, compiler and problem domain are not separated, and the confusion is expressed in the tangled syntax of "imperative" style languages. So the Lisps (and in some similar way Smalltalk) separate those concerns, then automate and hide away as much of them as possible. I think I begin to understand. Thanks guys, very enlightening. -- KeithBraithwaite

It is a bit abstract for me. I need an example that shows the separation between execution environment, compiler and problem domain. I wonder how it looks in Lisp, Smalltalk and Java.

Do you know methods/tools designed to achieve this separation? I am sure Lisp 'compiler' does not give enough guidance? Can we gain this separation in Java/C++ using the right discipline?

It appears to be the other way around to me. It is in languages like C++ that the execution environment, compiler, and problem domain are separate. So you don't have to gain it. When working with C++, it is necessary to first work in the problem domain, by drawing up elaborate UML diagrams and design documents. This is because the finished source code usually bears no resemblance to an expression of a solution to the original problem. Every time you begin to express the solution, it immediately becomes apparent that you need to go out and declare some more temporary variables, define another class or two, and allocate some more memory before continuing.

Then, it's on to the compiler, writing the code and getting the compiler to accept that code. No matter what the problem, it must be forced to fit within the object-oriented paradigm. There's always a semicolon missing somewhere, or an illegal cast, or a missing mandatory cast, etc., and once you fix those, there are mysterious header compile errors and "undefined references" to functions you swore you compiled in. When you are working with the compiler, you are not thinking about the original problem domain, so it can be said that the problem domain is well separated from the compiler. In fact, the compiler has a problem domain of its own.

When the compiler is done, you are left with a file that contains binary instructions that are very roughly equivalent to what you wrote in the source file. This binary file is undeniably and totally separate from the compiler which produced it. It is this binary file that you work with in the execution environment. Working in the execution environment is very different from working in the other two areas, no matter how you go about it.

In Lisp, all these elements are combined, not separated: The compiler, linker, execution environment, editor, and debugger are all mixed together. At least some of the elements reside in the same binary image. You don't necessarily know when compilation is taking place (only some Lisps require an explicit command to invoke the compiler). All errors seem to occur in the same domain, not umpteen different domains. The same debugger comes up when there's a compile-time error as when there's a runtime error. In a Lisp IDE, programs are compiled incrementally, as they're being written, a little bit at a time. Even while the program is executing, it is still possible to go back, change things, recompile things, and have the results be fully integrated with the running program. It is possible to create a notation that means exactly what you want to say. In Lisp, everything sort of blurs together.