Stumbling Blocks For Domain Specific Languages

[moved from LanguageOrientedProgramming, edited slightly]

As for whether LanguageOrientedProgramming (and DomainSpecificLanguages and MetaProgramming in general) are a good idea, I see two big stumbling blocks:

The first is cultural. Programmers don't want to learn new languages or new paradigms. Lisp has provided the ability to create DSLs since the 1960s, and the arguments against it are never "because it only compiles to Lisp" and only rarely "because it doesn't let you define new syntax" (which is false, BTW. The semantics of Lisp are described in terms of nested lists, and it's not a huge deal to write a shift/reduce parser that outputs Lisp program fragments, which could then be macroexpanded/compiled/evaluated like any Lisp program. The reason it hasn't happened is that most Lisp programmers like EssExpressions and don't really want to switch to infix syntax. A more restricted form of adaptable syntax, ReaderMacro?s, is part of the CommonLisp standard and is fairly widely used.)

Instead, programmers usually argue that Lisp gives the programmer too much freedom. They worry that someone else will create a language that they'll then have to learn in order to maintain the code. Take a look at WhyWeHateLisp and SocialProblemsOfLisp. People argue that with Java and C++, then at least the control structures will be familiar, even if they have to learn huge class libraries.

It may be simpler to learn a DSL than an equivalent class library, but many people would prefer learning the class library. Class libraries give you something familiar (paradigm + basic language structures) to hold on to, while a DSL is likely completely new. That perception - that a class library is learnable while a new language is a big risk - is a big stumbling block to LanguageOrientedProgramming.

The second problem is technical. Defining a DSL is basically an exercise in compiler writing. There's a certain inherent difficulty to writing compilers which you can't paper over with editors and other DSLs. I really doubt that you can make it simpler than LispMacros. Lisp takes away all the syntax issues, gives you raw program structures in terms of forms and symbols, and then gives you a well-defined target language (Lisp) that already has a substantial number of features. Despite this, writing good Lisp macros is considered an exercise for the advanced programmer.

Now imagine that instead of compiling to high-level Lisp, which may already have been extended with macros, you want to compile to Java or machine code. Do you really think the average corporate programmer, who has enough trouble just writing Java, is going to be able to compile to it? According to the paper, "Language Oriented Programming: The Next Programming Paradigm

http://www.onboard.jetbrains.com/articles/04/10/lop/

the big timesink in ordinary programming is mapping your conceptual model of the problem into the implementation model of the target language. When you construct a DSL, you not only have to map your conceptual model of the problem onto the implementation language, but you have to formalize that map sufficiently to let the computer automatically transform between the two of them. The paper seemed to suggest that the transformation step is fairly simple with the help of editors and LOP DSLs, but as anyone who's taken a compiler-design course can tell you, there's a lot of conceptual trickiness that you need to deal with regardless of the tools.

I think LanguageOrientedProgramming is a good idea, but I really think one should begin with Lisp or ML and then try to extend that rather than reinventing the wheel from (ick) Java. Java was not designed for LOP; in fact, it took out most of the available LOP features in C++ (the preprocessor, OperatorOverloading, destructors, templates, etc.) It's probably necessary to provide Java bindings and extensions for marketing needs, but if you try designing based on what Java gives you, or even what IntellijIdea provides on top of Java, you'll be way behind where the Lisp folks were in the 1970s. -- JonathanTang

I think the paper being referred to is the one at the top of LanguageOrientedProgramming as this material was moved from that page some time ago. -- JohnFletcher

One might say that comprehension of a language is measured by knowing the syntactic constructs and how they translate to semantic components, the ability to predict evaluation orders, etc.

DSLs, syntax extensions, and macros violate this expertise, allowing for strange new syntax with strange new evaluation orders and thus introducing a learning burden on later maintainers. This is not an insignificant price to pay for the Meta features, so it must be weighed carefully... and avoided if a feasible solution (with acceptable characteristics) is readily available in the host language.

The problem of MetaProgramming proliferation can very likely be heavily curbed by use of community and convention. E.g. One can require by convention that the MetaProgramming proponent obtain from a few other programmers (that have stake in the code) endorsement for each new macro, syntax extension, DSL, use of reflection or MetaObjectProtocol, and even creation of a new framework (frameworks share many of the negative properties of DSLs). This should raise the bar just far enough to prevent the useless, unintuitive, and obnoxious uses of MetaProgramming, and would also help ensure discussion towards other possibilities the proponent might not have seen.

[Moved from MetaProgramming]

In addition, there are other reasons why writing a good DomainSpecificLanguage is difficult.

Debuggers and other tools will likely be non-existent. In most cases, you will have the target language's toolset to fall back on, but you'll be debugging the generated code, and not the meta-language code that the developer wrote (and thus exposing the application developers to the junk that the meta-language was intended to hide them from). If a runtime interpreter is written--the debugger will be useful in debugging the interpreter, but almost useless in debugging the program in the meta-language.
- This can be mitigated a great deal if the same mechanism is used to define and extend the target language syntax as is used to describe the DSL, allowing debugger information to properly be traced to the original source. The existence of a 'target language' (as opposed to an AbstractSyntaxTree) would be less relevant in debugging a LanguageOrientedProgramming system - one designed for the task of defining and utilizing DSLs.
- Note that this mitigation does assume something more of EmbeddedDomainSpecificLanguage or 'domain specific extensions' represented by a formal mechanism within the language. Lack of debuggers, syntax highlighting, etc. certainly are problems that will plague any stand-alone DSLs. With most modern IDEs, it will also be a problem even for Embedded DSLs, but the problem is far more readily solvable by having an IDE aware of the semantics and syntax for describing the EDSL within the host language. It seems DrScheme annotates reader macros with original source information for this purpose, though I'm not certain of the exact mechanism. I'm certain the Eclipse support for XMF (described in ExtensibleProgrammingLanguage) also possesses similar features.
The language will be proprietary. No outside expertise will be available. Documentation for the DomainSpecificLanguage, if any, will likely be sparse. Much of the knowledge of the system will be folklore.
- Fortunately, this is also paired with the domain itself possessing limited expertise. It is counter-intuitive that this would be beneficial, but the real benefit is that a well designed DSL can help teach the necessary aspects of the domain to its users via examples, perhaps more intuitively than could observing usage of a domain-specific library with all the semantic noise associated with integrating domain-specific tasks into a GeneralPurposeProgrammingLanguage. Thinking about it another way: when you have cause for a DSL, you'll need to solve it somehow.. and no matter what approach you take will be 'proprietary', be short on documentation (at least from most programmers), and experts in comprehending the API and how to use it will be sorely lacking. A DSL at least has some potential of reducing noise and artifacts introduced by the implementation language and making more obvious the intent of the program. Even better, if the domain-experts aren't programming-experts, they'll likely be able to utilize a DSL more readily than they could deal with the full power of the language.
When bugs do crop up (they will); there are three sources to consider: the parser, the translator/interpreter/compiler, and the the program written in the DomainSpecificLanguage. With native code, worrying about compiler bugs is usually a concern of last resort.
- This is true, though a bit optimistic. In native code you'll end up with library-bugs in the domain-specific library implementation, which has its equivalent in those compiler-bugs in the DSL translator, so you only cut it down to one extra source of errors (the parser). If the DSL happens to be for easier data specification, you likely won't be avoiding a parser in any case (though this claim is a bit language dependent; some languages don't force you to describe complex data procedurally, so embedding data isn't always a problem).
One common theme that must be watched for in any case--is the problem of senior programmers engaging in MetaProgramming with the purpose of protecting the junior programmers from themselves--i.e. "We can't let those guys code in C++! They'll create all sorts of wild pointers! Let's give them a (mostly undocumented, proprietary, ad-hoc, custom, and without debugging tools of any sort) new BondageAndDisciplineLanguage of our own superior design, to keep the junior staff from tripping over their own feet!" If the programming staff can't competently program in the selected language, then you need to either a) train the programmers, b) replace the programmers, or c) replace the language. If they aren't trainable to an industry-standard language with lots of teaching aids available; they likely won't be trainable to an ad-hoc language with none whatsoever. No matter how simple-minded the system architect who wrote it thinks it might be.
- Having greater power from any source, including MetaProgramming, introduces the possibility for greater abuse. One should also fear the junior puppy programmers who get excited at the idea of MetaProgramming and start peeing over everything (ooh! ooh! I can write a Japanese version of Java inside XMF! and I can ride my bike with no handlebars, no handlebars, no handlebars...). This sort of problem is best met by community and cultural development. Powerful meta-facilities are excellent for teaching programmers about language design, and are equally useful for language-designers, but actual application needs to be limited to those places that offer the greatest benefit.

That said, if there is a significant amount of domain knowledge that can be expressed much more clearly in a meta-language than in a domain-independent language (including libraries written using the native facilities of that language), or if you need to let users customize stuff (without requiring them to learn a complete programming language) then MetaProgramming might be a good idea.

Regarding protecting the junior programmers from themselves:

In a large project you need both leaders and followers. Followers will carry out 50-80% of the project code. In such a case, you will appreciate order and punctuality over creativity. Please remember, it is not easy to manage too many creative people. Not to mention our salaries, and the time it takes to find us.

The followers do not care about changing the system architecture. They want to be productive. They will be more productive in a secure, well defined and measured environment. They need guidance and they will bless anyone who gives them more productivity for less programming power.

Personally, I think these 'followers' should be 'technical followers' and the leaders in domain expertise. -- ShaiBenYehuda.

Writing DSLs is not the task of junior programmers. Criticizing DSL-building tools in the hands of juniors is wrong-headed, IMO. Typically it's the technical lead(s) who define a new object language that the juniors use to express solutions to domain problems. This works well with pyramid team structures since the few experts at the apex provide the small, highly-factored object languages that the many juniors at the base use to describe the program.

When it comes to DSLs, the technical leads define the languages that the juniors use to implement the user stories. -- BryanEdds

In my opinion, one should try domain-specific (DS) API's first. That way you don't have to reinvent a language and compiler/interpreter from scratch. However, some existing languages don't seem up to this tasks. For example, named optional parameters are very handy for DS API's in my opinion. But many languages have awkward or missing support for this. Use of OOP attributes as a substitute can sometimes be too verbose (although having the option of both syntax techniques, or equivalent, based on specific instances is handy.) -t

CategoryMetaprogramming