Extensible Programming Language

The phrase 'ExtensibleProgrammingLanguage' has been around since the 70s, maybe even the 60s. At that time it referred in general to languages with extensible syntax or semantics; one of the big notions was that of a modular compiler that could be controlled through the language script itself.

While the original movement died a choking death, the modern movement towards ExtensibleProgrammingLanguages is generally called 'LanguageOrientedProgramming' which involves the embedding of DomainSpecificLanguages or DomainSpecificTweaks within a host language (but see StumblingBlocksForDomainSpecificLanguages). There is, of course, a large focus on both the ability to manipulate syntax and on performing MetaProgramming (in order to avoid AutomatedCodeGeneration provided by utilities outside the language). If you're interested see RealMacros (mostly seen in Lisp, Forth, and Scheme; see CompileTimeResolution and PartialEvaluation), and HomoiconicLanguage. Syntax extensions can have various forms. Some languages (e.g. Algol 68) allow user defined operator symbols with priority and associativity. In other languages (e.g. Seed7 (http://seed7.sourceforge.net)) a pattern with keywords and places for parameters is used to define the syntax of (imperative language) statements. Syntax extensions can be done purely statically, but it also isn't difficult to include flexible parser support in the runtime to allow for runtime evaluations (e.g. on an ReadEvalPrintLoop, or for creation of an 'eval()' function).

Beyond syntax and language manipulation, there is also the MetaObjectProtocol and handling classes as first-class objects in order to allow runtime manipulation and specialization of language interactions. Ruby and Smalltalk are especially well know for manipulating their own language via the MetaObjectProtocol.

The notion of IntentionalProgramming seems to be a class of ideas towards implementing, representing as 'source', and living with ExtensibleProgrammingLanguages in general.

Tragedy of Extensible Programming

Extensible programming is all about adapting the language to the problem. This seems to work really nicely at the scale of individual projects and applications. However, the language matures. We see more applications, and common bits get refactored into libraries. Five or ten years later, we have a lot of libraries, and we start thinking it wiser to adapt our problems to fit the existing libraries. Problem is, the ExtensibleProgrammingLanguage wasn't designed for that. Libraries from different projects don't fit together very well because they depend upon independently developed, redundant, and subtly incompatible extensions to the language - e.g. different assumptions about concurrency, error handling, dispatch, memory management, reactive updates, side-effects. So you end up spending much code adapting or reinventing old libraries, and peeking inside library code in order to understand what is happening under-the-hood.

Alternatively, we could design a language that focuses on composition rather than extension. Such a language would be less flexible, and seems somewhat painful to start - the language is never a perfect fit for the problems, and chunks of our applications are less direct than we'd prefer. But five or ten years later, we have many libraries, and they all follow the same basic rules, and the interface is sufficient to determine composition.

Of course, most languages are designed for neither extension nor composition.

LanguageDesign is always quite subtle. The difference between lazy vs. strict evaluation is a few lines of code in the interpreter, but huge from a programmer's POV. The benefits of ExtensibleProgrammingLanguage seem obvious in toy examples, but falter for mature languages. There are approaches (macros, for example) to obtain some of the benefits of extension, but cross-cutting changes or code-walking macros are going to hurt composition.

XMF is an industry strength ExtensibleProgrammingLanguage designed for LanguageOrientedProgramming. All aspects of XMF can be easily extended or redefined at run-time allowing the dynamic construction of domain specific languages (DSLs) which can be used either standalone or weaved into existing DSLs. XMF is also a mature full-featured stand alone language that has been used in industry for modeling and data intelligence applications. Take a look at the site (http://www.ceteva.com/xmf.html).

(Moved from CapArray.)

Context: One of Python's better known philosophies - There should be one, and preferably only one, obvious way to do it. (And the corollary: that way should be correct.)

[... I'm] seeing hypocrisy - meta programming and languages that can be extended beyond original form is unconstrained. When decisions are offered to the programmer such as being able to choose slightly different language syntax (due to extension capabilities), it makes us wonder whether these are really useful features of a language or whether they are too unconstrained.

I've said nothing hypocritical. I expressed above the belief that power is important - the power to make decisions, or to create the modules you wish, or to create constructs and syntaxes exactly on par with the language's own, is extremely valuable. But that doesn't mean it would be wise to create fourteen versions of CapArrays each with slightly different dependencies and algorithmic performance. Having the power to do so if I wanted to do so? Excellent. Actually doing so? Foolish. There is a huge difference between "can" and "should".

The ability to extend syntax in particular "can" be applied to create 'Java in XMF' and 'Lisp in XMF' and 'C in XMF'. One might even do 'Brainfuck in XMF' or 'Intercal in XMF'. And doing these things would be a fine exercise or convincing proof of the syntax extension system. But there isn't any good reason to do so - there is almost never a good reason to write a general-purpose language inside a general-purpose language. The ability to extend the syntax is intended primarily for specializing it: embedding DomainSpecificLanguages (and DomainSpecificTweaks) that make it easy to describe business rules, database schema, scene graphs, video-game map and event-systems, etc. with access to the type system and without the continuous overhead that would be involved with a general-purpose language.

Niklaus Wirth warned people that extensive language syntax capabilities means that it can reduce the chances of people easily sharing and reading each others' code. In a module system, adding a new algorithm and structure with the same language in place means people do not have to relearn a new language - they just have to swap in a module. In some cases, extension to the language is nice - my point is there may be some signs of hypocrisy here - if in python everything should be done a certain way, then what will XMF do?

I do expect some XMF programmers to go wild and have fun creating arbitrary new syntaxes and customizing it either for themselves or simply to learn more about language design. Such things would not make it into any 'standard' library of modules (though I imagine a WikiIde won't be so easily controlled - thousands of programmers doing their own thing and proudly stealing/sharing services and code will be a very interesting experiment).

Agreed, I still have bad dreams about what could happen in a Wiki Ide. I once had a Lisp dream about people editing code in a wiki ide, for some reason.. The security/vandalism implications are to be studied still more yet. Odd GrammarVandals? and code vandals and other loony bins might need to be constantly monitored too.

Another issue in extensible languages is OperatorOverloading.. it is powerful, extensible.. but also can be abused. Ultimately, it seems about compromises.

OperatorOverloading is a special case of polymorphism. I'll agree it is powerful. And 'power' always can be abused, but any abuse that can be inflicted with polymorphism alone pales in comparison to what is possible by freely manipulating syntax, so I wouldn't really concern myself with it as a special case. What is important is that security concerns aren't readily violated by syntax manipulation, which, while 'twould be a concern if one didn't have the ability to bind to 'approved' and checkpointed code, isn't a significant concern even on WikiIde.

AprilZeroEight