Multiple Independent Languages

Imagine if we could embed all code (Java, Perl, SQL, config data etc) in freeform text pages [...]

Text initially drawn from ProgrammingInWiki; some specific mentions of WikiIde from context.

Q: Could MarkupLanguageNine be used for this?

A: Probably, though MarkupLanguageNine possesses a strong ErlangLanguage bias when it comes to values descriptions (in the metadata), and yet lacks strong referential semantics between meta-data components or from the meta-data into the source (creating the barrier discussed below). Something can be provided by the interpreter module, but making it meaningfully common across different interpreter modules would be a chore. Fortunately for ML9 (and, I'm sure, considered at great length in its design), the vast majority of languages don't reserve '@' in column 1 for anything - even those that do leverage the '@' symbol can place it in column 2 without problem. From what I've read, it wouldn't be a bad option as far as options go. That said, we'd need to extend it to support anchors and linking efforts, dependency specifications, etc. and that semantics barrier (especially across Java/Perl/SQL/config data/etc.) is something that needs looking at, as would be providing IDE support that crosses so many different languages in a single wiki (e.g. syntax highlighting, debugging, etc.)

ML9 also lacks, from what I can tell, the ability to nest tags. Maybe I'm missing something? --SamuelFalvo??

ML9 provides full Erlang-value support for the meta-data associated with each tag. (@tag {name1=value1,name2=value2,...}). that is, every except the 'data' meta-value, which must be a flat string. This allows for arbitrarily complex data structures via use of tuples and/or lists. The 'data' meta-value is generally loaded from the text between the tags (though it may be loaded explicitly - (@tag {data="a flat string"})) and must be a flat string because it could be a description in any language (including text commentary) and even when describing Erlang code, this text is NOT describing an Erlang value). The author advertises this as being 'completely the opposite of XML' since in XML the attributes are flat while the 'data' section between tags has structure - and I have to agree that, regarding the purpose for which ML9 is intended, this is a good thing.

Multi-Language + Coordinator vs. Single Extensible Language with integrated InternalDomainSpecificLanguages?

The scenario above ("imagine if we could embed all code (Java, Perl, SQL, config data etc) in freeform text pages [...]") favors implicitly the use of multiple languages. Ignoring consideration of whether ML9 could accomplish it for a moment (though I don't doubt it could - ML9 was designed for exactly this purpose), the question of whether using multiple independent languages is a GoodThing deserves some extremely critical consideration of its own. Further, the use of a 'coordinator' language, be it ML9 or not, also deserves scrutiny. My own conclusion is that the decision to coordinate multiple independent languages would be a BadThing (for a number of reasons), but that the extensible single-language solution can (if certain provisions are made) be quite elegant yet still provide the vast majority of the power you'd be looking for with multiple languages (and possibly even more). But I'll attempt to present the evidence for you to make your own judgement.

There are five implications in coordinating multiple independent languages.

First, the set of tools (or plugins) you require for the coordination grows significantly. If we were to have a single 'WikiIde', it would need to support syntax highlighting, compilation, debugging, code-stepping, error reports, etc. for each of the languages. Even if the tools were perfect and possessed no errors, it will be a challenge to coordinate all of them and integrate them usefully with the WikiIde. This challenge grows further if there are multiple versions of languages in use.

Second, coordinating and, especially, integrating parts written in different languages becomes very difficult. You can't just write an algorithm in one language, reference that algorithm with a a chunk of code in another language, and use it. Instead, the algorithm is written in some independent language (LanguageA) and if you're writing in LanguageB, you must have not only a ForeignFunctionInterface to LanguageA, but the ability to reference the compiled object formed from the chunk of code containing TheAlgorithmInLanguageA in order to access it (which results in non-trivial naming and reference problems - to where does TheAlgorithmInLanguageA get compiled? should we use a WikiWord to reference it? or some semantic UniformResourceIdentifier understood by the underlying OS/filesystem and injected via the coordinator language?) Lazy programmers will violate OnceAndOnlyOnce simply to implement algorithms in each language.

Third, a coordinator language that also has the task of marking up and formatting text for display must ultimately integrate with the code-descriptions whilst not interfering with the processor of the independent language. This can be provided via some common interpreter-module of the coordinator language, of course, but it is worth considering the direction being taken: just one step beyond providing markup annotations for display of code is starting to provide macro annotations and functions over the code itself. After all, any coordinator language is essentially one massive language preprocessor. One could allow macro-support even in languages that don't support it, using WikiWord macros with parameters as a sort of HyperText MetaProgramming in languages that don't support it naturally. (Even ML9 - @inject {macro="MyWikiWordMacro?", param1="embedded-code",param2="more-embedded-code"} - would, with a little switch to the preprocessor module, allow for macros.} However, this lands us one step shy of creating a 'single extensible language', but without access to the semantic context or environment coordination provided by a single language.

Fourth, consider a goal of writing the various language-tools within the WikiIde. This could be useful for a number of reasons - among them, tools are unlikely to be perfect and ARE likely to evolve over time, so the ability to debug them (or reference them by version) would be useful. In addition, one might wish to reference tools with WikiWords to provide for easier coordination (as opposed to, for example, strings to be identified by a shell). WikiWords could either reference pages that contain code for the tools, or contain code for accessing the tools. E.g. the compiler for Java could be written on a page called 'TheJavaCompiler?'. But if the tools themselves are to be written in the WikiIde, we end up with two basic choices: (a) either part of the tool is written in a coordinator language, or (b) the tool is written wholly in independent languages coordinated by the coordinator language (possibly itself - TheJavaCompiler? written in Java). The latter results in a bootstrap problem (since every independent language is written in itself or another independent language that must be bootstrapped) - and such a bootstrap problem must be somehow resolved by the coordinator language (or in yet another independent language). The former requires a TuringComplete coordinator language. Either way, one ends up just one step shy of diving into the 'single extensible language' (since one is self-extending via WikiWords referencing languages compiled into the Wiki), except that we went about getting there in the least intuitive and least efficient way possible - instead of automatically gaining the syntax highlighting and debugging and compilation of every sub-language, we must write specialized debuggers and highlighters for every independent language we add.

Fifth, and in general, there are simply a ton of barriers between languages (including the coordinator language) creating all sorts of AccidentalDifficulty and adding to the PainOfProgramming. There are referential and namespace barriers: one can't simply reference an algorithm written in another language, or even written in the same language on another part of the same WikiIde - not without extending the language and its tools to support WikiWords (which results in a specialized form of the vaunted 'independent' language). There are semantic barriers: one can't expect types, representations of values, objects, etc. to have the same meaning across languages, nor can one expect the coordinator language to have any 'real' idea of what is going on beneath, and a refactoring browser would also require significant per-language plugins. There are communications and debugging barriers; any debugger tool or plugin that must cross a ForeignFunctionInterface into another language must somehow make this transparent to the WikiIde and handoff the debugger to the other specialized debugger plugin, which must support the virtual language-runtime-environment created by the original language (which may itself be running in a virtual language-runtime-environment from yet another 'original' language); also, there is communications in the cross-process sense, but that tends to be easier to handle (since the WikiIde can be aware of both the message source and destination runtime environments). And then there is the lack of cross-language optimization and code-verification; barriers are bad things where automated optimizers and provers are concerned.

Even if we limited everything to one single language plus coordinator, the Wiki coordinator language itself becomes a layered preprocessor for that language with semantics barriers and referential barriers reaching into the core language.

If I were ProgrammingInWiki, what I'd want is a single, common language in which all code AND all commentary and annotations are written. Obviously this imposes some requirements on this language; among them is required support for markup annotations that affect only the display of code and commentary, not the meaning. Another is the need for configuration-builder support - i.e. the linker and set of available services needs to be controlled from within the language. In addition, on a per-configuration basis it would be useful to control all sorts of aspecting (to allow for aspect-oriented programming).

The main benefit is that the WikiIde suddenly gets a single, common processor, a common reference space, common syntax highlighting, the ability to reference any piece of code or commentary from any other piece of code or commentary (especially with anchors), a single debugger, code-stepper, compiler, etc., a single bootstrap, OS independence (no need to modify or intercept calls to the OS to support injected referential URIs back into the Wiki when referencing some pieces of code from others; no need for a panoply of shell tools to support 'makefile' configuration specs, etc.), the ability to reference any algorithm on the WikiIde and know it will work in your piece of code, and the ability to do so without naming a language (as opposed to needing to reference TheAlgorithmInLanguageX, or needing to use complicated references through obtuse ForeignFunctionInterfaces??, or need to perform communications to other processes through pipes or sockets, etc. - all with combined comms, security, type-safety, and optimization costs.)

But this doesn't exclude the possibility for getting your java/erlang/smalltalk/perl/SQL/Prolog/makefile/pseudo-code syntax where you want it. One could use a language with extensible syntax.

CategoryProgrammingLanguage