Framework Is Language

This is a position statement that frameworks are (in all essential manners) languages. This is intentionally a weaker claim than ApiIsLanguage. See ApiIsLanguage for context.

RE: "You don't have to learn new syntax or idioms to use an API. APIs aren't language. They're vocabulary."

I've heard this opinion expressed by many different people, but, in general, I disagree.

When the designer of an API has expectations or requirements on how calls to an API are ordered and otherwise synchronized, then the API is not merely a vocabulary; it is a FrameWork that embodies various protocols and imposes behavioral requirements on the user... sometimes these behavioral requirements are fairly shallow (create comes before delete, don't delete the same thing twice, query before you expect a response, etc.), and sometimes they are very pervasive (special protocols for accessing shared resources to avoid DeadLock, communications context for security or compression or DeltaIsolation, contracts with preconditions and postconditions and invariants, temporal and spatial demands to achieve realtime performance or network quality assurance on embedded systems, and so on), but they always exist.

The degree to which an API can be considered a dedicated language is really the degree to which these behavioral concerns are imposed upon the user of the API. Some of these behavioral concerns are derived of EssentialDifficulty - a necessary evil in order to achieve properties of the protocol that are not provided directly by the language. Some of the behavioral concerns are derived of AccidentalDifficulty - an unnecessary evil that exists wherever the API is dodging around or accommodating limitations of the language, or leaking implementation details.

Examples of EssentialDifficulty include the need to construct a shared key (e.g. DiffieHellman?) to participate in secure communications with an unknown remote system, or the need to publish data to a named topic before it can be distributed, or the need to register a service before others can find it. Examples of AccidentalDifficulty include working around the sequential programming model of a language to add concurrency and other synchronization patterns, taking extra steps to ensure an object won't be accessed in one part of the code before destroying it in another, demands on code composition to support PrevalenceLayer, requiring the programmers carefully ensure that the call stack leading to a particular API call not be involved in certain mutexes for risk of deadlock, sandboxing of code to eliminate as much ambient authority as possible before running scripts and such provided remotely, BoilerPlateCode to 'initialize' a library (as opposed to individual uses of a protocol), and so on. AccidentalDifficulty often requires violating OnceAndOnlyOnce because you need to implement something 'once' in a manner suitable for ABC and 'once' in an incompatible manner suitable for XYZ and so on. Most complexity we experience today is still AccidentalDifficulty.

This AccidentalDifficulty is very problematic. Protocols, services, and contracts are difficult enough to compose based purely on EssentialDifficulty. When AccidentalDifficulty born of various MissingLanguageFeature?s is added to the mix, well, there is a reason FrameworksConsideredHarmful: PrimitivesAndMeansOfComposition is given one very harsh kick to the nads then left crying and whimpering on the floor. So long as you only need the one framework, you'll do well enough, but the framework as a whole is SecondClass: not abstract, not composable, full of gotchas and idioms that you and any MaintenanceProgrammer will need to learn and understand, very little obvious from the API itself, and effectively needing its own FooOneOhOneInSevenDaysForDummiesInaNutshellSuperBibleUnleashed.

It is in these more extreme cases, where the behavior imposed on the user of the API is pervasive and frustrating, that ApiIsLanguage is clear and obvious. So what's on the other end of that spectrum? Well, at a guess I would say that the other side is where the API feels like a straightforward extension to the language in which it was written, where each element of the API feels and acts and perhaps even looks (syntactically) like a simple or primitive extension to the language. SymmetryOfLanguage isn't (further) violated: the result is just as FirstClass, abstract, and composable as everything else in the language (which, for some languages, might not be a very impressive claim). Users don't need to know anything about the API that they wouldn't also need to know about other language primitives. Etc.

I suspect that, when dealing with APIs at the FrameWork and pervasive DesignPatterns end of the API spectrum, use of TypefulProgramming, ObjectCapabilityModel, RealMacros, and other MetaProgramming mechanisms to guide proper expression of the API is highly justifiable. I don't have numbers, but I suspect both original developers and MaintenanceProgrammers will have an easier time learning, reading, writing, and verifying the resulting 'language' after such extension efforts. Further, with the TypefulProgramming approach, or with FirstClass RealMacros (as supported in Scheme), the IDE and compiler can get in on the action by detecting errors statically, providing syntax highlighting for the extended language, allowing one to drill down into expanded views of macros when debugging, etc. (DrScheme provides these mechanisms.)

Still, I do understand the folks who resist such efforts, who want to believe "You don't have to learn new syntax or idioms to use an API. APIs aren't language, they're vocabulary". It would be nice to believe that, no matter what the problem, it can be solved via an API that tosses a few new vocabulary words at it without any unusual requirements being imposed on the user.

If we ever have a language where 100% of frameworks and protocols and unanticipated DomainValues and entirely new DomainModels and ad-hoc MetaModels with their associated inference engines and MultiValuedLogics can all be expressed without any unusual behavioral impositions or gotchas relative to the rest of the language - and we achieve this without LanguageIdiomClutter and with acceptable EconomyOfExpression (such that it isn't "usual" to jump through hoops) - then maybe people will stop writing APIs that impose behavioral requirements on the user.

But it isn't possible. Even if we captured all the AccidentalDifficulty imposed by existing language designs for all known problems, there would still be both EssentialDifficulty of known problems and AccidentalDifficulty of the unknown problems. The theoretical best any LanguageDesigner with intentions towards MinimalDesign can do is eliminate individual classes of difficulty (one at a time, in an effort with rising combinatorial complexity) then provide the language with a clear path for upgrade while supporting forward compatibility of existing programs. Useful ExtensibleProgrammingLanguage mechanisms include: some form of extensible syntax, ability to annotate code (HotComments) and modify the PostProcessor? to operate over them, support for TypefulProgramming, a standard HomoiconicLanguage representation of the AST for reflective operation by said PostProcessor?, low-level DependencyInjection at the code layer (related to HyperSpaces and AspectOrientedProgramming), and requiring a clear statement at the top of each page of code describing the exact version of syntax in use.

Once these ExtensibleProgrammingLanguage mechanisms are in place, they act as a buffer allowing extensions to the language without upgrade to it, and allowing certain classes of upgrades to the language without breaking existing programs or requiring pervasive changes to existing code (since the same ExtensibleProgrammingLanguage mechanisms can be used to downgrade a syntax to an earlier version). Further, such systems obviate the need for many classes of language upgrade, especially those related to DomainModelling. If one wants an InformLanguage style declaration of FirstClass InteractiveSceneGraphs for a VirtualWorldInterconnect?, one writes up the appropriate extensions to the set of communications services, adds some new syntax for describing and composing SceneGraphs and the available interactions, and provides the defaults and assumptions in order to hide semantic noise. (That said, as with all LanguageDesign work, it isn't "trivial" to provide composability, flexibility, static verification, avoid boilerplate, etc. - there are huge differences between well designed language extensions and poorly designed ones just as there are huge differences between well designed APIs and poorly designed ones.)

ExtensibleProgrammingLanguage support reduces the need to perform 'actual' language upgrades to features that either must be pervasive to be useful (such as DistributedTransactions, subscription to mutable state, automatic distribution of code, ObjectCapabilityModel, failover redundancy, system-wide GarbageCollection, or end-to-end ProcessAccounting policy via ExplicitManagementOfImplicitContext in a distributed system, etc.) and various forms of expression that cannot be optimized through composition of other primitives (such as supporting 'sets' with their disordered communication rather than just 'lists').

Actually, a greater problem than upgrading a language with new features is upgrading it with LiberatingConstraints. I would suggest that any LanguageDesigner consider those at time of BigDesignUpFront because there is no way to add them after the language becomes popular (that is a penalty of premature success; related is AlanKayOnMessaging). ObjectCapabilityModel, compile-time confinement, removal of AmbientAuthority?, some classes of static TypeSafety, guarantees of Termination (of the sort sought by TotalFunctionalProgramming), shunning of mutable state (KillMutableState) and shunning of names (NominativeAndStructuralTyping) to support distribution, purity in the lower layers, and much more beside all fall into the whole 'LiberatingConstraint' concept.

Well, it seems I've wandered onto subjects that interest me but aren't entirely topical... so I'll stop there.

The point is that ApiIsLanguage for almost any API that solves a difficult problem. Arguing that one "needs to learn new syntax" may be true but is nonetheless not (in general) a point against use of an extensible syntax mechanism... because, for APIs, one (in general) "needs to learn the gotchas, idioms, DesignPatterns, and protocol, that the RealMacros can guide you in avoiding or utilizing respectively."

Those who argue, "oh, this new language feature eliminates the need for macros or extensible syntax,", should bear in mind that the new feature likely just moves the need for macros a bit further towards the vast language frontier, allowing one to deprecate and transition out some language extensions, and perhaps optimize or improve composition of existing language extensions.