Trans Clusion

What's a transclusion?

A means of providing OnceAndOnlyOnce storage and addressing of amorphous shared persistent data, including tracking of temporary copies. -- JayOsako

A transclusion is when you import a bunch of stuff here from over there so that it appears as if it were native to here while in reality still being over there. A transclusion is an extreme way of breaking the identity of object representation, a way of making a single object appear to be in many places at once (ie, both here and there simultaneously). Any change to any of the representations of the object will change the object everywhere. Examples of transclusions include union directories in PlanNine, NameSpace imports in SmalltalkLanguage, and so on and so forth.

Transclusions are also useful for text documents. So for example, when an author wanted to quote a document, then instead of copying the quoted part of the document over into their own document, they would make the quoted part of the other document actually be in the new document. The advantage of using transclusions is that any change in the original material (such as fixing typos) would be reflected in the quoted material automagically.

This advantage can be a disadvantage as well, since the transcluded original may be changed to something that the transcluder may not have intended nor anticipated. For instance, a picture of a graph may be transcluded in a page on this wiki. If the owner of the transcluded picture edits it to mean sometthing else, it may no longer serve its purpose on the page. An extreme example would be a picture of some graph, which the owner edits ('replaces' is more like it) to show 'adult content'. The 'curve' the page was about now takes on quite a different meaning. Some sort of transclusion contract could of course be devised to fix this, but this would then have to bee too strict to be useful, since almost all of the content would have to be copied in. For instance, transcluding a part of a document containing a spelling mistake to point out the mistake. If the transcluded document is fixed, the transcluding document now contains an error.

Exactly my thought five minutes ago. I guess some clean versioning control is required for this to work as expected.

Implementing transclusions

At a fundamental level, what's required to implement transclusions is:

an editor/viewer that understands a referential document type
a strict append-only addressing policy for that document type

A referential document type is defined as a (possibly recursive) list of pointers to contents. So instead of having an array of characters, you have a list of pointers to paragraphs. If someone wants to transclude a paragraph, they link directly to that paragraph, so that a future rearrangement of those paragraphs within that former document won't affect the latter one any.

Note that "modification" is antithetical to transclusion. If you transclude a whole document then you're assuming that document will never be modified, that its meaning will remain the same in subsequent versions. Conversely, if you want to be able to transclude individual spans of characters within a paragraph, then text within individual paragraphs has to remain fixed and must remain strictly append-only. This doesn't mean that you can only add to the end of a paragraph when modifying it. It does mean that when you ask for characters #50 to #80 of a paragraph, you'll always get the same answer, no matter how many revisions have been made to it. (When you insert something into a paragraph, it will have higher addresses, even if inserted at the beginning.)

Note that this scheme isn't in any way object-oriented. Nor can it ever be made object-oriented. Transclusions are fundamentally about destroying OO, so no one should expect them to be nice or elegant.

To be useful, a transclusion system must have a way for someone to:

find the exact version of a document a pointer came from
find the latest version of a document a pointer came from

For example, suppose you link to a paragraph in Document A version 1. Then at any later time, it must be possible for you to find both version 1 of Document A, and also the latest version in existence of that document.

If you implement a referential editor/viewer on top of a clean capability system, you get transclusions for free. Any fine-grained history in the capability system will translate to the referential document system automatically. If you don't build transclusions over some form of capabilities then you get broken transclusions and a hell of a time implementing the above features.

XanaduProject was an early failed attempt at implementing transclusions. After 40 years, they came up with nothing. One of the problems may have been that TedNelson dogmatically sought "transcopyright" (the ability of authors to prevent viewing of their work, wherever it might be transcluded, until the viewer pays them) which requires an implementation based on AccessControlLists. The problem with that is that ACLs are provably fundamentally broken, as explained on AccessControlList. While this doesn't necessarily cast doubt on the usefulness of transclusions, it does illustrate one poor way to implement them.

Are you sure it requires an implementation based on AccessControlLists? It's interesting to note the overlap between people who worked on Xanadu and those now working on capabilities: MarkMiller? (EeLanguage), JonathanShapiro? (ErosOs), DeanTribble? (JouleLanguage), MarcStiegler?, AnnHardy? (http://portal.acm.org/citation.cfm?id=692230), ChipMorningstar? (http://www.fudco.com/habitat/), ChrisHibbert. In particular MarkMiller? designed UdanaxGold?'s security model (see JayOsako). My bet is that we'll get all the useful features of Xanadu eventually, they will be thoroughly capability-based, but they won't be implemented in anything called "Xanadu" (probably not Udanax either). -- DavidSarahHopwood

Examples of transclusive documents

Some wikis have been extended to Purple Wikis, to offer a little bit of transclusion. See http://www.burningchrome.com:8000/~cdent/wiki.cgi?TransClusion (broken 1-Sep-09)

The idea here is that each Wiki paragraph is given a unique Purple Number and it can be referred to outside of its original page. When it is referred to, it is said to be "transcluded". Now, if anyone should change the paragraph in the original page, the change will be reflected in the page that has transcluded it. Further, when the item is moved around (for example, if items are inserted in front of it), the reference to it will not be lost in the transcluding page. Once you link to an item, you should be able to automatically find it, no matter what further development it goes through. (One can see some problems with this, which Ted Nelson does not really appear to address, to the best of my knowledge)

You can test this out with an example I set up on the above page. The above page refers to the page http://www.burningchrome.com:8000/~cdent/wiki.cgi?TranscludedLinksCanBeMoved using a transclusion. This second page has a transclusion to that reference, so if you go to this second page, it will show you how you can move the transcluded text around and how it will still be tracked.

If transclusion is to be 'deterministic', as you suggest, then there's little reason to bother with transclusion: instead, introduce a copy. If space savings and structure-sharing is the goal, consider use of CopyOnWrite techniques and other designs that support structure-sharing. I think you were on the mark when you said that the 'meaning' must be the same - not the data, or the contents, but rather what the contents represent must remain consistent. And if this is to be the case, perhaps some sort of semantic addressing might be more appropriate... e.g. one can label or derive much semantic meta-data including both position and content. I.e. one could label a sentence 'a sentence about municipal fence-height regulations in Montana' then transclude all such sentences that happen to also be part of, say, a particular document (another piece of meta-data). Then one could declare: "behold, here are the code-xyz-123 laws for fence-heights in Montana:" followed by the transclusion. Transclusion itself isn't enough, of course. One should also be able to automatically derive from the information to support concerns for layout, combining graphs, etc.

That said, transclusion was introduced to XanaduProject for another reason entirely: issues of copyright and royalty.

Other words might apply better to the idea of 'transclusion' described above, such as 'mashup' (http://en.wikipedia.org/wiki/Mashup_%28web_application_hybrid%29) or DataBinding.

CvWiki claims to support TransClusion.

Most wiki support in-line images, which are a kind of TransClusion (right?).

See http://www.usemod.com/cgi-bin/mb.pl?TransClusion for a description. (If this Wiki had transclusion, I would have used it just now instead of merely referring you to MeatballWiki.)

Related to SymbolicLink

The more I come to study the XanaduProject, the more I realize that my earlier concept of transclusions was in error. I used to think that the great part about transclusions was that, if the transcluded content was changed, it would be updated everywhere at once, similar to the "include" feature of the CeeLanguage preprocessor. Now, I understand better that the whole point of transclusion, as TedNelson uses the term, is that the transcluded content never, ever changes. The point is that I can make changes to a document which are very storage efficient. Suppose I have a 10 megabyte text file, and I change one character in it, but I still want to retain the old version. Without transclusion, I must now store 20 megabytes in order to have both versions. With transclusion, I can merely transclude all the repeated content, and only change that which needs to be changed. This only costs me one character plus all the overhead of the transclusion. This is much more efficient. Transclusion is more about reusing content within multiple versions of a document than it is about connecting content between different documents.

If what you say is right, then 'transclusion' is merely a structure-sharing optimization - and a rather cumbersome one without support for GarbageCollection and such. The use of transclusion in XanaduProject, however, seems focused upon royalty models - i.e. supporting 'micro-payments' for sharing text.

It is both. In UnixOs terms, a transclusion is a HardLink whose owner is tracked. This same object can appear any number of times anywhere in the whole XanaduProject address space. Any user can access a publicly-available transcluded object by purchasing the right to view it via a 'micro-payment'. This same object can appear in multiple revisions of the same document by the same user through the same HardLink mechanism. The mistake I was trying to address was viewing a transclusion as a SymbolicLink.