Universal Catalog Discussion

Questions from an interested party

Catalog... of what?

Every kind of object that's static enough for people to want to organize it in a coherent form. Fiction, movies, software, scientific articles, pictures, collections of any of these.

What is "Unity of Content"?

10,000 Usenet newsgroups is not unity. While NNTP is a useful mechanism for distributing information, it is best used by machines, for example, to spread arbitrations and recent updates to everybody. This must be done in the background, at a low level, and not at all exposed to users.

Unity means more than having the same user interface or using the same software to access different things. It means that there are no arbitrary divisions which a user must know (or even know about) in order to access the information they want. It means that if you want to find something, you only need to know what it is instead of where it is. You don't need to know that a novel is hosted by such and such an archive, you only need to know what kind of novel it is. There is no multitude of sites, channels, groups, users' filetrees or mailing lists. The only notion of "location" of an object ever visible to a user is one derived from the object's semantic meaning (e.g., an old academic article in psychology, recent nude pictures of Cindy Crawford, novel I read about two weeks ago) not on any syntactic information (eg, domain name, machine name, the last name of a minor character in the last chapter of the novel). The only notion of location ever made available to the user, the only notion ever necessary for navigation, identification, retrieval, etc, is information which the user cares about. See SyntaxVsSemantics.

As a concrete example, there are 5 separate valuable resources for Highlander fanfiction: the old mailing list archive few people know about, the central archive that isn't indexed by google and so can't be searched, Janeen's page, Rhiannon's page, Sylvia Volk's page. Unified means that there aren't separate repositories for old stories, new stories, stories over a certain size, good stories, stories by Janeen, stories set in the OH universe, et cetera. Unified means that you can check if there's anything new from the dozen authors whose web sites you'd otherwise have to check *separately* on a semi-regular basis. And that you don't have to wade through a mountain of dreck just to know if there's anything you're really interested in.

Unity of user interface in no way conflicts with diversity of storage. Storage schemes are low-level implementation details while user interfaces are high level design attributes. While the pervasiveness of broken systems makes it seem that the two are related in some way, this is only true in the wastelands of incompetent programmers' imaginations.

What is "resilience against sloth"?

A concrete example is in order. The first and for a long time most important fandom online was the Star Trek fandom. Fans wrote lots and lots of stories and there was even a central archive where you were supposed to be able to find stories. Unfortunately, every story submitted to that archive went through one or two people who had to verify, format and approve it, with a backlog of several months. Eventually, it got to the point where the archive was a joke and everybody bitterly complained about it. And of course, it got upstaged by the X-Files fiction archive, which was much better run, and was completely eclipsed within a couple years. Sloth killed the Star Trek fanfiction archive.

Ignoring the technical and delving into issues of policy? How "open" ought such a thing to be, both to initial submissions and subsequent modifications? WikiWiki is an example of a completely open forum; many other forums are "append only" in that content can only be added by the general public (only moderators may delete/modify previously added content).

In particular, UseNet has been destroyed by spammers. Email may be next. Assuming a UniversalCatalog could work, how to keep 'em out?

It should be extremely open but there should also be different degrees of open (not gonna explain that one though). Keep in mind that the catalog contains only static metadata, and so it's perfectly feasible for it to be append-only. By which I mean that it should be structured like an append-only Log and its objects versioned, the way ALL data storage by now ought to be.

So anyways, my initial idea on the technical aspect of update deliberation was that it would be done over usenet newsgroups. But I can see that ain't going to work, is elitist, non-uniform and inelegant to boot. But that was before I came up with the idea of a super-wiki attached to the Universal Catalog. So you can use the super-wiki to fight about updates. It's non-hierarchical, uniform, and only the people interested in a subject will see fights about updates of it. There will be no global Recent Changes but multiple local Recent Changes.

The technical aspect of that will be a vast automated WebOfTrust. Clients will reject unsigned updates, signed updates whose originators are untrusted, trusted updates that conflict with a much more trusted update, and trusted updates from sources vastly less trusted than the previous version's originator. Each client or group has its own unforgeable identity and gains trust from other clients by faithfully and promptly serving requests for the index and content. After having visibly served a sufficient amount of requests, a client will be trusted to vouch for other clients. After having accumulated sufficient trust either directly or indirectly, by being vouched for, a client will be trusted to sign updates. Even then, ultimate authority will be held by explicitly trusted groups of people; human trust beats machine trust. These human arbiters will be organized in a hierarchy.

Persistent Versioned Append-only Log.

Further, there needs to be multiple logs, but not split along ontology. The current custom on the web of having separate archives for different kinds of things is insane. Rather, there needs to be multiple logs for security reasons. There is only one Universal log. The others are private. Despite this, they must all be seamlessly integrated together.

What really matters is that while the log's replication and distribution is open and unlimited, the insertion points are tightly controlled and limited by cryptography.

You hadn't said that (or made that clear) before; reading the above made it sound like it would be similar to UseNet; hence the suggestions of using an architecture similar to UseNet. Agreed that ID uniqueness is probably the least of your worries.

Now tell us - if the insertion points are "tightly controlled" and "limited by cryptography" - tightly controlled in what fashion? Enough of 'em so they don't become a bottleneck, but not so many that they become an anarchy (uh-oh, I said the A-word in front of Richard) like UseNet? Controlled by who - you? Some corporation? The freemasons? And what purpose does the crypto serve - authentication of authors? Keeping out undesirables? Secure anonymous/pseudonymous posting? Limiting read access to approved individuals?

Read and write are completely independent operations. You can control writes without restricting reads in any way.

Let's be very clear here. A universal catalog is a few-to-many publication model. Updates to it have to be controlled and restricted in order to preserve the integrity of the log. Yes, authentication serves this function. Peers who try to send out bad data will simply be dropped by everyone else as unreliable. And given that updates can be done on a FreeNet model, this does lend itself to secure anonymous "posting". However, it's not anything like posting in the sense of Usenet. That's a completely separate, and much more involved, functionality. (especially if you want principled distributed secure posting)

Preserving the integrity of the log means preventing spam and other malicious attacks. This serves to protect freedom and anarchy, not restrict it. The only restriction to publication will be whether the object is correctly identified and well-categorized. It will work by levels (trial, approved, permanent) and probably be automated. Whoever holds the keys will be a fugitive hunted down by the US government, for obvious reasons.

[Who controls the universal catalog? Who gives out crypto keys? What are the politics of writing?]

The universal catalog will be controlled by the cabal that writes the software, same as always. The keys will be given out by that same cabal to a shadow group of rabid libertarians and anarchists. Spammers and trolls will be suppressed.

Any news? (2004-05-13)

next to do: after that: big bang should include: after that: after that:

Google?

Google sucks. YOU try googling for "the star trek story I read yesterday which I really liked". It doesn't work.

So you want to eliminate privacy, to make it easier for you to find stuff? And how is "I really liked" recorded?

First, no. And second, easily. There's absolutely no reason why a user can't annotate a media object with their own personal 1-5 rating.

And even if you don't let subjectivity enter the picture, there are many, many problems with google due to the fact that it doesn't distinguish between media objects and documents. So if you google for Star Trek looking for a specific episode, movie or fanfic, you'll get a lot of crap.

The problem gets much, much worse when you're trying to google from more than one archive site. For example, Buffy the Vampire Slayer fanfic has literally hundreds of separate archive sites. And then there are the archives that don't let themselves be searched by google like the central Highlander fanfic archive.

And then of course there's the copyright problem. IMDB will never, ever provide bittorrents for movies because this would be "illegal".

The central reason for a universal catalog is this:

While I find your notions of IdeaOwnership? abhorrent, your thinking on this subject is sufficiently developed to warrant my participation. I don't own this idea, but I have been fermenting something similar for several years. I will freely admit that my thoughts aren't as well developed as yours, but I will not be some serf watching you take credit for an idea whose time has come. If you insist on owning this idea, I'll take my design AND programming skills and walk.

What I've been working on I call the all knowledge network which sits on top of the n2er.net. The reason I use these names is because I have registered the domains http://allk.net and http://n2er.net to serve as the loci for the development projects. I am not attached to these names if something better can be procured. The n2er.net is a peer-to-peer overlay network (ala freenet) that rides on top of the internet. The allk network is a distributed wiki flavored overlay on top of n2er.net Allk turns on edit mode for the entire web.

I am excited about the UniversalCatalog because its media focus complements my focus on editable spaces.

This is not my only project right now, so my commitment is constrained by other demands on my time.

-- BrandonCsSanders

I just added the map of how all the projects I'm working on fit together at: http://brandon-cs-sanders.com/map/

It is of course badly in need of revision. -- BCSS

Cute. Not very informative but cute.

If your editable spaces amounts to wiki or something very similar to wiki, then it's not very interesting. You'd do well to remember that wikis have failed. It's been more than 10 years since C2 has gone online and the web is still largely what it was before wikis came into existence. In fact, blogs (an inferior, less powerful, technology) were developed after wikis.

My thinking on editable spaces is just as extensively developed as my thinking on a UniversalCatalog. And I too have long since realized that the two perfectly complement each other. The former is a shared repository of all static objects, while the latter is a repository of all dynamic objects.

If you've registered domain names at all then I doubt your projects are as ambitious as my own. Mine obsolete DNS so there's no point at all to my registering any domain names.

As for idea ownership (or what in academia is called 'taking rightful credit') this page has been up for easily 2 years for all to see. You're the first person to come along to even recognize its significance, let alone to have developed it independently.

And the funny thing is that a UC is only one of my many projects, all of which are equally ambitious, and that unlike the others, I developed the UC idea over a single summer instead of over years and years. The main reason I'm so attached to an idea which I freely admit whose time has long since come is $$$. Or at least, that was my position until I fully appreciated the dearth of design skills among programmers. You and I may think that it's more than time for a UC, but if we both died then nobody would develop it for another 10 to 20 years.

Now onto brass tacks, what language are you using for development?

(And for curiosity's sake, when did you come up with a UC? Oh, and if I was serious about idea ownership, I would have a patent pending.)

-- RK

Projects don't develop in a vacuum. I'll use DNS, wikis, and even the plain old web until they are actually obsolete. Vaporware (mine or yours) doesn't make anything obsolete. http://allk.net and http://n2er.net will host the development of these projects until they develop into a better hosting alternative.

I recognize that you are a thinker because I have some capacity there myself. I'm willing to grant that you've developed your thinking **skills** and your knowledge base to a level I haven't yet attained. I'm even open to the possibility that your native analytical and design capacities are greater than mine.

AND

Big deal ... where has your thinking gotten you? You also must be able to inspire doers to collaborate with you.

If you don't intend to build everything yourself, what is your plan for recruiting and involving others who are better doers than you are?

-- BCSS

Until now, I didn't have any channels for recruiting and involving others. Wikis, newsgroups and web sites are not such channels, not for Smalltalk projects at any rate. Now that I do, I've been doing hardly anything but. -- RK

EditText of this page (last edited October 29, 2014) or FindPage with title or text search