Massive Distribution

How does one coordinate the update of a massively distributed log?

The solution should:

be decentralized; no central authority for anyone to attack or sue.
scale up to a billion nodes divided into 100,000 distinct synced-groups.
minimize synchronization overhead between the groups. So no voting.
have low latency; every one of the 100,000 groups should have the opportunity to write within a week. So no token system.
not fragment the system into inconsistent logs.
prevent hoarding, spamming and spoofing by fragmenting off any group that tries to pull off that kind of stunt.
be resilient and fail gracefully.
be completely automated, even merging inconsistent updates.
use local knowledge to affect global change.

By way of contrast, WikiWikiWeb solves the distributed writing problem by being highly centralized and requiring massive human intervention. Neither of these are options.

Why are you trying to solve this problem? In the past when I've seen this question asked, there has usually been a better solution to the true, underlying problem. Perhaps you could give us a context.

Should this be moved to QuickQuestions?

The problem is creating a useful, integrated, persistent and resilient catalog-index for the hundreds of thousands of stories and books floating around on the net. Add to that all of the articles in all the online journals, all the images floating on web sites, all the software pirated on IRC, all the mp3s floating on Napster clones, all the movies shared on eDonkey. And then allow the catalog-index to scale by a million so it can handle all the people on Earth getting connected on the net.

The goal is to build something vaguely like Google`s directory service, but a million times more useful. The goal is to sink the shrink-wrapped software and entertainment industries by making unauthorized content and services more convenient to get online than walking down to the corner store. Also targeted are the publishing industry (especially scientific journals, private archives and citation indexes) and library systems. But enough megalomania. Can you do that in a non-distributed manner?

Anyways, I already came up with a solution to the distributed writing problem myself. As it turns out, it's already been implemented in a multi-million node network. So I know it works.

Do you mean USENET?

You gave the secret away! Oh well, there's at least one or two secrets to the system left.

CategoryWishList