Worlds Largest Database

The Largest database will be measured, not in terabytes, but in petabytes.

According to http://www.usatoday.com/news/washington/2006-05-10-nsa_x.htm?csp=34 , GeorgeBush's NSA has been analyzing a database constructed of all the call-records of all domestic American telephone calls "ever".

Well, let's be charitable and imagine this means "since 9/11/01". About how much data is there in that database there? 300 million people * say an average of 5 calls a day * 365 days * 5 years = about 3 trillion records. Each record would have, at a minimum, 2 * 9 digit numbers + a timestamp of 12 digits = say 30 bytes.

Aw heck, that's only 90 terrorbytes. Um, terabytes. Sic semper ... um, fi. No, really, I was going to type "fi". Hey! Let go of me! Hey!! -

Yes, but some of those numbers could be foreign you know - they're more likely to be terrorists. So to call the UK, you've got to have 0044xxxxxxxxxxx. Thats potentially 15 bytes. Which makes 42 bytes per record. Which is the answer to life the universe and everything. Which is scary.

Yeah, but Americans don't actually call overseas much. Nowhere near the data volume. And GeorgeBush said they only monitored 3,500 overseas calls last year. That's really peanuts compared to his domestic spying. But remember the NSA's technology is always about 15 years ahead of the commercial state of the art. Average new PC disk drive now is, what, 250 gigs? Invoking MooresLaw - doubling every 18 months - that means the average NSA disk drive is 250 terabytes. Average commercial RAID is about, what, 10 terabytes? Then the WorldsLargestDatabase should comfortably fit in 10 petabytes. Room in there for a lot more detail than a few measly call records. And since the NSA must have real working QuantumComputers to analyze it all, they're likely working with complete voice recordings ...

The NiMd? program described at http://72.14.207.104/search?q=cache:B22mr6Qy2y4J:www.ic-arda.org/Novel_Intelligence/mass_data_struct_org_norm.htm+NIMD+site:www.ic-arda.org/&hl=en&gl=us&ct=clnk&cd=4&lr=lang_en talks about databases made of many different data types including audio and video and comprising "several petabytes" ...

Growth of Large Databases:

The size of the world’s largest databases has tripled every two years since 2001:

Other Large Databases and InformationSharing?:

Business Intelligence and FastSharing?:

Size of commercial databases Handling data at the petabyte level: "In the near future scientific endeavors such as the Human Genome Project and the Earth Observing System will need to handle huge databases. The growth of computer power averages about 1 percent per month. The improvement of cost/performance over the last 35 years has been on the order of 100 million. Imagine where we'll be a hundred years from now." "Just a few years ago, a data warehouse or transactional database that approached a terabyte was considered big. Today, "big" means tens of terabytes. Here's the story behind four of the largest data systems in the world, plus a government database project expected to reach up to 5 petabytes (or 5,000 terabytes) within several years and up to 50 petabytes in 20 years. All are examples of organizations pushing the edge of what's possible with database technologies."

CategoryDatabase

EditText of this page (last edited May 12, 2006) or FindPage with title or text search