Data persistence in a digital world

John L. Miller — Fri, 29 Jun 2007 09:27:00 GMT

A while back I read a news article pointing out an issue largely overlooked, namely the transience of digital data.

For thousands of years, institutional and personal memory were stored solely in physical written form, on paper, papyrus, wax, stone, you name it. The systems for writing these memories are well known and easily accessible: anyone who can read the language can read virtually anything. The down side is that accessing a piece of data in its native form requires a physical token, such as the book containing that data. How many of you remember taking a trip to the library when you needed to know something 30 years ago? And waiting for a book from inter-library loan if the local library didn't have it, sometimes being shipped from hundreds of miles away...

We've switched ease of access and permanence. Thanks to computers and networking, virtually any information you're interested in is no further away than your desktop, or perhaps your pocket if you're especially well connected. This comes at a price, though: most institutional and personal memory will be lost in just a few years, thanks to rapidly evolving digital storage formats and the lack of a long-lived, easily accessible physical store.

It's not uncommon to find written diaries from both common folks and historical figures going back hundreds of years. However, I challenge you to find an electronic diary of someone from the 80's that can still be read today. For example, I did a lot of writing on my Atari 520ST, and stored it on 3.5" floppies. Assuming the floppies are even working today, I don't have any machines with a floppy drive. I certainly don't have any that can read the data files created by whatever word processor I used, and it didn't save in plain text.

Is the web a temporally localized phenomenon, or will it last for centuries? It's too early to tell. But if it *doesn't* last, how much information will be lost when it fades into obscurity? Will digital archaeologists be able to make heads or tails of what came even a few decades before?

With luck, we'll never know. But our grandchildren will.

Cyberspace

John L. Miller — Thu, 28 Dec 2006 18:34:00 GMT

What exactly is cyberspace? For me, it's what I read about in William Gibson's Neuromancer and Neal Stephenson's Snow Crash. It's an immersive datasphere which provides utility information to the common man. It's a playground to the fringe-dweller, where anything can be learned about anything. Tactile representation of this data was always a key feature of the fictional cyberspace, allowing intuition, interpretation, and artistic mining and synthesis beyond that afforded by a naked page of numbers. So, cyberspace is a marriage of a pervasive data plane and an immersive, intuitive interface.

Some people no doubt consider Second Life the ideal platform for cyberspace. It's an artificial world which provides a great deal of control to the user. You can specify your appearance and control your interaction with the world. Even more, it's a social environment where people can meet and exchange views (and information) in relative anonymity. It even includes a currency with an exchange rate, in case part of that information exchange requires a little economic lubricant.

I have to disagree.

Second Life falls short of my personal definition of cyberspace. The interface is there, sure, but is it the right interface? And what about the data? The internet and the web already provide the data aspect of cyberspace. Asimov posited creation of a Universal Library with all the world's data, and I think that day is already here. An ever growing fraction of mankind's knowledge finds its way to the web every day, eagerly transcribed and hosted by the teeming masses. Even better, the barrier to publication is practically nonexistent: anyone can say anything. When was the last time you needed to physically visit a real library to do some research or read an article?

A better answer to creating cyberspace is a purpose-built interface to the world's data. We've already got a lot of the data out there in a standard, mineable format, thanks to pioneers like Tim Berners-Lee and global adoption of standards he (among others) proposed. Which begs the question, if there's already a format AND a consumer for that format (web browsers), why not use it? It's a great way to see the data as the presenters intend.

I'm a fringe-dweller. I want to see the data in a way which serves my own nefarious ends, not have it filtered and spoon-fed to me.

The internet has an amazing amount of data on it, both immediately available as html, and as data in other protocols such as P2P file sharing, instant message, VOIP, and SQL. There's a fascinating (though slightly dated) study called How Much Information? 2003 which summarized information generation and availability for a wide variety of mediums. It estimates 167 terabytes (thousands of gigabytes) of static content ('surface web'), and another 91,850 terabytes of 'deep web' (generated content based on obscured databases). Add in email and instant messaging, and you're up to half an exabyte, 532,897 gigabytes.

Given the amount of data available for mining, is there a better way to view all of it than a web browser and a stable full of one-off applications?

The Challenge

Design an interface to allow ready access to the world's data. It can use formatting information (such as html tags) provided as part of the data, but shouldn't be restricted the presentation specified by that formatting information. The electronic world should be navigable as a vast sea of data, organized in a fashion that enables new insight.

Key questions:

Is a 3D, immersive, or haptic interface desirable, or is good old-fashioned 2D the best way to go?
What new navigations mechanisms do you provide for trolling through data?
How do you prove your interface is better than the status quo, good old-fashioned web search?
Can a useful interface be made interesting and entertaining to use?

If you're already working on this, I'd enjoy hearing about any non-confidential efforts you've made.

Networking, Games, and Virtual Environments : The Web

Data persistence in a digital world

Cyberspace