How can I backup large amounts of digital data for a period of 100
years, avoiding the consequences of hardware failure and software
obsolescence?

I run a large information website, made from 2,000 files using 280 Mb
of data (30 Mb of text, the rest images). This represents about 7,000
hours of
my time and I am concerned about the long-term prospects for
conventional data storage on PC hard-disc backed up onto data CD. Is
it true that the rapid advances of software and PCs imply that data
CDs will not be readable in 20 years time because the necessary
hardware and software will be obsolete and unobtainable? This has
already happened to me once, I started using a state-of-the-art
Amstrad PCW8256 in 1983, which used CP/M (before the MSDOS era), now
20 years later it is very difficult to get hardware or software to run
the system because it is obsolete.
        Can you advise the best way to store digital data so that it
will be easily accessible by the public in 2103? Film may be an idea,
but it would require 5 million words printed out and photographed onto
microfiche. They solved the problem in 1600, I regularly work from
documents written in ink on manuscript that are easily readable after
400 years.
        I believe that Scientific American ran an article on this
subject several years ago, but I have never been able to trace it, can
you help? Also, is anyone backing up the Internet as it exists today,
it would be a wonderful time-capsule.

Hello,

While I have not found the specific article that you are looking for,
I have found numerous references to it online. I have sent an e-mail
to their science and technology address at experts@sciam.com, and am
awaiting a reply which will hopefully tell me which issue the article
appeared in. Long term digital storage is a problem which more and
more companies and institutions are coming to terms with. In fact, by
contract of the National Gallery of Art, a consulting firm in
Bethesda, MD (link below) have a patent-pending invention which
promises to provide a "solution for archiving all file sizes, from
megabytes to zettabytes." There isn't much detail in the article, but
it's good that others are realizing and coming up with solutions to
deal with this problem.

It's true that older CDRs may have only had a lifespan of 5-10 years,
but today's CDRs have been tested to a archival life of 50 or more
years, some commercial brands pushing even 100 years. The National
Media Lab came up with these numbers in a directed study to
counterpoint an article written up in US News & World Report entitled,
"Whoops, there goes another CD-ROM". Both are linked below for your
convenience.

The United States is not the only country interested in this dilemma
though, the National Library of Australia has a website devoted long
term data storage, and has a robust bibliography to continue reading
should you be interested.

But even if a media is invented which can stand up to the wear and
tear of a century or more, data retrieval may be the problem. A huge
report (linked below) lists technological obsolescence as the real
problem, not long term data storage. For example, many programs
created for the first computers stored on paper punch cards are no
longer able to be read by ANY current computer system. This is
digitally stored data which is not even 40 years old! Often museum
relics are fixed up in order to restore and convert data into storable
formats. Periodic data migration is the troublesome solution which
seems to be the only viable one.

So what is the nearest short-term solution? DVD-R or high quality CDR
discs which may last upwards of 100 years, even if the technology
required to play them back may not. Or you could use this as a short
term goal until holographic storage becomes a reality. The transparent
cubes used to store the data are engineered for durability (at least
in theory). For a primer on holographic storage, I've directed your
attention to a couple articles, one of which is on Scientific
American's website.


Search Strategy:

"scientific american" "long term data storage" on google:
://www.google.com/search?q=%22scientific+american%22+%22long+term+data+storage%22

long term data storage "100 years" on google:
://www.google.com/search?q=long+term+data+storage+%22


Additional Links:

Accessible, Long-term, Error-free Archival Storage of Digital Data
Files:
http://www.storagesearch.com/morris-ward-art1.html

"Data Storage and CD-R Longevity" (references Scientific American
article as well):
http://www.beauphoto.com/frames/digital/news/cdr.html

"Whoops, there goes another CD-ROM" on US World News & Report:
http://nl3.newsbank.com/nl-search/we/Archives?s_hidethis=yes&p_product=UW&p_theme=uw&p_action=search&p_maxdocs=200&s_dispstring=19980216003232&p_field_advanced-0=&p_text_advanced-0=(%2219980216003232%22)&p_perpage=10&p_sort=YMD_date:D&xcal_useweights=no

National Media Laboratory press release & response to US World News &
Report article:
http://www.cd-info.com/CDIC/Industry/news/letter-190298.html

"Preserving Digital Information" commissioned by The Commission on
Preservation and Access and The Research Libraries Group, Inc.:
http://www.rlg.org/ArchTF/tfadi.index.htm

"Holographic Memory" on How Stuff Works:
http://www.howstuffworks.com/holographic-memory.htm

"On the Horizon: Holographic Storage" on Scientific American:
http://www.sciam.com/article.cfm?articleID=0003B887-9065-1CBD-B4A8809EC588EEDF&pageNumber=1&catID=9


I hope I addressed the points in your questions to your satisfaction.
As to your internet archiving mini-question, please point your browser
to http://www.archive.org/ otherwise known as the Wayback Machine.
They've been taking periodic snapshots of many websites as far back as
the mid 1990's. It's quite interesting to see how old versions of
popular websites looked.

Thank you for the opportunity to answer your question, if you require
more information, please clarify the question, or if you find this
answer satisfactory, please feel free to rate it. Thank you!
 
skermit-ga

---

Clarification of Answer by skermit-ga on 16 Jan 2003 16:51 PST

Still doing background research and found some more interesting
things, as with my other research, all suggestions point to periodic
data migration as the only solution for long term data storage.


Additional links:

Nice infographic on storage media and lifespans:
http://www.businessweek.com/archives/1998/A16MAC20.GIF

"Data Storage: From Digits To Dust" on Business Week:
http://www.businessweek.com/archives/1998/b3574124.arc.htm

"Fading Bits Of History" on ABC News:
http://abcnews.go.com/sections/scitech/DailyNews/preservation010708.html

"Recording History" on ABC News:
http://abcnews.go.com/sections/scitech/DailyNews/preservation_efforts010710.html

"Glimpses Through Time" on ABC News:
http://abcnews.go.com/sections/scitech/DailyNews/preservation_whysave010709.html


skermit-ga

---

Request for Answer Clarification by doxeyman-ga on 16 Jan 2003 17:03 PST

Hi Skermit,

        Thanks for your long, interesting and quick answer, boy you
guys can move! Can I ask whether you know of a UK firm offering data
migration, I would like to follow up this suggestion.

Doxeyman

---

Clarification of Answer by skermit-ga on 16 Jan 2003 18:06 PST

I'll do some research, but I'm not sure any services offer long term
data storage (or I haven't seen any that advertise this).

skermit-ga

Thanks "Skermit" for a great answer to a problem thats worried me for years.

wonder if "vacuum packing" a CURRENT computer/technology with your
archived data would be a feesable thing....kind of like cryonics in a
sense.

Hah, nice idea... reminds me of the secret decoder rings in Cracker Jacks!

You mention microfilming as one possible (partial) solution; you may
not know that COM (direct computer output to microfilm) is possible.
Your data would not have to be printed out to paper before being
microfilmed. This is a well-established technique and has excellent
longevity.

Owain

If you had archived your data 50 years ago (or even 20 years ago) what
method might you have used?  How many of the methods that you might
have used are still in use today?

If you want to put your data in a vault and walk away for 100 years,
paper (or microfilm) is about the only thing you can rely on.  If you
want to store it electronically, plan to periodically transfer it to
new media.  In principle you could store a readout device with the
stored data, but you'd better include a complete computer, because
input/output methods may have changed in 100 years.  And there's no
guarantee that the readout device and the computer will still work
after being in storage for 100 years.

Hi doxeyman-ga

  Here is the link to the Scientific american article that you've
requested:

http://www.informatics-review.com/classic_reviews/long.pdf

I do have my own ideas(developed over last few years) of longterm
inexpensive digital archival that is fast inexpensive and does not 
depend on hardware/software obselence and still storage efficient
(apart from easily being tranfered inexpensively using current 
technology).  Unfortunately I cannot provide you with any further
information since the idea is much more worth than the price for 
an answer :-).

Regards
ldavinci-ga

Thanks a million "IdaVinci", I've been looking for that Scientific
American
article for years. Now I've recorded it on paper(!). If your idea is
half as good as you describe, you'll make a lot of money and please
add me to your list of future customers.
And thanks "Owain", I didn't know about COM (direct computer output to
microfilm), but I aim to find out a lot more, it sounds like what I am
looking for.
And thanks to "Skermit" for such a good answer to start with. What a
great Forum this is.

http://www.chron.com/cs/CDA/story.hts/tech/1739675

Has is a new article on the pitfalls of digital evolution, and people
in charge of data migration not keeping up.

skermit-ga