Re: [ARSCLIST] long range file storage

[Damien Moody <dmoody@xxxxxxx>]

Yes, there will be both audio and video. My understanding is that we expect our, as of yet not specifically defined "preservation quality" files could be up to 4 TB max. We're currently planning a disk/tape hybrid where the long-term storage will be done on tape and intermediary "derivative quality" files prepared from hard drive systems. I would just love to see a more efficient system, but we're breaking almost all-new ground here, so perhaps we'll be fortunate enough to develop one over time. 

Damien J. Moody
Information Technology Services
Library of Congress

>>> ArcLists@xxxxxxxxxxxxxxx 08/09/05 5:27 PM >>>

Hello, Damien,

I'm sure we'll all be interested in hearing the solution for
Culpepper when it gets fully designed, implemented, debugged, and
running--it's a fascinating project.

Many (most?) archives don't grow 8PB per year. University of
Toronto's TSpace system is one facility I've discussed using as a
possibility for a campus project. This is modeled after work done at
MIT and Cambridge, as I understand it. It's a disc/tape hybrid.

An hour-long stereo audio CD, as you know, is about 0.6 GB. So, 1TB
can hold about 1600 one-hour audio CDs. Your growth sounds like
adding 12.8 million CDs to the archive annually. I realize that the
LoC archive includes video so that is what really adds up. Are you
planning on including high-resolution (4K?) film scans in this system?

I do think that many of the people struggling here (i.e. on this
list) have archives in the <10TB region (16,000 hours of stereo CD
quality recording). I could be wrong

Cheers,

Richard

At 05:11 PM 8/9/2005, you wrote:
>But what would be the optimum system if you had an archive you
>expect to grow at, say, 8 petabytes per year? Wouldn't spinning
>disks be rather expensive or prohibitive in other ways?
>
>Damien J. Moody
>Information Technology Services
>Library of Congress
>
> >>> ArcLists@xxxxxxxxxxxxxxx 08/09/05 4:45 PM >>>
>
>Hi, Russ,
>
>I think there are some archives who are not ready to make this step.
>
>Personally, I've made the step to spinning discs as my sole storage
>medium. I have at least three copies of each file, soon to be in two
>separate buildings, linked by fiber optic 100 Base FX. The two main
>stores are 1TB each and then there is additional storage amounting to
>more-or-less another 1TB on individual machines (that hold the third
>copy). There is a fair amount of expansion space left in the systems
>I have. I could probably go to 3TB each with the architecture I have.
>I only retain client files for the short term.
>
>The cloning software does NOT propagate deletes and, in the instance
>of digital images, does not propagate updates to all copies (some
>copies are marked "digital negative," essentially).
>
>Long ago and far away, I made CD and then DVD copies of everything.
>It took forever. Now, I check the backup logs a few times a week to
>see if there are any abnormal error messages (I always get a few
>error messages on email as files change during the compare/copy latency).
>
>My Brother-In-Law has about 7,000 slides that he would like to
>digitize. He's been photographing architecture to illustrate his
>teaching of history. I just looked at the scans that he had done at
>the college, and they ranged from 837x564 to a few at 1500x2242. I
>suggested that these were probably not the best scans for
>preservation. He wants CDs. He's not ready yet to move to spinning
>disks. I suggested we could put the PSD files on disks and we could
>burn high-rez JPEGs into gold CDs. I'd hate to put the raw PSDs on
>CD! (I am anticipating PSDs > 20MB/image in the final archival
>scanning and JPEGs~3MB per image).
>
>Two mindsets/paradigms need to be brought into focus:
>(1) It's all data
>(2) Use data center management techniques to make sure you don't lose it
>
>Cheers,
>
>Richard
>
>At 04:07 PM 8/9/2005, you wrote:
> >I've been following the discussion on long-range file storage, and it seems
> >that with all the complexities of burning and storing optical media as well
> >as concerns about being able to play the media decades down the line
> >(storing original player devices, etc.) it may not be impractical to
> >consider the alternative of redundant arrays of independent hard disks and
> >tape backups - along the business model of data storage?
> >
> >Yes, a plastic CD or DVD in itself is cheap (even at $1), but might it not
> >be more efficient, even more economical to set up systems like this? Once
> >the system is engineered and set up, the technicians just create and save
> >the audio files, concerning themselves only with file management, naming,
> >metadata, and so on. Any thoughts on this?
> >
> >Russ Hamm
> >Ed Tech Specialist
> >National School District
> >San Diego, CA
> >http://nsd.us
> >
>
>Richard L. Hess                           email: richard@xxxxxxxxxxxxxxx
>Vignettes
>Media                           web:   http://www.richardhess.com/tape/
>Aurora, Ontario, Canada             (905) 713 6733     1-877-TAPE-FIX
>Detailed contact information: http://www.richardhess.com/tape/contact.htm

Richard L. Hess                           email: richard@xxxxxxxxxxxxxxx
Vignettes
Media                           web:   http://www.richardhess.com/tape/
Aurora, Ontario, Canada             (905) 713 6733     1-877-TAPE-FIX
Detailed contact information: http://www.richardhess.com/tape/contact.htm