Volume 17, Number 7
Dec 1993

Preservation of Electronic Formats

Report of a Week-Long Course by Michael Spring at the Preservation Intensive Institute,

August 1-6, 1993, University of Pittsburgh

We were bitted, byted, and nibbled until we grumbled. We were ASCIIed and UNICODEd until we groaned. We were TIFFed, JPEGed, and SMGLed until we screamed. And we were RLE, Huffman, and LZW compressed until we exclaimed "Shoot me, please!" But when the volcano of information that is Dr. Michael Spring stopped, the swirling lava of detail became stone, the world became calm. . . and we saw meaning. The purpose of this report is not to summarize the week but to reflect on the lessons learned. The reflections that follow are the outcome of one of the three discussion groups that met on the final day of class, in combination with my own ideas. Where these reflections seem correct please credit the discussion group, and where they seem misdirected please blame me.

As people concerned about the implications of electronic technology for preservation, we need to understand three basic concepts: 1) all digital information is coded information; 2) digital technology can be used as a tool for preservation; and 3) regardless of whether we exercise the preservation options presented by digital technology, we have to cope with analog and digital electronic records that already exist or will be produced. A more detailed consideration of each topic fol-lows.

1. Digital information is coded information

The first concept, that all digital information is coded, means that in order to preserve digital information and provide future access to this information, we must also preserve the key that translates the code. Digital information is mere-ly a series of ones and zeros (bits) gathered into parcels of eight (bytes) that are gathered into collections called files. The meaning of these bits, bytes, and files can be, and often is, arbitrarily determined by the information's creator. Though we may come to excel in preserving the physical media on which digital information is stored, unless we can decipher the information preserved, future access will require the services of a cryptographer.

The solution for those faced with the responsibility of transferring information to or preserving information in electronic formats is an awareness of and support for standards that define the meaning of digital information. We need to know who creates standards and how we can influence their development. We need to be aware of existing standards such as the ASCII standard (American Standard Code for Information Interchange) for text and TIFF (Tagged Image File Format) for graphics, as well as the hundreds of proprietary formats established by software vendors. Finally, we need to look to the future and support both the development and the use of emerging universal standards such as UNICODE (ASCII code + a possible several billion English and non-English national characters), TIFF and SGML (Standard Generalized Markup Language--a standard used to describe documents which may include textual data, image data, or other data in predefined formats).

These reflections led to the following recommendations:

a) A national repository should be established to preserve both public and proprietary standards for interpreting digital information. This recommendation is based on the importance of this information for the preservation of digital information and the realization that such a task would be impossible for any individual library to assume.

b) As a profession, librarians and preservation profes-sionals need to develop a forum (e.g., a journal, electronic journal. . .) where digital standards can be discussed and explained in terms comprehensible to members of the profession.

2. Digital technology as a preservation tool

Digital technology can be used as a tool for preserving information that is currently in nondigital format. The uses of this technology include analog-to-digital conversion (for sound and video recordings), image-to-digital (for documents, books, photos, etc.) and text-to-digital (OCR/ICR--optical or intelligent character recognition).

In order to understand conversions to digital format, we must understand how the digital record relates to the original. What do we gain and what do we lose? All digital conversions are based on a series of discrete samples of the original information, whether an analog recording, a page from a book, or a photograph. The completeness with which the original information is captured is determined by the distance of these samples from one another, either in time or space, (e.g. dots per inch, samples per second) and the quality of each sample taken. The more samples taken and the high-er the quality of the sample, the higher the potential resolu-tion of a digitally reproduced copy.

The quality of the sample directly relates to the size of the scale by which each sample is measured. For example, if a color photograph is scanned into digital format, we could sample at three quality levels ranging from low to high. We can sample it as a black and white image (using only two values for any given sample), a gray scale image (up to 256 different values of gray per sample) or a full color image (up to 6.7 million different color values per sample).

Once we understand the relationship between the original information and its digitized copy we need to understand the limits and costs of the technology. What are the costs of information input, information storage, and information output? What information might be lost, or enhanced? What are the limitations of image to digital or audio to digital conversions? Once information is digitized, how do we store it?

The quantity of information digitized represents its own limitations. The more information we save (higher sampling rate and/or more values per sample), the higher the cost of processing and storing that information. Storage requirements can be tempered by a variety of data compression schemes. As preservation specialists we must understand that compression algorithms can be lossless (no information lost on decompression) or lossy (decompressed information different from the original compressed information). What do we gain and what do we lose in such schemes? Once our information is digitized and compressed, how do we organize the data (see discussion on standards in Part 1 above), and how do we retrieve it? What are the limitations of converting a graphic image of text, such as the scanned image of a page from a book, into keyword-searchable, character based information? What are the advantages or disadvantages of CD-ROM? How stable is the physical medium?

Only when we understand the technology involved in the hardware and software can we make decisions concerning the uses of digital technology as a preservation tool. These decisions must be guided by answers to the following questions:

By whom and how will the information be used?

• Can digital technology achieve the quality required by these perceived users and uses at a cost we can afford?
• Will access to a digital copy increase or decrease demand on the original?
• Should textual information be digitized as an image, as is currently done with microfilm, as text that can be indexed and searched on a computer, or should it be digitized in both formats?
• How can we index or catalog the digitized information?
• The answer to these questions will help us to answer questions like the following:
• Is digital technology the answer to this particular application or should more traditional means be used?
• At what resolution should an image be scanned or an analog audio recording sampled? --Should basic black and white printed text be scanned as a black and white image or as a gray-scale image?
• Should we use a lossy or lossless compression to store our data?

3. Coping with electronic records

As preservation specialists we must learn to cope with existing electronic records as well as those we produce through our preservation efforts. Preservation of electronic formats requires that we know and understand a) the logical format or code by which the information is translated to human terms; b) the technology that can read the information on the particular medium on which the electronic record exists; and c) the life of the medium on which the information is stored.

a) Logical format. The importance of the encoding scheme or format of digital information places two requirements on preservation specialists. First, it requires that as digital information is collected, we must acquire knowledge of the format in which it is stored. This knowledge must be inextricably tied to the electronic record through cataloging or other means. Further, it is necessary to insure that the specifics of the format are preserved somewhere (see Part 1 above). If the format is peculiar to the records in hand, as may be the case with a custom-designed software application, then we must also obtain a detailed record of the encoding of that format and ensure that it remains tied to the data. Second, we must support the development and use of universal standards so that the problems associated with the existing standards lessen with time.

b) Reading technology. Whereas the format tells us how information is arranged within a digital file, it tells us nothing about the mechanism required to read the digital information from the particular medium on which it is recorded. Information regarding this reading technology, like the format or coding information, should be tied to the electronic record through cataloging or other means. We need to know and document the hardware or combination of hardware and software that enables us to read the electronic record from the particular medium in our possession. We cannot assume that all similar media require similar technology to read. Magnetic tapes, for example, can contain digital or analog information and can only be read by an appropriate machine. Floppy discs are an example of media that can be physically identical yet incomptible. Disks formatted on an Apple computer are not easily read on IBM-compatible computers, nor are IBM disks easily read in Apple computers.

c) Life of the medium. Finally we need to understand that the life of the medium (magnetic tape, floppy disk, CD-ROM, etc.) on which electronic information is stored depends on a combination of two factors. The first is the rate of the medium's physical decay. How long is the medium capable of maintaining its information intact? The second factor is the life expectancy of the technology used to write to and read from that medium. Should this technology disappear, the information on the medium becomes inaccessible.

This combination of factors affecting the life of an electronic medium requires that the preservation specialist be diligent on two fronts. He or she must act in a traditional sense and moniter the condition of the artifact on which the electronic information is stored. When the artifact can no longer sustain the information, like a brittle book unable to support its printed message, it must be copied or its electronic image refreshed on the existing medium.

Unlike a brittle book, however, the preservation of electronic media requires that the specialist monitor the technology that placed the information on the electronic medium. The obsolescence of an electronic reading technology can mean loss of access to the information stored with that technology. Consequently, in addition to monitoring the artifact, the preservation specialist must monitor both the reading technology connected with the artifact and emerging technologies that will supersede that technology. It becomes his or her responsibility to migrate data to the newer technology before the old technology disappears.

Fortunately, a significant advantage of digital electronic data (though not analog data) is its ability to be refreshed, without loss, on its existing medium or transferred, also without loss, over a wire from one computer to another regardless of hardware and/or software differences. If the reading technology is properly documented (see Part b above), any data produced by a given technology can be quickly identified en masse and transferred to a newer medium.

Note: Special thanks to Michael Spring, Shannon Zachary, Karen Motylewski, Barclay Odgen and my wife, Mary Jermann, for reviewing the draft of this essay and offering comments, criticisms and encouragement, all of which have made it better than it otherwise would have been.