Digital Archive Directions (DADs) Workshop

(A part of the ISO Archiving Workshop Series)


Position Paper

Digital Archive Directions (DADs) Workshop

DATE: June 22-26, 1998

HOST: The National Archives and Records Administration
Archives II
8601 Adelphi Road
College Park, MD 20740-6001



1. Identification of Proposed Topic [Required]

1.1 Title

Data Ingest Methodology and Tools

1.2 Contributor(s)

Mike Martin, Steven Hughes, Jet Propulsion Laboratory

1.3 Description of Proposed Project

Over the past 8 years, the Planetary Data System has developed a methodology for the production of archive quality data sets. It is used by all the Planetary Data System Data Engineers as a guide in negotiating agreements with data producers (generally space missions like Galileo or Cassini) and performing all the steps required to define, document, produce and evaluate digital archive submissions. While the current workbook is oriented to the mission interface normally encountered in PDS operations, it could be extended into a general workbook for archive summission procedures. The following chart illustrates the activities involved in the end-to-end process. There are also a number of data validation tools that are used by the data engineers to validate incoming products. The foremost is the volume verifier which assures all components of an archive quality products are included on an archival submission and validates all metadata entries included in the products.

1.4 Justification

Most of the archiving and data processing systems we are familiar with have no formal methodology for negotiating with data producers. This can lead to a wide diversity in the quality and type of documentation and data formats which the archive receives and stores.

1.5 Definitions of Concepts and Special Terms

archive quality data product
A data product which is accompanied with sufficient descriptive material to be interpretable by a user who is not familiar with the circumstances of its production. Carrying all the information needed to be useful over a very long time span.
Volume verifier
A software program that automatically tests archive volumes to see that they conform to the standards of the archive system. Project Data Management Plan - A negotiated agreement between the archive and a producer which identifies and describes all products of potential archival value.
Archive and Transfer Plan
A negotiated agreement between the archive and the producer which identifies specific data products and a schedule for their delivery to the archive.

1.6 Expected Relationship with OAIS Reference Model

This activity would support the administration and ingest functions.



2. Scope of Proposed Standard

2.1 Recommended Scope of Standard

Enhance the PDS Data Preparation Workbook to be a general guide to the practices and procedures of negotiating, receiving, reviewing and accepting archival products. Include relevant material from other similar efforts. Develop a standard set of data validation tools built on the PDS.

2.2 Existing Practice in Area of Proposed Standard

We are not aware of any other rigorous methodology for archive data ingest.

2.3 Expected Stability of Proposed Standard with Respect to Current and Potential Technological Advances

It is expected that the methodology and tools will need revisions as the ingest process becomes more web-based (many procedures still focus on cd-rom volumes), but the basic techniques involved should be permanent.

Wider Views

Overview of the DADs Workshop
Overview of US Effort
Overview of International Effort


A service of NOST at NSSDC. Access statistics for this web are available. Comments and suggestion are always welcome.

Author: Mike Martin ( +1.818.354.9342
Author: Steve Hughes ( +1.818.354.9338
Curator: John Garrett ( +1.301.286.3575
Responsible Official: Code 633.2 / Don Sawyer ( +1.301.286.2748
Last Revised: May 18, 1998, Mike Martin (May 26, 1998, John Garrett)