Digital Archive Directions (DADs) Workshop

(A part of the ISO Archiving Workshop Series)


Position Paper

Digital Archive Directions (DADs) Workshop

DATE: June 22-26, 1998

HOST: The National Archives and Records Administration
Archives II
8601 Adelphi Road
College Park, MD 20740-6001



1. Identification of Proposed Topic [Required]

1.1 Title

Data Archiving at EOSDIS

1.2 Contributor(s)

Jeanne Behnke, Mail Code 423, GSFC, 301-614-5270, 301-614-5326 Ben Kobler, Mail Code 423, GSFC, 301-614-5270, 301-614-5231

1.3 Description of Proposed Project

Mission to Planet Earth (MTPE) is a long-term NASA research mission to study the processes leading to global climate change. The Earth Observing System (EOS) is a NASA campaign of satellite observatories that are a major component of MTPE. The EOS Data and Information System (EOSDIS) is another component of MTPE that will provide the Earth science community with easy, affordable, and reliable access to Earth science data. EOSDIS is a distributed system, with major facilities at six Distributed Active Archive Centers (DAACs) located throughout the United States. The EOSDIS software architecture is being designed to receive, process, and archive several terabytes of science data on a daily basis. Thousands of science users and perhaps several hundred thousands of non- science users are expected to access the system. While there are many segments in EOSDIS (e.g., flight operations, network) this position paper discusses the development of the EOSDIS archive. In the development of the archive, key issues development challenges, experiences integrating COTS, and the difficulties of managing a complex system development effort. The EOSDIS data collection begins at the ground systems. From the ground systems, the data is forwarded to the science data processing system (SDPS) where the data is captured, processed, archived and distributed. The SDPS is being developed as a fully distributed and heterogeneous system using object-oriented (OO) development methodologies. The focus of this paper are the subsystems that most closely map to the OAIS Reference Model. 1. INGEST subsystem - receives data from external and internal sources and submits them for archive 2. SCIENCE DATASERVER subsystem - inserts and acquires data from the archive 3. STORAGE MANAGEMENT subsystem - manages the access to the mass storage and peripheral hardware and software 4. DATA DISTRIBUTION subsystem - provides the software infrastructure for distributing data via electronic or media

1.4 Justification

It is easier to appreciate the magnitude of the design problem, if you look at the scope of the data volumes to be processed. EOSDIS, at all DAACs, will have to support at least 260 different data products and sets of raw instrument data. These DAACs can expect at least 480 GB of raw instrument data and subsequent processing will create approximately 1.6 TB of data, all to be stored daily. At one of the largest DAACs, located at Goddard Space Flight Center, the permanent archive is expected to reach about 245 TB with more than 15 million files. This DAAC will expect to ingest around 574 GB per day (a throughput rate of 6.64 MB/s into the archive) and disseminate more than 368 GB/day (a throughput rate of 4.26 MB/s out of the archive). In addition to providing a comprehensive data retrieval and processing system, the SDPS is being tasked to provide a flexible, scaleable and reliable system. The architecture must be capable of supporting: * new data types with minimal software modifications * new data centers that will not require new code and software agreements * standard interfaces (HDF-EOS) enabling coordinated data analysis * data access from a wide variety of users (e.g., kindergarten teachers, as well as college professors) * technological advances and the infusion of new COTS products and techniques (e.g., data mining) * inevitable change and new requirements

1.5 Definitions of Concepts and Special Terms


1.6 Expected Relationship with OAIS Reference Model

The ECS architecture has many of the same architectural components as the OAIS reference Model. These include Ingest, Archival Storage, Data Management, and Access and Dissemination. This paper will contrast the ECS architecture with the OAIS Reference Model, describing the common services and functions, Data flow, and Information model of the ECS architecture.



2. Scope of Proposed Standard [Desired]

2.1 Recommended Scope of Standard

2.2 Existing Practice in Area of Proposed Standard

2.3 Expected Stability of Proposed Standard with Respect to Current and Potential Technological Advances

Wider Views

Overview of the DADs Workshop
Overview of US Effort
Overview of International Effort


A service of NOST at NSSDC. Access statistics for this web are available. Comments and suggestion are always welcome.

Author: Jeanne Behnke ( jeanne.behnke@gsfc.nasa.govi) +1 301-614-5270
Ben Kobler ( +1.301.286.3575
Responsible Official: Code 633.2 / Don Sawyer ( +1.301.286.2748
Last Revised: June 4, 1998, Jeanne Behnke (June 9, 1998, John Garrett)