NSSDC Archive Plan, 2000-2002

Introduction. This archive plan summarizes the expected data inflow to NSSDC for the years 2000-2002, and is the successor to several earlier 3-year plans. This plan, however, differs from earlier plans by distinguishing for the first time data held at NSSDC for, and usable by, the general research community and data held at NSSDC for, and usable by, a very limited community. As discussed below, this latter category has somewhat different meanings across the three different discipline areas that NSSDC supports: astrophysics, solar and space physics, and planetary and lunar science.

In general, only data independently usable by the general research community are considered to enter NSSDC’s permanent archive. The permanent archive presently consists of off line data volumes including 9-track and 3480 tapes holding older legacy data, 4-mm and 8-mm tapes and CD-R disks holding more recently arrived data, CD-ROM disks wherein 2-3 copies are in the archive and 10’s to 100’s of copies are held for distribution, and non-digital media (film, etc.) almost always holding old data.

NSSDC has begun efforts to gradually digitize at least selected ones of its non-digital data sets and to migrate its digital permanent archive from offline to a “nearline” DLT-tape-based jukebox.

NSSDC provides electronic access to independently usable data from current and recent missions which data are typically not electronically accessible from other sources. Three “levels” of electronic accessibility are: graphical-browse/subset/retrieve capabilities to CDF-formatted (Common Data Format) data via such systems as CDAWeb and OMNIWeb (ASCII output is a user option) and to data held as ASCII via “ftphelper”; basic file-level ftp access to the CDF’s underlying CDAWeb, etc., to the ASCII files accessible through ftphelper and to various other mostly-ASCII data sets; web-based SPYCAT and WISARD interfaces for file-level access to the large data volumes on NSSDC’s NDADS nearline data store. Special file-level web page access to astrophysics data on NDADS (IRAS) or magnetic disk (COBE, SWAS) is also offered. NDADS will be phased out imminently, with most NDADS data to become simply ftp-accessible and some TBD subset to become accessible via the higher functionality interfaces.

In general, the data that are not independently usable by the general research community are received on media and are held on those media until the earlier of: (1) the data come to be supported by sufficient ancillary material (documentation, software, etc.) that they become independently usable and then become part of NSSDC’s permanent archive or (2) the reasons why NSSDC held the data are no longer applicable and the data are released or (3) the media approach their end-of-life in which case a community-involved judgement is made on the cost-benefit ratio of migrating the data to new media.

Astrophysics. NASA’s astrophysics data environment is primarily characterized as a set of Science Archive Research Centers (SARC) which provide electronic access to NASA (and some non-NASA) mission data along with documentation about the data and tools for accessing and using the data. These include the High Energy Astrophysics SARC (HEASARC) at Goddard for gamma-ray and X-ray data and the Multimission Archive at Space Telescope Science Institute (MAST) for ultraviolet and optical data. Long wavelength data have their SARC functions done by a combination of Caltech/Infrared Processing and Analysis Center’s (IPAC) Infrared Science Archive (IRSA) and NSSDC according to whether the data providing mission is more JPL-oriented or Goddard-oriented.

Some data provided by these SARCs to NSSDC are in the nature of backup media of the system through which the SARC provides data access to researchers. In such cases, only the SARC itself is a potential customer for the data from NSSDC. Other SARC-provided data are accompanied into NSSDC by supporting material adequate to make them usable independent of further SARC involvement. NSSDC typically does not provide electronic access to such data as long as a SARC continues to provide such access.

The only astrophysics data to which NSSDC provides electronic access are data from SWAS and COBE for which NSSDC has SARC responsibility, IRAS data of the 1980’s which have a unique IPAC/NSDC history behind them, and the astronomical source catalogs and journal tables of the NSSDC’s Astronomical Data Center which will not be further discussed herein.

HEASARC provides a full system backup to NSSDC each year on DLT tapes and is expected to continue in this mode through at least 2002. In 1999, HEASARC provided about 1.3 TB of data on 102 DLT tapes. Incremental data to be added to these cumulative annual submissions are:

It has yet to be determined whether non-U.S. Guest Observer data from XMM will be managed at HEASARC and archived at NSSDC. The above table assumes it will be. Otherwise the XMM numbers should be cut by 60%.

In addition to these “backup mode” data for which HEASARC is NSSDC’s only customer, the HEASARC also produces and provides independently usable static data and supporting material on CD-ROM. The plan for the providing of such CD’s for 2000-2002 is somewhat indeterminate at this point. For the purposes of this archive plan, we assume 10 GB will be delivered on CD to NSSDC in 2000, 20 GB in each of 2001 and 2002. Note that the contents of these CD’s are electronically accessible from HEASARC, so these will not be held in NSSDC electronic data dissemination areas.

From MAST NSSDC expects to receive the following system-backup-mode data in the 2000-2002 period:

The next major data delivery expected at NSSDC from IRSA is the multi-TB 2-MASS data set. However, this data set is not expected to start arriving at NSSDC until 2003 at the earliest. IRSA is supporting public access to these data.

As a SARC itself, NSSDC will receive and make electronically accessible the following data volumes:

Finally, as a special case, NSSDC receives EUVE data from U. California, Berkeley.

These data are made e-accessible from HEASARC mass storage through interfaces at both HEASARC and MAST. In the past, NSSDC has received about 6 GB of independently usable “science archive” data and about 100 GB/year of near-telemetry data virtually unusable outside the UCB environment. The near-telemetry data have not been transmitted to NSSDC in most recent years, and it is unclear whether this flow will resume. For purposes of this archive plan, we assume only the following:

Summarizing all these astrophysics data sources, we have (NIU = not independently useful, IU = independently useful):

Space and Solar Physics. Space Physics missions are virtually all Principal Investigator/Co-Investigator mode missions wherein the following “data products” become available for archiving: low processing level (LPL) data not yet irreversibly transformed; independently usable and analyzable digital data, typically but not necessarily irreversibly transformed; browse-only data (GIF’s, film products, etc.)

The LPL data typically arrive at NSSDC on media and are inadequately documented to make their use outside the framework of the data-providing project a realistic option. Such data are not made electronically accessible to the external community. The latter two product types are typically made electronically, except for data sets for which there is sufficient electronic access provided elsewhere. Electronic access may be provided through CDAWeb (managed by Goddard’s Space Physics Data Facility, an NSSDC partner) or through one of the systems managed directly by NSSDC (e.g., COHOWeb, basic ftp, the retiring NDADS/SPyCAT system).

The following data inflows are expected from active missions:

The following inflows are expected from various community-based data restoration projects. All these data are independently usable.

The Solar Data Analysis Center (SDAC) at Goddard functions much like an astrophysics SARC for the solar remote sensing data from current and recent solar missions (except that the voluminous helioseismology data are managed at, and made electronically accessible from, Stanford U.). Key current missions supported at SDAC are Yohkoh, SOHO and TRACE. Data from the latter two are managed by SDAC in both an online-accessible mode and an offsite media-storage mode. Copies will be provided to NSSDC after conclusion of those missions. SDAC will provide Yohkoh data on CD-R’s as follows

These data are made electronically accessible by SDAC.

Planetary and Lunar Science. NSSDC’s main source of these data is the multi-nodal Planetary Data System (PDS). PDS “publishes” data sets in that they accumulate data and needed supporting material onto media which are peer reviewed for adherence to PDS standards and for their correct and independent usability. These data arrive at NSSDC on CD-ROM or (soon) on DVD and are not made electronically accessible by NSSDC because PDS provides such access. (NSSDC provides electronic access to selected popular images as part of its outreach to the general public; this will not be further considered herein.)

PDS presently expects to deliver to NSSDC 450, 590, and 400 CD-ROMs (unique titles) in 2000, 2001, and 2002, respectively. This corresponds approximately to 280, 370 and 250 GB.

Added note:
In November, 2000, PDS made the following projections of CD's
and DVD's to be created by PDS and delivered to NSSDC.

NSSDC’s not independently usable data. Summarizing the discipline-dependent significance of some of NSSDC’s data not being generally usable if obtained from NSSDC, we have: In astrophysics, the data-providing SARC may be the only customer but the SARC can and does provide the data and supporting material in an independently usable way to researchers. In space physics, data-providing projects are typically the only NSSDC customer for Low Processing Level data. Such data are typically not usable outside the project framework for want of sufficient (a great deal of!) supporting material.

The following summarizes the information of the preceding paragraphs. All numbers have Gbytes as units. “NIU” designates “not independently usable.” Lines not so marked represent data that are nominally independently usable.