NSSDC/WDC-A-R&S 98-01

1997 ANNUAL STATISTICS AND HIGHLIGHTS REPORT FOR THE NATIONAL SPACE SCIENCE DATA CENTER

Joseph H. King
National Space Science Data Center
Greenbelt, Maryland 20771
 

Table of Contents

PREFACE

1. INTRODUCTION

2. HIGHLIGHTS

3. DATA MANAGED AT NSSDC, AND 1997 INFLOW AND OUTFLOW

3.1. DATA INFLOW

3.2. DATA OUTFLOW

4. ADDITIONAL NSSDC SERVICES

4.1. NSSDC Information Systems

4.2. NASA/Science Office of Standards and Technology (NOST)

Glossary

Tables

Figures 1-4

1997 Annual Report of the National Space Science Data Center

PREFACE

The National Space Science Data Center is pleased to issue this 1997 Annual Report describing (1) the 1997 growth and evolution of NSSDC's data archives, access pathways, and other tools and services, and (2) the 1997 access to those data and services by NSSDC's customer communities. This report has been made WWW-accessible in the hope that readers will avail themselves of the opportunity to link to the services reported herein.

The scope of this report is that of the traditional NSSDC as defined by the NSSDC budget. It should be noted that some of the activities thereby supported are the responsibilities of the Astrophysics and Space Physics Data Facilities, organizational peers of the formal NSSDC within Goddard's Space Science Data Operations Office.

I welcome suggestions for user-benefiting improvements to this Annual Report and to NSSDC services.

Joseph H. King
Head, National Space Science Data Center

1. INTRODUCTION


This report characterizes NSSDC's data holdings, metadata holdings, access pathways, and value-added data products, tools, and services at the end of 1997, with a focus on the 1997 activities leading to that end-of-year state. In addition, this report characterizes the nature and amount of 1997 access to NSSDC's data and services by its multi-component customer community.
 
 

2. HIGHLIGHTS


NSSDC ingested 2.1 TB of data from 28 space science spacecraft in 1997, to bring its total holdings of space science data to 10.0 TB. In all, these data plus NSSDC's legacy Earth science data (awaiting ESDIS directions on transfers; see Glossary for acronyms) represent 4,352 distinct data sets from 1,330 investigations that have flown on 379 different spacecraft. During 1997, about 0.9 TB of astrophysics and space physics data were made newly network accessible. Also during 1997, NSSDC's users downloaded via network 640,000 data files and received about 4 TB of data on mailed media (mostly CD-ROMs). More detailed characterizations of these numbers, by science discipline by mission, as measured in both byte counts and volume (tapes, etc.) counts, are found in section 2 of this report.

During 1997, NSSDC participated in the definition phases of the Space Science Data Service, intended as an umbrella entity encompassing, and facilitating interoperability among, the public-data management and dissemination sites of NASA's Office of Space Science. One concrete result of the SSDS emergence was the signing of a Memorandum of Understanding between NSSDC and Goddard's High Energy Astrophysics (HEA) Science Archive Research Center (HEASARC) by which the HEASARC is designated the mass store for network access to HEA mission data by the research community and other potential users, and NSSDC is designated as the site for permanent archiving and for offline distributions of HEA data. As the result of this MOU, NSSDC took about 0.7TB of HEA data offline, freeing up capacity for other data not otherwise network-accessible.

Much support was provided to the International Ultraviolet Explorer Project in its sunset months as it retrieved data from NSSDC, performed a Final Archive reprocessing using revised calibration coefficients and algorithms evolved from years of working with IUE data, and provided the Final Archive data back to NSSDC where they were ingested to NDADS. Another effect of the SSDS emergence was an agreement between Goddard and the Space Telescope Science Institute (STScI) whereby STScI will become the primary user access and user support point for IUE data; NSSDC will retain IUE data in its nearline environment as there is no present plan that STScI will host in its mass storage facilities any but the more frequently accessed IUE data.

NSSDC's support for the public's access to Mars Pathfinder imagery generated 11 million hits to NSSDC's WWW pages in July, about double the normal rate.

The CDAWeb system, first made public is mid 1996, matured into an extremely important tool for the ISTP community in their access to and display and retrieval of key parameter data from the great many investigations carried on many ISTP and ISTP-related spacecraft data. Every working day, approximately 150 plots were created by CDAWeb users, and 25 data files were created and downloaded. During the year, data in addition to ISTP key parameter data were ingested to CDAWeb to exploit more generally CDAWeb's ability to provide a data preview (browse) function and to provide just the set of parameters desired by users for just the times specified by the users. CDAWeb's predecessors, OMNIWeb and COHOWeb, had their near-Earth and deep-space solar wind databases extended and continued to grow in usage.

A new WWW-based CD-ROM Catalog and interface were developed. This enables customers to browse NSSDC's CD-ROM holdings and to build an order just as a supermarket customer fills a shopping basket walking up and down the aisles. A secure software package allows customers to submit credit card information with confidence. Every working day, about six CD-ROM orders are received at NSSDC, with about four exploiting this WWW interface and providing credit card numbers.

NSSDC began the migration of data from its 9-track tapes to Digital Linear Tapes (DLT). This process, when completed, will end NSSDC's decades-long usage of standard 7-track and 9-track magnetic tape.

Additional WWW interfaces were built to provide access to NSSDC's metadata stores, geophysical models, and spacecraft position data. The discipline pages accessible from the NSSDC Home Page (http://nssdc.gsfc.nasa.gov/ ) provide pathways to the descriptive material about numerous spacecraft and experiments as held in the NSSDC Master Catalog. This extends the prior year's effort which yielded WWW access to the NASA Master Directory.

NSSDC has enriched its WWW access to a great many geophysical models and to pages through which key atmospheric, ionospheric, and geomagnetic and magnetospheric magnetic field models could be executed. Users can now obtain graphical in addition to tabular output. This has been implemented via Java applets downloaded to users' computers. These models are accessible from http://nssdc.gsfc.nasa.gov/space/model/models_home.html .

Readers are encouraged to exercise the multiple options on the hierarchical array of WWW pages starting with NSSDC's home page. There are several more functionalities beyond those called out in the preceding paragraphs.
 
 

3. DATA MANAGED AT NSSDC, AND 1997 INFLOW AND OUTFLOW


There are several ways to characterize the multi-disciplinary NSSDC archive. Byte counts are a common metric for modern archives, and will be reported herein. Numbers and diversity of media volumes managed, and numbers of distinct data sets, are also very important. (In NSSDC's terminology, a data set is typically all the data from a given instrument at a given processing level in a given format.) The diversity of data sets and of media types relate to the intellectual heterogeneity and technical heterogeneity of the archive, respectively, and we shall report on these also.

At the end of 1997, NSSDC had 4352 distinct data sets and accompanying documentation packages being managed. Table 1 indicates the disciplines from which these data sets come, and a breakout as to whether the data sets are digital or non-digital (film, etc.). The table shows that these data sets come from 1330 experiments which have flown on 379 mostly-NASA spacecraft. By data set count, space physics is the dominant discipline, accounting for over half NSSDC's data sets. This reflects the fact that in its early years, NASA launched a preponderance of multi-experiment space physics missions.

Note from the table that NSSDC manages roughly equal numbers of digital and non-digital (mostly film) data sets, although it should also be noted that NSSDC has been acquiring almost no non-digital data in recent years.

Table 2 is a different characterization of the NSSDC archive, by byte counts and media volume counts. The table shows 13.2 TB of total data, a 2.3 TB subset which is network-accessible, and 67,740 digital media at NSSDC. The byte counts are estimates, involving for some data sets assumptions about the mean numbers of bytes on various media types.

Note that only astrophysics data and space physics data are network-accessible from NSSDC. That planetary data are not network-accessible is the result of the emphasis of the Planetary Data System (through which virtually all planetary data has entered the NSSDC archive for the past several years) on CD-ROM production and dissemination, and because PDS nodes are now making the most important planetary data network-accessible from their nodes. [An exception is NSSDC's WWW-accessible planetary image catalog, at http://nssdc.gsfc.nasa.gov/imgcat/ , which is largely oriented towards the general public.] NSSDC's Earth science data are gradually being migrated to the EOSDIS data management infrastructure, hence are not good candidates for network-accessibility from NSSDC.

Table 3 better characterizes NSSDC's network-accessible data, by Project and by whether the data are immediately accessible from magnetic disk (online) or from the robotics-based NDADS system (nearline). Data held online are typically of relatively small volume and very high interest. Figure 1 shows the growth of the nearline data volume over the past five years, and indicates which projects' data ingest started in which years. It should be noted that the near line archive reached ingest rates in the 80 GB/month range in the second half of 1997.

Figure 2 characterizes the data volumes made network accessible via NDADS during 1997. Most of these data first arrived at NSSDC during this year, while a small amount was promoted to network accessibility from NSSDC's offline archive. Data inflow to NSSDC during 1997 is discussed in more detail subsequently.

Table 4 characterizes the digital media managed at NSSDC, not including back up copies. This table is an expansion of Table 2 in which total numbers of unique digital media volumes were given. It should be noted that most volumes are replicable and have one backup volume. However, for "CD-ROM (Titles)" which are not locally replicable, NSSDC typically holds between 20 and 200 copies of each title. For these, NSSDC must replenish stock through a commercial vendor as request activity drives NSSDC stock down.

Table 5 identifies NSSDC's cumulative non-digital archive, by disciplines by form factor. Note that NSSDC has large volumes of non-digital data for each of the discipline areas it supports. It should be noted, however, that very little new data are arriving at NSSDC in non-digital form in recent years. In particular, in 1997 NSSDC received in non-digital form only some hardcopy ion composition data plots from the ISEE 1 spacecraft.
 
 

1. DATA INFLOW


Tables 6 and 7 characterize the inflow of digital data to NSSDC during 1997. In particular, Table 6 shows that NSSDC received approximately 2.1 TB of new data in 1997, via a combination of networks and hard media. Table 6 shows data volumes by project, with the astrophysics (gamma rays) and space physics subsets of ISTP/Wind data attributed to their respective disciplines. Note the dominance of IUE, FAST and ISTP missions. Table 7 characterizes the inflowing media types by discipline. Several points should be noted. CD-WO media are the dominant input media type overall and for space physics individually. Planetary and space physics data inflow was dominated by Compact Disk (both mass-replicated and write-once) and astrophysics by 8-mm tapes.

During 1997, NSSDC received approximately 780 GB of data electronically, in addition to the data arriving on the media reported in Table 7. The electronic inflow was dominated by the IUE mission (540 GB), with 87 GB from the GSFC ISIS data restoration effort, and lesser amounts from a number of other missions.

By data set count, which as noted earlier marks the intellectual heterogeneity of NSSDC, increments or totalities of 100 distinct data sets from 84 distinct experiments arrived at NSSDC during 1997. Of these 84 experiments, first data were received from 13 experiments, four each on the FAST and (Russian/French) ARCAD spacecraft plus one each on the Clementine, Mariner 4, Mariner 5, Ulysses, and OGO 6 spacecraft.
 
 

2. DATA OUTFLOW

NSSDC provides user access to its data holdings through multiple electronic interfaces and, in addition, through a user support infrastructure for the mailing of offline digital and non-digital data volumes. Most electronic interfaces are accessible through NSSDC's WWW home page and include: (1) special WWW-based interfaces to specific data sets or groups thereof; (2) anonymous/FTP pathways to a range of data files maintained permanently on NSSDC magnetic disk as well as files staged from NSSDC's nearline mass storage environment; (3) WWW-based and other interfaces to that nearline environment for the purpose of selecting data files and having them staged or downloaded.

The dominant special WWW-based data access interfaces NSSDC offers relate to ISTP key parameter and selected other space physics data (CDAWeb), the OMNI and uniformized-COHO solar wind datasets (through OMNIWeb and COHOWeb, respectively), the Infrared Astronomical Satellite (IRAS), and the Astronomical Data Center astronomical source catalogs and journal tables.

The OMNI data set is a 34+ year compilation of multi-spacecraft near-earth solar wind magnetic field and plasma data and energetic particle data, while the COHOWeb database is a uniformized set of files of merged magnetic field, plasma, and position data for each of many deep space spacecraft. Table 8 shows high level annual statistics for the CDAWeb, OMNIWeb, and COHOWeb systems. Note the significant growth in CDAWeb and OMNIWeb usage from 1996 to 1997, while COHOWeb has a 1996-7 plateau. Ulysses data are the most popular of the multi-spacecraft data that COHOWeb supports.

In the Astronomical Data Center, 2200 astronomical source catalogs and journal tables are maintained online for easy access via WWW based tools for browsing and manipulating the contents of the catalogs and tables and via FTP. During 1997, the system experienced approximately 15,000 accesses per month. This was nearly triple the access rate of each of the previous two years!

A great many NSSDC data sets and other information services are held permanently on magnetic disk for Anonymous/FTP access from NSSDC's VMS and UNIX computers. The reader is invited to review all these services from the FTP link on NSSDC home page. Table 9 gives the annual counts of files downloaded from the combination of these computers, both overall (184,000 files in 1997) and for the directories with the most activity. Note that these include public interest options (Photo Gallery), standard formats information and tools (CDF, FITS), "real" data (COHO, COBE), and geophysical model information, coefficients, and software. These counts do not include data files FTP-downloaded after having been staged from NSSDC's NDADS nearline system; relevant statistics on NDADS access are discussed in the following paragraphs.

Table 10 summarizes the 1997 accesses to NSSDC's nearline mass storage environment (NDADS) through the multiplicity of available interfaces. Note that three measures are offered for each mission supported. A request is the specification of one or multiple "entry ID's" each of which specifies one or more data files. Both request counts and counts of files are reported. Finally as a measure of the breadth of interest in the data from any given mission, we count on a monthly basis, and report annual means of these monthly counts, the numbers of unique electronic addresses (username @ site) accessing each mission's data.

NDADS hosted a totality of 10,681 requests in 1997, for a total of 0.46 million data files. By all measures, IUE data and the IUE user community is the dominant beneficiary of NSSDC's nearline environment. Other broadly based communities accessed data from IMP 8, ISTP, and IRAS.

By way of trends, Figure 3 shows the gradual upward trend in counts of data files downloaded from NDADS over the past 5 years. While the dominant mode of dissemination of data to the astrophysics and space physics research communities is via the internet, NSSDC continues to provide a high level of offline data dissemination. The customer community for this offline data dissemination is to a large extent the general public, including educators and students. Table 11 characterizes this user community of NSSDC's offline data services.

Table 12 gives the counts of requests for offline data sets from various disciplines in 1997, and as integrated over NSSDC's history. (A small fraction of requests which are multi-disciplinary are double counted in this table.) Note particularly the dominance of planetary data over both time scales. This is largely associated with lunar and planetary image data which are widely requested by the general public. The high level of astrophysical offline activity reflects requests by the amateur and professional astronomical communities for ADC catalogs on CD-ROM.

Table 13 shows the most recent 5-year history of NSSDC's offline data request activity by media type. Several points are noteworthy. Requests for data on magnetic tape have been declining uniformly and dramatically over this interval. That tapes-disseminated numbers are not declining as fast as tape-request numbers is due to the inclusion in these statistics of Earth-science-tape out-migration, and long-standing (but newly concluded) IUE 3-agency tape exchange.

The dominant mode of offline digital data dissemination is now via CD-ROM. It is of interest to note that every working day of 1997, NSSDC mailed about 25 CD-ROMs to 5 requesters.

Also significant from Table 13 is the fact that requests to NSSDC for film data have not declined over the past 5 years; NSSDC still finished one film request almost every working day.

Finally, very noteworthy is the distribution of about 4400 copies of the Multi-wavelength Milky Way poster created in 1996 by the SSDOO/Astrophysics Data Facility as a special education/outreach product. These distributions are not included in the counts of Tables 11 and 12.

4. ADDITIONAL NSSDC SERVICES


In addition to its archive of scientific data characterized in the preceding part of this Annual Report, NSSDC offers a number of additional services which are described in this Section.
 
 

1. NSSDC Information Systems


The Automated Internal Management (AIM) database identifies virtually all launched spacecraft, the experiments carried by many of those spacecraft, and data sets primarily as archived at NSSDC. Table 14 identifies the numbers of spacecraft, experiments, and data sets described in the AIM File, along with the numbers of new entries made in 1997. The database serves as the source of information for many of NSSDC's WWW information pages. The NSSDC Master Catalog (NMC) and a number of discipline project pages retrieve information from AIM and build WWW pages "on the fly" so that the latest information is presented to the user. Figure 4 shows the continuing rapid growth in WWW-based access to AIM/NMC descriptions over the 1995-1997 interval, with over 40,000 accesses in 1997 and a near doubling each year of the access rate.

The AIM database also serves as a useful source of comparative information about spacecraft, experiments, etc. such as launch dates, time spans of data, orbital characteristics, time resolution, data availability, etc. The presence of comparative information in a consistent format within a single database makes the database especially valuable for survey reports.

The NSSDC Supplementary Data File (NSDF) is similar to AIM, but slightly different from it. It tracks non-spacecraft data, multi-source spacecraft or other data, models and programs, and other NSSDC-held data sets that do not fit the AIM spacecraft/experiment/data set hierarchy. See Table 15 for NSDF statistics.

The NASA Master Directory (NMD) is an open, network-accessible system for brief, overview information about data sets of interest in the space sciences (astrophysics, planetary science, solar physics, and space physics) both within NASA and outside of it. A number of the NMD data set entries contain URL references to important sites on the World-Wide Web where further information may be obtained. This includes hyperlinks to the AIM database for further information about the data sets and their associated spacecraft and experiments. The WWW interface to the NMD also includes the capability to retrieve entries based on simple text retrieval search techniques.

During 1997, the mean monthly number of WWW accesses to the NMD was 946, up nearly 200 from 1996. An average of 285 entries per month were retrieved by the users following their use of the search mechanisms.

The Technical Reference File (TRF) tracks individual published and unpublished papers associated with space flight experiments. The NSSDC ID for the experiment is attached to the reference information so lists of papers relevant to a particular experiment can be reported, and/or provided to persons accessing data from a given experiment from NSSDC. TRF also contains references to selected standards relevant to space science data management as well. TRF statistics are given in Table 16.

The Interactive Request and Name Directory (IRAND) tracks people who have interacted with NSSDC over the years. It includes full names, one or more addresses, telephone and email information, and what NSSDC distribution lists they are on. The database contains approximately 54,000 entries and this information is also accessed and made available through the PIMS interface on the NSSDC WWW Home Page. Further IRAND statistics are available as Table 17. Note especially that almost 10% of the records were updated in 1997, reflecting volatility of such information items as Internet addresses.

The Interactive Data Archive (IDA) is another database of interest. IDA tracks the inventory of NSSDC's digital data volumes (tapes, disks, etc.). IDA had 156,245 records at the end of 1997, with 2804 records having been added during 1997.
 
 

2. NASA/Science Office of Standards and Technology (NOST)


The NOST's mission is to facilitate the recognition and use of standards to reduce the cost/benefit ratios in the exchange and management of scientific data among NASA entities and the scientific communities they serve. NOST's Web Home Page is at http://ssdoo.gsfc.nasa.gov/nost/. The NOST strategy is to play a coordinating role in helping the science disciplines identify new standards requirements. NOST participates in partnerships with them, other agencies, and industry on facilitating the adoption of leading-edge technologies with national or international visibility that can be tailored to meet NASA science information management and exchange requirements, and it assists in the process of moving these technologies toward standards with commercial support.
 
 

1. Consultative Committee for Space Data Systems (CCSDS)


NOST operated NASA's highest level Control Authority office in accordance with the applicable Consultative Committee for Space Data Systems (CCSDS) and ISO standards to formally archive data descriptions for interchange and long term preservation. Forty-three (43) new descriptions were registered, and 315 descriptions were under maintenance as of December 31, 1997. The primary missions supported were: DARN, EQUATOR-S, FAST, INTERBALL, Hawkeye, and POLAR. This is about double the number of new descriptions from last year. NOST participated in the development of draft CCSDS/ISO standards applicable to multi-discipline and sub-discipline information interchange. The primary standards and their usage categories were:

EAST Data Description Language: Completed formal NASA and international review to become an approved CCSDS standard. It provides a standard way to describe a typical record format, including any special primitive data types and records that vary their structure depending on the presence of certain values within the record. This is useful for the long term preservation of such data structures (including level 0 data) and for rapid validation of record/file contents when the information is to be archived. Data Entity Dictionary Specification Language (DEDSL): Was held for revision to be harmonized with the conceptual data element standard ISO 11179 and the emerging implementations in other federal agencies. DEDSL goes beyond ISO 11179 to support units and to provide a concrete syntax for the exchange of data elements.

Reference Model for an Open Archival Information System (OAIS): The reference model is nearing the end of its 3-year development period and has been getting favorable review both within the space agencies and in the broader digital archive communities including the National Archives and the Research Libraries Group. The model establishes initial criteria for recognition of a true archival function and should lead to improved archival implementations, provide a basis for further standardization, and provide more cost-effective vendor support.

 

NOST provides access to various software tools in support of selected CCSDS/ISO standards. These include:

- Parsers for Standard Formatted Data Units (SFDU) Structures

1. SFDU Workbench
2. SFDU Toolkit

- Parsers for the Parameter Value Language (PVL)

1. PVL Statement Parser
2. PVL Toolkit

- Support software for registering, archiving, and disseminating data descriptions from CCSDS Control Authority Offices

1. Portable Control Authority Office System

- Tools for using Enhanced ADA Subset (EAST) description language

1. OASIS
2. EAST Interpreter
3. EAST Generator
4. EAST Reformatter
5. EAST ASCII Dump
6. EAST Viewer

2. Flexible Image Transport System (FITS) Technical Panel and Support Office


The Flexible Image Transport System (FITS) is the standard format for astronomical data transport, endorsed and supervised by the International Astronomical Union (IAU). NOST participates in the evolution of FITS by commenting on proposed new FITS extensions and conventions and by the development of a formal FITS specification document using the NOST standards accreditation process. The FITS Support Office (FSO) of NSSDC/ADF supports the NOST-convened FITS Technical Panel (of Goddard and extra-Goddard FITS experts) in its work and supports the community by providing a number of services, such as current versions of all documents, WWW information pages, test software, and technical consulting.

The FITS Technical Panel completed text for a Draft Standard version 2 of the "Definition of the Flexible Image Transport System." The version of the standard incorporates the image and binary table extensions and the blocking agreement endorsed by the IUA FITS Working Group. The Draft Standard has been distributed to the public for review as part of the NOST certification process.

Version 4 of the FITS Users' Guide was completed by the FSO and distributed to the community. This version expands the discussion of conventions, provides more sample headers, and emphasizes online FITS information sources. The WWW pages of the FSO were upgraded, including the addition of a new section on FITS keywords with links to various sites.

During 1997, the community accessed the WWW home page of the FITS Support Office over 8000 times. Nearly 1400 networked retrievals each of the Standard and the Users" Guide were logged, 90% from the FSO's site and 10% from NRAO. This represents a 250% growth in retrievals since 1996.
 
 

3. Common Data Format


The NSSDC Common Data Format (CDF) is a self-describing data abstraction for the storage and manipulation of multidimensional data in a discipline-independent fashion. CDF is comprised of three parts, the CDF data files that contain both the actual data values and metadata, the CDF software library that is used to create, access, manage, manipulate, etc. CDF files, and a well-defined Applications Programming Interface (known as the CDF Interface) that provides transparent access to underlying software and data. CDF provides the essential framework for which generic applications (e.g., visualization, statistical) can easily be created.