FEP - Format Use by a Project - ACE - HDF

1. Format (Format System) Identification

HDF Version 4.1r2

2. Data Model Extensions by Project

HDF has not been extended or specialized in any way by the ACE Science Center (ASC).

3. Original Format Selection Rationale

Tom Garrard, the original leader of the ASC, has passed away, so I can only give a second-hand account of the reasons HDF was chosen as the format for the ACE project. This choice was made in late 1996.

Tom was a strong advocate of the Tennis format, which is an evolution of the Chapter/Verse format used for HEAO and Voyager. Tennis is used for SAMPEX data, and was developed at Caltech Space Radiation Lab (SRL).

However, when the nine ACE instrument teams were surveyed, most of them did not favor the use of Tennis, since it is not a standard format supported by vendors of analysis software, and the Tennis software library had not been ported to a sufficient number of computing platforms.

HDF and CDF were therefore considered as alternatives to Tennis, and HDF was finally chosen. Some of the considerations which went into the decision-making process were:

• a. C-programming-language-friendly API, and easy translation of C data structures into HDF data records. HDF is similar to Tennis in this sense.
• b. Facilities for embedding documentation within data files (HDF Annotations and attributes).
• c. Support from NCSA for many computing platforms, and portability of data between platforms. Mature support for UNIX platforms was a requirement.
• d. Suitable for storage of large numbers of short logical records derived from the raw spacecraft telemetry data stream, and also archiving of highly processed science data.
• e. Availability of a library of utility programs for user-manipulation of the data, and support from third-party tool vendors (IDL, NOESYS, etc.).
• f. Support for browsing of HDF data files via the web (the DIAL HDF data server, from NCSA/Hughes STX).
• g. Several members of the ACE science team were familiar with HDF, and advocated its use.

4. Data Types

    Processing Level:
Level 1A (Reversibly processed instrument outputs)
Level 1B (Non-reversibly processed instrument outputs)
Higher level physical parameter data

Object Types:
Spectra
Time Series 
Multi-variate distributions 
    

5. Support

The National Center for Supercomputing Applications develops and supports HDF. During the early stages of ACE data processing software development, ASC personnel interacted directly with NCSA personnel, reporting bugs in earlier versions of HDF, and ensuring that the HDF format meets the specific requirements of the ACE data.

6. Project Software

Custom software developed at the ASC in the C programming language is used to ingest raw (level 0) spacecraft telemetry data and convert these data to HDF format (level 1 data). Similarly, custom software converts level 2 data submitted by the instrument teams into level 2 HDF data format.

To simplify the process of converting data to HDF format, a perl program (hdfgen.pl) has been written at the ASC. hdfgen.pl creates the C subroutines necessary to initialize and perform reads and writes of an HDF file, using a C structure(s) as a template. Thus, minimal knowledge of HDF is needed to access the data in an ACE Level 1 file. hdfgen.pl can be used to convert any data to HDF format, once one has defined a set of C data structures to contain the data. At ASC, hdfgen.pl is used in the creation and manipulation of Level 1, Level 2, browse and ancillary HDF data files.

NCSA/Hughes STX provided the ASC with the source code for their DIAL HDF WWW data browser, which the ASC has subsequently customized. The ASC uses this software to present ACE data to the scientific community on the web. Users can browse ACE data online and download data subsets in ASCII format, or create online plots.

7. Experience

The ASC finds HDF to be adequate for archiving and distribution of ACE Level 1 data. ACE level 1 data is used only by the ACE instrument teams, and it has not been difficult to support this small community in their use of the HDF format. There have been no platform-related problems, and people are using both IDL and software written in C to access the level 1 data.

The general scientific community needs access to ACE Browse and Level 2 data, and most are not equipped to extract data directly from HDF files. To serve these users, the ASC provides a web browser interface to the ACE data. Users can browse ACE data online and download data subsets in ASCII format, or create online plots. The ASC continues to improve/customize the software used to provide this web browser interface to the ACE data.

8. Desired Functions

A formal framework for describing each element of an HDF data structure (e.g. to specify the units for a physical quantity). HDF annotations are useful, but they are loose in the sense that their content is free-form. We need something which an external application can key on to provide, for example, useful labels for plots, etc. Perhaps HDF attributes provide this functionality. The ASC will be working on this issue as we continue to improve our level 2 data products.

9. Selection Criteria

See section 3 above. I think those requirements are still valid. However, I would add compatibility with data products from similar/related projects as another important consideration. i.e. CDF might have been a better decision for ACE, since CDF is used by ISTP and other Sun-Earth-Connection projects.

Another related consideration is the level of support for a data format by the long-term archiving facility for the project. Long-term archiving of ACE data will be the responsibility of the NSSDC, and current indications are that the NSSDC will be able to support value-added tools/applications for CDF, but not necessarily for other data formats, such as HDF.

10. Handling Plans for Format Evolution

The ASC has no plans for migrating to HDF 5 at this time. The apparent lack of backward compatability between HDF 5 and HDF 4.1r2 is a serious issue which we may have to contend with in the future.

11. Other Comments

The ACE Science Center plans to deliver ACE Level 2 data in HDF format to the NSSDC for long-term archiving. What the NSSDC can do with these data to make them easily available to the scientific community is TBD. The ASC will work with the NSSDC to resolve this issue. In the meantime, the ASC is the active archive for ACE data, and the data are available via the ASC web site www.srl.caltech.edu.ACE/ASC.

Wider Views