FEP - Format Use by a Project - GOES ISTP KP and other SEC projects - Common Data Format (CDF)

1. Format (Format System) Identification

Common Data Format (CDF) version 2.6 is the format that the Space Environment Center (SEC) uses for most projects. For the data the SEC provides for the ISTP project, SEC uses CDF version 2.5.

2. Data Model Extensions by Project

The Space Environment Center has not modified or extended the CDF in any way. However, SEC staffers have asked CDF Support for various features over the years, and some of these features have been added to the current CDF software.

3. Original Format Selection Rationale

The Space Environment Center selected CDF as its official format system in the early 1990s for several reasons. Several other Space Physics organizations that SEC works closely with had already selected CDF as their format standard. These agencies include the International Solar Terrestrial Physics (ISTP) program and the Interagency Consultative Group (IACG). The Space Environment Center wanted a format standard that would save both satellite data and the metadata associated with it. CDF provides such a structure. CDF is platform independent, which was another reason SEC selected it.

4. Data Types

The Space Environment Center creates CDFs containing instrument data at processing level 0, level 1A, level 1B, and at higher level physical parameter data. The CDF format contains time series data.

5. Support

The Space Environment Center has needed technical advice for the CDF, in particular when SEC first began using CDF, when new staffers began using CDF, and when SEC changed from R-variable to Z-variable CDF. The CDF User Support office at the National Space Science Data Center has provided this technical advice. The support that the Space Environment Center received from CDF Support has been outstanding. This group provided quick response to problems and to other questions, and worked with SEC staff until problems were resolved.

6. Project Software

The Space Environment Center Retrieval and Analysis of Scientific Data (SELRAS) is a database and analysis system that contains both historical and current datasets. It is a sophisticated system which allows data to be displayed both textually and graphically using IDL. The SELRAS database uses CDF exclusively for its database, which has up to 15 years of satellite data on-line.

The Space Environment Center uses the Interactive Data Language (IDL) software for data visualization projects. There is a close connection between the Interactive Data Language (IDL) and CDF. IDL supports CDF, as well as a few other scientific data formats. Without that support, SEC would not be able to use CDF in SELRAS or any other data visualization project.

SEC has created many other programs, and some larger systems, to process raw and corrected scientific data, and to store the data in SELRAS-readable CDF. This includes processing of Space Environment instrument data from the GOES, NOAA/TIROS, ACE, and WIND satellites. SEC processes and sends GOES data, in CDF format, to the ISTP site daily.

7. Experience

One of the strengths of CDF is the way it stores data and associated metadata in the same file, thus making the metadata available for present and future research. The CDF software ("distribution") is easy to install and use on the UNIX and Window NT machines in the Space Environment Center, and the resulting CDF files are portable between platforms. There is software to create CDF and to read from and write to CDF. Other software provides a set of CDF utilities for examining the contents of the CDF. A set of user and programmer manuals is available. Manuals from 3rd party vendors are not available, but the CDF Support Office provides the answers to any questions, The software and support are free.

Scientists at SEC value the display of data in CDF format in plots, using SEC's in-house plotting system, SELRAS. Data can be viewed graphically and rapidly. Raw instrument data is usually stored as flat binary files at SEC, but much processed research data and averaged satellite data is stored in CDF.

8. Desired Functions

If calls to the CDF software could be made from Java and from perl, this would speed up development time, and also speed up some processes that run on SEC web pages. Currently, research data is converted to CDF prior to running Web processes.

It would be very useful to be able to remove a set of records from a CDF, using starting record number and number of records to remove, and to have the gap closed up. If possible, it would be great to have the option to keep the CDF the same number of records, with the "emptied" records put at the end, with their pad values.This would be extremely helpful in cases where data is being added, say, daily, to a monthly file, and a single day's data has accidentally been added twice. It would allow us to simply remove the duplicate records.

Currently, if one uses "extend records", the records are added to the end of the CDF, but no pad values are put in. We cannot use extend records for this reason. Pad values should be put into the "extend" records.

9. Selection Criteria

The Space Environment Center has invested in CDF, and it meets the Center's needs as well today as it has in the past.

10. Handling Plans for Format Evolution

The Space Environment Center begins using new format features in CDF software soon after the changes are available. Thus far, CDF format evolution has resulted in better performance, and software staffers welcome such new features. Those programmers in SEC who create CDF files most often are alert to those features that may not be supported in future versions of CDF; they avoid using them in new code, and they re-write old code as needed.

11. Other Comments

Wider Views