FEP - Format Use by a Researcher - McGlynn - FITS

1. Format (Format System) Identification

The Flexible Image Transport System (FITS) is nearly universally adopted as an interchange format within the astronomy community (both space- and ground-based). It is also used extensively as an archival format. The HEASARC uses FITS files for storage of all its scientific data and as the internal file format for data processing. See http://heasarc.gsfc.nasa.gov/docs/heasarc/fits.html for further information.

2. Original Motivation

The HEASARC was mandated to support FITS as a transport format for delivery of data to users. Given this requirement we evaluated the use of FITS internally. The advantages were machine independence -- very important in our heterogeneous environment, a well defined structure for associating meta-data and data, and excellent long-term stability. The perceived disadvantages were inefficiencies in reading data in non-native binary representations, and concerns about flexibility and the archaic structures in FITS. Benchmarking dispelled our concerns with inefficiency and the wide-spread adoption of the binary tables format within FITS has alleviated the second although limits on FITS metadata and lack of a standard mechanism to define relationships among FITS files remain a concern.

3. Data Types

The HEASARC and associated data centers process data at all levels and types within FITS. Typically the first step in analysis is to wrap the telemetry stream within FITS and all subsequent processing uses FITS format data. Images, spectra, event lists, time series and engineering data are some of the types of data used within FITS.

4. Support

The FITS standard is maintained by the International Astronomical Union. The HEASARC has taken a leading role in developing FITS libraries and software. We have also worked extensively with other organizations, notably the Smithsonian Astrophysical Observatory, in defining FITS metadata standards for high-energy astronomy.

5. Software

FITS is supported by a wide variety of packages and software libraries. The major astronomy analysis environments, IRAF, AIPS, and the HEASARC's FTOOLS fully support FITS data. Software libraries are available in many languages including at least C, Fortran, IDL, Java, TCL, Perl and MatLab (see http://heasarc.gsfc.nasa.gov/docs/heasarc/fits/fits_libraries.html). FITS readers for at least the simplest FITS formats are available even within some very widely distributed tools such as XV.

6. Environment

Personally I prefer doing analysis within the IDL. FITS data can be trivially converted to and from data structures in this environment. The HEASARC's FTOOLS environment is our standard analysis system and is supported on Unix, VMS and Win-32 platforms.

7. Usage

FITS is used as our internal format -- the F in FTOOLS stands for FITS!

8. Experience

Strengths:

• Portability of data -- no dependence on machine architectures.
• Universal acceptance allows easy integration of data from multiple sources.
• Large infrastructure of FITS resources in the astronomy community.
• Long-term stability of FITS -- commitment that FITS formats are never made obsolete.
• Well defined process for adopting modifications.
• Data format standard clearly defines syntax of the data but allows tailored software implementations of reading/writing.

• Old format with fairly arcane restrictions, e.g., 8 character keyword names.
• A number of important features are not fully within the definition of the standard.
• FITS does not completely specify data. It is easy to write data that another user cannot use even if both use 'FITS'. A lot of work is needed to ensure compatibility within FITS files. I suspect this is true of most other formats as well.

9. Desired Functionality

As mentioned above there are a number of annoying historical restrictions in FITS. There has been some effort to get around this by adoption of non-sanctioned conventions, but this is currently being done piecemeal.

10. Selection Criteria

The astronomy community is extremely broad, space-based observations are only a small part of the entire community. The growing importance of multi-wavelength analysis means that adopting community formats for data interchange is paramount. For archiving data the primary issue is the long-term stability of the data standard.

11. Impact on Research

The wide-spread adoption of FITS within astronomy has greatly benefited our ability to perform multi-wavelength campaigns and data analysis. Within the astronomy community the issue is not so much what data format to adopt, but the specific details of FITS files to be used for particular applications, i.e., the specification of the profiles of FITS data to be used. FITS provides us with a common grammar but we still speak many dialects.

12. Other Comments

Wider Views