FEP - Format Use by a Researcher - Louis Giglio - FITS

1. Format (Format System) Identification

FITS

2. Original Motivation

A combination of things led us to begin using the FITS format. These were: 1) desire for a simple, portable, self-describing format that is easy to work with, 2) a great deal of inconvenience and wasted time using HDF over the course of several years, and 3) desire for a standard format that is stable and well-documented at the physical level (as opposed to software interface level).

An additional, less critical goal was that the format be sharable with colleagues having limited hardware resources (in parts of Africa, for example).

Our choice of FITS has so far been quite satisfactory. We've met every one of our original needs, and been able to accomodate a couple of unexpected requirements as well.

Note that in our case using FITS is somewhat of an anomaly -- we use it for non-astronomical data, mostly terrestrial remotely-sensed data sets. However, many of the FITS-related issues ironed out by by the astronomical community are directly applicable to our needs. Examples of this include date and time formats, checksum conventions, and image projections.

3. Data Types

We currently use FITS for Level 1B and higher data levels. Our object types are mainly images, but include some time-series as well. A few of our applications require tabular data, which is well suited to storage in FITS binary tables.

4. Support

The predominant type of support we've needed has been software.

The CFITSIO library from HEASARC has been extremely useful. We have also used many other resources developed within the astronomical community, such as "saoimage", "fv", and FTOOLS.

I also view the comments and suggestions we've received from the astronomical community as support.

5. Software

We usually write our software in C, and use the CFITSIO library to simplify reading and writing FITS files. We also do some work in IDL, and use many of the freely available FITS-related IDL routines. The bulk of our code is for remote- sensing related applications, such as retrieving biophysical surface properties. We haven't had to write many FITS utilities because a lot have already been written by others (saoimage and FTOOLS for example).

We have had to write some code to import FITS files into commercial applications that do not support FITS directly. Examples include ENVI and PCI.

A few times I have hacked Perl, REXX, or C shell scripts to manipulate FITS files directly (i.e. without using specialized FITS libraries). This is sometimes the fastest way of doing things.

6. Environment

Unix on SGI and HP workstations, supplemented by a few Linux boxes.

7. Usage

We use FITS as a research format, and have started using it as an operational processing and archive format as well. As such, we try to leave data in FITS as much as possible.

8. Experience

Strengths:

1. Taken care of our metadata needs.
2. I/O performance is excellent -- very close to what we can get from straight binary reads.
3. Writing code to read/write FITS is simple and straightforward. Keyword convention is simple and works well.
4. In addition to meeting our research needs, so far meets our operational and archival needs as well (no need for multiple formats at different stages).
5. Data can be accessed within FITS files directly if desired.
6. Monster library not required read and write FITS files. Users with limited hardware can read FITS.
7. Format version problems don't crop up as they do with HDF amd other formats.

Weaknesses:

1. 16- and 32-bit unsigned integer images are not described as such directly, but rather signed images with an appropriate bias. This does not help convey information about the image, and can confuse users.
2. Standard does not specify some things that it should. (Example: EXTNAME keyword not allowed in primary array, so what keyword should be used instead if someone wants to name it?)

9. Desired Functionality

1. EXTNAME alternative for primary array should be specified in standard.
2. Keyword unit representation should be explicity defined in standard. Currently most folks use an ad-hoc convention using brackets, but this should be made part of the standard.

10. Selection Criteria

Simple, portable, consistent format defined at the physical file level. I personally do not want another format that is defined in terms of a particular library's software interface.

11. Impact on Research

We've been able to spend more time concentrating on research, because the code to manipulate FITS files is much simpler than equivalent code that works with other formats we've used, and we need not spend time dealing with multiple file formats.

I've found that I personally take better care to embed and look at descriptive metadata and comments within FITS files because it's so easy to do.

12. Other Comments

Wider Views