FEP - Format Use by a Researcher - John Sahr - netCDF

1. Format (Format System) Identification

netCDF

2. Original Motivation

We wanted a convenient, standard, self-describing, platform independent format for our data transport. We have used custom binary formats which have the advantage of minimal size and fast I/O, but suffer documentation and portability problems. After looking around, we experimented with HDF and netCDF, ultimately settling upon netCDF. netCDF interfaces other tools (i.e. matlab) fairly easily, and recently UCAR has added a very easy-to-use C++ interface on top of the C library.

3. Data Types

We use netCDF for level 1A and 1B and higher level presentations of data. We do not intend to store much "raw" (level 0) data, so the conversion to level 1A will be rapid.

Object types are time series and multivariate distributions derived from the radar data.

typical dimensions

level 1A: 1e6 time points (complex valued short (16 bit) signed ints)

level 1B: 1e6 time points, 2-8 time series from combination of several synchronous receivers

higher level: 2000 ranges x 256 velocities (typ) x 1 - 6 other dimensions of real floats

4. Support

We have been able to support netCDf very easily ourselves. I and my several grad students have all successfully installed and used netCDF software tools, and libraries. It runs on my laptop, my home computer, and (of course) the computers at school.

5. Software

we write C and C++ to create, destroy, and manipulate netCDF files. We have written a few tools to translate into forms for Matlab, and have prepared visualizations with GMT ("Generic Mapping Tools").

6. Environment

Our preferred environment is Linux. We have also used Solaris. For a variety of reasons MS operating systems are not acceptable for this project.

7. Usage

netCDF on its own has no analysis component; it is just a (very convenient) holder of information. netCDF handles our several data formats very cleanly, and permits a very robust annotation of the data which propogates forward in time.

8. Experience

netCDF has mostly solved our data handling problems. There are a couple issues.
(1) netCDF forces users to think more logically about their data than they may have previously. There is a non negligible learning curve associated with learning to use the netCDF format (very similar to HDF, of course).
(2) The file I/O for netCDF is much slower than a binary image would be. In C, a single write() or read() statement could rapidly fetch or store a very large file, in comparison to netCDF. Again, this is a learning curve issue, because code development time is often far shorter with netCDF than with a custom binary format --- especially when the custom format goes out of scope in a significant software revision.

9. Desired Functionality

A few additional "native" data formats would be very handy. For example, we would be very happy to have a native 4 bit + 4 bit complex integer (low precision), or perhaps more generally, a native "complex" (re, im) data type.

These features are relatively easy to encapsulate in C++ however, so their absence will not be a great hindrance to us.

10. Selection Criteria

We would still use netCDF.

• it's free
• it's functional
• it solves our problems welli

11. Impact on Research

Our research is enabled by being able to recover the circumstances of old data (old = > 6 months) in a safe and sane fashion.

The structure that netCDF requires is *good* practice for data taking.

We will spend less time (re)writing data load/store routines and more time doing data analysis.

12. Other Comments

We're a small fish, and while we'd be willing to switch to some other standard, if the solution imposes significant cost, we will not be able to implement it.

A technical point: netCDF permits exactly one dimension to be "unspecified" in size, which is handy for those whose experiments do not produce fixed size data sets. However, it does not naturally tolerate data which grows in *two* dimensions. This is apparently a problem for some instruments that I've heard of.

Wider Views