Formats Evolution Process (FEP)- Objective and Approach

Significant benefit has accrued to specific science communities from their use of standard formats within those communities. Examples include the worldwide astronomical community in using FITS (Flexible Image Transport System), the International Solar Terrestrial Physics (ISTP) community in using CDF (Common Data Format) and the NASA Planetary community in using Planetary Data System (PDS) Labels.

However, scientific progress continues to be impeded within some science communities, and across the boundaries of traditional discipline domains, by the lack of, or excessive multiplicity of, available standards for data formats and structures. A somewhat more subtle adverse effect of this situation is that the commercial software sector is unable to perceive what formats/structures it should support (e.g., with applications software) leading to the situation whereby developers and/or users of the many available "standard" formats must develop/maintain/evolve tools that the commercial sector might otherwise do.

In order to address this as it impacts in particular the NASA/OSS space science domain, the NASA/Science Office of Standards and Technologies (NOST) at the National Space Science Data Center is revitalizing a process initiated several years ago and largely dormant since. This process has two time horizons, one on the scale of a few years and the other on the scale of a year or so.

On the scale of a few years, we want to review what "data object types" (images, spectra, time series, etc.) and accompanying metadata have to be managed and scientifically used and what relatively common functionalities need to be associated with such management and use. The objective at this scale is to enable (through consensus on requirements) a coordinated evolution of then-current format standards and associated data management and analysis tools, either within the NASA/OSS and related communities or within the commercial sector (or by some combination of the two).

On the nearer time scale, and in a way consistent with the longer-time- horizon effort, we want to have a focus on the needs of the OSS/SEC community where formats multiplicity is a significant data exchange and exploitation impediment. We want to review the SEC data management requirements and the levels of success of extant standard formats (IDFS, CDF, netCDF, HDF, etc.) and their accompanying software in satisfying the various requirements. From the effort at this scale we would hope for (1) more coordinated and complementary development work on the part of the groups responsible for the various formats in the near term, and for (2) a consensus set of recommendations to new projects needing to make formats selections for various object types and levels of processing in the time frame of 1-3 years hence.

To get these activities underway, we are initiating a dialogue involving the science community through these Web pages. This is an opportunity to significantly affect the evolution and usage of science data formats. It could be in your best interest to participate in this dialogue, either to defend what you like or to promote what you need, and thereby influence the consensus positions that will be generated. Whether you are a science researcher, archivist, project data manager, format developer, tool developer, or early applier of new standards and technologies, you may have valuable insights to contribute or significant positions to defend. We expect the insights gained from these Web pages will significantly affect the future development and support of science data formats.

Wider Views