Volume 14, Number 4, December 1998
By Nancy Laubenthal and Joseph King
The National Space Science Data Center and its host organization at Goddard, the Space Science Data Operations Office (SSDOO), have been actively working to update computer software and systems to be ready for correct operations well before the year 2000 arrives.
In keeping with NASA guidelines, all currently used software and systems had to be characterized as being intended for use, or not, past January 1, 2000. All software and systems intended for use across the millennial boundary then had to be declared as mission critical (MC) or otherwise. Staff had to do repairs for and testing and certification of mission critical software for Y2K compliance at an accelerated pace relative to non-mission-critical (NMC) software.
The designation "mission critical" was initially intended to keep NASA's spacecraft operational across the millennial boundary, but each NASA organization, including NSSDC and SSDOO, designated an often arbitrarily selected set of its software and systems judged the most important as mission critical.
Testing was designed to ensure operability not only across the 1999-2000 boundary but was also intended to ensure the correct handling of February 29, 2000. Note that while most years divisible by four have a February 29, most years divisible by 100 do not have a February 29, except that years divisible by 400 (for example, the year 2000) do have a February 29.
The testing was to ensure Y2K compliance of NSSDC/SSDOO's computers and operating systems, its various purchased and shareware commercial off-the-shelf (COTS) software, and its inhouse-developed applications. Testing was done on the one hand with extra copies of time tagged data and information having their time tags artificially shifted to the year 2000 and on the other hand by shifting the actual computer system clocks so that they would cross critical time boundaries (12/31/1999 to 01/01/2000; 02/28/2000 to 02/29/2000).
Unlike some other organizations, NSSDC/SSDOO does not have a separate Y2K-testing hardware environment. Its tests were done with operational machines normally used in the ingestion of data from data providers and in the delivery of data and services to its users. Prior to the November 6-15 clock-shift period for the main VMS NASA Data Archive and Distribution Service ([NDADS], etc.) environment, staff backed up all disks and then restored those disks after the clock-shift period, such that normal operations resumed in the post-test period as if the test period had not occurred. Fortunately, the clock-shift period passed apparently without major inconvenience to NSSDC/SSDOO's providers and users, in part because of advanced notice issued to those communities.
Virtually all of SSDOO's mission critical software and systems and many of its non-mission-critical systems have now been certified as being Y2K-compliant.
The following mission-critical systems have been certified:
|1. MC elements of the Astronomical Data Center (ADC).|
|2. Advanced Satellite for Cosmology and Astrophysics (ASCA) data system.|
|4. Common Data Format (CDF).|
|5. International Solar-Terrestrial Physics (ISTP) ingest pipeline.|
|6. NDADS: Several anomalies surfaced during the time-shift testing, and SSDOO is working to resolve these problems.|
|7. NSSDC ingest.|
|8. MC elements of NSSDC operations and data bases.|
|9. Rossi X-Ray Timing Explorer (RXTE) data system.|
|11. Space Physics Catalog (SPyCAT).|
|12. Web Interface for Searching Archival Research Data (WISARD).|
The following non-mission-critical software were also tested and certified:
|2. NASA Science Office of Standards and Technology (NOST) software.|
|3. NSSDC information systems applications.|
|4. NMC elements of NSSDC operations and data bases.|
|5. Space physics Bills of Lading (BOL).|
|6. Space Physics Data Availability Catalog (SPDAC) software.|
|7. Voyager MAG data, Ulysses GRB data, Ulysses SEDR ephemeris data.|
The Y2K testing revealed a few minor problems that were fixed immediately and a few additional problems noted for subsequent fixing. These are being addressed and fixed as this article is being written.
Y2K problems encountered follow:
|1. Setting up a test Sybase data base and taking the original data base off line created problems that took a few hours to resolve.|
|2. With the system clock set to the year 2000, the Perl system routine "localtime()" returns the year as 100 instead of 2000. (This routine worked as documented: It returns the current year minus 1900.) Perl code had to be modified to properly handle the Perl system routine "localtime()".|
|3. In the custom software "Stocker Form" one date call used a custom "C" function and date validation that had to be modified to work correctly with dates after 2000.|
|4. The EID_LIST option of FST_REPORTS does not work when the system clock is set to the year 2000. This report option uses the key_id (EID) field to perform searches and sorts of the File Store/Stage (FST) data base. The issue is that the searches are by date and the key_id field is a 32-character text field. This data base design and implementation issue will have to be fixed.|
|5. The FST log files incorporate the current date/time in the file names (FST_yyyyhhmm-nn...). When the system clock is set to the year 2000, the file names incorporate the string "1900" vs. "2000" (FST_1900hhmm-nn...).|
|6. The list file created by the ISTP BOL generator also had the string "1900" vs. "2000" incorporated into the file name.|
|7. Staff were unable to open a SONY platter in the Cygnet Jukebox when the system clock was set to the year 2000. Specifically, staff were unable to use the SOAR OINIT utility to mount the newly created magfile. This will be investigated and fixed, possibly requiring additional limited downtime for testing.|
There were a number of lessons learned from this Y2K testing exercise.
|1. The testing will take longer than you think. Remember Murphy's law?|
|2. There will be unanticipated problems not related to Y2K testing; for example, a disk fan heated up and had to be replaced (a two-hour delay),creating an SGI bootable disk was not straightforward.|
|3. Testers should perform a "shakedown" run of their test sequences prior to the official pre-shift (control) run and time shift tests. This shakedown run should be performed under conditions that are as close to the live test conditions as is practical.|
|4. Be methodical in the extreme. Do not worry about speed, especially on the first run. You likely will have to modify your test plan's details somewhat. It is not worth saving a few minutes up front if it increases the chances of having to repeat the whole series of tests.|
|5. Turn the network time daemon off, or it will reset the system clock.|
|6. On UNIX systems it is impractical to set the system to an earlier date during a series of time shift tests. This fact means that testers should be careful to complete all phases of tests for a given shift date BEFORE the date is shifted forward.|
|7. Demonstration licenses (and others) may expire when you shift the time forward. Temporary Interactive Data Language (IDL) licenses must be obtained for any dates you test.|
|8. When printing Web pages for documentation purposes from Macintoshes or PCs, Internet Explorer works better than Netscape. Internet Explorer prints forms with entries filled in; Netscape leaves entries blank.|
|9. If possible, testers should work from a location other than their office to minimize interruptions.|
|10. Testers of the core NDADS software have observed a problem where, in some cases, bit compares of original data files did not match bit-for-bit with the stored files. Staff determined that the detected file differences are caused when copying files between disks with different blocking factors. The last block of the new file does not seem to have been initialized to zeros, but this does not affect the contents of the good data or the data usability. This phenomenon is seen with the VMS COPY command as well as the File Store/Stage-System for Optical Archiving and Retrieval (FST/SOAR) software.|
|11. Balloons and refreshments lead to more successful testing!|
See URL http://ssdoo.gsfc.nasa.gov/~nancy/y2k/ for Y2K COTS and custom software
inventories, Y2K test schedules, information, test plans, and test results.
NASA home page GSFC home page GSFC organizational page