NDADS FTP Service

Data on NDADS are organized by project (e.g., IUE, IMP-8). For a given project, data are organized by "datatype," which identifies groupings of one or more data file types. For a given project and datatype, data are sequenced into Entry Ids (EIDs) that correspond to the file, or group thereof, for a specific observation or time interval. For a given project, datatype, and EID, there are the constituent file(s). NSSDC has provided certain interfaces to its NDADS holding for three years now, the E-mail-based ARMS interface being the most widely known and used of these. Very recently, NSSDC has provided an FTP interface that makes the NDADS jukeboxes act as a large ANONYMOUS-type magnetic disk. This article describes the basics of this FTP access.

NSSDC's newly developed FTP server to the NDADS archive allows users to access the data in our jukeboxes using the FTP network interface. Because most of the NDADS data are kept in a nearline jukebox and not on online magnetic disk, the user needs to consider several choices in making an FTP request for NDADS data. Requests are processed on a first-come, first-served basis. Therefore, waiting on NDADS to service an FTP request can be a lengthy process, if the user is the last one in the queue. Therefore, the NDADS FTP server allows requests to be made in a batch mode as well as in an interactive mode.

Below is an example of an FTP command into NDADS:

shell% ftp ndadsd.gsfc.nasa.gov 9121 (UNIX)

-or- $ftp ndadsd.gsfc.nasa.gov/port=9121 (VAX/VMS)

(The IP address for NDADSD is 128.183.36.20)

The user should login as ANONYMOUS with and enter a 'userid' (e.g. jbehnke) at the PASSWORD prompt. At the top level directory, users will find four subdirectories and a README file. Users should 'get' the README for more information about using NDADS FTP and look in the 'holdings' subdirectory for more information on each project in NDADS.

Here is an example of an NDADS FTP server run:

   $ ftp ndadsd.gsfc.nasa.gov/port=9121
   NSSDCA.GSFC.NASA.GOV MultiNet FTP user process 3.3(109)
   Connection opened (Assuming 8-bit connections)
   <NDADS/FST archive server.
   <  System version: Alpha
   <  Session id: 941205-113932-60600698
   <VMS 6.0 upgrade had taken place and Server is still being tested.
   <Login as 'anonymous' and get file 'readme' for complete info.
   <Archive server ready for new user.
   NDADSD.GSFC.NASA.GOV>login
     Foreign username: anonymous
   <Guest login okay, send email address as password:
   Password:
   <Guest user (jbehnke) logged in.
   NDADSD.GSFC.NASA.GOV>ls
   <ASCII data connection (128.183.36.23,3351):
   archive
   holdings
   misc
   readme
   staged
   <ASCII transfer complete.

Use cd archive/PROJECT to transfer to project-specific subdirectories, wherein the datatypes are listed. Presently, NDADS cannot give lists of the EIDs associated with a given project and datatype, as this could be a prohibitively long list. Rather, it is expected that the user will come to understand the EID-naming and file-naming conventions from elsewhere in the NDADS information environment, and will then get the desired EIDs and files.

There are several ways to order data. One simple method is to simply 'get' the entry-id or even 'get' files within entry-ids. The specification is built from the four components mentioned above, as:

	   /archive/project/datatype/eid/filename

For example, consider a file that was ingested into NDADS with the following characteristics:

• Project ADC
• Datatype ALL
• Entry id A1079
• Filename A1079_PM.DOC

	cd  /archive/adc/all/a1079
	get  a1079_pm.doc

Users may also get files directly. Note that when getting from a directory that is not the current working directory, users have to be careful to specify the local name of the file; otherwise, the NDADS FTP program may try to place it in a (most likely nonexistent) local directory (in this case, /archive/adc/...):

	get  /archive/adc/all/a1079/a1079_pm.doc  local.doc

One advantage of the 'interactive' method is that NSSDC datafiles now have Uniform Resource Locators (URLs). For example, the URL for that file would be:

	ftp://ndadsd.gsfc.nasa.gov:9121/archive/adc/all/a1079/a1079_pm.doc

So, users can embed (possibly slow) links to data right inside WWW documents.

The interactive get method is simple, uses only standard FTP commands, and is amenable to any program that knows how to FTP files. However, it can mean that users are stuck waiting for the optical jukebox to come free, i.e. several minutes spent in 'get.' Because of the potential for delays, several other methods are available for using FTP in a mode that allows users to submit an FTP request and have the data sent to the appropriate location.

One method is to 'script' commands to the NDADS FTP server into a batch file. This method allows users to retrieve several entry-ids at a time. The extended set of FTP commands are used in this case for alternative transfer modes, i.e. moving data across the network. The following transfers all the files associated with several Astronomical Data Center (ADC) entries to a staging area. ADC is a project at the NSSDC that stores its data on the NDADS nearline devices.

The user supplies the RHOST (which stands for the remote host that you want the data delivered to), the RUSER (the remote user on that host who will receive the data) and the RPASS (the user's password on that host). If desired, users can supply the RDIR, the directory where the files are to be placed. Users can chose the type of transfer, FTP or DECNETCOPY, by specifying it after the RTYPE directive. For example, %quote RTYPE DECNETCOPY. The default transfer mode is FTP.

Next, use the SELECT command for the data you want by specifying the entry ids. Then enter the RSTAGE command, which effectively places all operations into a transaction queue. This command responds quickly and signals to the user (or client process) that the server plans to 'shut down' or disconnect, after which the server will begin the staging and delivery of data.

	quote  RHOST  magrathea.gsfc.nasa.gov
	quote  RUSER  zaphod
	quote  RPASS  zz9pza
	quote  RDIR  hotstuff/data
	quote  SELECT  adc  all  A1079
	quote  SELECT  adc  document  A1080
	quote  SELECT  adc  document  A1081
	quote  RSTAGE
	Remote server has closed the connection.
	quit

       shell%

The example above shows the transfer of all the files associated with several ADC entry-ids to the given remote destination in an asynchronous manner. The commands are shown in uppercase for clarity, but, in fact, they are case-insensitive and should therefore be used in lower case during your session. Note that 'quote' is part of the extended FTP command set. After submitting the FTP request, the user goes on to other activities, and then receives an E-mail message when the data transfer has been completed.

Some features of ARMS are not available in the FTP server yet. For example, ARMS has the ability to create, on the fly at the user request, an ASCII version of the IMP-8 MAG15 data which are physically stored in NDADS only in IBM-binary format. Using FTP at this time, only the binary version is available.

Many enhancements are planned for the NDADS FTP server over the next few weeks. For more information or comments on the FTP server, please contact:

Jeanne Behnke, behnke@nssdca.gsfc.nasa.gov, (301)286-8340
Code 633, NASA Goddard Space Flight Center
Greenbelt, MD 20771

NASA home page

GSFC home page

GSFC organizational page