Different subjects were used in each phase of testing to avoid any learning curve associated with their ability to use the archive system. Subjects were selected to represent the potential user community (70% research scientists, 10% NASA managers, 10% educators, and 10% students) and were selected from a list of volunteers gathered from the American Society for Gravitational and Space Biology, the Aerospace Medical Association, and the Space Life Sciences Training Program.
The testing was performed by the LSDA Evaluation Committee led by a team from the Uniformed Services University of the Health Sciences (USUHS) with representatives from NSSDC, Johnson Space Center's Mission Science Office, and Ames Research Center's Data Archive Office. The USUHS team, led by Dr. Joel Leonard, was responsible for definition and coordination of the evaluation and the LSDA Evaluation Committee. Elise Blease of Hughes STX participated in the LSDA Evaluation Committee for NSSDC.
The goal of the evaluation is to deliver a product that is easy to use and will provide users information and data without their having to spend a lot of time learning how to use the system. The tests results for Phases 1-5 have been extremely helpful in pinpointing usability problems in the LSDA prototype system. Evaluators hope that the results from Phase 6 will give the project an idea of where users ran into problems when using the system on their own.
Phases 1, 2, and 3 were conducted as follows. Phase 1 consisted of a questionnaire sent to ten volunteer subjects (scientists, graduate students, NASA managers) throughout the country to determine typical "tasks" for which the user community would use the LSDA. Phase 2 consisted of a detailed test of the archive components by four paid evaluators specializing in software testing and interface. Phase 3 tested ten volunteer subjects (scientists, graduate students, NASA managers) from the Washington, D.C., Metro area. This phase was an observation test where two to three LSDA evaluators watched subjects use the system. Evaluators recorded all actions and problems and conducted the test in silence, telling subjects to use the system as if being unobserved. They were given specific tasks to perform; these tasks were generated from the Phase 1 results.
In Phases 4 and 5, Phase 5 was a duplicate of Phase 4 performed on a larger scale and after the May-June 1995 redesign of the products resulting from Phases 1-4. Phase 4 tested ten volunteer subjects (scientists, graduate students, NASA managers) from the Washington, D.C., Metro area and was a "thinking out loud" test in which two to three LSDA evaluators watched subjects use the system while the subjects spoke in stream of consciousness. The test subjects were also asked to perform specific tasks as they had been in Phase 3. Evaluators recorded the test on audio tape and took notes on problems encountered and comments made by subjects. Phase 5 tested 27 volunteer subjects (scientists, graduate students, NASA managers, undergraduate students, educators at three different sites throughout the country (ten at NASA/Kennedy Space Center, nine at NASA/Ames Research Center, and eight at NASA/Johnson Space Center).
Phase 6 is currently underway. Questionnaires were mailed to 40 volunteer subjects (scientists, graduate students, NASA managers, undergraduate students, educators). This mailing consisted of set tasks, questionnaires, and instructions for accessing the on-line system. The tasks were designed so the users would get to a point at which they needed to order LSDA CD-ROMs to complete their tasks. Accompanying the CD-ROMs were questionnaires and further tasks to be completed.
After the completion of Phases 5 and 6, the project anticipates that any major problems found will be fixed and any critical redesign performed before the projected public release of the LSDA later this year.