Lecture Review
Greg Davidson
Metrics for Measuring NASA Performance

By Lisa Kaspin and Bob Brodsky

Greg Davidson, Deputy Program Manager for the James Webb Space Telescope Project at Northrop-Grumman Space and Technology, spoke about how he designed Metrics for Measuring NASA Performance for an appreciative audience at the November 30^th dinner meeting of the American Institute of Aeronautics and Astronautics at the LAX Crowne Plaza Hotel.

Davidson was charged with developing standards for the quality and quantity of NASA scientific achievements while at NASA's Office of Space Science in Washington, DC from the late 1980's to the early 1990's. The work was in response to Congress' passing the Government Performance and Results Act, which required government agencies to assess their performance in terms of their goals.

Davidson researched possible metrics for the following:

• Increase in capability over previous alternatives
• Quantity of data obtained (at the same time, its significance)
• Cost efficiency or scientific information yielded per dollar
• Number of papers in refereed scientific journals
• Most importantly, whether the data answer fundamental questions and at the same time stimulate a new line of questioning

Davidson developed metrics using the journal Science News: specifically, the end-of-the-year list of what Science News deems the most important stories in all fields of science. For the past 13 years, Davidson has published the Science News metric, assigning points to NASA missions based on the number of stories listed in the Science News summary.

The Science News metric revealed the following:

• Among the most important stories, 7.2% reported on space science: the majority of those stories described NASA achievements! Up to 10% of worldwide scientific achievements per year were attributable to NASA over the period 1973-2004.
• Discoveries from the Hubble Space telescope alone (from 1990-2004) accounted for most (53) of the stories: more than Voyager (15.7), Viking (15.2), Galileo (11.2), and a combination of Apollo, Skylab, ATM, and Apollo-Soyuz (9.8) over a 30-year period!

What do these achievements mean in terms of the type of missions that NASA should undertake? There has been debate over whether to spend more for bigger, more complex missions, or to go for "faster, better, cheaper." To address this debate, Davidson used the Science News metric as a function of mission cost (Development plus Mission Operations and Data Analysis budgets) for performance in the 1990's

• Larger missions (eg, Galileo, Mars Observer) averaged twice as much science knowledge per dollar as did the smaller missions (eg, TOMS-EP, Deep Space 1, SSTI Lewis and SSTI Clark.)
• In addition, one must consider that some of these missions had been extended or were in prime mission life, indicating that their value could increase as more science knowledge comes rolling in.

There are some caveats to the use of the Science News metric. Science News is not peer-reviewed, meaning that their assessment of importance was reviewed by an editorial staff and not by a panel of research scientists. In addition, the timing of events may affect their importance ratings.

Still, Science News fulfills an important need: it shows the significance of NASA's contributions in the context of science-wide and worldwide achievements, comparing NASA with organizations such as the National Science Foundation and the European Space Agency. This reporting of other countries' achievements, together with the fact that this metric is derived independently of NASA, minimizes the possibility of bias. In addition, the Science News metrics trends are comparable with trends shown by other publications, citations, and textbook content.

At the same time, other metrics did not fare as well as a predictor of performance.

The rates of subsystem failures do not indicate the quality of a mission. If they did, large, complex missions which are more prone to losing a subsystem yet are also more resilient such as the Hubble would be considered failures.

The "bathtub curve" of failures plotted over time does not always predict a mission's success. Data from the 1992 OSSA Spacecraft Subsystem Reliability Historical Database showed that while the number of failures was the highest at the beginning of missions (so-called "infant mortality"), the number of failures thereafter was random over time. This contradicted the expectation of a plateau in failures until a point when many systems fail at once (the other side of the bathtub.)

In light of the importance of metrics for NASA's accomplishments, Davidson ended his talk by discussing one of the most important factors in the success of any mission: the human factor. How many projects have ended up in failure, or even in a disaster such as the Challenger, because of human errors? Failure review boards have found that the root causes stemmed from "leadership" or "culture" issues. Davidson showed a system created by 4 Dimensional Systems (Boulder, CO) for measuring and addressing human factors such as relating, directing, valuing, and visioning. He asserted that such systems can increase the number of successful missions and even save lives with human exploration.

Copyright © 1998-2007 Organization for the Advancement of Space Industrialization and Settlement. All Rights Reserved.