Techniques for Evaluating New Scientific Instruments

Thumbnail Image
Clark, Aleia
Mote, Jonathon
Associated Organizations
Organizational Unit
Organizational Unit
Supplementary to
This paper proposes an evaluation technique for evaluating the impacts of new scientific instruments in a way that avoids the pitfalls of "economic-only" cost-benefit analysis and meets the needs of the customer organization and Congress. On several occasions, Congress has rejected the requested appropriation of 700 million dollars to place a new suite instruments, the Hyperspectral Sounder (HES), on the GOES T, a new weather satellite to be launched in 2016, by the National Oceanographic and Atmospheric Agency (NOAA).  Part of the reason for the rejection is that the economic cost-benefit analysis completed by NOAA indicates only marginal gains from this investment. At the macro level of predicting the national weather, this is true. Over the last twenty years there has been considerable gain in the accuracy of the sixth and seventh day weather predictions. Of course, you feel quite differently about this at the local or meso (regional level) because you focus on the many errors--the unannounced snowstorm that shuts down your city for three days, the unexpected tornado that kills people in your state, or the sudden rainstorm that drowned your picnic. The specific strength of this new set of instruments is that it can be more specific about where severe weather will occur. Another reason why Congress has been reluctant to move ahead, outside of the enormous budget deficits, is the lack of support of the National Weather Service (NWS) within NOAA. Again, there are some understandable reasons for this resistance. NWS uses a global model from which meso models are derived. In fact, the local weather predictions are made by various meteorologists who work for the radio and television stations in your community. The macro models presently push the limits of super computers and cannot easily absorb real time data over short periods (say six hours or less). To do so, requires totally reworking the entire set of complex equations used for weather prediction involving inputs from around the world. This is a very expensive and time consuming task and it is not clear that there is a large enough computer to handle this new information. This is a classic example of path dependency because of an existing technology and cognitive model along with considerable sunk costs in the present system. Therefore, our new evaluation techniques have to accomplish three objectives: 1) Develop new criteria that avoid the pitfalls of the economic cost-benefit analyses 2) Create a case that would convince Congress despite budget deficits that investment in the HES is valuable for their constituents 3) Propose a political strategy that overcomes the resistance of NWS. In addition, we want our discussion of these new evaluation techniques to be in terms that allow others to apply these ideas to situations other than weather forecasting. As indicted above, the strong point of HES is that it predicts sudden weather changes (within six hours) on a meso scale (region or even locality). It can do this because it is measuring the following weather parameters: vertical moisture profiles, vertical temperature profiles, derived stability indices, derived motion of winds, moisture flux and ozone total. Together these measure sudden inversions, the major cause of unexpected instabilities of weather associated with tornados, severe winter storms, floods, and the like. Therefore, the solution to the first problem is to focus on improvements in warning time for each of these severe weather events. To take one example, to move from 13 minutes to 13 hours of warning time for a tornado saves lives. Note, that warning time does not reduce the destruction of property. This is still there. But this is one reason why economic benefits are largely but not always beside the point and especially for extreme weather events. The damage still exists although it might be slightly mitigated but one can save lives. Besides tornados, thunderstorms including flooding, and winter storms, we also want to focus on sudden decreases in air quality and its health consequences. The political solution for Congress is to chart every unanticipated severe weather event in each congressional district and state during the past year to indicate how many of their constituents would be affected by earlier warning times. Of course, the question is what evidence can be marshaled for a system that is not operational, one of the major parts of our presentation. The strategy for handling the NWS is to advocate the creation of separate meso models, which can be much simpler and adapted to the specific geographical regions of the United States and of course constructed to absorb real time data. In other words, the argument is to supplement and not to argue that the NWS should change its modeling.
Date Issued
Resource Type
Resource Subtype
Rights Statement
Rights URI