Component-oriented design studies for efficient processing of hyperspectral infrared imager data

Future meteorological sounding instrumentation for aircraft and satellite platforms will include hyperspectral imaging infrared spectrometers with high time and space resolution, capable of providing terabytes of raw data per day. In tandem with the development of the instruments themselves, corresponding software must be architected to be capable of timely, efficient and accurate processing of the raw data produced. Design candidates for such a software architecture must respond to use cases including deployment in large-scale distributed production environments with stringent reliability specifications; phasing of research algorithms through testing and validation into production use; marshalling of data product views to metadata-aware analysis and archival systems; maintenance of software supporting multiple similar instrument systems over the course of decades; and most importantly, delivery of fully annotated datasets to end-users with real-time latencies. Consistent techniques in the specification and propagation of metadata for both algorithm software and data content are of paramount concern in manipulating large quantities of data over long stretches of time. Further, long-term maintainability and cost-effectiveness of the system can be assured by improving reusability of both systems software and science software, through defining well-specified interfaces for software components and implementing automated mechanisms for integration and testing. We illustrate current design work, avenues of research and lessons learned on a software component architecture and corresponding development practices addressing the aforementioned concerns.