An architecture for real-time warehousing of scientific data

The management and processing of terabyte-scale data sets is time-consuming, costly, and an impediment to research. Researchers require rapid and transparent access to the data without being encumbered with the technical challenges of data management. The contribution of this work is an architecture for archiving and analyzing real-time scientific data that isolates researchers from the complexities of data storage and retrieval. Data access transparency is achieved by using a database to store metadala on the raw data, and retrieving data subsets of interest using SQL queries on metadata. The second component is a distributed web platform that transparently distributes data across web servers. We have successfully implemented the architecture on a data set produced by the United States National Weather Service's network of WSR-88D weather radars.