Data Processing and Information Retrieval of Atmospheric Measurements

The product of this research work takes raw atmospheric science data as input, and generate clean, standardized, and redundancy-free data as output. There are two major tasks involved for the work: data processing and information retrieval. Data processing involves removing inaccuracies and resolving inconsistencies among data. Information retrieval involves identifying the information needed, extracting the data, and consolidating similar entries. Given the complexity of the process, various techniques have been used in the development. In particular, fuzzy matching and fuzzy rulebased inference engine have been used for removing inconsistencies among data entries, retrieving information from certain sections of data files, and consolidating information from different sources. A rule-based forward chaining system is chosen to represent the factors that associate with the type of measurement as well as their interrelationships, and then make decisions on the category of the measurement. The retrieval of instrument information is aided by a preprocessor based on the natural language processing (NLP) technique. The output of NLP is used to match with an entry in the instrument dictionary using a fuzzy rule-based system to determine the instrument type. A software package based on the algorithms presented in this paper has been developed using a single programming language; the package has been deployed for real-world applications.