A statistical and signal processing based system for data quality management

Research in data quality is getting more important as databases in research centers and companies get larger. Therefore, developing new mechanisms to discover knowledge in large data bases is as urgent as finding ways to measure and assure the quality of the data. This work describes the development of a Quality Control System, intended to operate on dynamic and large data bases. Techniques used vary from standard statistical tests to signal processing methodology, such as filtering, wavelet transformation and neural processing. Data used for this work consist of asocial database and of time series for five years of stock data for companies present in the SP500 index. Generalization tests show that feed forward neural networks represent a suitable tool for tracking pre-processed (filtered) financial series, and can be used to define a corridor inside which one may consider new data as acceptable. For this data, we were also able to develop a model for the distribution of the differences between consecutive days, which can be combined to neural processing for data acceptation. Tests performed on the social data allowed us to identify probabilistic density functions for a set of variables, making it possible to create a objective test of data quality assessment.