Semi-automatic mining of correlated data from a complex database: correlation network visualization

In previous work we have addressed the issue of frequent ad-hoc queries in deeply-structured databases. We wrote a library of functions AutodenormLib.py for issuing proper JOIN commands to denormalize an arbitrary subset of stored data for downstream processing. This may include statistical analysis, visualization or machine learning. Here, we visualize the content of the Thalamoss biomedical database as a correlation network. The network is created by calculating pairwise correlations through all pairs of variables, whether they be numerical, ordinal or nominal. We subsequently construct the network over the entire set of variables, clustering variables with similar effects to discover group relationships between the various biomedical characteristics. We use a semi-automatic procedure that makes the selection of all pairs possible and discuss issues of dealing with different types of variables. This is done either by limiting the analysis to numerical and ordinal ones, or by binning their values into intervals of values. Knowledge extracted from the data in this mode can be used to select variables for statistical models, or as markers of medically interesting conditions.