Run-Time Correlation Engine for System Monitoring and Testing

Today's enterprise applications can produce vast amounts of information both during system testing and in production. Correlation of this information can be difficult as it is generally stored in a range of different event logs, the format of which can be application or vendor specific. Furthermore these large logs can be physically distributed across a number of different locations. As a result it can be difficult to form a coherent understanding of the overall system behaviour. This has implications for a number of domains (e.g. autonomic computing, system testers), where an understanding of the system behaviour at run-time is required (e.g. for problem determination, autonomic management etc.) This paper presents an approach and implementation of run-time correlation of large volumes of log data and symptom matching of known issues in the context of large enterprise applications. Our solution provides for automatic data collection, data normalisation into a common format, run-time correlation and analysis of the data to give a coherent view of system behaviour at run-time and a symptom matching mechanism that can identify known errors in the correlated data on the fly.