Real-time Grid monitoring based on complex event processing

Monitoring systems in Grid infrastructures typically collect and aggregate data originating in distributed agents and store it in global, periodically refreshed repositories. However, in some scenarios access to real-time streams of monitoring information, rather than persistent data sets, would be beneficial. In this paper, we evaluate the Complex Event Processing (CEP) approach applied to real-time Grid monitoring and argue that CEP enables us to achieve two goals, otherwise difficult to combine: real-time access to monitoring data and advanced query capabilities. Monitoring for the purpose of dynamic allocation of Grid resources serves as a case study to demonstrate powerful real-time query capabilities provided by CEP. In addition, we show how to employ CEP for data reduction. For practical verification of our solution, a CEP-based Grid monitoring infrastructure, GEMINI2, has been developed. We have measured the overhead due to CEP-based monitoring and conclude that real-time Grid monitoring is possible without excessive intrusiveness for resources and network. (C) 2011 Elsevier B.V. All rights reserved.