Recurrent neural network based real-time failure detection of storage devices

Studies have revealed that the failure rates of storage devices can often be as high as fourteen percent. To make matters worse, there are frequently no warning signs for precaution before catastrophic failure of storage devices occurs. A real-time predictive maintenance system that provides an automatic means for predicting when maintenance should be performed to ultimately eliminate unexpected breakdowns needs to be developed. Unlike traditional regression predictive modeling, the failure detection of storage devices is a problem of time series prediction, which adds the complexity of a sequence dependence among the input variables. The proposed LSTM (Long Short-Term Memory) network is a branch of RNN (Recurrent Neural Network) used in deep learning, which presents a very large architecture that can be successfully trained. LSTM is good at extracting patterns in input feature space, where the input data spans over long sequences. With the gated architecture of LSTM, it is capable of learning the context required to make predictions in time series forecasting. It is ideal for generating responses that depend on a time-evolving state; for example detecting the condition of storage devices over time. This paper describes our development of an LSTM (Long short-term memory), a special kind of RNN (Recurrent Neural Network)—based real-time predictive maintenance system (RPMS) built on top of Apache Spark for detecting storage device failure. By streaming real-time data into a RPMS directly from the device itself, the issues can be revealed and addressed early before they cause costly downtime.