Effective Detection of Rare Anomalies from Massive Waveform Data Using Heterogeneous Clustering

Today's measurement instruments are capable of capturing and processing massive amount of waveform data. High sampling rate Analog to Digital Converters (ADCs) and low-cost storages make it relatively easy to collect "big measurement data" at massive scale. More and more measurement instrument users acquire tera-byte-scale waveform data which are essential for hard-to-find failure detection and prediction. However, conventional analysis techniques focus on small fragments of signals and largely lag behind today's test and measurement data assets' processing demands. Most of these techniques are inadequate for coping with the massive data volume and the complexities of the analysis tasks. A previous report by the authors introduced a heterogeneous waveform clustering framework to break the technical barriers. The present paper demonstrates the effectiveness of the proposed framework with real-world application examples at tera-byte data scale. The framework consists of the real-time tagging for pre-sorting incoming data, quick clustering for summarizing data overviews from long-duration recording, and detail clustering for deeper analyses. The tagging process is the critical performance link for satisfying the processing time and hardware constrains. We share theoretical analysis on the degree of freedom involved in the waveform and the tagging results. The data is pre-sorted into tag database with highly efficient retrieval characteristics, allowing the system to provide results quickly and flexibly. Three real-world waveform analysis examples are demonstrated, namely power line voltage, mechanical relay stick error, and Bluetooth device current consumption. Our framework allows efficient and robust exploration of complex signal signatures for detecting extremely rare anomalies. The detected anomaly patterns not only show straightforward engineering usages, but also demonstrate a predictive analysis power of related signal events.