Uncertainty-Aware Topological Persistence Guided Knowledge Distillation on Wearable Sensor Data

In applications involving analysis of the wearable sensor data, machine learning techniques that use features from the topological data analysis (TDA) have demonstrated remarkable performance. Persistence images (PIs) generated through TDA prove effective in capturing robust features, especially to signal perturbations, thus complementing classical time-series features. Despite its promising performance, utilizing TDA to create PI entails significant computational resources and time, posing challenges for applications on small devices. Knowledge distillation (KD) emerges as a solution to address these challenges, as it can produce a compact model. Using multiple teachers one trained with raw time-series and another with topological features, is a viable approach to distill a single compact student model. In such a case, the two teachers will have different statistical characteristics and need some form of feature harmonization. To tackle these issues, we propose uncertainty-aware topological persistence guided KD. This approach involves separating common and distinct components between the teachers and applying varying weights to control their effects. To enhance the knowledge provided to a student, uncertain features from the teachers are rectified using the uncertainty scores. We leverage feature similarities to offer more valuable information and employ relationships computed based on the orthogonal properties to prevent excessive feature transformation. Ultimately, our method yields a robust single student that operates solely on the time-series data at test time. We validate the effectiveness of the proposed approach through the empirical evaluations across various combinations of the models and data sets, demonstrating its robustness and efficacy in different scenarios. The proposed method enhances the classification performance of a student model by approximately 4.3% compared to a model learned from scratch on GENEActiv.