Interactive Machine Learning for the Internet of Things: A Case Study on Activity Detection

The advances in Internet of Things lead to an increased number of devices generating and streaming data. These devices can be useful data sources for Activity Recognition by using Machine Learning. However, as the set of available sensors may vary over time, e.g. due to mobility of the sensors and technical failures, the feature space might also change over time. Moreover, the labelled data necessary for the training is often costly to acquire. Active Learning is a type of Interactive Machine Learning where the model is given a budget for requesting labels from an oracle, and aims to maximize accuracy by careful selection of what data points to label. It is generally assumed that a query always gets a correct response, but in many real-world scenarios this is not a realistic assumption. In this work we investigate different Proactive Learning strategies, which explore the human factors of the oracle and aspects that might influence a user to provide or withhold labels. We implemented four proactive strategies and hybrid versions of them. They were evaluated on two datasets to examine how a more proactive, or reluctant, user affects performance. The results show that a more proactive user can improve the performance, especially when the user is influenced by the accuracy of earlier predictions. The experiments also highlight challenges related to evaluating performance when the set of classes is changing over time.