A Queueing Analysis of Multi-model Multi-input Machine Learning Systems

A multi-model multi-input machine learning system (MLS) is an architectural approach to improve the reliability of the MLS output by using multiple models and multiple sensor inputs. While the errors in MLS output can be reduced by redundancy with diversity, the performance overhead/gain caused by the employed architecture may also be concerned in safety-critical applications such as a self-driving car. In this paper, we proposed queueing models for analyzing a multi-model multi-input MLS performance in two architectures, namely a parallel MLS and a shared MLS. The parallel MLS architecture runs two different machine learning models in parallel, while the shared MLS architecture runs a single machine learning model but uses two different sensor inputs. We model the behavior of the parallel MLS by a quasi-birth-death process. On the other hand, we model dynamics of the shared MLS as a continuous-time Markov chain of GUM/1 type. The numerical experiments on the proposed models show that the parallel MLS generally achieves better throughput performance than the shared MLS under the same parameter settings. We also show that the throughput performance of the shared MLS can be improved when the input data arrival rates are sufficiently high.