Toward Active Sequential Hypothesis Testing with Uncertain Models

This work introduces an active learning approach for hypothesis testing with uncertain likelihood models. Uncertain models appear when hypotheses' parameters are built from finite and limited training data. As a result, hypothesis testing performance is limited by the dearth of training data even as the number of observations increases asymptotically. Even with large amounts of observational data, decision-making at desired error rates can be impossible. This work proposes various active learning methods to collect as little additional training data as possible and still guarantee desired error bounds. These methods attempt to reduce the amount of observational and training data required sequentially and adaptively for each hypothesis until only one hypothesis is accepted. Finally, various active learning methods are compared in terms of their data collection costs to achieve the desired false rejection rate through simulations.