Mode-matching control policies for multi-mode Markov decision processes

We consider a Markov decision process (MDP) with a two-dimensional state vector, (S, D), where S is interpreted as the system state, and D is interpreted as the probability distribution of the system operating mode, denoted by M. The mode M determines the probability transition and reward structures for S. If the mode were known and constant, a constant-mode optimal controller for controlling the evolution of S could be computed off line. We are interested in knowing if and when the set of constant-mode optimal controllers can be used to control the system effectively when the mode evolves stochastically. We propose a mode-matching control policy under which the controller applied to the system at each epoch is the constant-mode optimal controller for the current most likely mode. We consider the case when the current mode is directly observable that is, D is the trivial distribution as well as the case when only the probability distribution of the current mode is available at each control epoch. Sufficient conditions under which the mode-matching control policies are optimal are derived. We also derive bounds on the performance degradation from the optimum when the non-optimal mode-matching control policies are used. The problem formulation, sufficient conditions and performance bounds are illustrated by a numerical example.