Learning Parameterized Prescription Policies and Disease Progression Dynamics using Markov Decision Processes

We develop an algorithm for learning physicians' prescription policies and the disease progression dynamics from Electronic Health Record (EHR) data. The prescription protocol used by physicians is viewed as a control policy which is a function of an underlying disease state in a Markov Decision Process (MDP) framework. We assume that the transition probabilities and the policy of the MDP are parameterized using some known features, such that only a small portion of them are informative. Two l(1)-regularized maximum likelihood estimation problems are formulated to learn the transition probabilities and the policy, respectively. A bound is established on the difference between the average reward of the estimated policy under the estimated transition dynamics and the original (unknown) policy under the true transition dynamics. Our result suggests that by using only a relatively small number of training samples, the estimate can achieve a low regret. We validate our theoretical results on a test MDP motivated by a disease treatment identification application.