Transient and Average Markov Reward Chains with Applications to Finance

The article is devoted to Markov reward chains with finite state space. Since the usual optimization criteria examined in the literature on Markov reward chains, such as a total discounted, total reward up to reaching some specific state (called transient models) or mean (average) reward optimality, may be quite insufficient to characterize the problem from the point of a decision maker. It seems that it may be preferable if not necessary to select more sophisticated criteria that also reflect variability-risk features of the problem. Perhaps the best known approaches stem from the classical work of Markowitz on mean variance selection rules, i.e. we optimize the weighted sum of average or total reward and its variance. In the article explicit formulae for calculating the variances for transient and average models are reported along with sketches of algorithmic procedures for finding policies guaranteeing minimal variance in the class of policies with a given transient or average reward. Application of the obtained results to financial models is indicated.