A Sensitivity-Based Construction Approach to Sample-Path Variance Minimization of Markov Decision Processes

We study the limiting average variance along the sample path as the secondary criterion for Markov decision processes, with the long-run average performance as the primary criterion. By applying the sensitivity-based approach, we intuitively construct the difference formula for the sample path variance under different policies. Thereby, a sufficient condition for the sample-path variance optimality can be easily derived. This work extends the sensitivity-based construction approach to the Markov decision processes with the nonstandard performance criterion. Compared with the pure mathematical verification, the sensitivity-based construction approach shows more intuition and provides insights on the sample-path structure of Markov decision processes.