Stationary policies with Markov partition property

This paper treats the infinite horizon discounted cost control problem for partially observable Markov decision processes (POMDPs). Sondik [14] studied the class of finitely transient policies and showed that their value functions over an infinite time horizon are piecewise linear (p.w.l) and can be computed exactly by solving a system of linear equations. However the condition for finite transience is stronger than is needed to ensure p.w.l value functions. In this paper we introduce alternatively the class of periodic policies whose value functions turn out to be also p.w.l. Moreover we examine a more general condition than finite transience and periodicity. We implement these ideas in a replacement problem under Markovian deterioration, investigate for periodic policies and give numerical examples.