Domain decomposition methods for large Markov chain control problems and nonlinear elliptic-type equations

This paper deals with domain decomposition methods for the numerical solution of two classes of related problems. The first class consists of Markov chain control problems on large state spaces. The second class can roughly be characterized as the possibly degenerate nonlinear elliptic partial differential, partial differential integral equations, or variational inequalities, which arise (perhaps only formally) in optimal stochastic control problems with diffusion-, jump-diffusion-, or reflected-diffusion-type models. At least in a formal sense, the optimal cost functions are presumed to satisfy such PDEs. The two problems are connected since an effective solution procedure for the first class involves approximations by the solutions of controlled or uncontrolled Markov chain problems. This paper concerns methods for such chains. Whether or not the PDE has only a formal meaning, the solutions to the approximating Markov chain problems converge to the desired cast Functional. Owing to many nonstandard features in such problems and to the nonlinearity of the equations which are to be solved, the decomposition problems for the Markov chain control problems often require special methods. We prove that appropriate adaptations of current decomposition techniques converge under conditions which are typical of many classes of applications, Probabilistic interpretations of the various algorithms in terms of functionals of Markov chains (controlled or not) are used heavily to simplify the analysis, point the way to other applications, and to provide a good intuitive understanding. An important application is to problems in four or more dimensions where, without decomposition, the size and structure of the state space and associated data structures often lead to very poor performance on cache machines.