On the Structure of Decentralized Controllers in Networked MDPs

This work considers the problem of constructing optimal decentralized controllers for networked Markov Decision Processes (MDPs). A number of subsystems, each modeled as an MDP, are dynamically coupled through a network and affect one another with delay. Each subsystem has a corresponding controller which has perfect knowledge of the local state and may communicate its state to the other controllers over a noise-free channel with fixed delay. The problem is framed in terms of a coordinator with access to the shared information of the controllers, leading to a dynamic programming problem to calculate optimal policies. Further, under the criteria of partial nestedness the dynamic program exhibits additional factored structure, leading to more efficient synthesis of the optimal policy.