Action-dependent bidirectional contrastive predictive coding for neural belief representations

The key to solving the complex, partially observable Markov decision process (POMDP) with high dimensional observations lies in explicit belief representations. However, the existing methods generally adopted the black-box model for shaping beliefs, which is inefficient and lacks interpretability. Due to this reason, the action-dependent bidirectional contrastive predictive coding (BCPC|Action) is proposed in this paper, in which the observation features are extracted efficiently through self-supervised contrastive learning. Owing to the bottleneck belief constraints in the bidirectional model, the upper bound of prediction errors is effectively reduced. Besides, the forward prediction is optimized by the guidance of an easier trainable backward prediction; thus, the bidirectional match regularization (BMR) could be derived for stabilizing the training process. More importantly, the interpretability of the learned belief representation is thoroughly explored based on the gradient truncation. Simulation results verify the effectiveness of the presented method; apart from achieving highly accurate belief tracking, the state uncertainties could be characterized reasonably, which provides a guarantee for solving the POMDP optimal policy for downstream tasks.(c) 2022 Elsevier B.V. All rights reserved.