Reinforcement Learning Based Markov Edge Decoupled Fusion Network for Fusion Classification of Hyperspectral and LiDAR

Hyperspectral images (HSIs) and light detection and ranging (LiDAR) are two critical and frequently used types of remote sensing data, each containing rich spectral and elevation information. Fusing HSI and LiDAR can exploit the complementary properties of the two modalities for ground object classification. The performance of existing fusion classification methods is often limited by the difficulty of adapting feature extraction operators to complex spatial distributions, and the correlation and specificity between different modalities are not reasonably exploited. Therefore, the reinforcement learning-based markov edge decoupled fusion network (MEDFN) is proposed. This network can intelligently compose graphs based on different modal characteristics and tasks to adapt to complex spatial distributions; it can also suppress noise to complete fusion classification while fully utilizing complementary information of different modalities. First, a reinforcement learning-based graph construction subnetwork (RLGN) is proposed to learn a two-modal graph construction strategy suitable for classification tasks by transforming regular multimodal data into irregular graph data. Second, a multimodal edge attention module (MEAM) is proposed to extract edge features between spatial neighboring nodes and model the importance of each node, thereby capturing the spatial topology information encompassed in the multimodal data. Finally, the decoupled multimodal fusion module (DMFM) is proposed to decouple multimodal features into shared and unshared parts and enhance the model's ability to distinguish features by targeting the modal-shared feature between modalities and modal-specific feature. The experimental results based on three well-known HSI and LiDAR datasets demonstrate the effectiveness of the proposed MEDFN in fusion classification tasks.