Learning and aggregating principal semantics for semantic edge detection in images

Learning features that contain both local variations and non-local semantics is crucial yet challenging for Semantic Edge Detection (SED), which involves the joint localization and categorization of edges in images. Directly fusing multi-stage detail-to-abstract pixel features from encoders as done in existing SED methods leads to feature representation with either blurred details or limited semantics. In this paper, we present a novel Transformer-based framework for SED, named SEDTR. Our approach employs an efficient semantic aggregation strategy achieved through the design of the Principal Semantics Learning (PSL) module and the Principal Semantics Aggregation (PSA) module. Specifically, the PSL module distills concise principal semantics from high-level features. Subsequently, the multi-stage pixel features are spatially tuned by aggregating semantic clues from the learned principal representations in PSA. Benefiting from the concise principal representation, noise is effectively suppressed while edge points are distinctly highlighted. Additionally, we enhance the pixel features with cross-level complements before PSA to facilitate the semantic aggregation. The PSA-tuned multistage features are summed to form the final features for SED. Extensive comparative experiments conducted on the SBD, Cityscapes and ADE20K datasets demonstrate that the proposed model outperforms existing approaches in both edge localization and categorization.