A transformer-based neural ODE for dense prediction

Neural ordinary differential equations (ODEs) represent an emergent class of deep learning models exhibiting continuous depth. While they have shown promising results across various machine learning tasks, existing methods for dense prediction tasks have not fully harnessed their potential, often due to employing sub-optimal architectural components and limited dataset studies. To address this, our paper introduces a robust neural ODE architecture specifically tailored for dense prediction tasks and performs an extensive evaluation across a broad range of datasets. Our approach draws upon proven design elements from top-performing networks, integrating transformer blocks as core building blocks. Unique to our design is the retention of multiple concurrent representations at varying resolutions throughout the network. These representations continually exchange information, ensuring they remain updated. Our network achieves unrivaled performance in tasks such as image classification, semantic segmentation, and answer grounding. We conduct several ablation studies to shed light on the impacts of various design parameters. Our results affirm the effectiveness of our approach and its potential for further advancements in dense prediction tasks.