Inductive Bias Integration for Transformer Enhancement in Small-scale Segmentation Tasks

Transformers have made substantial contributions to various computer vision tasks, leveraging their attention mechanisms. However, their performance tends to be deficient on small-scale datasets. This shortfall primarily arises because Transformers lack the inherent inductive bias that forms a cornerstone of Convolutional Neural Networks (CNNs). To address this, we propose an innovative model called ConTransNet, which encapsulates a Convolution-Transformer synergy for enhanced learning in data-limited environments. A trained CNN is used to impart feature knowledge to a Transformer, which significantly enhances the Transformer's performance. Our strategy includes a unique positional localization technique designed for optimal feature transfer. Compared to pure Transformer models, our method increased mIoU by 10.3% on the half Cityscapes dataset. This finding highlights our method's effectiveness in enhancing Transformer performance on small-scale datasets and point towards potential advancements through blending CNNs and Transformers for such scenarios.