Colorectal cancer is a disease with a very high mortality rate, making timely diagnosis and prevention a crucial task. In clinical practice, the detection of intestinal polyps presents challenges due to their small size, complex background, and large span in target scale, making it difficult to accurately detect small targets. Existing detectors struggle to adaptively extract discriminative features, and their detection heads lack specificity, leading to difficulties in accurately identifying polyps against similar intestinal wall backgrounds. To address this issue, we propose a Bi-level Routing Attention and Asymmetric Dual-Head based framework for intestinal polyp detection(BRALADH). This approach enhances the detection capabilities of the branches through strategies such as global information fusion and adaptive sample selection. It dynamically sets thresholds for each target, ultimately solving the problems of missed and false detections caused by high similarity between the foreground and background and large variations in target scale, thus improving detection accuracy. Qualitative and quantitative experiments on three datasets demonstrate that this method outperforms existing ones, showing strong stability and generalization performance.