Small object detection in remote sensing faces significant challenges, such as their offset sensitivity caused by the small area coverage, the dim targets in images, and their vulnerability to complex backgrounds, which often result in missed detections and false alarms. In this work, we propose a noise-adaptive context-aware detector (NACAD) to alleviate the above problems, which mainly consists of a region proposal network (RPN) with noise-adaptive module (NAM), a context-aware module (CAM), and a position-refined module (PRM). The main contributions are threefold. First, we leverage the information around small objects as positive-incentive noise (also known as pi-noise); through enlarging the range of small objects by NAM, more anchors of them are preserved as positive samples, thus stimulating the model to detect small objects. Second, the CAM is designed to provide multiple observation perspectives and abundant contextual representations for the enhancement of object features. Third, to reduce the interference of pure noise in the complicated backgrounds around small objects, the spatial calibration along two coordinate axes is devised by PRM to optimally use information beyond object regions. The effectiveness of our proposed detector, particularly on small objects, has been validated by the experiments on two public datasets, ITCVD and HRRSD. In particular, the NAM improves the recall of small objects, CAM enhances small object features, and PRM helps address the pure noise in complicated backgrounds around small objects.