In this paper, a novel complex mask for the implementation of fractional differentiation is deduced. By the combination of fractional differentiation and integration, a new compound derivative is proposed, in which the fractional-order derivative mask is employed. The compound derivative, together with the complex mask, is applied to edge detection, forming a new edge detection operator. The performances of the compound derivative, in terms of detection effectiveness and noise immunity, are demonstrated through 1D examples. The 2D experimental results indicate that, without the contamination of noise, the new edge detection operator can accurately detect edges; while with the noise contamination, the new operator can effectively suppress noises. Finally, quantitative analysis demonstrates that the new operator can outperform Canny operator.