Kernel estimation in transect sampling without the shoulder condition

We consider the estimation of wildlife population density based on line transect data. Nonparametric kernel method is employed, without the usual assumption that the detection curve has a shoulder at distance se ro, with the help of a special class of kernels called boundary kernels. Asymptotic distribution results are included. It is pointed out that the boundary kernel of Zhang and Karunamuni (1998) (see also Muller and Wang (1994)) performs better (for asymptotic mean square error consideration) than that of the boundary kernel of Muller (1991). But both of these kernels are clearly superior to the half-normal and one-sided Epanechnikov kernel when the shoulder condition fails to hold. In practice, however, for small to moderate sample sizes, caution should be exercised in using boundary kernels in that the density estimate might become negative. A Monte Carlo study is also presented, comparing the performance of four kernels applied to detection data, with and without the shoulder condition. Two boundary kernels for derivatives are also included for the point transect case.