A maximum-likelihood method for the estimation of pairwise relatedness in structured populations

A maxinum-likelihood estimator for pairwise relatedness is presented for the situation in which the individuals under consideration come from a large outbred subpopnlation of the population for which allele frequeticies are known. We demonstrate via simulations that a variety of commonly used estimators that do not take this kind of misspecification of allele frequencies into account will systematically overestimate the degree of relatedness between two individuals front a subpopulation. A maximum-likelihood estimator that includes as a parameter is introduced with the goal of producing the relatedness estimates that would have been obtained if the subpopulations allele frequencies had been known. This F-ST is misspecified. Bootstrap confidence estimator is shown to work quite well, even when the value of E intervals are also examined and shown to exhibit close to nominal coverage when F-ST is correctly specified.