On distances between phylogenetic trees

Different phylogenetic trees for the same group of species are often produced either by procedures that use diverse optimality criteria [18] or from different genes [12] in the study of molecular evolution. Comparing these trees to find their similarities(e.g. agreement or consensus) and dissimilarities, i.e. distance, is thus an important issue in computational molecular biology. The nearest neighbor interchange (nni) distance [25, 24, 32, 4, 5, 3, 16, 17, 19, 29, 20, 21, 23] and the subtree-transfer distance [12, 13, 15] are two major distance metrics that have been proposed and extensively studied for different reasons. Despite their many appealing aspects such as simplicity and sensitivity to tree topologies, computing these distances has remained very challenging. This article studies the complexity and efficient approximation algorithms for computing the nni distance and a natural extension of the subtree-transfer distance, called the linear-cost subtree-transfer distance. The linear-cost subtree-transfer model is more logical than the (unit-cost) subtree-transfer model and in fact coincides with the nni model under certain conditions. The following results have been obtained as part of our project of building a comprehensive software package for computing distances between phylogenies. 1. Computing the nni distance is NP-complete. This solves a 25 year old open question appearing again and again in, for example, [25, 32, 4, 5, 3, 16, 17, 19, 20, 21, 23] under the complexity-theoretic assumption of P not equal NP. We also answer an open question [4] regarding the nni distance between unlabeled trees for which an erroneous proof appeared in [19]. We give an algorithm to compute the optimal nni sequence in time O(n(2) log n+n . 2(O(d))), where the nni distance is at most d. The algorithm allows us to implement practical programs when d is small. All above results also hold for linear-cost subtree-transfer. 2. Biological applications require us to extend the nni and linear-cost subtree-transfer models to weighted phylogenies, where edge weights indicate the length of evolution along each edge. We present a logarithmic ratio approximation algorithm for nni and a ratio 2 approximation algorithm for linear-cost subtree-transfer, on weighted trees.