Enumeration of coalescent histories for caterpillar species trees and p-pseudo caterpillar gene trees

For a fixed set X containing n taxon labels, an ordered pair consisting of a gene tree topology G and a species tree topology S bijectively labeled with the labels of X possesses a set of coalescent histories-mappings from the set of internal nodes of G to the set of edges of S describing possible lists of edges in S on which the coalescences in G take place. Enumerations of coalescent histories for gene trees and species trees have produced suggestive results regarding the pairs (G, S) that, for a fixed n, have the largest number of coalescent histories. We define a class of 2-cherry binary tree topologies that we term p-pseudocaterpillars, examining coalescent histories for non-matching pairs (G, S) in the case in which S has a caterpillar shape and G has a ppseudo caterpillar shape. Using a construction that associates coalescent histories for (G, S) with a class of "roadblocked" monotonic paths, we identify the p-pseudo caterpillar labeled gene tree topology that, for a fixed caterpillar labeled species tree topology, gives rise to the largest number of coalescent histories. The shape that maximizes the number of coalescent histories places the "second" cherry of the p-pseudo caterpillar equidistantly from the root of the "first" cherry and from the tree root. A symmetry in the numbers of coalescent histories for p-pseudo caterpillar gene trees and caterpillar species trees is seen to exist around the maximizing value of the parameter p. The results provide insight into the factors that influence the number of coalescent histories possible for a given gene tree and species tree. (c) 2021 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).