Ecological Specialization and Evolutionary Reticulation in Extant Hyaenidae

During the Miocene, Hyaenidae was a highly diverse family of Carnivora that has since been severely reduced to four species: the bone-cracking spotted, striped, and brown hyenas, and the specialized insectivorous aardwolf. Previous studies investigated the evolutionary histories of the spotted and brown hyenas, but little is known about the remaining two species. Moreover, the genomic underpinnings of scavenging and insectivory, defining traits of the extant species, remain elusive. Here, we generated an aardwolf genome and analyzed it together with the remaining three species to reveal their evolutionary relationships, genomic underpinnings of their scavenging and insectivorous lifestyles, and their respective genetic diversities and demographic histories. High levels of phylogenetic discordance suggest gene flow between the aardwolf lineage and the ancestral brown/striped hyena lineage. Genes related to immunity and digestion in the bone-cracking hyenas and craniofacial development in the aardwolf showed the strongest signals of selection, suggesting putative key adaptations to carrion and termite feeding, respectively. A family-wide expansion in olfactory receptor genes suggests that an acute sense of smell was a key early adaptation. Finally, we report very low levels of genetic diversity within the brown and striped hyenas despite no signs of inbreeding, putatively linked to their similarly slow decline in effective population size over the last similar to 2 million years. High levels of genetic diversity and more stable population sizes through time are seen in the spotted hyena and aardwolf. Taken together, our findings highlight how ecological specialization can impact the evolutionary history, demographics, and adaptive genetic changes of an evolutionary lineage.