Large ancestral effective population size explains the difficult phylogenetic placement of owl monkeys.

The phylogenetic position of owl monkeys, grouped in the genus Aotus, has been a controversial issue for understanding Neotropical primate evolution. Explanations of the difficult phylogenetic assignment of owl monkeys have been elusive, frequently relying on insufficient data (stochastic error) or scenarios of rapid speciation (adaptive radiation) events. Using a coalescent-based approach, we explored the population-level mechanisms likely explaining these topological discrepancies. We examined the topological variance of 2,192 orthologous genes shared between representatives of the three major Cebidae lineages and the outgroup. By employing a methodological framework that allows for reticulated tree topologies, our analysis explicitly tested for non-dichotomous evolutionary processes impacting the finding of the position of owl monkeys in the cebid phylogeny. Our findings indicated that Aotus is a sister lineage of the callitrichines. Most gene trees (>50%) failed to recover the species tree topology, although the distribution of gene trees mismatching the true species topology followed the standard expectation of the multispecies coalescent without reticulation. We showed that the large effective population size of the common ancestor of Aotus and callitrichines was the most likely factor responsible for generating phylogenetic uncertainty. On the other hand, fast speciation scenarios or introgression played minor roles. We propose that the difficult phylogenetic placement of Aotus is explained by population-level processes associated with the large ancestral effective size. These results shed light on the biogeography of the early cebid diversification in the Miocene, highlighting the relevance of evaluating phylogenetic relationships employing population-aware approaches.