Over the last two decades, an emerging body of research has demonstrated that non-human animals exhibit the ability to combine context-specific calls into larger sequences. These structures have frequently been compared with language's syntax, whereby linguistic units are combined to form larger structures, and leveraged to argue that syntax might not be unique to language. Currently, however, the overwhelming majority of examples of call combinations are limited to simple sequences comprising just two calls which differ dramatically from the open-ended hierarchical structuring of the syntax found in language. We revisit this issue by taking a whole-repertoire approach to investigate combinatoriality in common marmosets (Callithrix jacchus ). We use Markov chain models to quantify the vocal sequences produced by marmosets providing evidence for structures beyond the bigram, including three-call and even combinations of up to eight or nine calls. Our analyses of these longer vocal sequences are suggestive of potential further internal organization, including some amount of recombination, nestedness and non-adjacent dependencies. We argue that data-driven, whole-repertoire analyses are fundamental to uncovering the combinatorial complexity of non-human animals and will further facilitate meaningful comparisons with language's combinatoriality.