The evolutionary origin of Primates' exceptionally large brains is still highly debated. Two competing explanations have received much support: the ecological hypothesis and the social brain hypothesis (SBH). We tested the SBH in (n = 82) baboons (Papio anubis) belonging to the same research centre but housed in groups with size ranging from 2 to 63 individuals. We found that baboons living in larger social groups had larger brains. This effect was driven mainly by white matter volume and to a lesser extent by grey matter volume but not by the cerebrospinal fluid. In comparison, the size of the enclosure, an ecological variable, had no such effect. In contrast to the current re-emphasis on potential ecological drivers of primate brain evolution, the present study provides renewed support for the social brain hypothesis and suggests that the social brain plastically responds to group size. Many factors may well influence brain size, yet accumulating evidence suggests that the complexity of social life might be an important determinant of brain size in primates.