Quantum chaotic systems are often defined via the assertion that their spectral statistics coincides with, or is well approximated by, random matrix theory. In this paper we explain how the universal content of random matrix theory emerges as the consequence of a simple symmetry-breaking principle and its associated Goldstone modes. This allows us to write down an effective-field theory (EFT) description of quantum chaotic systems, which is able to control the level statistics up to an accuracy ${\cal O} \left(e^{-S} \right)$ with $S$ the entropy. We explain how the EFT description emerges from explicit ensembles, using the example of a matrix model with arbitrary invariant potential, but also when and how it applies to individual quantum systems, without reference to an ensemble. Within AdS/CFT this gives a general framework to express correlations between "different universes" and we explicitly demonstrate the bulk realization of the EFT in minimal string theory where the Goldstone modes are bound states of strings stretching between bulk spectral branes. We discuss the construction of the EFT of quantum chaos also in higher dimensional field theories, as applicable for example for higher-dimensional AdS/CFT dual pairs.