Short sleep is held to cause poorer brain health, but is short sleep associated with higher rates of brain structural decline? Analysing 8,153 longitudinal MRIs from 3,893 healthy adults, we found no evidence for an association between sleep duration and brain atrophy. In contrast, cross-sectional analyses (51,295 observations) showed inverse U-shaped relationships, where a duration of 6.5 (95% confidence interval, (5.7, 7.3)) hours was associated with the thickest cortex and largest volumes relative to intracranial volume. This fits converging evidence from research on mortality, health and cognition that points to roughly seven hours being associated with good health. Genome-wide association analyses suggested that genes associated with longer sleep for below-average sleepers were linked to shorter sleep for above-average sleepers. Mendelian randomization did not yield evidence for causal impacts of sleep on brain structure. The combined results challenge the notion that habitual short sleep causes brain atrophy, suggesting that normal brains promote adequate sleep duration-which is shorter than current recommendations.