Compulsive drug use despite adverse consequences defines addiction. While mesolimbic dopamine signaling is sufficient to drive compulsion, psychostimulants such as cocaine also boost extracellular serotonin (5-HT) by inhibiting reuptake. We used SERT Met172 knockin (SertKI) mice carrying a transporter that no longer binds cocaine to abolish 5-HT transients during drug self-administration. SertKI mice showed an enhanced transition to compulsion. Conversely, pharmacologically elevating 5-HT reversed the inherently high rate of compulsion transition with optogenetic dopamine self-stimulation. The bidirectional effect on behavior is explained by presynaptic depression of orbitofrontal cortex–to–dorsal striatum synapses induced by 5-HT via 5-HT_1B receptors. Consequently, in projection-specific 5-HT_1B receptor knockout mice, the fraction of individuals compulsively self-administering cocaine was elevated.