A search for leptons displaced from the primary vertex is performed with the ATLAS detector at the Large Hadron Collider. The search includes the full proton-proton collision dataset collected during Run 2 at $\sqrt{s}=13$ TeV and a partial dataset collected during Run 3 in 2022-2023 at $\sqrt{s}=13.6$ TeV, corresponding to integrated luminosities of 140 fb$^{-1}$ and 56.3 fb$^{-1}$, respectively. Final states with displaced electrons or muons are considered, and novel triggers introduced in Run 3 are employed that use large impact parameter tracking to reconstruct displaced tracks with low momentum. In addition, photon reconstruction and multivariate techniques are employed to broaden the sensitivity to channels with large background rates or highly displaced electrons, respectively. The results are consistent with the Standard Model background expectations and are used to set model-independent limits on the production of displaced electrons and muons. The analysis is also interpreted in the context of a gauge-mediated supersymmetry breaking model with pair-produced long-lived sleptons, and a dark sector model with pair-produced chargino-like states. The results include 95% CL exclusions of selectrons with lifetimes from 4 ps to 60 ns and a mass of 150 GeV, and exclusions of selectrons, smuons, and staus with a lifetime of 0.3 ns for masses up to 740 GeV, 830 GeV, and 440 GeV, respectively. Dark charginos with masses up to 380 GeV are excluded for a mass difference with the neutral state of 40 GeV, and mass differences down to 17 GeV are excluded for dark charginos with a 100 GeV mass.