A search for lepton-flavor-violating $Z\to e\tau$ and $Z\to\mu\tau$ decays with $pp$ collision data recorded by the ATLAS detector at the LHC is presented. This analysis uses 139 fb$^{-1}$ of Run 2 $pp$ collisions at $\sqrt{s}=13$ TeV and is combined with the results of a similar ATLAS search in the final state in which the $\tau$-lepton decays hadronically, using the same data set as well as Run 1 data. The addition of leptonically decaying $\tau$-leptons significantly improves the sensitivity reach for $Z\to\ell\tau$ decays. The $Z\to\ell\tau$ branching fractions are constrained in this analysis to $\mathcal{B}(Z\to e\tau)<7.0\times10^{-6}$ and $\mathcal{B}(Z\to \mu\tau)<7.2\times10^{-6}$ at 95% confidence level. The combination with the previously published analyses sets the strongest constraints to date: $\mathcal{B}(Z\to e\tau)<5.0\times10^{-6}$ and $\mathcal{B}(Z\to \mu\tau)<6.5\times10^{-6}$ at 95% confidence level.