Clusters of topologically connected calorimeter cells around cells with large absolute signal-to-noise ratio (topo-clusters) are the basis for calorimeter signal reconstruction in the ATLAS experiment. Topological cell clustering has proven performant in LHC Runs 1 and 2. It is, however, susceptible to out-of-time pile-up of signals from soft collisions outside the 25 ns proton-bunch-crossing window associated with the event’s hard collision. To reduce this effect, a calorimeter-cell timing criterion was added to the signal-to-noise ratio requirement in the clustering algorithm. Multiple versions of this criterion were tested by reconstructing hadronic signals in simulated events and Run 2 ATLAS data. The preferred version is found to reduce the out-of-time pile-up jet multiplicity by ${\sim }50\%$ for jet $p_{\textrm{T}}\sim 20$ GeV and by ${\sim }80\%$ for jet $p_{\textrm{T}} \gtrsim 50$ GeV, while not disrupting the reconstruction of hadronic signals of interest, and improving the jet energy resolution by up to 5% for $20< p_{\textrm{T}} < 30$ GeV. Pile-up is also suppressed for other physics objects based on topo-clusters (electrons, photons, $\tau $-leptons), reducing the overall event size on disk by about $6\%$ in early Run 3 pile-up conditions. Offline reconstruction for Run 3 includes the timing requirement.