Motivated by neutron scattering experiments, we investigate the decay of the fidelity with which a wave packet is reconstructed by a perfect time-reversal operation performed after a phase-space displacement. In the semiclassical limit, we show that the decay rate is generically given by the Lyapunov exponent of the classical dynamics. For small displacements, we additionally show that, following a short-time Lyapunov decay, the decay freezes well above the ergodic value because of quantum effects. Our analytical results are corroborated by numerical simulations.