We provide a theoretical explanation for two recent experiments on decoherence of Aharonov-Bohm oscillations in two- and multi-terminal ballistic rings. We consider decoherence due to charge fluctuations and emphasize the role of charge exchange between the system and the reservoir or nearby gates. A time-dependent scattering matrix approach is shown to be a convenient tool for the discussion of decoherence in ballistic conductors.