The optical conductivity of MgB2 has been determined on a dense polycrystalline sample in the spectral range 6 meV–4.6 eV using a combination of ellipsometric and normal incidence reflectivity measurements. σ1(ω) features a narrow Drude peak with anomalously small plasma frequency (1.4 eV) and a very broad ‘dome' structure, which comprises the bulk of the low-energy spectral weight. This fact can be reconciled with the results of band structure calculations by assuming that charge carriers from the 2D σ-bands and the 3D π-bands have principally different impurity scattering rates and negligible interband scattering. This also explains a surprisingly small correlation between the defect concentration and Tc, expected for a two-gap superconductor. The large 3D carrier scattering rate suggests their proximity to the localization limit.