We present the anisotropic optical conductivity of MgB2 between 0.1 and 3.7eV at room temperature obtained on single crystals of different purity by spectroscopic ellipsometry and reflectance measurements. The bare (unscreened) plasma frequency ωp is almost isotropic and equal to 6.3eV, which contrasts some earlier reports of a very small value of ωp. The data suggests that the σ bands are characterized by a stronger electron-phonon coupling λtr but smaller impurity scattering γimp, compared to the π bands. The optical response along the boron planes is marked by an intense interband transition at 2.6eV, due to which the reflectivity plasma edges along the a and c axes are shifted with respect to each other. As a result, the sample spectacularly changes color from a blueish-silver to the yellow as the polarization is rotated from the in-plane direction toward the c axis. The optical spectra are in good agreement with the published ab initio calculations. The remaining discrepancies can be explained by the relative shift of σ bands and π bands by about 0.2eV compared to the theoretical band structure, in agreement with the de Haas-van Alphen experiments. The widths of the Drude and the interband peaks are both very sensitive to the sample purity.