The magnetic circular dichroism and the Faraday rotation are the fundamental phenomena of great practical importance arising from the breaking of the time reversal symmetry by a magnetic field. In most materials, the strength and the sign of these effects can be only controlled by the field value and its orientation. Furthermore, the terahertz range is lacking materials having the ability to affect the polarization state of the light in a non-reciprocal manner. Here we demonstrate, using broadband terahertz magneto-electro-optical spectroscopy, that in graphene both the magnetic circular dichroism and the Faraday rotation can be modulated in intensity, tuned in frequency and, importantly, inverted using only electrostatic doping at a fixed magnetic field. In addition, we observe strong magneto-plasmonic resonances in a patterned array of graphene antidots, which potentially allows exploiting these magneto-optical phenomena in a broad THz range.