We analyse the generation of tensor metric perturbations by the anisotropic stresses of a primordial magnetic field. First we present the implication of causality on the initial spectrum of the magnetic field, and then derive strong limits on its amplitude by applying the nucleosynthesis bound on the gravitational waves (GW) induced by the field. Finally we discuss the CMB tensor spectrum generated by the magnetic field, presenting analytical approximations for the temperature anisotropies and polarisation signal by a possible helical component in the magnetic field.