We present a novel measurement method for the relative group delay of chirped fiber Bragg gratings, based on the time-of-flight of single photons. In contrast to existing techniques, the light source is a pulsed broad-band LED. This simple and inexpensive source allows for measurements over a large spectral range with a high spectral resolution (8 pm) obtained from a tunable Fabry-Perot filter. The photons are detected by an InGaAs avalanche photodiode operated in photon-counting mode, resulting in a precision for the group delay of 15 ps. We observed an unusual asymmetry in the group delay curves of one of the commercial gratings measured with our method. This asymmetry can be qualitatively understood, taking into account a nonmonotonous variation of the grating period.