The three-pulse photon echo is a well-known technique to store intense light pulses in an inhomogeneously broadened atomic ensemble. This protocol is attractive because it is relatively simple and it is well suited for the storage of multiple temporal modes. Furthermore, it offers very long storage times, greater than the phase relaxation time. Here, we consider the three-pulse photon echo in both two- and three-level systems as a potential technique for the storage of light at the single-photon level. By explicit calculations, we show that the ratio between the echo signal corresponding to a single-photon input and the noise is smaller than one. This severely limits the achievable fidelity of the quantum state storage, making the three-pulse photon echo unsuitable for single-photon quantum memory.