Low-temperature magnetic force microscopy was used to study the threshold of nucleation of spontaneous vortex-antivortex structures in superconductor/ferromagnet (S/F) hybrid systems. We investigated S/F heterostructures composed of Py as the magnetic material and Nb as the superconductor, with different thicknesses of Py and Nb. The condition for nucleation of spontaneous vortex-antivortex structures depends on fundamental parameters such as the superconducting penetration depth and the coherence length, as well as on the thickness of the superconducting film and the magnetic domain width. We compare our experimental results with those of existing theoretical models and provide an estimate of the threshold of the local out-of-plane component of the magnetization for different Py film thicknesses.