The observation by scanning tunnelling spectroscopy (STS) of Abrikosov vortex cores in the high-temperature superconductor YBa$_2$Cu$_3$O$_{7-delta}$ (Y123) has revealed a robust pair of electron-hole symmetric states at finite subgap energy. Their interpretation remains an open question because theory predicts a different signature in the vortex cores, characterised by a strong zero-bias conductance peak. We present STS data on very homogeneous Y123 at 0.4~K revealing that the subgap features do not belong to vortices: they are actually observed everywhere along the surface with high spatial and energy reproducibility, even in the absence of magnetic field. Detailed analysis and modelling show that these states remain unpaired in the superconducting phase and belong to an incoherent channel which contributes to the tunnelling signal in parallel with the superconducting density of states.