We report a detailed study of the noise properties of a visible-to-telecom photon frequency converter based on difference frequency generation (DFG). The device converts 580 nm photons to 1541 nm using a strong pump laser at 930 nm, in a periodically poled lithium niobate ridge waveguide. The converter reaches a maximum device efficiency of 46 % (internal efficiency of 67%) at a pump power of 250 mW. The noise produced by the pump laser is investigated in detail by recording the noise spectra both in the telecom and visible regimes and measuring the power dependence of the noise rates. The noise spectrum in the telecom is very broadband, as expected from previous work on similar DFG converters. However, we also observe several narrow dips in the telecom spectrum, with corresponding peaks appearing in the 580 nm noise spectrum. These features are explained by sum frequency generation of the telecom noise at wavelengths given by the phase-matching condition of different spatial modes in the waveguide. The proposed noise model is in good agreement with all the measured data, including the power dependence of the noise rates, both in the visible and telecom regimes. These results are applicable to the class of DFG converters where the pump laser wavelength is in between the input and target wavelength.