While pulsar timing array experiments have recently found evidence for the existence of astochastic gravitational wave (GW) background, its origin is still unclear. If this background isof astrophysical origin, we expect the distribution of GW sources to follow the one ofgalaxies. Since galaxies are not perfectly isotropically distributed at large scales, but followthe cosmological large-scale structure, this would lead to an intrinsic anisotropy in thedistribution of GW sources. In this work, we develop a formalism to account for this anisotropy,by considering a Gaussian ensemble of sources in each realization of the universe and then takingensemble averages over all such realizations. We find that large-scale galaxy clustering has noimpact on the expectation value of pulsar timing residual correlations, described by theHellings-Downs curve. However, it introduces a new contribution to the variance of theHellings-Downs correlation. Hence, the anisotropic distribution of sources contributes to theamount by which the measurements of pulsar timing residual correlations, in our single realizationof the universe, may differ from the Hellings-Downs curve.