We provide a detailed derivation of the spectral density of the stochastic background generated by the superposition of coalescing compact binaries. We show how the expression often used in the literature emerges from an average over the extrinsic parameters of the binaries (times of arrival, polarization angles, orbit inclinations and arrival directions) and how the Stokes parameters related to circular and linear polarization are set to zero by such averaging procedure. We then consider the effect of shot noise, i.e. the fact that for the superposition of a finite number of sourcesthese averages are only approximate, and we show how it generates circular and linear polarizations (even for isotropic backgrounds) as well as spatial anisotropies, and we compute them explicitly for a realistic population of binary black holes and binary neutron stars.