Colloidal stability of polymeric latex particles was studied in the presence of oppositely charged layered double hydroxide (LDH) platelets of different interlayer anions. Adsorption of the LDH particles led to charge neutralization and to overcharging of the latex at appropriate concentrations. Mixing stable colloidal suspensions of individual particles results in rapid aggregation once the LDH adsorption neutralizes the negative charges of the polymer spheres, while stable suspensions were observed at high and low LDH doses. The governing interparticle interactions included repulsive electrical double layer forces as well as van der Waals and patch-charge attractions, whose strength depended on the amount of LDH particles adsorbed on the latex surface. The type of the LDH interlayer anions did not affect the colloidal stability of the samples. Structural investigation of the obtained latex–LDH composites revealed that the polymer spheres were completely coated with the inorganic platelets once their concentration was sufficiently high. These results are especially important for designing synthetic routes for hybrid systems in suspensions, where stable colloids are required for uniform film-formation and for the homogeneous distribution of the inorganic filler within the composite material.