Background Pathological accumulation of cholesterol in late endosomes is observed in lysosomal storage diseases such as Niemann-Pick type C. We here analyzed the effects of cholesterol accumulation in NPC cells, or as phenocopied by the drug U18666A, on late endosomes membrane organization and dynamics. Methodology/Principal Findings Cholesterol accumulation did not lead to an increase in the raft to non-raft membrane ratio as anticipated. Strikingly, we observed a 2–3 fold increase in the size of the compartment. Most importantly, properties and dynamics of late endosomal intralumenal vesicles were altered as revealed by reduced late endosomal vacuolation induced by the mutant pore-forming toxin ASSP, reduced intoxication by the anthrax lethal toxin and inhibition of infection by the Vesicular Stomatitis Virus. Conclusions/Significance These results suggest that back fusion of intralumenal vesicles with the limiting membrane of late endosomes is dramatically perturbed upon cholesterol accumulation.