The thiolate-for-thiolate ligand exchange was performed on well-defined gold nanoparticles under an inert atmosphere without any modification of the core size. This reaction is faster than the well-known core etching. Surprisingly, if a chiral thiol is exchanged for its opposite enantiomer, the optical activity in the metal-based electronic transitions is reversed although the form of the CD spectra remains largely unchanged. The extent of inversion corresponds to the overall ee of the chiral ligand in the system. This shows that the chiral arrangement of metal atoms in the metal particle (surface) can not withstand the driving force imposed by the ligand of opposite absolute configuration. If the incoming thiol has a different structure, the electronic transitions in the metal core are slightly modified whereas the absorption onset remains unchanged. These results emphasize the influence of the thiols on the structure of the gold nanoparticles and give insight on the ligand exchange pathways.