Ligand exchange reactions on size-selected Au38(2-PET)24 and Au40(2-PET)24 clusters (2-PET: 2-phenylethylthiol) with mono- and bidentate chiral thiols was performed. The reactions were monitored with MALDI mass spectrometry and the arising chiroptical properties were compared to the number of incorporated chiral ligands. Only a small fraction of chiral ligands is needed to induce significant optical activity to the clusters. The use of monodentate camphor-10-thiol (CamSH) leads to comparably fast exchange on both clusters. The arising optical activity is weak. In contrast, the use of bidentate 1,1'-binaphthyl-2,2'-dithiol (BINAS) is slow, but the optical activity measured is strong. Moreover, a non-linear behaviour between optical activity and number of chiral ligands is found in the BINAS case for both Au38 and Au40, which may indicate different exchange rates of enantiopure BINAS with the enantiomers of inherently chiral clusters. This is ascribed to effects arising from the bidentate nature of BINAS. This is the first study where chiroptical effects are directly correlated with the composition of the ligand shell.