We investigate from a modelling perspective how lexical structure can be grounded in the underlying speech and visual categories that infants have already acquired. We demonstrate that the formation of well-structured categories is an important prerequisite for successful generalisation of cross-modal associations such that even after a single presentation of a word-object pair, the model is able to generalise to other members of the category. This ability to generalise a label to objects of like kinds, commonly referred to as the taxonomic assumption, is an emergent property of the model and provides an explanatory framework for understanding aspects of infant word learning. Furthermore, we investigate the impact of constraints imposed on the Hebbian associations in the cross-modal training phase and identify the conditions under which generalisation does not take place.