Scientific article
Open access

Natural language instructions induce compositional generalization in networks of neurons

Published inNature neuroscience, vol. 27, no. 5, p. 988-999
Publication date2024-03-18
First online date2024-03-18

A fundamental human cognitive feat is to interpret linguistic instructions in order to perform novel tasks without explicit task experience. Yet, the neural computations that might be used to accomplish this remain poorly understood. We use advances in natural language processing to create a neural model of generalization based on linguistic instructions. Models are trained on a set of common psychophysical tasks, and receive instructions embedded by a pretrained language model. Our best models can perform a previously unseen task with an average performance of 83% correct based solely on linguistic instructions (that is, zero-shot learning). We found that language scaffolds sensorimotor representations such that activity for interrelated tasks shares a common geometry with the semantic representations of instructions, allowing language to cue the proper composition of practiced skills in unseen settings. We show how this model generates a linguistic description of a novel task it has identified using only motor feedback, which can subsequently guide a partner model to perform the task. Our models offer several experimentally testable predictions outlining how linguistic information must be represented to facilitate flexible and general cognition in the human brain.

Citation (ISO format)
RIVELAND, Reidar, POUGET, Alexandre. Natural language instructions induce compositional generalization in networks of neurons. In: Nature neuroscience, 2024, vol. 27, n° 5, p. 988–999. doi: 10.1038/s41593-024-01607-5
Main files (1)
Article (Published version)
Secondary files (1)
ISSN of the journal1097-6256

Technical informations

Creation05/13/2024 12:40:42 PM
First validation05/13/2024 2:11:09 PM
Update time05/13/2024 2:11:09 PM
Status update05/13/2024 2:11:09 PM
Last indexation05/13/2024 2:11:29 PM
All rights reserved by Archive ouverte UNIGE and the University of GenevaunigeBlack