Natural language instructions induce compositional generalization in networks of neurons

Article Article scientifique Article Natural language instructions induce compositional generalization in networks of neurons eng 2024-03-18 2024-03-18 author Riveland Reidar 0000-0003-1510-290X 880766 Département de neurosciences fondamentales (NEUFO) author Pouget Alexandre 0000-0003-3054-6365 758806 Département de neurosciences fondamentales (NEUFO) 988-999 10.1038/s41593-024-01607-5 38499855 1097-6256 https://www.nature.com/articles/s41593-024-01607-5 <p>A fundamental human cognitive feat is to interpret linguistic instructions in order to perform novel tasks without explicit task experience. Yet, the neural computations that might be used to accomplish this remain poorly understood. We use advances in natural language processing to create a neural model of generalization based on linguistic instructions. Models are trained on a set of common psychophysical tasks, and receive instructions embedded by a pretrained language model. Our best models can perform a previously unseen task with an average performance of 83% correct based solely on linguistic instructions (that is, zero-shot learning). We found that language scaffolds sensorimotor representations such that activity for interrelated tasks shares a common geometry with the semantic representations of instructions, allowing language to cue the proper composition of practiced skills in unseen settings. We show how this model generates a linguistic description of a novel task it has identified using only motor feedback, which can subsequently guide a partner model to perform the task. Our models offer several experimentally testable predictions outlining how linguistic information must be represented to facilitate flexible and general cognition in the human brain.</p> Dewey 616.8 53 Département de neurosciences fondamentales 21A false 893 Groupe Alexandre Pouget 938 Nature neuroscience 27 5 1bbc98d5-fd62-4918-9948-2ce2ea2e44a4 Article (Published version) Public Article-scientifique-from-unpaywall.pdf 5036856 application/pdf CC BY-4.0 9e772e8ac3cc5179fd81a26f7e1bba8dc5dd56026b45d382c890c403311ff51c e3e7ba66-d40f-4682-a77b-2b8ec296e783 Supplemental data Public 41593_2024_1607_MOESM1_ESM.pdf 25781391 application/pdf 99340bf44a7e9b3de317ba14d9720e80921f976748127e11c63bc27d366b89e2 openAccess Faculté de médecine / Section de médecine fondamentale / Département de neurosciences fondamentales

Title	Published in	Access level	OA Policy	Year	Views	Downloads

Natural language instructions induce compositional generalization in networks of neurons	Nature neuroscience			2024	77	36

Archive ouverte UNIGE

Riveland, Reidar