Reliable Enough ? Benchmarking LLMs for Clinical Concept Extraction

Article Article scientifique Article Reliable Enough ? Benchmarking LLMs for Clinical Concept Extraction eng 2026-05-21 author Pignat Johann Division of Precision Oncology, Hôpitaux Universitaires de Genève, CH.,Division of Medical Information Sciences, Hôpitaux Universitaires de Genève, CH.,Department of Radiology and Medical Informatics, Université de Genève, CH. 880351 author Liakopoulos Petros Division of Precision Oncology, Hôpitaux Universitaires de Genève, CH. 405989 Département de médecine (DEMED) author Bonjour Jonatan Division of Precision Oncology, Hôpitaux Universitaires de Genève, CH. 419099 Département de médecine (DEMED) author Cheng Phil 0000-0003-2940-006X Division of Precision Oncology, Hôpitaux Universitaires de Genève, CH. 418075 author Varelogiannis Sokratis Division of Precision Oncology, Hôpitaux Universitaires de Genève, CH. 432599 author Lovis Christian 0000-0002-2681-8076 Division of Medical Information Sciences, Hôpitaux Universitaires de Genève, CH.,Department of Radiology and Medical Informatics, Université de Genève, CH. 29016 Département de radiologie et informatique médicale (RADIO) author Bjelogrlic Mina 0000-0002-6922-3283 Division of Medical Information Sciences, Hôpitaux Universitaires de Genève, CH.,Department of Radiology and Medical Informatics, Université de Genève, CH. 918489 Département de radiologie et informatique médicale (RADIO) author Gaudet-Blavignac Christophe 0000-0001-6527-5898 Division of Medical Information Sciences, Hôpitaux Universitaires de Genève, CH.,Department of Radiology and Medical Informatics, Université de Genève, CH. 645533 author Cuendet Michel 0000-0002-0754-3425 Division of Precision Oncology, Hôpitaux Universitaires de Genève, CH. 418547 Département de médecine (DEMED) author Michielin Olivier 0000-0003-4926-6355 Division of Precision Oncology, Hôpitaux Universitaires de Genève, CH. 903944 Département de médecine (DEMED) 786-787 10.3233/SHTI260285 42174960 0926-9630 Language models (LMs) offer promise for automating concept extraction, a task traditionally reliant on manual curation and earlier natural language processing (NLP) methods. We present a benchmarking approach to systematically evaluate on-premises LM performance. Using identification of first-line pharmacological treatments in melanoma patients as a test case, we demonstrate how this method supports structured comparison and error analysis for local models. Results indicate the importance of prompt design, and that small models struggle with layered reasoning tasks, such as sequencing interventions. These findings suggest that LMs are best deployed as supportive tools requiring careful evaluation. Benchmarking Computational Linguistics Medical Informatics Responsible AI Natural Language Processing Benchmarking / methods Humans Melanoma / drug therapy Data Mining / methods Data Mining / standards Reproducibility of Results Electronic Health Records Dewey 616.0757 Dewey 616 62 Département de radiologie et informatique médicale 227 56 Département de médecine 221 false 344 Interfaces Homme-machine en milieu clinique 610 1432 Découverte de biomarqueurs numériques pour l'oncologie de précision 1059 Studies in health technology and informatics 336 517c4c9b-43dd-43d6-aaaa-cd6d227b361e Article (Published version) Public SHTI-336-SHTI260285.pdf 154762 application/pdf CC BY-NC-4.0 dd3bb06ae35ce5d1b598fb4b384c1ed493779930dd37a5b49988cedafc1b8ad4 openAccess Faculté de médecine / Section de médecine clinique / Département de radiologie et informatique médicale Faculté de médecine / Section de médecine clinique / Département de médecine

Title	Published in	Access level	OA Policy	Year	Views	Downloads

Reliable Enough ? Benchmarking LLMs for Clinical Concept Extraction	Studies in health technology and informatics			2026	9	25

Archive ouverte UNIGE

Pignat, Johann