Item-Level Evaluation of Multimodal Large Language Models in Neuroradiology : Generational Performance and Execution Variability

Article Article scientifique Article Item-Level Evaluation of Multimodal Large Language Models in Neuroradiology : Generational Performance and Execution Variability eng 2026-06-13 author Ojeda Esparza Jose Federico From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 445021 author Yun Yeong-Chul From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. author Meinzer Clara From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. author Santarosa Corrado From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 418060 author Botta Daniele From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 417457 author Fitsiori Aikaterini From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 919174 Département de radiologie et informatique médicale (RADIO) author Pucci Marcella From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 405947 author Scheffler Max 0000-0003-2274-5129 From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 686304 Département de radiologie et informatique médicale (RADIO) author Vrancsik Nora From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 446512 author Reith Nicolas From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 857526 author Lövblad Karl-Olof From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. 135154 Département de radiologie et informatique médicale (RADIO) author Kurz Félix Tobias 0000-0001-7616-7147 From the Division of Neuroradiology (J.F.O.-E., C.S., D.B., A.F., K.-O.L., F.T.K.), Radiology (M.P., M.S., N.V., N.R.), Geneva University Hospitals, Geneva, Switzerland; Faculty of Medicine (J.F.O.-E., C.S., D.B., A.F., M.P., M.S., K.-O.L.), University of Geneva, Geneva, Switzerland; Division of Radiology (C.M.), German Cancer Research Center, Neuroradiology (Y.-C.Y.), Heidelberg University Hospital, Heidelberg, Germany. Felixtobias.kurz@hug.ch. 419472 ajnr.A9477 10.3174/ajnr.A9477 42288333 0195-6108 http://www.ajnr.org/lookup/doi/10.3174/ajnr.A9477 Background: Multimodal large language models have demonstrated consistent generational improvements on medical benchmark tasks, including radiology applications. However, whether these gains represent meaningful convergence toward the expert reference performance in subspecialty imaging domains such as neuroradiology remains uncertain, particularly when evaluated using item-level, human-referenced designs. Methods: We compared expert neuroradiologists, radiology residents, and four vision-capable large language models (GPT-4, GPT-5, Gemini 1.5, Gemini 2.5) using 106 image-based neuroradiology multiple-choice questions derived from Radiopaedia. Analyses were conducted at the question (item) level, preserving within-question pairing across groups. Mean accuracy differences were estimated using non-parametric bootstrap 95% confidence intervals, and statistical inference was performed using paired permutation tests restricted to pre-specified contrasts with false discovery rate correction for primary comparisons. Repeated model executions were analyzed to characterize execution-level variability. Performance was further contextualized against Radiopaedia community accuracy as a community-level reference. Results: Expert neuroradiologists achieved the highest mean item-level accuracy (0.915; 95% confidence interval, 0.877-0.953). Second-generation models demonstrated improved mean accuracy relative to earlier versions and approximated or exceeded resident-level performance in selected comparisons. However, a substantial gap relative to the expert reference persisted. GPT-5 and Gemini 2.5 underperformed the expert reference by mean per-item differences of -0.236 and -0.217, respectively. When contextualized against the community-level reference, advanced models aligned more closely with aggregate learner performance than with the expert reference accuracy. Improvements in mean accuracy were not uniformly accompanied by improved execution consistency. Conclusions: Although multimodal large language models show meaningful generational gains on neuroradiology tasks, these improvements do not constitute convergence toward the expert reference performance. Item-level, paired, human-referenced evaluation provides critical context for interpreting benchmark performance and helps distinguish apparent performance gains from true alignment with the expert reference. Importantly, relative stability in aggregate accuracy does not necessarily imply reliability at the level of individual decisions, underscoring the need to assess both performance magnitude and execution-level consistency. Multimodal large language models Neuroradiology Diagnostic accuracy Inter-run variability Dewey 616.0757 62 Département de radiologie et informatique médicale 227 false 470 Ischémie cérébrale et connectivité 801 926 Imagerie per-opératoire multi-modalité en radiologie interventionnelle 949 1470 Laboratoire de neuroimagerie quantitative et computationnelle 1078 American journal of neuroradiology 7dd30003-6061-40b1-aeef-e9d5ece1c83f Article (Submitted version) Restricted Draft_LLM.docx 1185299 application/vnd.openxmlformats-officedocument.wordprocessingml.document 106ac6848bafc19154513fe8f3dce449b6dd5794c53ed96bf2e0bc908aad8d85 restrictedAccess Faculté de médecine / Section de médecine clinique / Département de radiologie et informatique médicale

Title	Published in	Access level	OA Policy	Year	Views	Downloads

Item-Level Evaluation of Multimodal Large Language Models in Neuroradiology : Generational Performance and Execution Variability	American journal of neuroradiology			2026	1	0

Archive ouverte UNIGE

Vrancsik, Nora