MedFrenchmark, a small set for benchmarking generative LLMs in medical french

Article Article scientifique Article MedFrenchmark, a small set for benchmarking generative LLMs in medical french eng 2024-08-22 author Quercia Amandine Alizée Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland.,Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland. 880350 author Zaghir Jamil 0000-0002-8209-6098 Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland.,Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland. 956784 Département de radiologie et informatique médicale (RADIO) author Lovis Christian 0000-0002-2681-8076 Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland.,Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland. 29016 Département de radiologie et informatique médicale (RADIO) author Gaudet-Blavignac Christophe 0000-0001-6527-5898 Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland.,Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland. 645533 601-605 10.3233/SHTI240486 39176814 0926-9630 https://ebooks.iospress.nl/doi/10.3233/SHTI240486 Generative Large Language Models (LLMs) have become ubiquitous in various fields, including healthcare and medicine. Consequently, there is growing interest in leveraging LLMs for medical applications, leading to the emergence of novel models daily. However, evaluation and benchmarking frameworks for LLMs are scarce, particularly those tailored for medical French. To address this gap, we introduce a minimal benchmark consisting of 114 open questions designed to assess the medical capabilities of LLMs in French. The proposed benchmark encompasses a wide range of medical domains, reflecting real-world clinical scenarios' complexity. A preliminary validation involved testing seven widely used LLMs with a parameter size of 7 billion. Results revealed significant variability in performance, emphasizing the importance of rigorous evaluation before deploying LLMs in medical settings. In conclusion, we present a novel and valuable resource for rapidly evaluating LLMs in medical French. By promoting greater accountability and standardization, this benchmark has the potential to enhance trustworthiness and utility in harnessing LLMs for medical applications. Generative AI LLM NLP Benchmark Benchmarking France Computer Simulation Dewey 616.0757 62 Département de radiologie et informatique médicale 227 false 344 Interfaces Homme-machine en milieu clinique 610 Studies in health technology and informatics 316 f05d6db8-13bb-4515-be18-0bfae20e4818 Article (Published version) Public Article_scientifique_from_unpaywall.pdf 179473 application/pdf CC BY-NC-4.0 2fe6ca478ef7a110a01f6cd408c8668755038a6159ce89d33b668b2aacd5eab4 openAccess Faculté de médecine / Section de médecine clinique / Département de radiologie et informatique médicale

Title	Published in	Access level	OA Policy	Year	Views	Downloads

MedFrenchmark, a small set for benchmarking generative LLMs in medical french	Studies in health technology and informatics			2024	79	23

Archive ouverte UNIGE

Quercia, Amandine Alizée