Scientific article
OA Policy
English

Benchmarking Open-Source Large Language Models in Medical French

Published inStudies in health technology and informatics, vol. 336, p. 844-848
Publication date2026-05-21
Abstract

Large Language Models (LLMs) are increasingly applied in healthcare, yet their evaluation in medical French remains limited. Building on the MedFrenchmark study by Quercia et al. (2024), this work assesses 15 open-source models through a subset of 77 medical questions spanning multiple specialties and reasoning types. All 1,155 generated responses were manually rated on a 0-100 scale for factual accuracy, contextual relevance, clarity, and potential clinical usability. Model performance ranged from 36% to 81%, showing notable differences. While the largest model, GPT-OSS:120B, achieved the highest score, smaller models such as Qwen3:8B and Gemma3:27B reached comparable levels of precision and coherence. Overall, the strongest models demonstrated substantial progress in reasoning and linguistic fluency, although some inconsistencies in terminology and semantic control were still observed. This study offers an updated and transparent overview of open-source LLMs in medical French, underlining both their promising development and their growing potential for future clinical applications.

Keywords
  • Benchmark
  • Clinical AI
  • LLM
  • Medical French
  • NLP
  • Benchmarking
  • France
  • Humans
  • Language
  • Semantics
  • Natural Language Processing
  • Terminology as Topic
  • Large Language Models
Citation (ISO format)
TCHEREPANOVA, Maria et al. Benchmarking Open-Source Large Language Models in Medical French. In: Studies in health technology and informatics, 2026, vol. 336, p. 844–848. doi: 10.3233/SHTI260298
Main files (1)
Article (Published version)
Identifiers
Additional URL for this publicationhttps://ebooks.iospress.nl/doi/10.3233/SHTI260298
Journal ISSN0926-9630
3views
2downloads

Technical informations

Creation01/06/2026 08:55:44
First validation01/07/2026 09:14:20
Update01/07/2026 09:14:20
Status update01/07/2026 09:14:20
Last indexation01/07/2026 09:14:21
All rights reserved by Archive ouverte UNIGE and the University of GenevaunigeBlack