Generative LLMS have become ubiquitous across various fields, including healthcare and medicine. Consequently, there is growing interest in leveraging LLMs for medical applications, leading to the daily emergence of novel models. To identify gaps in medical question answering benchmarks in languages other than English, such as French, we propose a narrative review to synthetize existing research and identify gaps in the literature. Results highlights the lack of a quality benchmark in medical French. In conclusion, while LLMs, represent a significant advancement in medicine, their implementation into clinical practice still requires thorough validation, especially in non-English medical language and standardized evaluations.