Scientific article
OA Policy
English

Human-Level Extraction of Modified Rankin Scale Scores from Real-World Neurosurgical Clinical Notes Using a Locally Deployed, Quantized Large Language Model

Published inMachine learning and knowledge extraction, vol. 8, no. 8, 248
First online date2026-08-16
Abstract

This study conducts a clinical evaluation of a secure, locally deployed, quantized large language model (LLM) for automating modified Rankin Scale (mRS) score extraction from unstructured neurosurgical notes. We retrospectively selected 103 authentic clinical letters (2007–2025) from aneurysm patients at a tertiary neurosurgical centre. To comply with data privacy constraints, an open-source reasoning LLM (Qwen3-32B) with 4-bit quantization was deployed entirely on-premises. The LLM extracted mRS scores using a zero-shot approach with custom logits processors to enforce strict JSON formatting. Performance was compared to a reference standard (attending neurosurgeons’ consensus) and parallel scoring by medical residents and students. The LLM achieved excellent agreement with the attending consensus (QWK 0.95), matching the reliability of medical students (QWK 0.95) and residents (QWK 0.93). Exact agreement was 75%, and agreement within ±1 mRS point was 96%. Bayesian analysis strongly supported statistical equivalence between the model and human raters. The computationally optimized LLM demonstrated human-level classification reliability without task-specific fine-tuning. This approach successfully addresses key patient data privacy barriers and the formatting inconsistencies typical of open-ended generative models. Securely deploying a general-purpose, quantized LLM provides a scalable pathway for extracting functional outcomes and supports FAIR-aligned data systems.

Keywords
  • Artificial intelligence
  • Large language models
  • Information extraction
  • Neurosurgery
  • Modified Rankin Scale
Funding
Citation (ISO format)
BORNET, Alban et al. Human-Level Extraction of Modified Rankin Scale Scores from Real-World Neurosurgical Clinical Notes Using a Locally Deployed, Quantized Large Language Model. In: Machine learning and knowledge extraction, 2026, vol. 8, n° 8, p. 248. doi: 10.3390/make8080248
Main files (1)
Article (Published version)
Secondary files (1)
Supplemental data
accessLevelPublic
Identifiers
Additional URL for this publicationhttps://www.mdpi.com/2504-4990/8/8/248
Journal ISSN2504-4990
1views
0downloads

Technical informations

Creation18/08/2026 00:31:31
First validation09/09/2026 13:50:43
Update09/09/2026 13:50:43
Status update09/09/2026 13:50:43
Last indexation09/09/2026 13:50:44
All rights reserved by Archive ouverte UNIGE and the University of GenevaunigeBlack