Scientific article
Data paper
OA Policy
English

A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models

Published in

Scientific data, vol. 11, no. 1, 455

Publication date

2024-05-04

First online date

2024-05-04

Abstract

Due to the complexity of the biomedical domain, the ability to capture semantically meaningful representations of terms in context is a long-standing challenge. Despite important progress in the past years, no evaluation benchmark has been developed to evaluate how well language models represent biomedical concepts according to their corresponding context. Inspired by the Word-in-Context (WiC) benchmark, in which word sense disambiguation is reformulated as a binary classification task, we propose a novel dataset, BioWiC, to evaluate the ability of language models to encode biomedical terms in context. BioWiC comprises 20'156 instances, covering over 7'400 unique biomedical terms, making it the largest WiC dataset in the biomedical domain. We evaluate BioWiC both intrinsically and extrinsically and show that it could be used as a reliable benchmark for evaluating context-dependent embeddings in biomedical corpora. In addition, we conduct several experiments using a variety of discriminative and generative large language models to establish robust baselines that can serve as a foundation for future research.

Keywords

  • NLP
  • Biomedical NLP
  • LLM
  • Concept representation
  • Language
  • Natural Language Processing
  • Semantics

Citation (ISO format)

ROUHIZADEH, Hossein et al. A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models. In: Scientific data, 2024, vol. 11, n° 1, p. 455. doi: 10.1038/s41597-024-03317-w

Main files (1)

Article (Published version)

Secondary files (1)

Appendix

Identifiers

Additional URL for this publication

https://pubmed.ncbi.nlm.nih.gov/38704422/

Journal ISSN

2052-4463
216views
75downloads

Technical informations

Creation

07/05/2024 15:18:00

First validation

05/07/2024 14:05:43

Update

21/11/2025 10:21:35

Status update

21/11/2025 10:21:35

Last indexation

27/08/2026 07:19:52
All rights reserved by Archive ouverte UNIGE and the University of Geneva
OpenAIRE Indexed & Validated