Scientific article
English

An analysis of ChatGPT’s performance in predicting countries and regions of origin from personal names

ContributorsSeboe, Paulorcid
Published inScientometrics, vol. 131, no. 3, p. 1731-1754
First online date2026-03-19
Abstract

This study evaluated the performance of ChatGPT-4o in predicting the country and region of origin from personal names. A cross-sectional analysis was conducted using 2515 names from 63 countries of origin, generated by randomly combining popular first and last names from a public GitHub dataset. Predictions were obtained in two rounds, conducted five days apart (25 and 30 November 2024). Performance metrics included errorCoded (proportion of incorrect or non-classifications), errorCodedWithoutNA (proportion of incorrect classifications, excluding non-classifications), and naCoded (proportion of non-classifications). Cohen’s Kappa was used to assess agreement between rounds. ChatGPT correctly identified the country of origin for 69.7% and 70.3% of names in the first and second rounds, respectively (errorCoded = 0.303 and 0.297). Regional predictions were more accurate, with 95.8% of names correctly assigned in both rounds (errorCoded = 0.042). No non-classifications occurred (naCoded = 0), and agreement between rounds was high (Cohen’s Kappa: 0.841 for countries and 0.986 for regions). Performance varied by linguistic complexity, with high accuracy for linguistically distinct countries (e.g., Japan) and lower accuracy for countries with overlapping linguistic, cultural, or geographic features (e.g., English, French, and Spanish-speaking countries). In conclusion, ChatGPT-4o demonstrated high accuracy in predicting regions of origin and moderate performance at the country level when classifying names from a diverse international dataset. These findings highlight the potential and current limitations of large language models for name-based origin detection and suggest promising avenues for future applications in research, demography, and public health.

Keywords
  • Accuracy
  • AI
  • Artificial intelligence
  • ChatGPT
  • Classification
  • Country
  • Large language model
  • LLM
  • Misclassification
  • Name-based inference
  • Onomastics
  • Origin
  • Performance
  • Region
  • Romanization
  • Scientometrics
Citation (ISO format)
SEBOE, Paul. An analysis of ChatGPT’s performance in predicting countries and regions of origin from personal names. In: Scientometrics, 2026, vol. 131, n° 3, p. 1731–1754. doi: 10.1007/s11192-026-05587-0
Main files (1)
Article (Published version)
accessLevelRestricted
Secondary files (2)
Supplemental data
accessLevelRestricted
Supplemental data
accessLevelRestricted
Identifiers
Journal ISSN0138-9130
10views
1downloads

Technical informations

Creation30/03/2026 12:14:18
First validation09/04/2026 07:23:01
Update09/04/2026 07:23:01
Status update09/04/2026 07:23:01
Last indexation09/04/2026 07:23:02
All rights reserved by Archive ouverte UNIGE and the University of GenevaunigeBlack