en
Proceedings chapter
Open access
French

TURKOISE: a Mechanical Turk-based Tailor-made Metric for Spoken Language Translation Systems in the Medical Domain

Presented at Reykjavik (Iceland), 26th May 2014
PublisherKeith J. Miller, Lucia Specia, Kim Harris, Stacey Bailey
Publication date2014
Abstract

In this paper, we will focus on the evaluation of MedSLT, a medium-vocabulary hybrid speech translation system intended to support medical diagnosis dialogues between a physician and a patient who do not share a common language (Bouillon et al, 2005). How can the developers be sure of delivering good translation quality to their users, in a domain where reliability is of the highest importance? With MedSLT sentences are usually translated freely and, as a consequence of spoken input, they are often short. These characteristics entail low BLEU scores (Starlander and Estrella, 2011) as well as poor correlation when using human judgments. In the present paper we will describe the path that led us to using Amazon Mechanical Turk (AMT) as an alternative to more classical automatic or human evaluation, and introduce task-specific human metric, TURKOISE, designed to be used by unskilled AMT evaluators while guaranteeing reasonable level of coherence between the evaluators.

Research group
Citation (ISO format)
STARLANDER, Marianne. TURKOISE: a Mechanical Turk-based Tailor-made Metric for Spoken Language Translation Systems in the Medical Domain. In: Workshop on Automatic and Manual Metrics for Operational Translation Evaluation - within LREC 2014. Reykjavik (Iceland). [s.l.] : Keith J. Miller, Lucia Specia, Kim Harris, Stacey Bailey, 2014. p. 38–44.
Main files (1)
Proceedings chapter (Accepted version)
accessLevelPublic
Identifiers
  • PID : unige:42660
510views
149downloads

Technical informations

Creation10/22/2014 1:57:00 PM
First validation10/22/2014 1:57:00 PM
Update time03/14/2023 10:21:53 PM
Status update03/14/2023 10:21:53 PM
Last indexation01/16/2024 2:55:52 PM
All rights reserved by Archive ouverte UNIGE and the University of GenevaunigeBlack