UNIGE document Chapitre d'actes
previous document  unige:42660  next document
add to browser collection
Title

TURKOISE: a Mechanical Turk-based Tailor-made Metric for Spoken Language Translation Systems in the Medical Domain

Author
Published in Workshop on Automatic and Manual Metrics for Operational Translation Evaluation - within LREC 2014. Reykjavik (Iceland) - 26th May 2014 - Keith J. Miller, Lucia Specia, Kim Harris, Stacey Bailey. 2014, p. 38-44
Abstract In this paper, we will focus on the evaluation of MedSLT, a medium-vocabulary hybrid speech translation system intended to support medical diagnosis dialogues between a physician and a patient who do not share a common language (Bouillon et al, 2005). How can the developers be sure of delivering good translation quality to their users, in a domain where reliability is of the highest importance? With MedSLT sentences are usually translated freely and, as a consequence of spoken input, they are often short. These characteristics entail low BLEU scores (Starlander and Estrella, 2011) as well as poor correlation when using human judgments. In the present paper we will describe the path that led us to using Amazon Mechanical Turk (AMT) as an alternative to more classical automatic or human evaluation, and introduce task-specific human metric, TURKOISE, designed to be used by unskilled AMT evaluators while guaranteeing reasonable level of coherence between the evaluators.
Full text
Structures
Research group TIM/ISSCO
Citation
(ISO format)
STARLANDER, Marianne. TURKOISE: a Mechanical Turk-based Tailor-made Metric for Spoken Language Translation Systems in the Medical Domain. In: Workshop on Automatic and Manual Metrics for Operational Translation Evaluation - within LREC 2014. Reykjavik (Iceland). [s.l.] : Keith J. Miller, Lucia Specia, Kim Harris, Stacey Bailey, 2014. p. 38-44. https://archive-ouverte.unige.ch/unige:42660

223 hits

86 downloads

Update

Deposited on : 2014-12-06

Export document
Format :
Citation style :