Introduction: Refugees seeking medical help in busy emergency rooms face language barriers when interacting with doctors. One possibility to lower the burden of emergency rooms is performing triage for minor health disorders in community pharmacies. So far, no specialized machine translation system has been developed to help pharmacists and their team with the language barrier. Recent research shows that Google Translate (GT) is used in healthcare, although it is not trained for a medical setting. The research presented here aims to evaluate whether BabelDr (BD), a flexible speech-enabled phraselator developed for the medical domain, is better suited to perform triage in a pharmacy setting than GT with Arabic-speaking patients.
Methods: We adopted an experimental design inviting 8 pharmacists to orally perform triage in French with both systems on three scenarios (cystitis, headache, and sore throat) with two standardized Arabic patients. We collected system usage (time, interactions, diagnoses) and user satisfaction. In addition, both standard Arabic and Arabic dialects speakers evaluated the adequacy of the translations on a scale: correct, mistranslation, and nonsense. Translations were considered incorrect if rated as mistranslation or nonsense.
Results: Concerning diagnoses, all pharmacists reached a correct diagnosis with both systems. The time taken to complete the scenarios was similar across systems, with more total interactions and fewer unsuccessful interactions in GT. For satisfaction, pharmacists preferred BD to GT. For the standard Arabic speakers, 95% of BD's translations were rated as correct compared to 90% of correct translations with GT. For the Arabic dialect speakers, 92% of BD's translations were evaluated as correct as opposed to 85% of correct translations with GT.
Conclusions: Our pilot study suggests that BD is better suited than GT to perform triage in pharmacies for user satisfaction. For adequacy, BD’s translations are slightly better for speakers of standard Arabic and Arabic dialects. For successful interactions, GT is the most usable system. Nonetheless, pharmacists reached a correct diagnosis with both systems.
