The aim of this paper is to further compare two versions of our spoken language translator MedSLT which differ in terms of grammatical coverage; the first version is restricted to Yes-No answers and short elliptical sentences, while the second version allows all of the above but also full sentences. Previous work in this direction has focused on the relation between Word Error Rate(WER) and task performance (Starlander et al., 2008). In this paper this comparison is extended to study in more detail the task performance with respect to the number of successful interactions and to explain why some utterances could not be correctly recognized. Additionally this paper will explore how these results correlate with the quality of the translation produced by both versions of the systems.