Background: This study was designed in the context of a new educational program based on standardized-patients (SPs). The study objectives were (1) to evaluate the reliability of the assessment form used by SPs and (2) to compare global ratings to checklists reproducibility.

Method: History taking, physical examination and communication skills were assessed by SPs at the end of clinical encounters with year 3 medical students, using generic global rating scales and case-specific checklists. The validation process followed four steps: (1) correlation analysis between global rating and their relative checklist of specific items, (2) internal consistency estimation, (3) validation of the questionnaire dimensions, (4) estimation of the reliability of SPs' ratings compared to medical teachers' ratings.

Results: A total of 3322 consultations have been performed by 444 year 3 medical students. Statistical analysis showed a good internal reliability (Cronbach α was greater than 0.7) and an acceptable inter-judge reproducibility except for communication skills. Case-specific checklists did not prove to be more reliable than global ratings. Reproducibility was lower with SPs' than with medical teachers' assessment.

Conclusion: Global rating-based assessment should be preferred to checklists because they are faster and easier to use (shorter duration of SPs training). As SPs proved to be acceptable examiners, no third person seems to be required as external observer.