UNIGE document Scientific Article
previous document  unige:79613  next document
add to browser collection

How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations

Gerbase, Margaret W.
Published in Teaching and learning in medicine. 2015, vol. 27, no. 4, p. 395-403
Abstract Construct:The study compares paper and online ratings of instructional units and analyses, with the G-study using the symmetry principle, the response rates needed to ensure acceptable precision of the measure when compliance is low. Background: Students' ratings of teaching contribute to the quality of medical training programs. To date, many schools have replaced pen-and-paper questionnaires with electronic forms, despite the lower response rates consistently reported with the latter. Few available studies have examined the effects of low response rates on the reliability and precision of the evaluation measure. Moreover, the minimum number of raters to target when response rates are low remains unclear. Approach: Descriptive data were derived from 799 students' paper and online ratings of 11 preclinical instructional units (PIUs). Reliability was assessed by Cronbach's alpha coefficients. The generalizability method applying the symmetry principle approach was used to analyze the precision of the measure with a reference standard error of mean (SEM) set at 0.10; optimization models were built to estimate minimum response rates. Results: Overall, response rates were 74% and 30% (p < .001) and PIUs ratings were 3.8 ± 0.5 and 3.6 ± 0.5 (p = .02), respectively in paper and online questionnaires. Higher SEM levels and significantly larger 95% confidence intervals of PIUs rating scores were observed with online evaluations. To keep the SEM within preset limits of precision, a minimum of 48% response rate was estimated for online formats. Conclusions: The proposed generalizability analysis allowed estimating the minimum response needed to maintain acceptable precision in online evaluations. The effects of response rates on accuracy are discussed.
Full text
Article (Published version) (309 Kb) - document accessible for UNIGE members only Limited access to UNIGE
(ISO format)
GERBASE, Margaret W. et al. How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations. In: Teaching and learning in medicine, 2015, vol. 27, n° 4, p. 395-403. doi: 10.1080/10401334.2015.1077126 https://archive-ouverte.unige.ch/unige:79613

473 hits



Deposited on : 2016-01-19

Export document
Format :
Citation style :