Objective: To validate the Non-Technical Skills for Surgeons (NOTSS) system for assessment of the collective surgical teams' nontechnical skills after observing recordings of actual OR environment.

Background: The NOTSS system is a widely accepted tool to measure nontechnical skills of individual surgeons, and has mostly been used in the simulated setting. Surgical procedures are rarely performed by a single surgeon, but by a surgical team of attending surgeons, surgical assistants, and surgical trainees. Therefore, assessment of nontechnical skills may benefit from holistic assessment of the collective surgical teams.

Methods: Five trained participants assessed surgical team and attending surgeon using the NOTSS system after watching ten 20-minute long videos obtained from live OR. A set of reference ratings was provided by a multidisciplinary expert committee. We performed analyses to assess system sensitivity; examine inter-rater reliability of ratings; investigate concurrent construct validity; and assess feasibility and acceptability of using the NOTSS system to measure surgical team performance.

Results: There was adequate system sensitivity when comparing participants' and reference ratings. Inter-rater reliability among the participants' ratings was good except for decision-making category. The level of inter-rater reliability was similar when rating teams and attending surgeons. There was strong positive correlation between teams' and attending surgeons' NOTSS ratings at category [Pearson coefficient 0.86, 95% confidence interval (CI) 0.82-0.89] and element levels (0.83, 95% CI 0.80-0.85), demonstrating evidence of concurrent construct validity. The participants felt that the use of NOTSS system to measure teams' nontechnical skills was acceptable and feasible to a fair extent.

Conclusion: The NOTSS system, although developed for assessment of individual surgeons, is a useful tool for observing and rating surgical teams.