BACKGROUND: Trainee surgeons would benefit from regular, formative assessments to ensure they learn the nontechnical aspects of surgical performance. Non-Technical Skills for Surgeons in Denmark (NOTSSdk) is a tool to assess surgeons' nontechnical skills (NTS) during an operation. The aims of this study were to explore which parts of NOTSSdk supervisors use to assess trainee surgeons' NTS, to determine the internal consistency reliability of NOTSSdk, and to estimate how many operations were needed to obtain reliable ratings of a trainee surgeon's NTS.

METHODS: A total of 12 supervisors from 2 hospitals assessed 13 trainees in 48 procedures including laparoscopic cholecystectomies, inguinal hernia repairs, and laparoscopic appendectomies.

RESULTS: NOTSSdk showed good internal consistency reliability (R(2) = 0.95). Supervisors used both categories and elements of the tool to provide assessments, although 2 elements of leadership ("supporting others" and "coping with pressure") were considered irrelevant in 27% to 31% of the cases. Assessments of 5 procedures were sufficient to gain reliable ratings (Generalizability coefficient > 0.80) of a trainee surgeon's NTS.

CONCLUSION: As supervisors used the full tool, the internal consistency reliability was high and assessment of 5 procedures could be achieved in clinical practice, we suggest that NOTSSdk could be implemented as a formative assessment tool to facilitate the training of surgeons' NTS.