Evaluation of Deep Learning Models for Identifying Surgical Actions and Measuring Performance

There is a pressing need for an intraoperative assessment tool that meets high standards of reliability and validity to use as an outcome measure for different training strategies. The aim of this study was to develop a tool specific for laparoscopic skills and to evaluate its reliability and validity. The Global Operative Assessment of Laparoscopic Skills (GOALS) consists of a 5-item global rating scale. A 10-item checklist and 2 visual analogue scales (VAS) for competence and case difficulty were also used. During laparoscopic cholecystectomy, 21 participants were evaluated by the attending surgeon, by 2 trained observers and by self-assessment while dissecting the gallbladder from the liver bed. The intraclass correlation coefficient (ICC) for the total GOALS score was .89 (95% confidence interval [CI] .74 to .95) between observers, .82 (95% CI .67 to .92) between observers and attending surgeons, and .70 (95% CI .37 to .87) between participants and attending surgeons. The ICCs (observers) for the VAS (competence) and the checklist were .69 and .70, respectively. The mean total GOALS score (observers) for novices (postgraduate years [PGYs] 1 through 3) was 13 (95% CI 10.3 to 15.7) compared with 19.4 (95% CI 17.2 to 21.5) for experienced (PGY 4 through attending surgeons, P = .0006). The VAS demonstrated a difference in scores between novice and experienced participants (P = .001); however, the task checklist did not (P = .09). These data indicate that GOALS is feasible, reliable, and valid. They also suggest that it is superior to the task checklist and VAS for evaluation of technical skill by experienced raters. The findings support the use of GOALS in the training and evaluation of laparoscopic skills.

Purpose The education of surgical trainees should be based on an accurate evaluation of their surgical skill levels. In our hospital, the Objective Structured Assessment of Technical Skills (OSATS) is used for this purpose. We conducted this study to demonstrate the validity and accuracy of the OSATS for assessing surgical skills in the operating room (OR) setting. Methods Between January, 2007 and December, 2010, the OSATS global rating scale was used to assess several operations in which surgical trainees participated. We assessed ten surgical trainees who participated as the main surgeon or first assistant, and studied the correlation between their postgraduate year and their OSATS score. Results The median score of the global rating scale for each trainee improved with each year of experience. The median scores of all trainees in postgraduate years 3, 4, and 5 were significantly different (p < 0.001 for both the main surgeon and first assistant roles; Kruskal–Wallis test). Conclusion Using the OSATS global rating scale to assess the surgical skills of trainees in the OR was feasible and effective.