PS4-08: A VIDEO-ANCHORED RATING SCALE LEADS TO HIGH INTERRATER RELIABILITY OF INEXPERIENCED AND EXPERT RATERS IN THE ABSENCE OF RATER TRAINING
Ronit Patnaik, MD1, Nicholas Anton2, Eugene Ceppa, MD2, Annabelle Butler, MD2, Ambar Banerjee, MD2, Dimitrios Stefanidis, MD, PhD2; 1University of Texas Health Sciences Center, San Antonio, 2Indiana University School of Medicine
Background: Behaviorally anchored rating scales (BARS) are considered superior to traditional numerical scales for surgical performance assessment; however, they may still suffer from unreliability, leniency bias and lack of discriminant validity. Our objective was to assess the impact of incorporating videos in a BARS on the inter-rater reliability of novice, intermediate and expert raters.
Methods: The Intracorporeal Suturing Assessment Tool (ISAT), was modified to include short video clip anchors (<1 min) demonstrating poor, average, and expert performance. Blinded raters used this tool to assess fifteen laparoscopic suturing videos of trainees performing on a porcine fundoplication model. The raters included 4 novices, 4 surgery residents and 3 attending surgeons experienced in intracorporeal suturing; no rater training was provided. To determine inter-rater reliability, the intraclass correlation coefficient (ICC) was calculated and compared among raters. Ratings were also correlated with objective suturing scores of trainee performance.
Results: The ICC among all raters using the video-anchored ISAT tool was high at 0.80 (p<0.001). Ratings were also significantly correlated with trainee objective suturing scores for all rater groups (experts: R = 0.84, p <0.0001, residents: R = 0.81, p <0.0001, and novices: R= 0.69, p <0.0001).
Conclusions: Incorporating video anchors to define performance into a laparoscopic suturing assessment tool led to high inter-rater reliability and enabled less experienced raters without prior rater training to achieve similar consistency in their ratings as experts. These findings support the incorporation of video based anchors in performance assessment tools as an effective method to ensure reliable ratings independent of rater experience.