PS5-12: THE QUALITY OF OPERATIVE PERFORMANCE NARRATIVE FEEDBACK: A COMPARISON BETWEEN END OF ROTATION EVALUATIONS AND SIMPL
Samantha L Ahle, MD1, Mary S Schuller, MSEd2, Emily Carnes2, Mickyas Eskender, MD2, Jennifer Doyle, MA3, Jeanne Koehler, PhD4, Gabrielle Shaughness, BA5, Heeyoung Han, PhD4, Boyung Suh, PhD4, Xilin Chen, MPH6, Ahmed Latif6, Gregory Wnuk, MHSA6, Jonathan P Fryer, MD, MHPE2, John D Mellinger, MD4, Brian C George, MD, MAEd5; 1Yale School of Medicine, 2Feinburg School of Medicine-Northwestern University, 3Harvard Medical School, 4SIU School of Medicine, 5University of Michigan Medical School, 6Center for Surgical Training and Research, Department of Surgery, University of Michigan
Purpose: Many surgical training programs administer an evaluation with which faculty members may rate and comment on trainee operative performance at the end of the rotation (EOR). Programs who are members of the Procedural Learning and Safety Collaborative (PLSC) have also implemented SIMPL, a workplace based assessment tool with which faculty can rate and comment on a trainee’s operative performance immediately after a case. It is unknown how the quality of narrative operative performance feedback in SIMPL compares to those on EOR forms.
Methods: EOR evaluation and SIMPL narrative comments from a single academic year were collected from two university-based general surgery training programs, transcribed, and deidentified. Comments were partitioned by post graduate year (PGY) and data source (EOR vs. SIMPL) before being randomly sampled for inclusion in the qualitative analysis. Comments relating to operative skills were categorized by two surgeon raters as being specific or general and as encouraging and/or corrective. Using a previously defined rubric to combine those categories, comments were then classified as effective, mediocre, or ineffective. The frequencies with which comments were rated as effective were compared using X-square analysis.
Results: A total of 400 comments were analyzed (200 each from EOR evaluations and SIMPL). 25% (n = 50) of EOR and 46% (n = 92) of SIMPL operative performance evaluation comments were deemed effective (X2 = 18.08, p < .0001).
Conclusions: Evaluators give significantly higher quality operative performance feedback when using workplace based assessment tools rather than EOR evaluations, yet gaps still remain.