Plenary Session I
(S001) TOWARDS AUTOMATED ASSESSMENT: AI MODEL FOR PREDICTION OF TASK COMPLETION TIME FOR THE ADVANCED TRAINING IN LAPAROSCOPIC SUTURING (ATLAS) OFFSET CAMERA FOREHAND SUTURING
Sofia Garces Palacios, MD1, Huu P Nguyen, PhD1, Sharanya Vunnava1, Nicole Wise, MD2, Mika Lindsley, MD2, Darian Hoagland, MD3, Bryanna Stukes, MD1, Dmitry Nepomnayshy, MD2, Daniel J Scott, MD, FACS1, Ganesh Sankaranarayanan, PhD1; 1University of Texas Southwestern Medical Center, 2Lahey Hospital and Medical Center, 3Beth Israel Deaconess Medical Center
Introduction
To improve laparoscopic suturing skills, the Advanced Training in Laparoscopic Suturing (ATLAS), a proficiency-based curriculum was introduced. ATLAS Task 2 requires placing a forehand stitch through the two marked dots on the suturing model and then tying a knot intracorporeally with an offset camera angle. Performance measures include task completion time and errors, currently evaluated through in-person and human video review. This study aims to explore the use of an artificial intelligence (AI) model to accurately predict Task 2 duration.
Methods
Videos of Task 2 performances collected at two institutions were used for analysis. Trained raters assessed task duration from the videos. Further, the videos were annotated to mark the beginning and end frames of ten distinct phases of the task. To automate the detection of the phases, the X3D deep convolutional neural network model was applied for frame-level visual feature extraction and classification. For phase segmentation, a hybrid approach was introduced, integrating K-Nearest Neighbors (KNN) with a smooth averaging filter (SMA). This approach enabled unique segmentation of each phase, which allowed possible of time duration measurement from model predictions during post-processing.
Results
A total of 56 videos were included: 45 (~80%) for training and 11 (~20%) for testing the proposed model. In the phase segmentation task, the baseline model initially achieved an accuracy of 79.02% and F1 score 80.6%. This performance was significantly improved by the integration of a hybrid approach combining Smooth Moving Average (SMA) and Adaptive K-Nearest Neighbors (AKNN), yielding an increased accuracy of 87.27% and F1 score of 85.33%. For time duration measurement, the optimized model demonstrated robust performance, yielding an average error of approximately 1.52% when compared to the ground truth values (Figure 1).
Conclusion
The AI model exhibited high accuracy in predicting task duration for the ATLAS Needle Handling Task 2 and shows promise as a valuable tool for assessing proficiency while reducing reliance on human resources for evaluations. In the future, we plan to expand the model to assess additional ATLAS tasks and deploy it for assistance in real-time.
%20(1).png)
(S002) IMPLEMENTATION AND VALIDITY EVIDENCE FOR THE USE OF THE ENTRUST ASSESSMENT PLATFORM IN THE COLLEGE OF SURGEONS OF EAST, CENTRAL, AND SOUTHERN AFRICA (COSECSA) MEMBERSHIP OF THE COLLEGE OF SURGEONS (MCS) EXAMINATION
Cara A Liebert, MD, FACS1, Seno I Saruni, MBBS, FCSECSA, FACS2, Edward Melcer, PhD3, Jason Tsai, MS1, Yewon Son, BS1, Katherine Arnow, MS1, Lye-Yeng Wong, MD4, Hanna Getachew, MD, FCSECSA5, Abebe Bekele, MD, FCSECSA, FACS6, Dana T Lin, MD, FACS1; 1Stanford University School of Medicine, 2Moi Teaching and Referral Hospital, 3Carleton University, 4Oregon Health and Sciences University, 5Addis Ababa University, 6University of Global Health Equity, Rwanda
Background: The College of Surgeons of East, Central, and Southern Africa (COSECSA) has traditionally utilized oral Objective Structured Clinical Examinations (OSCE) to evaluate applied surgical knowledge and clinical decision-making competency of its trainees. While traditionally the gold standard, OSCEs are time and resource intensive. ENTRUST Assessment Platform is an online virtual patient simulation platform developed to objectively assess clinical decision-making across the spectrum of surgical care. This study describes implementation of ENTRUST in the Membership of the College of Surgeons (MCS) Examination and validity evidence for its use in high-stakes assessment in low- and middle-income countries (LMIC).
Methods: In close partnership with COSECSA, ENTRUST was integrated into the COSECSA MCS Exam in 2023 following prior pilot implementation and validity evidence in this population. MCS Exam candidates across 16 sub-Saharan countries completed an online 16-station exam consisting of 8 oral OSCE stations and 8 ENTRUST stations. The COSECSA Examination and Credentials Committee authored all cases and performed standard setting. Demographic and post-exam survey data were collected via Qualtrics survey. Descriptive statistics were calculated and performance on the OSCE and ENTRUST portions of the MCS Exam was compared using Spearman rank correlations.
Results: A total of n=150 MCS Examinees were included in the analysis. The mean (SD) age was 32.9 (4.6) and 68.8% were male; 97.5% reported English as the language used in their current training program. Examinees perceived both the ENTRUST and OSCE portions of the exam to be objective (4.15 vs 4.16, p=0.88). Mean (SD) for the ENTRUST Total Raw Score and OSCE Total Raw Score were 105.0 (13.7) and 113.6 (12.7), respectively. ENTRUST Total Raw Score was significantly correlated with OSCE Total Raw Score (rho=0.6, p<0.0001).
Conclusion: The ENTRUST Assessment Platform was successfully implemented for high-stakes assessment during the COSECSA MCS Exam across 16 sub-Saharan countries. There was strong correlation between score performance on the ENTRUST and OSCE portions of the examination, providing validity evidence for its relationship to other variables. The ENTRUST Assessment Platform is an accessible, objective, and scalable assessment tool to support competency-based medical education globally.
(S003) THE FUNDAMENTALS OF COMMUNICATION IN SURGERY (FCS): EARLY RESULTS FROM A MULTISITE STUDY
Carly G Sobol, MD1, Lauren Taylor, MD2, Katherine Hill, MD3, Courtney Morgan, MD1, Kimberly Kopecky, MD4, Emily Rivet, MD2, Adham Saad, MD5, Muneera Kapadia, MD6, Anna Newcomb, PhD7, Erin Strong, MD8, Robert Bynum, MFA1, Kyle Bushaw, MA1, Margaret L Schwarze, MD1, Amber Shada, MD1; 1University of Wisconsin, 2VCU, 3West Virginia University, 4UAB, 5USF, 6UNC, 7INOVA, 8Moffitt Cancer Center
Introduction: Advanced communication skills are essential to all surgical practices given the need to engage patients and families in decisions about surgery, and help them navigate burdensome treatments and unwanted postoperative events. The FCS is a 5-year curriculum centered on attending to emotion, supporting patients in deliberation, and managing uncertainty. With support from CESERT, the Macy Foundation, and the AAS we commenced a phase II ORBIT model multisite study to test the feasibility, acceptability, and fidelity of FCS delivery to assess scalability and impact on trainee outcomes.
Methods: The IRB declared this study exempt. After mailing physical and electronic materials to 5 pilot sites, including trainer and learner guides, PowerPoint videos, and props for skills activities, we conducted virtual train-the-trainer (TTT) sessions. We anticipate 175 resident learners will participate in academic year 2024-25. Data collection includes attendance, direct observation utilizing the Dane implementation fidelity framework, scoring rubrics, pre-post learner surveys, and stakeholder interviews. We used descriptive statistics and qualitative methods to analyze the data.
Results: All 15 trainers completed 4 hours of TTT. To date, 41 residents (93% attendance) have received one 2-hour FCS session. Residents reported the curriculum was worth their time and successfully met stated goals. Residents intended to apply what was learned when talking with patients and families, score mean 4.08±0.4 (max score 5). Trainers believed this was essential training and the curriculum was easy to teach. For example, “[FCS] doesn’t try to do 20 things, it narrows it down to really important basics.” As scored by peers, learners performed 97% of skills on the rubric during role play. The curriculum was delivered with fidelity. Specifically, trainers followed the training guides, participants engaged with intended content, and the FCS imprint was readily apparent. We estimate 134 additional residents will receive FCS this academic year.
Conclusions: The FCS curriculum is feasible, acceptable, and can be delivered with fidelity. We will use the data from this study to iteratively revise the curriculum and expand the cohort to 10 study sites in the 2025-26 academic year. We anticipate FCS will ultimately be accessible to all surgical trainees.
(S004) TRANSITIONING TO PRACTICE: EXPLORING HOW NEW-TO-PRACTICE SURGEONS NAVIGATE THE TENSIONS BETWEEN THE ROLES OF SURGICAL EDUCATOR AND PRIMARY SURGEON IN THE OPERATING ROOM.
Kimberly T Stewart, Kevin W Eva, Faizal Haji; Centre for Health Education Scholarship, University of British Columbia, Vancouver, Canada
Background: The transition to independent practice is a challenging time for physicians. For new surgeons in academic settings, managing an operating room (OR) is uniquely difficult, as they must assume primary responsibility for their patient’s surgical care while attending to the educational needs of surgical trainees. No empirical guidance exists regarding how to prepare new surgeons for these dual roles. The purpose of this study is to explore how new surgeons navigate the potential tensions between their roles as primary surgeon and surgical educator in the OR.
Methods: We conducted a qualitative study in the constructivist grounded theory (CGT) tradition. Using purposive and theoretical sampling, surgeons 6-36 months into practice in British Columbia participated in semi-structured interviews where role theory, role switching, and role conflict served as sensitizing concepts. Data collection and analysis occurred in iterative cycles using a constant comparative approach to thematic analysis, until we reached theoretical sufficiency.
Findings: Initial findings from interviews with ten surgeons (4 female, 6 males; representing 7 surgical sub-specialties; average of 20 months in practice) revealed that many surgeons feel unprepared for the responsibility of teaching in the OR. Individual surgeon qualities, trainee characteristics, health care systems limitations, and patient factors influenced the ability of surgeons to prioritize teaching in the OR. Rather than being abrupt all-or-none switches between surgeon and educator, new surgeons engage in a continuous fine-tuning of the balance between teaching and direct patient care throughout an operation. Participants described mitigation strategies when education cannot be prioritized, including setting preoperative expectations, arranging expert assistance, explaining unexpected changes in priorities to trainees, switching one’s teaching method, and post-operative debriefing. New surgeons drew on personal experiences in training to inform these strategies.
Conclusion: New surgeons working with trainees experience an inherent expectation to provide educational opportunities in the OR, which can be challenging due to competing workplace interests. Individually, they developed strategies to manage these challenges, drawing from their training experiences and by trial and error. These strategies could be the basis for more proactive faculty development to better prepare new surgeons as surgical educators in the OR.
(S005) IMPACT OF MODERN HEALTHCARE SYSTEMS ON SURGICAL EDUCATION
Martha Godfrey, MD, MS1, Emily Huang, MD2, Ami Shah, MD3, Brendan Scully, MD4, Minna Wieck, MD5, Candice Sauder, MD5, Jacob Peschman, MD6, Kenneth Lipshy, MD7, Priti Parikh, PhD8, Rebecca Hoffman, MD9, Nell Maloney Patel, MD4; 1University of Iowa Health Care, 2The Ohio State University, 3Rush University, 4Rutgers Health, 5University of California - Davis Health, 6Medical College of Wisconsin, 7WG Hefner VA Healthcare System, 8Wright State University, 9Geisinger Health
Introduction
Healthcare reorganizations, including consolidation of large-scale healthcare systems pose significant implications for surgical education. We sought to understand how these changes have affected surgical educators and trainee education nationally.
Methods
An IRB-approved survey about surgeon-educators’ current practice environments was administered via email to members of the ASE and APDS. Quantitative responses were characterized with descriptive statistics. Open-ended free text responses were coded inductively in an open thematic analysis approach.
Results
93 respondents reported employment by an academic medical center (40%), hospital system (5%), or medical school (6%). Few thought mergers were somewhat beneficial for UME (11%) or GME (14%). Of faculty, 64% reported salaried compensation; some reported a clinical RVU-based (40%), or educational RVU-based (1%) compensation component. Wellness and simulation were the most supported educational activities (21%, 21%). 81% reported national organization requirements (e.g., ACGME) impacting education program structure; 55% reported utilizing national guidelines to advocate for education resources. 81% of respondents answered open-ended questions, resulting in 392 applied codes falling into 4 major themes: clinical productivity over education, education as a resource-poor endeavor, exploitation of educators / learners, and burnout (Table).
Discussion
This is the first study to investigate how the shift towards large-scale healthcare systems impacts surgical education. Surgical educators reported prioritization of clinical productivity over educational objectives, with many expressing experiences of devaluation, exploitation, and burnout as a result. However, many institutions acknowledge the important role of national organizations in protecting education standards. Through these avenues, surgeon educators can advocate nationally to increase investment in educational initiatives.
(S1000) TEST
Test authors; Test Institutions