ASE 2024 Abstracts

Podium IA - Assessment

(S006) CASE LOGS AMONG GENERAL SURGERY RESIDENTS VARY BY GENDER AND PGY LEVEL
Erin M White, MD, MBS, MHS¹, Gurjit Sandhu, PhD²; ¹University of Washington, ²University of Michigan

Background: Prior studies of General Surgery residents have shown a disparity in the number of cases logged by men and women residents based on category minimum reports at time of graduation. We sought to explore these disparities at a more granular level by analyzing case logging patterns over the course of residency. 

Method: We analyzed the ACGME case logs of categorical residents, July 2016 to June 2023, from a convenience sample of six university-based general surgery residency programs. Trainee gender was determined by self-reported data in SIMPL. Data were analyzed using independent sample t-tests, grouping by gender. Subanalyses were performed for cases logged at each PGY level. 

Results: 537,454 ACGME case logs for 416 residents were included. 204 residents were men and 212 were women. 224 had completed their PGY5 year and 192 were currently in training. On average, female residents logged significantly fewer cases overall (1228.6 vs 1357.8, p=0.02) and fewer “teaching asst.” cases (47.2 vs 54.8, p=0.04), but they logged more cases as “first assistant” (61.0 vs 54.3, p=0.17). There was no gender disparity seen among PGY1 level case logs for either number of cases (208.4 vs 217.6, p=.27) or the proportion of cases logged as a “first assistant” (15.7% vs 15.9%, p=.45), however significant differences were observed among the higher PGY levels (see figure). 

Discussion: Beyond their intern year, female general surgery residents tend to log fewer overall cases and fewer teaching cases but a higher proportion of first assistant cases compared to their male peers. This first assistant disparity may contribute to the differences seen in overall case log numbers, as there is little incentive to log first assistant cases beyond the PGY2 year after the 250 junior case log minimum has been met. However, we noted many residents do continue to log first assistant cases even into their PGY3, 4 and 5 years. Additional research is needed to better understand the cultural and structural variability in case assignments and perceived roles in the operating room. 

(S007) TRAUMA RESUSCITATION LEADERSHIP IS MORE THAN NON-TECHNICAL SKILLS: REVISED NOTECHS SCORES SHOWS GOOD RELIABILITY, BUT SHOWS VARIABILITY ACROSS PATIENTS MANAGED BY THE SAME ATTENDING
Maria Sfakianos, MD, FACS¹, Daniel Jafari, MD¹, Susan Steinemann, MD, FACS², Eric Klein, MD, FACS¹, Cristy Meyer, MSNRN, CEN, TCRN¹, Manuel Beltran del Rio, MSc, PhD¹, Rashmeet Gujral¹, Matthew Bank, MD, FACS¹; ¹Northwell Health, ²University of Hawaii, John A. Burns School of Medicine

Background: Methods to evaluate teamwork and communications in high stress situations such as trauma includes the T-NOTECHS instrument to evaluate non-technical skills in five categories: leadership, cooperation, communication, assessment and decision, and situational awareness during trauma resuscitations. Video review assessments have been found to offer trainees opportunities for self-assessment of technical skills. T-NOTECHS performance stratified by trauma leader as well as assessors experience is understudied. We hypothesized that the revised T-NOTECHS scale (R-NOTECHS) would yield a reliable assessment of trauma leader, independent of assessors’ experience in the evaluation of non-technical skills in Level 1 trauma activations.  

Methods: Video recordings of all Level 1 (highest level) trauma activations at two Level 1 trauma centers from April 2023 to March 2024 were included. We utilized a modification of the validated T-NOTECHS instrument to measure five domains and using a 3-point scale (1= needs improvement, 2= moderately-executed, and 3= well-executed). A total R-NOTECHS score was calculated during trauma video review by various levels of experience, ranging from trauma and emergency medicine attendings to nursing students. Gwet AC2, a measurement of inter-rater reliability was used to evaluate the agreement stratified per video review and per trauma leader. An AC2 score were grouped as poor agreement (<0.4), some agreement (0.4-0.6), good agreement (0.6-0.8) and excellent agreement (0.8-1.0). 

Results: Nine hundred ninety-six R-NOTECHS scorecards from 133 video reviews with 19 trauma leaders were submitted. When assessing per video review, 81% of the AC2 agreement scores were good or better and 53% were excellent. When assessing per trauma leader there was 63% agreement in the good or better group and 21% in the excellent group. The median AC2 grouping by video review was 0.81 vs 0.65 grouping by trauma leader (p=0.01).  

Conclusions: The R -NOTECHS score is a reliable objective instrument in assessment of nontechnical skills when assessing individual video review cases. The same trauma leader score is likely determined more by other variables of the case. More studies may elucidate which variables can affect non-technical skill assessment.  

(S008) CONTROLLED AND AUTONOMOUS MOTIVATIONS OF HIGH AND LOW EPA USERS AMONG GENERAL SURGERY RESIDENTS
Ye Lim Sarah Lee¹, Alyssa Murillo, MD, MS, MAEd¹, Camilla Gomes, MD, MS¹, Kara Faktor, MD, MS¹, Riley Brian, MD, MAEd¹, Olle ten Cate, PhD², Adnan Alseidi, MD, EdM¹, Patricia O'Sullivan, EdD, MS¹, Lan Vu, MD¹; ¹University of California San Francisco, ²UMC Utrecht

Introduction:

Since the American Board of Surgery introduced Entrustable Professional Activities (EPA) as the primary assessment framework for general surgery residents, completion has varied. Although studies have identified systemic facilitators and barriers accounting for some of this variability, individual-level motivations merit exploration. We aim to describe the motivations of residents with high and low numbers of EPA assessments through the lens of self-determination theory (SDT), that is, to what extent the completion of EPA assessments seems to satisfy the basic psychological needs for autonomy, competence, and relatedness.

Methods:

We reviewed EPA assessments for general surgery residents (PGY1-4) at a quaternary academic institution from 06/2023-06/2024. We calculated the median number of assessments to classify residents as either high (at or above median) or low (below median) EPA users. Residents participated in a 15-minute semi-structured interview until we reached information sufficiency. Qualitative codes were generated deductively sensitized by SDT principles and by inductive coding for new ideas. We applied these codes in a directed content analysis to identify themes.

Results:

Of 32 residents, 19 were high EPA users (median=19) and 13 were low EPA users (median=11). We conducted 17 interviews with 11 high and 6 low users. Rather than identifying differences in motivation, we found that both groups shared similar perspectives on obtaining EPA assessments (Figure). Residents noted that faculty initiated most assessments, valuing factors supporting faculty engagement. When residents did seek assessments, their motivation was shaped by external regulators and the needs for autonomy, competence, and relatedness. The tension between viewing EPA assessments as learning tools for applying feedback and as sources of redundant, nonspecific feedback influenced residents’ autonomous initiation. Competence was driven by a desire for mastery but hindered by fear of negative evaluations that could challenge self-perceived competence. Completions were increased by factors of relatedness, including peer completion rates and social comparisons.

Discussion:

High and low EPA users share external regulators, faculty-related influences, and basic psychological needs underlying the motivation to obtain EPA assessments. Understanding these factors can help optimize the design and implementation of EPA assessments to better support resident engagement in feedback processes.

(S009) PATIENT EVALUATION OF WHITE VS NON-WHITE SURGICAL INTERNS AND SENIOR RESIDENTS
Tiffany R Bellomo, MD, Shaghayegh S Kermani, BS, Arian Mansur, BS, Roy Phitayakorn, MD, Sophia K McKinley, MD, MED; Massachusetts General Hospital

Background: Multisource feedback is a crucial component of resident education that is mandated by the ACGME. As such, there is increasing interest in incorporating non-faculty sources of feedback, such as patient feedback, into surgical resident evaluation. However, there is also recognition that systematic bias in any new forms of surgical resident evaluation could further disadvantage historically marginalized groups. This exploratory study investigated whether patient evaluations differ for white vs non-white residents to better characterize potential patient biases with regards to assessment of resident care.

Methods: Surgical inpatients who underwent elective gastrointestinal and oncologic surgery evaluated the quality of surgical resident care using a modified Consumer Assessment of Healthcare Providers and Systems Surgical Care Survey (S-CAHPS). Only patients who positively identified a resident caregiver by photo were permitted to complete the resident evaluation. The top-box method, which quantifies the frequency of the highest possible rating for each question, was used to describe patient satisfaction with surgical resident care in a manner consistent with S-CAHPS reporting guidelines. Patient recognition and evaluation of white vs non-white residents, stratified by resident level, were compared.

Results: A total of 324 patients agreed to participate in the study (91%, n=324/357). There were no significant demographic differences between patients cared for by white vs non-white residents at either the intern or senior level (age, gender, or education level). Patient recognition rate of white vs nonwhite residents did not differ at either the intern (white 80.2% [170/212] vs 87.5% [91/104] non-white, p-value=0.51) or senior level (white 84.6% [230/272] vs 85.1% [40/47] non-white, p-value=1.00).  Top-box S-CAHPS scores did not significantly differ between white and non-white residents for both intern (p>0.36) and senior (p>0.10) residents across all queried domains (Figure 1).

Conclusions: Patient recognition rate and quality of care evaluations were overall similar between white and non-white surgical interns and senior residents, suggesting that patients may be a potentially equitable source of resident assessment. Further study is needed to explore how to best use patient evaluation to support all trainees as they prepare for independent practice.

(S010) DEVELOPMENT AND VALIDATION OF A SIMULATION TASK FOR QUANTITATIVELY ASSESSING ORAL CANCER RESECTION SKILLS
Kayo Sakamoto, MD¹, Sohei Mitani, MD, PhD², Naoki Nishio, MD, PhD³, Takashi Kitani, MD, PhD², Eriko Sato, MD, PhD², Keiko Tanaka, MD, PhD¹, Toru Ugumori, MD, PhD⁴, Hiroyuki Wakisaka, MD, PhD⁵, Naohito Hato, MD, PhD¹; ¹Department of Epidemiology and Public Health, Ehime University Graduate School of Medicine, ²Department of Otolaryngology-Head and Neck Surgery, Ehime University Graduate School of Medicine, ³Department of Otorhinolaryngology, Nagoya University Graduate School of Medicine, ⁴Ugumori ENT Clinic, ⁵Ehime Prefectural University of Health Science

Purpose:
The 5-year survival rate of oral cancer patients has not improved during the past three decades and quality of the tumor resection is reportedly an important independent predictor of patient survival. However, no simulation training on oral cancerresection exists despite the technical difficulty. Therefore, this study aimed to develop and validate an oral cancer simulation task that could quantitatively assess tumor-resection skills.

Methods:
In this prospective study, Messick’s contemporary framework was used. A focus group of six board-certified head and neck surgeons facilitated articulation of experts' understanding of oral cancer resection into two primarycategories: "ensuring adequate margins" and "maintaining safety." According to the results, we developed a simulation task (Fig. A). "Ensuring adequate margins" in oral cancer resection was assessed using extracted phantom-model specimens (Fig. B); "maintaining safety" during electrosurgery was assessed by the degree of tumor-bed carbonization during electrosurgery (Fig. C). The validity evidence sources of the task were gathered from questionnaire responses and quantitative measurements obtained by performing the task.

Results:
Ten board-certified head and neck surgeons completed the questionnaire after performing the task. The replication of oral cancer resection was rated highly (4.40±0.52 out of 5 points), and experts agreed with quantitative measures used to assess surgeons' skills (4.80±0.42 out of 5 points), supporting content evidence in the Messick’s framework). Five experts and 12 students performed the task, and their skills were measured. All participants were confirmed to follow the instructions while performing the task, supporting response process. The internal consistency of the measures was good (Cronbach’s alpha: 0.803), supporting internal structure. Margin-error distances were significantly smaller for experts than for students (0.79 mm vs 2.45 mm; P<0.0001), and spectral colorimetric analysis revealed that lesser tumor-bed carbonization was achieved by experts than by students (2.33 vs 8.70; P<0.0001), supporting relations to other variables. Principal component analysis showed that experts’ performance was stable, whereas students’ skill was varied supporting consequences.

Conclusions:
We developed an oral cancer simulation task to quantitatively measureoral cancer resection skills. Using this simulation task may promote early acquisition of tumor-resection skills and improve outcomes for oral cancer patients.

(S011) DOES EVERYONE WHO STRUGGLES ALWAYS STRUGGLE?
Nicole K Roberts, PhD, V. Prasad Pola, MD, FACS, Margaret Boehler, RN, MS; Southern Illinois University School of Medicine

Introduction

When confronted with struggling residents, programs must balance the learner’s desire to complete training and enter a career in medicine with future patient safety. Most programs aim to keep their learners in the program, working to remediate those who have problems. They trust that learners with problems will straighten out, whether in their current program, in the next step of their training, or with the help of a mentor in practice. We sought to understand if identified performance problems during residency training indicate an increased risk for difficulty in practice.

Methods

We reviewed the records for 104 residents in our surgical training program from 1977-2014, using a previously developed checklist to identify and characterize resident performance problems. We used online resources to review various post-residency indicators suggestive of negative outcome for these residents. We tested ABSITEs and number of problems as outcome-predictors using ROC .

Results

Residents who had substantial problems (noted more than once or highly consequential) were at increased risk of board failure, negative online mentions, and overall negative post-residency indicators.  The ROC model quality on ABSITE scores was best for the fifth year. The wide range of scores limited its usability. The ROC analysis for number of unique problems revealed a decent model. More than four unique problems indicated a higher risk of negative post-residency indicators. 

Table: Relative risk of negative post-residency indicators

Conclusion

A resident with more than four unique problems is at higher risk for post residency failure and warrants serious discussion about their “goodness-of-fit” in surgical practice.

(S012) EVALUATING PATTERNS IN TECHNICAL AND NONTECHNICAL SKILLS FEEDBACK FOR SURGICAL RESIDENTS: INSIGHTS FROM SIMPL APP DATA
Madeline R Cloonan¹, Elizabeth R Maginot, MD¹, Kelsey R Tieken, MHPTT, MD¹, Shaheed Merani, MD, PhD¹, Nancy Schindler, MHPE, MD², Tiffany N Tanner, MD¹, Abbey L Fingeret, MHPTT, MD¹; ¹University of Nebraska Medical Center, ²Pritzker School of Medicine

Introduction:

Safety in the operating room (OR) requires both technical skills (TS) and nontechnical skills (NTS). Feedback residents receive on NTS remains underexplored. We sought to determine whether TS or NTS feedback is more commonly documented in SIMPL. We hypothesize that TS feedback will be most frequent, with feedback types varying by supervision type, resident performance level, and academic rank.

Methods:

This retrospective analysis reviewed SIMPL app data from a midwestern general surgery program (March 2022-April 2024). Variables included resident year, gender, academic rank, case complexity, performance level, and feedback type, categorized as TS, NTS, both, or comments without feedback. NTS feedback was further classified using the NOTSS framework.

Descriptive statistics summarized feedback types. A chi-square goodness-of-fit test assessed if TS feedback was more frequent. Associations between feedback type and categorical variables were examined using chi-square tests, with linear-by-linear association tests for ordinal trends.

Results:

Of 902 attending assessments, 566 (63%) included dictated feedback. TS feedback was significantly more common (p < 0.001), with 68% of entries as TS-only, 11% as both TS and NTS, 4.2% as NTS-only, and 14% as comments without feedback. NTS domains included leadership (7.6%), situational awareness (6.2%), decision-making (5.5%), communication (3%), and teamwork (2%). TS feedback was more likely with active help, while passive help and supervision-only were associated with combined TS and NTS feedback (p < 0.001). Intermediate performers received more TS-only feedback, while expert-level performers were more likely to receive both TS and NTS feedback (p < 0.001). Associate professors primarily provided TS-only feedback, while full professors tended to give feedback that was neither TS nor NTS (p < 0.001).

Feedback type was not significantly associated with PGY level (p = 0.399), case complexity (p = 0.068), attending gender (p = 0.418), resident gender (p = 0.073), or resident-attending gender concordance (p = 0.482).

Conclusion:

Our analysis indicates that SIMPL dictated feedback is associated with TS feedback more frequently than NTS feedback. Although NTS feedback may occur verbally, it is less often captured in SIMPL. Enhancing NTS documentation could provide residents with a more comprehensive view of their performance.

(S013) NON-CORE EPAS ENHANCE UNDERSTANDING OF RESIDENT ENTRUSTMENT PROGRESSION
Alyssa Murillo, MD, MS, MAEd¹, Sarah Lee¹, Camilla Gomes, MD, MSc¹, Kara Faktor, MD¹, Riley Brian, MD, MAEd¹, Olle ten Cate, PhD², Adnan Alseidi, MD, MAEd¹, Patricia O'Sullivan, EdD, MS¹, Lan Vu, MD¹; ¹University of California San Francisco, ²UMC Utrecht

Introduction:

Although the American Board of Surgery established 18 core general surgery Entrustable Professional Activities (EPAs), non-core procedures make up a substantial portion of residents’ case volumes. The role of non-core procedures as EPAs remains unclear. We aimed to identify non-core procedures with EPA assessments and to compare the entrustment supervision (ES) scores between the non-core and core EPAs.

Methods:

We retrospectively analyzed EPA data from a large quaternary referral center. Faculty completed EPA assessments for the 18 core EPAs and used an "other procedure" category for operations not included in the core EPAs, which we called non-core EPAs. The total number of core and non-core EPA assessments were stratified by postgraduate year (PGY) to assess completion across training levels. We developed a mixed-effects linear regression model to compare Entrustment-Supervision (ES) scores (1 (observation) -4 (practice ready)) between core and non-core EPA groups, accounting for the influence of individual residents’ performance.

Results:

We collected 1135 core EPA assessments and 558 non-core EPA assessments between June 2022 and July 2024. As residents advanced through training, they completed more non-core EPA assessments on average, with PGY5 residents showing the highest number of non-core EPA assessments (Figure). The five most common ‘other procedures’ were Pancreatectomy (n=74), Gastrectomy (n=67), Ostomy Reversal (n=60), Pediatric Central Line (n=52), and Hepatectomy (n=45). Mixed effects modeling demonstrated a small proportion of score variability (Variance = 0.05) is attributable to differences between residents. ES scores were significantly higher on the core EPA assessments (F = 9.37, p < 0.01). PGY level was a significant predictor of ES scores (F = 279.45, p < 0.001), with average scores on both the core and non-core EPAs increasing by PGY level (Figure). With each advancing PGY level, the gap between core and non-core EPA ES scores widened, with non-core EPA scores consistently remaining lower (F = 20.01, p < 0.001).

Conclusion:

Non-core procedures often represent complex operations, with lower ES scores compared to core EPAs. Non-core EPAs provide clarity on the progression of entrustment as case complexity increases, suggesting a promising adjunct for assessing senior level residents.