ASE 2024 Abstracts

Objective surgical skill assessment is critical for medical education. Existing evaluation methods largely depend on expert judgments, which are subjective and time-consuming. This study introduces a computer vision-based approach for automatic assessment of open surgical skill, specifically focusing on suturing proficiency.

We employed a suture pad simulator and engaged 23 participants to perform a running suturing task. Additionally, 28 participants with postgraduate experience greater than three years were invited to rate the expertise displayed in each image, on a 1-5 Likert scale. These ratings served as our ground truth. The study utilized two advanced machine learning algorithms, YOLOv8 and ResNet, for image processing. YOLOv8 was used to delineate the stitches by drawing bounding boxes, while ResNet determined the orientation of each suture within its respective bounding box. 

Three metrics were calculated to quantify the surgical skill: parallelism between stitches, measured as the average angle between sutures; variation in distances between each pair of adjacent sutures; and variation in the length of individual stitches. We hypothesized that higher-ranked images, deemed "expert", would exhibit smaller angles and lower variations in both suture distance and stitch length.

Spearman correlation tests revealed statistically significant correlations between the calculated metrics and the expertise ratings for each task. As the first metric got P-Value of 0.005, the second P-Value of 0.029 and the last P-Value of 0.008 – all were below the 0.05 statistical significance threshold. 

Our findings validate the potential of using computer vision algorithms to provide an objective, automated assessment of surgical skills in open suturing tasks. This approach could significantly enhance existing evaluation methods and contribute to more standardized and effective surgical training programs.

Introduction:

Self-assessment of technical skills stands as a fundamental pillar of surgical training, serving the dual purpose of precisely identifying and subsequently rectifying deficiencies. Here, we aimed to assess temporal trends in surgical interns’ ability to self-assess their technical skills compared to independent reviewers.   

Methods:

Between 2017-2019, general surgery residents at a tertiary care academic center self-assessed their technical performance in five inanimate tasks (knot tying, suturing, Fundamentals of Laparoscopic Surgery peg transfer, intracorporeal suturing, and vascular anastomosis) at the start of their intern year and at six to twelve months follow-up. Initial self-assessments were compared to blinded reviewer scores, and follow-up self-assessments and actual scores were compared to the initial attempt. Module scores were normalized on a 0 to 1 scale. Statistical comparisons were performed using the Mann-Whitney and Wilcoxon matched-pairs signed rank tests with significance set at p <0.05.

Results:

Twenty-nine residents completed both initial and follow-up self-assessments during their intern year. Median composite self-assessment scores (IQR), combining all five inanimate tasks, were 0.537 (0.453, 0.590) for the initial attempt and 0.547 (0.505, 0.626) on follow-up. Median composite actual scores for the initial and follow-up attempts were 0.659 (0.605, 0.746) and 0.739 (0.681, 0.797), respectively. Interns consistently underestimated their composite scores compared to actual scores on both the initial (p<0.0001) and follow-up attempts (p<0.001). Although self-assessment scores showed no significant difference between the initial and follow-up attempts (+1.86%, p=0.069), a significant improvement was observed in actual scores between these attempts (+12.14%, p<0.0001). 

Figure 1 illustrates the median score distributions for the initial and follow-up assessments across the five inanimate tasks. In contrast to composite outcomes, task-specific analysis revealed that residents accurately self-assessed their improvement (or lack thereof) over time in knot tying, intracorporeal suturing, and vascular anastomosis – modules representing the technical extremes among the completed tasks.

Conclusion:

Overall, interns’ self-evaluation of operative skills lags behind their actual technical abilities, and this gap persists over time, regardless of their training progression or experience. Implementing structured training programs, particularly for tasks that demand a more nuanced assessment, could perhaps enhance interns’ ability to identify areas in need of improvement.

Introduction: Resuscitative thoracotomy (RT) is a high-stakes yet low frequency procedure performed in the emergency room by trauma surgeons.  Due to the low volume of RT even among high volume level I trauma centers, there is variable trainee experience.  A plastic simulator was designed with a beating heart to simulate the procedure.  The aim of this study was to establish the bar of competency using this novel simulator.

Methods: A standardized introduction to the procedure and the simulator were carried out.  The previously published assessment form was used to assess the videos of trainees performing the procedure.  All the trainees underwent pre-procedural training followed by the procedure.  Each trainee was video recorded and the recording was reviewed and assessed asynchronously by the reviewers. Three reviewers reviewed each video independently.  Descriptive statistics were used to assess the scores and item analyses performed.  Correlation between key steps and general/global skills were analyzed and Cronbach alpha score calculated. 

Results: 15 senior level residents (PGY3-5) completed the pre-procedural training and the procedure.  All videos were included.  The mean score for the key steps and general skills, respectively was 3.09 (0.74 SD) and 2.67 (0.74).  The mean global score was 2.43 (SD 0.78).  Item analysis demonstrated that the opening of the chest/use of the rib spreader, time and motion, and handling of instruments had the highest level of discrimination 0.80, 0.80, and 0.75, respectively.  These items were effective in distinguishing different skill level and had the highest correlation with the global score.  The correlation between key steps and global score was 0.86 and between general skills and global score was 0.92.  The Cronbach alpha was 0.90.  A global score of 3.0 was deemed to be an appropriate bar for competency.

Conclusions: A score of three or higher correlated with what the experts deemed to be competent on the RT trainer using the previously published assessment tool.  The assessment tool effectively discriminated between various levels of skill used in conjunction with the RT trainer.  The RT trainer and the assessment tool are effective means to assess the basic level of competency of surgical residents.   

Introduction: Many surgical residency programs offer an intern bootcamp at the beginning of training that includes simulation of various surgical skills. Our institution has also implemented a retest of these same skills near the end of the intern year. In addition to measuring changes in skill-specific performance scores, we sought to examine learners’ perceptions of difficulty and confidence in skills at various timepoints.

Methods: During an annual simulation program for our 22 Surgery PGY1s, task performance scores were collected from 6 simulated tasks (simple suturing, open knot tying, fascial closure, laparoscopic peg transfer, laparoscopic circle cut, and laparoscopic suturing) at month 0 and month 10 of the intern year. Self-perception ratings of confidence and perceived difficulty were also collected from each participant at three intervals: before skill completion at month 0, after skill completion at month 0, and at month 10. These scores and self-perception ratings were compared using Wilcoxon signed rank tests.

Results: Performance scores for open skills increased throughout the year, while scores for advanced laparoscopic skills, specifically circle cut and intracorporeal laparoscopic suturing, did not increase significantly. However, intern confidence in all skills, including circle cut and laparoscopic suturing, increased significantly from 0 to 10 months. Additionally, when comparing perceived difficulty between post-skill 0 month and post-skill 10 month, we noted a significant decrease for all skills, meaning learners thought the skills were less difficult at month 10 than month 0.

Discussion: Our findings demonstrate that confidence levels and perceived difficulty of advanced laparoscopic skills in particular do not correlate with skill performance levels, and more importantly, trainees may have an inflated sense of ability that is not warranted. Inflated confidence in early learners can be harmful to their learning, as interns may attempt more advanced procedures that they are unequipped to perform too early, and neglect refinement of the more basic skills. These findings are important for faculty to recognize the gap that may exist between an intern’s reported self-confidence in a skill, and their ability to correctly perform said skill. This gap could be bridged with more dedicated simulation time, particularly for advanced laparoscopic skills.

Background

OPTIMAL (Optimizing Performance Through Intrinsic Motivation and Attention for Learning) theory proposes that enhanced expectancies, increased autonomy, and having an external focus of attention improve motor learning. Similar to findings during motor skill acquisition in sports, we hypothesized that enhancing expectancies by decreasing the perceived difficulty of a surgical task would improve acquisition, retention, and skill transfer.

Methods

We conducted a randomized controlled trial comparing the use of an optical illusion which decreased perceived difficulty (enhanced expectancies [EE]) versus no illusion (normal expectancies [NE]) while learning an arthroscopic targeting task. Medical students at one school consented to participate and were randomized to EE or NE groups. Both groups followed the same training sequence using a simulated arthroscopic model: pre-test, training, post-test (immediately post-training), and retention (24-hours later). During training, the EE group targets had an Ebbinghaus optical illusion (targets appeared larger than reality), while the NE group had no illusion. During post- and retention testing, performance (integrated measure of number of attempts and deviation from target) was assessed without illusion on the same and on a different (transfer) task. We compared post-test, retention, and transfer performance between EE and NE groups, controlling for pre-test performance, MS year, handedness, and gender.

Results

Overall, 67 medical students (52% male) volunteered to participate. Most students were in their first (43%) and fourth (42%) years. During training, the EE group had decreased performance compared to NE (mean difference=0.50, 95% CI=[-0.03,0.97], p=0.04). No difference in performance was noted based on study group (EE versus NE) during post-test (mean difference=-0.31, 95% CI=[-1.04,0.42]), immediate transfer (mean difference=-0.16, 95% CI=[-1.06,0.74]), retention (mean difference=0.44, 95% CI=[-0.21,1.09]), and retention transfer (mean difference=-0.42, 95% CI=[-1.26,0.42]).

Conclusion

We found no difference in performance, transfer, or retention of arthroscopic targeting skills when using the Ebbinghaus illusion to decrease perceived task difficulty during training (enhanced expectancy). Given benefits observed in sports motor skills, it is yet possible that other EE interventions could be beneficial in surgical skills learning; further study is warranted.

Background: Africa bears 25% of the world’s burden of surgical disease with only 2% of the global health workforce. Laparoscopic surgery is now the standard of care for many surgical conditions and yet laparoscopic surgical training remains inaccessible to practicing surgeons in low- and middle- income African countries (LMICs). Low cost, effective, and locally accessible training is critically needed in this region.

Methods: ALL-SAFE is a global surgical collaborative that designed a low-cost laparoscopy training system. Modules simulating laparoscopic salpingostomy for ectopic pregnancy and laparoscopic appendectomy for appendicitis were constructed using locally-accessible materials. A 2-day laparoscopic skills course was conducted by ALL-SAFE in a large national hospital in Abuja, Nigeria in October 2023. During this course, participants received didactic lectures on the physiology and mechanics of laparoscopic surgery and completed basic skills tasks (peg transfer and circle cut) as well as the ALL-SAFE ectopic pregnancy and appendectomy simulated procedures on their own box trainers. Participants completed pre- and post-course surveys to assess their confidence in the knowledge and skills required for laparoscopic surgery.

Results: Of the 34 course participants, 100% completed the pre-course survey and 67% (n=23) completed the post-course survey. 82% (n=27) of course participants were male. 45.5% (n=15) were general surgeons, 24.2% (n=8) were urologists, and the remaining represented other surgical subspecialties. While most participants had observed laparoscopic cases in the operating room (57.6%), 51.5% said that this ALL-SAFE course was their first exposure to laparoscopic simulation. Prior to the course, 48.5% of participants felt “not at all confident” and 36.5% felt “slightly confident” in performing a laparoscopic appendectomy. Following the course, 39.1% of participants felt “very confident” and 34.8% felt “moderately confident” [Figure 1].  All participants felt that they were successful at achieving their educational goals for the course (17.4% moderately successful, 56.5% very successful, 26.1% extremely successful).

Conclusion: A short and low-cost laparoscopy training course was successful at improving confidence in performing laparoscopic surgery and achieving the educational goals of a diverse group of Nigerian surgeons. Hosting similar courses in other African LMICs may improve dissemination of laparoscopic surgical skills in this region.

Introduction:

General surgery residents are now expected to excel in robotic surgery as part of their residency training. However, there is a significant gap in addressing emergency undocking training within these programs, particularly for high-stakes situations involving bleeding. To address this need, we have developed an interprofessional curriculum for teaching emergency undocking using simulation models. This curriculum is designed to seamlessly integrate into general surgery residency robotic training programs and emphasizes early exposure and structured practice for both surgeons and the entire operating room (OR) team. Our simulation models, including a high-fidelity cadaver model and a moderate-fidelity explant model, provide a controlled environment for residents to gain essential experience in effectively managing emergency situations.

Methods:

To evaluate the effectiveness of our curriculum and training approach, we conducted a study involving 52 participants, comprising 33 surgical residents, 15 nurses, and 4 scrub technologists. We assessed their confidence levels and conducted knowledge assessments tailored to their respective roles. These assessments were conducted both before and after completing the training lab simulation.

Results:

Our analysis of pre- and post-confidence levels within each staff category revealed significant improvements in confidence levels among residents (p < 0.001), nurses (p < 0.001), and scrub technologists (p < 0.008). Regarding the knowledge assessment, we also conducted separate statistical analyses for each staff category: residents, scrub technologists, and nurses. All these analyses yielded statistically significant results, with p-values of 0.006, 0.050, and <0.001, respectively.

Conclusion:

Our comprehensive assessment of the training program's impact on participants' confidence levels and knowledge across different staff categories has produced highly encouraging results. These findings demonstrate significant improvements in confidence levels among residents, nurses, and scrub technologists, emphasizing the program's effectiveness in boosting the confidence of all team members involved in robotic surgery emergencies. Additionally, our knowledge assessment reaffirms the value of our interprofessional approach in evaluating the performance of each staff position individually. The updated results, based on a larger dataset and a breakdown by position, further highlight the effectiveness of this interprofessional approach in assessing the performance of each staff position individually.