PS4-01: A NOVEL TOOL TO EVALUATE RESIDENT SURGICAL ENTRUSTABLE PROFESSIONAL ACTIVITIES (SEPAS) PERFORMANCE
Xiaodong (Phoenix) Chen, PhD, Alan Harzman, MD, Amalia Cochran, MD, E Christopher Ellison, MD; The Ohio State University
Background: Resident operative performance evaluation is an essential tool that helps programs prospectively assess, review and develop resident surgical competencies and hence assists residents with self-directed improvement of their surgical entrustable professional activities (SEPAs). We developed a novel resident evaluation tool aiming to provide multi-source multi-faceted feedback for resident performance improvement and entrustment appraisal based upon SEPAs. The purpose of this study was to examine the reliability and the validity of a novel SEPAs instrument.
Methods: A prospective evaluation of six procedure-specific SEPAs instruments derived from the validated OPRS evaluation tools was conducted between April and September 2018. Each instrument includes an open-ended feedback item and a series of Likert-Scale rating items, such as resident learning goal, preparation, procedure-specific performance, guidance, general skills, plan and judgment, team management, and resident prospective entrustment. Attending, resident and a constant 3rd surgeon-observer completed the same evaluation for the observed case within 3 days of each evaluated operation.
Results: A total of 40 cases performed by 10 residents and 11 attending surgeons were observed and evaluated. The SEPAs instruments were supported by strong validity evidence (Table 1). Internal reliability was high with Cronbach’s α ranging from 0.84 to 0.94 across the six procedures. Test-retest reliability varied from 0.74 to 0.93 in the study sample.
Conclusions: The SEPAs instruments are a reliable and valid tool to assess important aspects of resident learning and surgical competencies. They provide multi-source, multi-faceted evidence that is useful for both prospective resident feedback, self-directed improvement and appraisal of entrustment.
| Content Validity | Capture important SEPAs aspects: resident learning, patient care, medical knowledge, operative performance, autonomy, and prospective entrustment. |
| Criterion Validity | Consistent with literature: residents’ prospective entrustment scores highly (all p<0.0001) correlated with step-specific guidance (r=0.72), followed by overall guidance (r=0.67), and procedure-specific performance (r=0.63); case difficulty mildly limited resident entrustment (r=-0.31, p=0.0008). |
| Construct Validity | Factor analysis revealed three latent variables - "procedure-specific autonomy and entrustment", "transferable operative competencies", "resident learning efficacy" – conveyed more than half of the SEPAs overall information (58%) in the test sample. |