C4 - 01: IMPLEMENTATION OF A COMPETENCY-BASED ASSESSMENT FRAMEWORK IN ORTHOPEDICS
Portia Kalun, MSc, Natalie Wagner, BSc, Nathan Cupido, BSc, Bradley Petrisor, MD, FRCSC, Ranil Sonnadara, PhD; McMaster University
Introduction: The transition to competency-based medical education (CBME) will require more frequent, formative assessments, which in turn can help facilitate the integration of both clinical learning and service provision. We developed a new workplace-based assessment framework that targets specific competencies in a Canadian orthopedic training program. The goal of the framework was to ensure that trainees are continually assessed in terms of skill acquisition in their transition to independent practice. The framework provides procedure/task-specific tools that can assist faculty in providing trainees with these assessments as well as formative feedback. The primary purpose of this study was to investigate the reliability and validity of these tools.
Methods: Twenty-four tools were developed and reviewed using a consensus-building process. Content experts were involved and ensured the face and content validity of the tools. Each tool contained a task-specific checklist and a 5-point global rating scale (GRS) for surgical independence. Construct validity was examined by comparing performance of residents across different levels of training. Inter-rater reliability was explored using videos of resident performance for a subset of the tools (ankle open reduction and internal fixation [ORIF], knee arthroscopy, and shoulder arthroscopy). Controlling for order effects, each video was independently rated by four evaluators.
Results: The assessment tools were able to discriminate between post-graduate year on both the checklist (χ2(4)=10.14, p<0.05) and GRS (χ2(4)=14.11, p<0.01). The subset of tools evaluated also demonstrated moderate to high inter-rater reliability for ankle ORIF (ICC=0.63), knee arthroscopy (ICC=0.86), and shoulder arthroscopy (ICC=0.91).
Conclusion: These results suggest our approach to assessment demonstrates good reliability and validity, and consistently reflects trainee performance. Using a valid and reliable approach to assess individual performance, we can ensure that graduating trainees have truly attained all of the necessary skills required for independent practice. Furthermore, building a framework for assessment around workplace-based activities will help integrate trainee learning with patient care, limiting the impact of CBME on faculty workloads. Future work will examine the potential use of these tools in making end-of-rotation decisions, and to demonstrate how this new framework guides decisions on trainee competence.