WS2-05: PROTECTING THE QUALITY OF OUR ASSESSMENTS: UNDERSTANDING VALIDITY AND RELIABILITY IN SURGERY
Edmund W Lee, MD1, Tiffany N Anderson, MD1, Elizabeth M Huffman, MD2, John R Martin, MD2, Robert Naples, DO3, Dana T Lin, MD1, James N Lau, MD, MHPE1, Jeremy M Lipman, MD3, Yoon Soo Park, PhD4, Ara Tekian, PhD, MHPE4; 1Stanford University, 2Indiana University, 3Cleveland Clinic, 4University of Illinois at Chicago
Background: As Surgery moves towards competency-based education with the implementation of Entrustable Professional Activities (EPAs), the need for robust assessments is becoming ever more essential. For example, for assessment of medical knowledge through multiple-choice examinations, there is ample validity evidence ranging from rigorous question-writing guidelines to psychometric analyses. However, relevant noncognitive assessment tools lack such robust validity.
Goals and Objectives: The goal of this workshop is to discuss the concept of validity and empower learners to develop validity evidence to already-developed assessment tools. In addition, we hope to build a community of surgeons to collaborate on assessment projects.
By the end of this workshop, participants will be expected to:
- Define validity and reliability of assessment
- Understand validity as an argument and not an outcome
- Compare historical understandings of validity with Messick’s Framework of Unified Validity
- Design a plan to develop validity evidence for a variety of assessment tools
Workshop Structure and Content: To encourage inter-institutional collaboration, this workshop will be co-hosted by instructors representing four academic institutions including international experts in psychometrics and competency-based education. First, a short presentation will be delivered defining validity and its importance for assessment. Participants will then break into small groups. Each group will choose an assessment tool from a list provided and develop a strategy to build validity evidence using Messick’s Framework of Unified Validity. This will be a very interactive workshop, and participants will have an opportunity to brainstorm and discuss their ideas with each other. At the conclusion of the workshop, we will address the complexities, barriers, and threats to validity. Participants will leave with a completed worksheet and a set of guidelines to implement at their home institutions. Our expectation is for each group to use this plan to collaborate on a validity project in the future.