Shorter and Stronger: Revision of a Resident Performance Evaluation Survey

Introduction: In order to evaluate the outcomes of a USU School of Medicine (SOM) education, a program director (PD) evaluation survey was developed in 2005 and is completed annually by PDs regarding trainees who graduated from USU and are in their first post-graduate training year (PGY-1) and their third post-graduate training year (PGY-3). The survey was last reviewed and revised in 2010 to better align with the competencies established by the Accreditation Council for Graduate Medical Education but has not been further evaluated or revised. The objective of the study was to utilize 12 years of aggregated data to improve the psychometric properties of the survey, with an emphasis on shortening the survey. A secondary objective was to refine the verbiage of the existing questions and add new items to assess health systems science competencies. Methods: The survey was sent out to PDs who supervised USU SOM graduates in the classes of 2008 to 2019 (n = 1,958) with 997 responses for the PGY-1 PD survey and 706 responses to the PGY-3 PD survey. Exploratory factor analysis (EFA) was conducted on 334 complete responses of the PGY-1 survey and 327 responses of the PGY-3 survey. A working group of PDs, USU Deans, and health professions education scholars reviewed the results from the EFA and from a survey of experienced PDs and, in an iterative process, developed a proposed revised survey. Results: The EFA for both the PGY-1 data and the PGY-3 data yielded three factors, and a total of 17 items were identified with cross-loading across the factors in the PGY-1 and/or PGY-3 surveys. Items without clean loadings or that were determined to be unclear, redundant, or difficult to assess by PDs were revised or removed. Items were revised or added to address needs within the SOM curriculum, including the new health systems science competencies. The proposed revised survey had 36 items in place of the original 55 items and contained at least four items for each of the six competency domains (patient care, communication and interpersonal skills, medical knowledge, professionalism, system-based practice and practice-based learning and improvement, and military-unique practice, deployment, and humanitarian missions). Conclusions: The USU SOM has benefited from over 15 years of results from the PD surveys. We identified those questions that performed well, and these were refined and augmented in order to optimize the performance of the survey and fill gaps in our knowledge of graduates' performance. To determine how well the revised set of questions performs, efforts will be made to increase the response rate and completion of 100% of items on the survey, and the EFA should be repeated after about 2-4 years. Furthermore, the USU graduates should continue to be tracked longitudinally past residency to determine if PGY-1 and PGY-3 survey measures predict long-term performance and patient outcomes.