Do You See What I See? A Comparison of CCC and Self-Assigned Milestones Across Military Medical Specialties

Background: The Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to complete competency-based assessments of medical trainees based on nationally established Milestones. Previous research demonstrates a strong correlation between CCC and resident scores on the Milestones in surgery, but little is known if this is true between specialties. In this study, we investigated a variety of specialties and sought to determine what factors affect self-assessment of milestones. In addition, a post-hoc analysis was completed on the COVID-19 pandemic effects on self-evaluation. Methods: This is an IRB approved observational study on prospectively collected self-evaluation milestone data that is used within each ACGME program's Clinical Competency Committees. Medical trainees within the San Antonio Uniformed Services Health Education Consortium were approached for possible participation in this study with permission from program directors. Results: There was no significant difference between self-assessments and CCC-assessments based on self-identified gender or residency type (surgical versus nonsurgical) for any milestone domain. Within the postgraduate year (PGY) groups, the PGY5 and PGY6 tended to rate themselves higher than CCC. Chiefs (Internal Medicine PGY2/3, and General Surgery PGY5/6) tended to be more accurate in scoring themselves than the interns (PGY1) within the milestone of Interpersonal Skills and Communication (chiefs 0.5 vs. interns 0.62, p = 0.03). On post hoc analysis of self-rating, during the first wave of the COVID 19 pandemic, Post-Covid residents were more likely to underrate themselves in Systems-Based Practice compared to the Pre-Covid cohort (-0.49 vs 0.10; p = 0.007) and more likely to rate themselves higher in Professionalism (-0.54 vs. -0.10, p = 0.012). Conclusion: Unique to this study and our institution, there was no gender difference found in self vs CCC evaluations. With the change in learning environment from COVID, there was also a change in ability for some learners to self-assess accurately. As medical educators, we should understand the importance of both encouraging learners to practice self-assessment as well as give feedback to trainees on their progress. We also need to educate our faculty on the use of milestones for assessment to create a true gold standard in the CCC.