Do Standardized Patients Have Concerns About Students Not Captured by Traditional Assessment Forms?

Construct: Traditionally, standardized patients (SPs) assess students' clinical skills principally through numerical rating forms—an approach that may not fully capture SPs' concerns. SPs are students' closest approximation to real patients. To maximally benefit students' clinical training and evaluation it is important to find ways to give voice to the totality of SPs' perspectives. Background: SPs have been shown to be a reliable and valid means to assess medical students' clinical skills in clinical skills examinations. We noticed, however, that SPs often express “off the record” concerns about students, which they do not include on traditional assessment forms. Approach: To explore these “off the record” concerns, we designed a Concerns item and added it to the traditional assessment form for an end-of-3rd-year clinical skills examination shared by three medical schools. We asked SPs to use this Concerns item to identify students about whom they had any “gut-level” concerns and provided them with a narrative opportunity to explain why. SPs were informed that the purpose of the item was to help students with difficulties and was not part of the student's grade. Results: We analyzed the concerns data using quantitative and qualitative methods. Of 551 students at three schools, 223 (∼40%) had concerns recorded. Seventy students received two or more concerns. Qualitative analysis of SPs' comments revealed 3 major categories of concern: communication and interpersonal skills, history taking, and physical exam. Grouped under each were several subcategories. More than half of the written comments from the SPs related to the communication/interpersonal skills category and included subcategories commonly addressed in communications courses: lack of empathy, good listening skills, and lack of connection to the patient. They also included subcategories that in our experience are less commonly addressed: odd or off-putting mannerisms, lack of confidence, unprofessional behavior, domineering behavior, and biased behavior. Another 47% of concerns identified deficiencies in history taking and physical examination. Of the students with concerns noted by two or more SPs, SPs' narrative comments on 84%, 42%, and 48% of the students in the domains of communications, history, and physical exam respectively indicated potential problems not identified by scores on the traditional assessment form. Conclusion: The Concerns item is a narrative assessment method that may add value to traditional quantitative scoring by identifying and characterizing problematic student performance not captured by the traditional assessment form. It may thus contribute to giving fuller voice to the totality of SPs' perspective.