Identifying themes within a medical school admission committee's reviews of applicants

Background: Admissions committees attempt to select the most qualified applicants based on many cognitive and "noncognitive" factors. Purpose: Identify common themes cited in the admissions committee member summaries of medical school matriculants and determine the relative frequency and importance of these themes. Methods: After reviewing a convenience sample of 150 reviewer comments, 14 qualitative themes were identified. Utterances (thematic word strings) from each of the three reviewer comments for each matriculant for 7 academic years (1989-1996) were then categorized and coded as being positive, negative, or neutral. Intra-rater and inter-rater reliabilities were calculated. Results: Utterances (n = 9299) about 981 matriculants were categorized by theme and sorted as being positive, neutral, or negative. Intra-rater reliabilities were excellent (mean K = 0.98, range 0.90-1.00). Similarly, inter-rater reliabilities were also excellent (mean K = 0.94, range 0.55-1.00 and mean K = 0.90, range 0.08-1.00). Four themes (overall summarizing comments, academic, test scores, and motivation) accounted for more than half (56%) of the utterances. Conclusions: We were able to qualitatively identify themes and provide information about how one committee weighs both cognitive and "noncognitive" factors. Admission committees should consider reexamining their process and potentially expanding, eliminating, or modifying application components.