Reproducibility of literature search reporting in medical education reviews

Purpose: Medical education literature has been found to lack key components of scientific reporting, including adequate descriptions of literature searches, thus preventing medical educators from replicating and building on previous scholarship. The purpose of this study was to examine the reproducibility of search strategies as reported in medical education literature reviews. Method: The authors searched for and identified literature reviews published in 2009 in Academic Medicine, Teaching and Learning in Medicine, and Medical Education. They searched for citations whose titles included the words "meta-analysis," "systematic literature review," "systematic review," or "literature review," or whose publication type MEDLINE listed as "meta-analysis" or "review." The authors created a checklist to identify key characteristics of literature searches and of literature search reporting within the full text of the reviews. The authors deemed searches reproducible only if the review reported both a search date and Boolean operators. Results: Of the 34 reviews meeting the inclusion criteria, 19 (56%) explicitly described a literature search and mentioned MEDLINE; however, only 14 (41%) also mentioned searches of nonmedical databases. Eighteen reviews (53%) listed search terms, but only 6 (18%) listed Medical Subject Headings, and only 2 (6%) mentioned Boolean operators. Fifteen (44%) noted the use of limits. None of the reviews included reproducible searches. Conclusions: According to this analysis, literature search strategies in medical education reviews are highly variable and generally not reproducible. The authors provide recommendations to facilitate future high-quality, transparent, and reproducible literature searches.