Estimating the 5-Year Publication Potential for Grant Awardees: Analysis of the Peer Reviewed Orthopaedic Research Program

Introduction Research-funding agencies are sometimes criticized for their ineffective review of grant proposals and for prioritizing competition for grants over project outcomes. Deficiencies in the review process for grants may limit the diversity of a program's applicant pool and restrict opportunities for publication. Our study asked what features of Peer Reviewed Orthopaedic Research Program (PRORP) grants and grant recipients were associated with successful grant outcomes, with success defined as publication of results within 5 years of receipt of funding. Materials and Methods Using data from all PRORP grants from 2009 to 2017, we built machine-learned predictive models to estimate publication within 5 years. Features included in the analysis were principal investigator characteristics (sex, degree, and institution type) and grant characteristics (research grant mechanism, primary and secondary research topics, and amount awarded). We evaluated model performance using calibration plots and then determined the models' discriminatory ability by estimating the area under the receiver operator curve and c-statistic. Then we used Brier scores to obtain an overall assessment of each model's accuracy. We ultimately selected 1 model for administrative use based on its performance measures. Results The Bayesian generalized linear model performed best (area under the curve, 0.82 [95% CI, 0.68-0.95]; Brier score, 0.17 [95% CI, 0.10-0.23]) and therefore was selected. It showed administrative utility in decision curve analysis. Awardee features most strongly associated with publication were previous award winner (no), principal investigator specialty (anesthesiology), sex (male), degree (PhD), and institution type (university). Grant features most strongly associated with publication were primary topic (stem cell therapy and chemoprevention), secondary topic (biologics), research type (in vitro study and animal validation), award mechanism (translational research), award amount > median amount ($598,000), and award years 2010-2011. Conclusions This model can aid the PRORP in identifying awardees who will successfully publish, and funding agencies and policymakers likewise can use it to apportion grants in a manner that promotes diversity across the applicant pool.