Relevance and focus shift: New metrics for the grant evaluation process pilot tested on nih grant applications (rip)

Among the challenges faced by program staff in research funding organizations is obtaining an early assessment of the suitability of grant applications received in response to new funding announcements. Here we present two new metrics that use text mining to provide rapid and objective characterization of grant applications. This pilot study assesses the relevance and focus shift of grant applications submitted to the National Cancer Institute's (NCI) Provocative Questions (PQ) Initiative (RFA-CA-11-011 and RFA-CA-11-012). Relevance is measured by comparing the titles and abstracts of PQ grant applications to the background text on the PQ website summarizing the intent, goals and feasibility of each PQ. Focus Shift measures the similarity between PQ applications and prior applications submitted to the National Institutes of Health (NIH). Our results found the majority of applications to be relevant, but the relevance scores varied significantly by topic. Of the applications with very low focus shift scores, manual review of a subset found that 25-50% were very similar in scientific approach to previously submitted grant applications to other funding opportunities. The primary limitations of our automated approach are that similarity measurements are sensitive to the comparison text and often unable to distinguish subtle text differences.