Nonlinear mixed effects dose repsonse modeling in high throughput drug screens: Application to melanoma cell line analysis

Kuan Fu Ding, Emanuel F. Petricoin, Darren Finlay, Hongwei Yin, William P.D. Hendricks, Chris Sereduk, Jeffrey Kiefer, Aleksandar Sekulic, Patricia M. LoRusso, Kristiina Vuori, Jeffrey M. Trent, Nicholas J. Schork*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Cancer cell lines are often used in high throughput drug screens (HTS) to explore the relationship between cell line characteristics and responsiveness to different therapies. Many current analysis methods infer relationships by focusing on one aspect of cell line drug-specific dose-response curves (DRCs), the concentration causing 50% inhibition of a phenotypic endpoint (IC50). Such methods may overlook DRC features and do not simultaneously leverage information about drug response patterns across cell lines, potentially increasing false positive and negative rates in drug response associations. We consider the application of two methods, each rooted in nonlinear mixed effects (NLME) models, that test the relationship relationships between estimated cell line DRCs and factors that might mitigate response. Both methods leverage estimation and testing techniques that consider the simultaneous analysis of different cell lines to draw inferences about any one cell line. One of the methods is designed to provide an omnibus test of the differences between cell line DRCs that is not focused on any one aspect of the DRC (such as the IC50 value). We simulated different settings and compared the different methods on the simulated data. We also compared the proposed methods against traditional IC50-based methods using 40 melanoma cell lines whose transcriptomes, proteomes, and, importantly, BRAF and related mutation profiles were available. Ultimately, we find that the NLMEbased methods are more robust, powerful and, for the omnibus test, more flexible, than traditional methods. Their application to the melanoma cell lines reveals insights into factors that may be clinically useful.

Original languageEnglish
Pages (from-to)5044-5057
Number of pages14
Issue number4
StatePublished - 2018
Externally publishedYes


  • Bioinformatics
  • Cancer
  • Drug response
  • High throughput drug screen
  • Nonlinear mixed effect models


Dive into the research topics of 'Nonlinear mixed effects dose repsonse modeling in high throughput drug screens: Application to melanoma cell line analysis'. Together they form a unique fingerprint.

Cite this