Adaptive fitting of linear mixed-effects models with correlated random effects

Linear mixed-effects model has been widely used in longitudinal data analyses. In practice, the fitting algorithm can fail to converge due to boundary issues of the estimated random-effects covariance matrix G, that is, being near-singular, non-positive definite, or both. Current available algorithms are not computationally optimal because the condition number of matrix G is unnecessarily increased when the random-effects correlation estimate is not zero. We propose an adaptive fitting (AF) algorithm using an optimal linear transformation of the random-effects design matrix. It is a data-driven adaptive procedure, aiming at reducing subsequent random-effects correlation estimates down to zero in the optimal transformed estimation space. Simulations show that AF significantly improves the convergent properties, especially under small sample size, relative large noise and high correlation settings. One real data for insulin-like growth factor protein is used to illustrate the application of this algorithm implemented with software package R (nlme).