Finite mixture of varying coefficient model: Estimation and component selection

Heterogeneous longitudinal data have become prevalent in medical, biological, and social studies. This paper proposes a finite mixture of varying coefficient models for handling heterogeneous populations. Each component of the mixture is modeled by a varying coefficient mixed-effect model that characterizes the longitudinal relations among variables. The identifiability of the mixture model is studied. Regression splines with equally spaced knots are used to approximate the varying coefficient functions, and a nested expectation maximization algorithm is developed to obtain the maximum likelihood estimation. We propose a penalized likelihood method based on the smoothly clipped absolute deviation (SCAD) penalty for the component selection of finite mixture of varying coefficient model. A modified BIC-based criterion based on the SCAD penalty, the BICSCAD, is proposed for selecting the penalty parameter and spline space simultaneously. The asymptotic properties of parameter estimation and component selection consistency are studied under mild conditions. Simulation studies are conducted to illustrate the component selection, parameter estimation, and inference of the proposed method. The model is then applied to a heterogeneous longitudinal data set from a study of the treatment effect on the use of heroin in the California Civil Addict Program. (C) 2019 Elsevier Inc. All rights reserved.