Quantifying individual variation in reaction norms: Mind the residual

Phenotypic plasticity is a central topic in ecology and evolution. Individuals may differ in the degree of plasticity (individual-by-environment interaction (I x E)), which has implications for the capacity of populations to respond to selection. Random regression models (RRMs) are a popular tool to study I x E in behavioural or life-history traits, yet evidence for I x E is mixed, differing between species, populations, and even between studies on the same population. One important source of discrepancies between studies is the treatment of heterogeneity in residual variance (heteroscedasticity). To date, there seems to be no collective awareness among ecologists of its influence on the estimation of I x E or a consensus on how to best model it. We performed RRMs with differing residual variance structures on simulated data with varying degrees of heteroscedasticity and plasticity, sample size and environmental variability to test how RRMs would perform under each scenario. The residual structure in the RRMs affected the precision of estimates of simulated I x E as well as statistical power, with substantial lack of precision and high false-positive rates when sample size, environmental variability and plasticity were small. We show that model comparison using information criteria can be used to choose among residual structures and reinforce this point by analysis of real data of two study populations of great tits (Parus major). We provide guidelines that can be used by biologists studying I x E that, ultimately, should lead to a reduction in bias in the literature concerning the statistical evidence and the reported magnitude of variation in plasticity.