Weighting Formulas for the Least-Squares Analysis of Binding Phenomena Data

The rectangular hyperbola, Y = abx/(1 + bx), is widely used as a fit model in the analysis of data obtained in studies of complexation, sorption, fluorescence quenching, and enzyme kinetics. Frequently, the "independent variable" x is actually a directly measured quantity, and y may be a simply computed function of x, like y = x(0) - x. These circumstances violate one of the fundamental tenets of most least-squares methods-that the independent variable be error-free-and they lead to fully correlated error in x and y. Using an effective variance approach, we treat this problem to derive weighting formulas for the least-squares analysis of such data by the given equation and by all of its common linearized versions: the double reciprocal, gamma-reciprocal, and x-reciprocal forms. We verify the correctness of these expressions by computing the nonlinear least-squares parameter standard errors for exactly fitting data, and we confirm their utility through Monte Carlo simulations. The latter confirm a problem with inversion methods when the inverted data are moderately uncertain (similar to 30%), leading to the recommendation that the reciprocal methods not be used for such data. For benchmark tests, results are presented for specific data sets having error in x alone and in both x and x(0). The actual estimates of a and b and their standard errors vary somewhat with the choice of fit model, with one important exception: the Demng-Lybanon algorithm treats multiple uncertain variables equivalently and returns a single set of parameters and standard errors independent of the manner in which the fit model is expressed.