Propensity score matching mitigates risk of faulty inferences in observational studies of effectiveness of restoration trials

Determining effectiveness of restoration treatments is an important requirement of adaptive management, but it can be non-trivial where only portions of large and heterogeneous landscapes of concern can be treated and sampled. Bias and non-randomness in the spatial deployment of treatment and thus sampling is nearly unavoidable in the data available for large-scale management trials, and the biophysical landscape characteristics underlying the bias are key but rare considerations in analyses of treatment effects. Treatment effects from large-scale management trials are typically estimated with multivariable regression (MVR) models. However, this method is unsuited to reliable estimations of treatment effects when treated and untreated areas differ in their underlying biophysical variability. An alternative to conventional regression is to use propensity score (PS) matching, which can limit the differences in confounding variables among treatment groups and assure the data collected or selected for analysis are more consistent with a randomized and unconfounded experiment. Thus, PS is expected to identify treatment effects more accurately. We used data from a large-scale monitoring effort of a megafire to evaluate the efficacy of PS matching in making inferences on treatment effects when treatments are applied non-randomly over a large heterogeneous area. We compared the resulting inference to both traditional MVR methods and to "naive" methods that do not consider treatment allocation bias. Treatment effects varied between the different statistical methods for controlling selection bias and confounding biophysical factors. The PS-matched model revealed a weaker treatment effect of drill seeding and a greater effect of herbicide spraying on the cover of perennial bunchgrasses when compared to MVR or naive modelled estimates. The inferences from the PS-matched model are considered more reliable because the treated and untreated plots are more similar in their underlying biophysical characteristics. Synthesis and applications. Failure to consider the non-random and selective deployment of restoration treatments by managers leads to faulty inference on their effectiveness. However, tools such as propensity-score matching can be used to remove the bias from analyses of the outcomes of management trials or to devise sampling plans that efficiently protect against the bias. Failure to consider the non-random and selective deployment of restoration treatments by managers leads to faulty inference on their effectiveness. However, tools such as propensity-score matching can be used to remove the bias from analyses of the outcomes of management trials or to devise sampling plans that efficiently protect against the bias.image