Data Sculpting: Interpretable Algorithm for End-to-End Cohort Selection

Many scientific and medical analysis involves fitting a parametric model over a heterogeneous data set. The model is often chosen to be low capacity (e.g. logistic regression) in order to make statistical inference about the association between each feature and the outcome (e.g. odds ratio). However the simple model often cannot capture the heterogeneity in the data. For example, a subset of the data might follow a clean logistic relation, but other data points could follow different relations so that the fitting a logistic regression over the entire set may not find any association. In this paper, we propose a novel algorithm, Data Sculpting, for simultaneously learning to select a subset of the data while fitting the desired parametric model on the selected cohort. Data Sculpting retains the statistical inference convenience of the original model, while leveraging end-to-end differentiable optimization (via the Concrete selector) to learn interpretable rules for selecting the cohort. Extensive experiments demonstrate that Data Sculpting is efficient, robust and substantially improves over the standard approaches.