On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes

We compare discriminative and generative learning as typified by logistic regression and naive Bayes. We show, contrary to a widely-held belief that discriminative classifiers are almost always to be preferred, that there can often be two distinct regimes of performance as the training set size is increased, one in which each algorithm does better. This stems from the observation-which is borne out in repeated experiments-that while discriminative learning has lower asymptotic error, a generative classifier may also approach its (higher) asymptotic error much faster.