Adaptive Feature Generation for Online Continual Learning from Imbalanced Data

Online continual learning (OCL) is the setting where deep neural network (DNN) incrementally learns new tasks with online data streams. The major problem in OCL is catastrophic forgetting, that DNN forgets the acquired knowledge on previous tasks quickly. Recently emerged studies tackle a more realistic problem that the data follows an imbalanced class distribution in OCL by storing particular exemplars. However, preserving exemplars causes memory burden and privacy issues. In this paper, we propose a non-exemplar based method-Adaptive Feature Generation (AdaFG) for OCL from imbalanced data, which tackles the class imbalance and catastrophic forgetting problems simultaneously. Specifically, we argue that one common reason for these problems is the decision boundaries of minority or old classes with few or no samples are affected by majority classes. Therefore, we first maintain a representative prototype for each class in the feature space, which dynamically changes with the streaming data to approximate the class mean feature. Then, we generate new features adaptively for old and minority classes based on their prototypes and train the DNN's classifier to adjust the decision boundaries. Experiments on three popular datasets demonstrate AdaFG's effectiveness in consolidating previous knowledge and addressing the class imbalance problem without preserving exemplars.