Universal Sound Separation Using Replay-based Data Sampling in Incremental Learning

In this paper, we investigate an incremental learning method in universal sound separation (USS). USS uses deep learning methods, which require learning large numbers of data. In most real-world situations, the number of available data increases, and it is necessary to learn additional data incrementally. One of the challenges of incremental learning is mitigating catastrophic forgetting, which is a performance degradation on past data. We used the replay-based method to reduce catastrophic forgetting in USS. We proposed three sampling methods for the replay-based method and compared their effectiveness for different properties of the reused data. We also compared the three proposed sampling methods with regularization-based and fine-tuning-based methods to investigate the effects of catastrophic forgetting. Experimental results show that many proposed methods reduce the performance degradation and improve the average separation performance by up to 0.36 dB compared with the fine-tuning-based method. The replay-based method is also found to be more effective than regularization-based methods.