Applying SoftTriple Loss for Supervised Language Model Fine Tuning

We introduce a new loss function based on cross entropy and SoftTriple loss, TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models. This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about 0.02-2.29 percentage points. Thorough tests on popular datasets using our loss function indicate a steady gain. The fewer samples in the training dataset, the higher gain-thus, for smallsized dataset, it is about 0.71 percentage points, for mediumsized-0.86 percentage points, for large-0.20 percentage points, and for extra-large 0.04 percentage points.