Adder Encoder for Pre-trained Language Model

BERT, a pre-trained language model entirely based on attention, has proven to be highly performant for natural language understanding tasks. However, pre-trained language models (PLMs) are often computationally expensive and can hardly be implemented with limited resources. To reduce energy burden, we introduce adder operations into the Transformer encoder and propose a novel AdderBERT with powerful representation capability. Then, we adopt mapping-based distillation to further improve its energy efficiency with an assured performance. Empirical results demonstrate that AddderBERT(6) achieves highly competitive performance against that of its teacher BERTBASE on the GLUE benchmark while obtaining a 4.9x reduction in energy consumption.