DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa (Decoding-enhanced BERT with disentangled attention) is a model architecture that improves the BERT and RoBERTa models with two novel techniques. The first is a disentangled attention mechanism in which each word is represented by two vectors that encode its content and position separately, with attention weights among words computed using disentangled matrices over their contents and relative positions. The second is an enhanced mask decoder that replaces the output softmax layer to predict masked tokens during model pretraining.

These techniques significantly improve both the efficiency of pretraining and downstream task performance. Compared to RoBERTa-Large, a DeBERTa model trained on only half of the training data performs consistently better across a wide range of NLP tasks, improving MNLI by +0.9% (90.2% to 91.1%), SQuAD v2.0 by +2.3% (88.4% to 90.7%), and RACE by +3.6% (83.2% to 86.8%). The authors stated that the code and pretrained models would be made publicly available.