CodeBERT: A Pre-Trained Model for Programming and Natural Languages

CodeBERT is a bimodal pre-trained model that jointly represents programming language and natural language to support downstream applications such as natural language code search and code documentation generation. It uses a Transformer-based neural architecture and is trained with a hybrid objective that adds a replaced token detection task, in which the model detects plausible token alternatives sampled from generators. This design lets CodeBERT learn from both bimodal NL-PL pairs and larger amounts of unimodal data.

After fine-tuning, CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation. To probe the knowledge it encodes, the authors build an NL-PL probing dataset and evaluate in a zero-shot setting with model parameters fixed, where CodeBERT outperforms previous pre-trained models. The results establish it as a strong general-purpose foundation for combining code and natural language.