BioBERT: a pre-trained biomedical language representation model for biomedical text mining

BioBERT is a domain-specific language representation model that adapts BERT for biomedical text mining, motivated by the observation that directly applying general-domain NLP models to biomedical text works poorly because of a word-distribution shift from general corpora to biomedical corpora. It is pre-trained on large-scale biomedical corpora and keeps almost the same architecture across tasks so it can be fine-tuned for different biomedical applications.

When pre-trained on biomedical text, BioBERT largely outperforms BERT and previous state-of-the-art models, improving biomedical named entity recognition by 0.62% F1, relation extraction by 2.80% F1, and question answering by 12.24% MRR. The results show that domain-specific pretraining helps the model understand complex biomedical texts, and the authors released the pretrained weights and fine-tuning source code freely.