SciBERT: A Pretrained Language Model for Scientific Text

The paper addresses the difficulty and expense of obtaining large-scale annotated data for NLP in the scientific domain by releasing SciBERT, a pretrained language model based on BERT. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. The authors evaluate it on a suite of tasks including sequence tagging, sentence classification, and dependency parsing, drawing on datasets from a variety of scientific domains.

Across these evaluations, SciBERT demonstrates statistically significant improvements over BERT and achieves new state-of-the-art results on several of the tasks. The code and pretrained models are made publicly available. This mattered because it provided the community with a strong, reusable domain-adapted language model for scientific text, reducing reliance on scarce labeled scientific data.