OPT: Open Pre-trained Transformer Language Models

OPT (Open Pre-trained Transformers) is a suite of decoder-only pretrained transformer language models ranging from 125M to 175B parameters, released to counter the fact that large language models are expensive to train and that the few available through APIs do not grant access to full model weights. The authors aim to share the models fully and responsibly with interested researchers so they can be studied directly rather than only through restricted interfaces.

The paper reports that OPT-175B is comparable to GPT-3 while requiring only one-seventh of the carbon footprint to develop, highlighting a more efficient and transparent path to building large models. Alongside the weights, the team releases a logbook documenting the infrastructure challenges they faced and code for experimenting with all released models, making large-scale LLM research more accessible and reproducible.