HighlightCurated by Aramai EditorialarXiv

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Introduces Self-Instruct, a framework that improves instruction-following in LLMs by bootstrapping instructions from the model's own generations.

Instruction-tuned language models generalize well zero-shot but depend heavily on limited human-written instruction data. Self-Instruct is a framework that improves instruction-following by bootstrapping off a model's own generations: it generates instructions, inputs, and outputs from the model, filters invalid or similar ones, and uses them to finetune the original model. Applied to vanilla GPT3, it yields a 33% absolute improvement on Super-NaturalInstructions, on par with InstructGPT-001, and leaves only a 5% gap behind it on expert-written novel-task instructions.

Based on: Self-Instruct: Aligning Language Models with Self-Generated Instructions · Annual Meeting of the Association for Computational Linguistics

HighlightCurated by Aramai EditorialarXiv

SciBERT: A Pretrained Language Model for Scientific Text

Releases SciBERT, a BERT-based language model pretrained on scientific text to improve downstream scientific NLP tasks.

Large-scale annotated data for scientific-domain NLP is expensive and hard to obtain. The authors release SciBERT, a pretrained language model based on BERT that leverages unsupervised pretraining on a large multi-domain corpus of scientific publications. SciBERT is evaluated on a suite of tasks including sequence tagging, sentence classification, and dependency parsing, using datasets from a variety of scientific domains. It shows statistically significant improvements over BERT and achieves new state-of-the-art results on several tasks, with code and pretrained models released publicly.

Based on: SciBERT: A Pretrained Language Model for Scientific Text · Conference on Empirical Methods in Natural Language Processing

HighlightCurated by Aramai EditorialarXiv

Efficiently Modeling Long Sequences with Structured State Spaces

Introduces S4, a structured state space sequence model that efficiently captures very long-range dependencies across modalities.

S4 is a sequence model for long-range dependencies where RNNs, CNNs, and Transformers struggle at 10,000+ steps. It builds on the state space model x'=Ax+Bu, y=Cx+Du, and introduces a parameterization that conditions state matrix A with a low-rank correction, enabling stable diagonalization and reducing computation to a Cauchy kernel. This makes prior SSMs far more efficient while preserving their strengths. S4 reaches 91% on sequential CIFAR-10, narrows the gap to Transformers while generating 60x faster, and sets SOTA on Long Range Arena, including the 16k-length Path-X task.

Based on: Efficiently Modeling Long Sequences with Structured State Spaces · International Conference on Learning Representations

HighlightCurated by Aramai EditorialAAAI Conference on Artificial Intelligence

Knowledge Graph Embedding by Translating on Hyperplanes

Proposes TransH, a knowledge graph embedding that models each relation as a hyperplane with a translation to capture complex mapping types.

The paper embeds knowledge graphs of entities and relations into a continuous vector space. The efficient TransE handles reflexive and one-to-many, many-to-one, and many-to-many relations poorly, while richer models fix this but sacrifice efficiency. TransH represents each relation as a hyperplane with a translation on it, preserving these mapping properties at nearly TransE's complexity, plus a sampling trick that cuts false-negative labels. On WordNet and Freebase—link prediction, triplet classification, fact extraction—it significantly improves accuracy over TransE.

Based on: Knowledge Graph Embedding by Translating on Hyperplanes · AAAI Conference on Artificial Intelligence

HighlightCurated by Aramai EditorialarXiv

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Proposes DeBERTa, improving BERT/RoBERTa with disentangled attention over content and position plus an enhanced mask decoder.

DeBERTa (Decoding-enhanced BERT with disentangled attention) improves BERT and RoBERTa via two techniques. Disentangled attention represents each word with separate content and position vectors, computing attention using disentangled matrices over contents and relative positions. An enhanced mask decoder replaces the output softmax to predict masked tokens in pretraining. Trained on half the data, DeBERTa beats RoBERTa-Large on MNLI (+0.9%), SQuAD v2.0 (+2.3%), and RACE (+3.6%). Code and models are released publicly.

Based on: DeBERTa: Decoding-enhanced BERT with Disentangled Attention · International Conference on Learning Representations

HighlightCurated by Aramai EditorialarXiv

Self-Refine: Iterative Refinement with Self-Feedback

Introduces Self-Refine, a training-free method where one LLM iteratively improves its own outputs via self-generated feedback.

Self-Refine improves LLM outputs through iterative self-feedback and refinement, mirroring how humans revise their writing. A single LLM generates an initial output, then provides feedback on it and uses that feedback to refine itself, requiring no supervised data, extra training, or reinforcement learning. Evaluated across 7 diverse tasks on GPT-3.5, ChatGPT, and GPT-4, Self-Refine outputs are preferred by humans and automatic metrics, improving performance by about 20% absolute on average. The work shows even top LLMs like GPT-4 can be improved at test time.

Based on: Self-Refine: Iterative Refinement with Self-Feedback · Neural Information Processing Systems

HighlightCurated by Aramai EditorialarXiv

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Introduces the Switch Transformer, a simplified sparse mixture-of-experts model that scales to trillion parameters at constant compute cost.

Unlike standard models that reuse the same parameters for all inputs, Mixture of Experts (MoE) selects different parameters per example, giving huge, sparsely activated models at constant compute. Adoption has been limited by complexity, communication cost, and training instability, which the Switch Transformer addresses by simplifying routing and reducing overheads. New techniques tame instabilities and enable bfloat16 training. Based on T5, it delivers up to 7x faster pre-training, gains across 101 languages, and trillion-parameter models with a 4x speedup over T5-XXL.

Based on: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · Journal of machine learning research

HighlightCurated by Aramai EditorialarXiv

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Investigates an early version of GPT-4, arguing it shows more general intelligence than prior models across many domains and tasks.

The paper reports an investigation of an early, still-in-development version of OpenAI's GPT-4, trained at unprecedented scale. The authors argue it belongs to a new cohort of LLMs with more general intelligence than earlier AI, solving novel, hard tasks across mathematics, coding, vision, medicine, law, and psychology without special prompting, often near or beyond human level. They suggest it is an early, incomplete form of AGI, stress its limitations, and discuss challenges ahead, including moving beyond next-word prediction.

Based on: Sparks of Artificial General Intelligence: Early experiments with GPT-4 · arXiv.org

HighlightCurated by Aramai EditorialarXiv

Pointer Sentinel Mixture Models

Introduces the pointer sentinel mixture architecture that lets neural sequence models copy words from recent context or use a softmax classifier.

Softmax neural sequence models reach top language modeling only with large hidden states and vocabularies, yet still fail on rare or unseen words even when context is unambiguous. The authors propose a pointer sentinel mixture that either reproduces a word from recent context or generates one via a standard softmax. Their pointer sentinel-LSTM reaches 70.9 perplexity on Penn Treebank using far fewer parameters, and they release the WikiText corpus for evaluating longer contexts and larger vocabularies.

Based on: Pointer Sentinel Mixture Models · International Conference on Learning Representations

HighlightCurated by Aramai EditorialarXiv

Generating Sequences With Recurrent Neural Networks

Shows how LSTM recurrent networks generate complex sequences with long-range structure by predicting one data point at a time.

Long Short-term Memory recurrent neural networks can generate complex sequences with long-range structure simply by predicting one data point at a time. The approach is demonstrated on discrete text data and real-valued online handwriting. It is then extended to handwriting synthesis by letting the network condition its predictions on a text sequence, producing highly realistic cursive handwriting across a wide variety of styles.

Based on: Generating Sequences With Recurrent Neural Networks · arXiv.org

HighlightCurated by Aramai EditorialarXiv

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Describes SentencePiece, a language-independent subword tokenizer and detokenizer that trains directly from raw sentences for neural text processing.

SentencePiece is a language-independent subword tokenizer and detokenizer built for neural text processing, including neural machine translation. Unlike existing tools that assume pre-tokenized word sequences as input, it can train subword models directly from raw sentences, enabling a fully end-to-end, language-independent pipeline. In an English-Japanese NMT experiment, training directly from raw sentences matches the accuracy of direct subword training. Open-source C++ and Python implementations are released under the Apache 2 license.

Based on: SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing · Conference on Empirical Methods in Natural Language Processing

HighlightCurated by Aramai EditorialarXiv

OPT: Open Pre-trained Transformer Language Models

Releases OPT, a suite of open decoder-only pretrained transformers from 125M to 175B parameters, matching GPT-3.

Large language models trained for hundreds of thousands of compute days show strong zero- and few-shot ability, but their cost makes them hard to replicate, and available ones expose no full weights for study. OPT (Open Pre-trained Transformers) is a suite of decoder-only pretrained transformers from 125M to 175B parameters, shared fully and responsibly with researchers. OPT-175B is comparable to GPT-3 while requiring only 1/7th the carbon footprint to develop. The authors also release a logbook of infrastructure challenges and code for experimenting with the models.

Based on: OPT: Open Pre-trained Transformer Language Models · arXiv.org