HighlightCurated by Aramai EditorialarXiv

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

FlashAttention: an IO-aware exact attention algorithm using tiling to cut GPU memory traffic, speeding Transformer training and enabling longer context.

Self-attention scales quadratically with sequence length, making Transformers slow on long sequences, and approximate methods cut compute but often lose quality without speedups. The authors argue the missing principle is IO-awareness, counting reads and writes between GPU memory levels. FlashAttention is an IO-aware exact attention algorithm that uses tiling to cut transfers between GPU HBM and on-chip SRAM, needing fewer HBM accesses. It trains Transformers faster (15% on BERT-large, 3x on GPT-2, 2.4x on long-range arena) and enables longer context and new capabilities.

Based on: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · Neural Information Processing Systems

HighlightCurated by Aramai EditorialarXiv

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

Shows word embeddings encode gender stereotypes and proposes a geometric method to remove them while preserving useful structure.

Word embeddings, even those trained on Google News, encode female/male gender stereotypes to a disturbing degree, and their widespread use tends to amplify these biases. Geometrically, gender bias is captured by a direction in the space, and gender-neutral words are linearly separable from gender-definitional words. The authors use these properties to modify embeddings, removing stereotypical associations like receptionist-female while keeping legitimate ones like queen-female. Evaluations show significant bias reduction while preserving clustering and analogy-solving ability.

Based on: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings · Neural Information Processing Systems

HighlightCurated by Aramai EditorialarXiv

Bidirectional LSTM-CRF Models for Sequence Tagging

Applies LSTM, BI-LSTM, LSTM-CRF, and BI-LSTM-CRF models to sequence tagging, reaching state-of-the-art on POS, chunking, and NER.

This paper proposes a range of LSTM-based models for sequence tagging, including LSTM, bidirectional LSTM, LSTM with a CRF layer, and bidirectional LSTM with a CRF layer (BI-LSTM-CRF). It is the first work to apply a BI-LSTM-CRF model to NLP sequence tagging benchmarks. The model uses both past and future input features via the bidirectional LSTM and sentence-level tag information via the CRF layer, producing state-of-the-art or comparable accuracy on POS tagging, chunking, and NER while being robust and less reliant on word embeddings.

Based on: Bidirectional LSTM-CRF Models for Sequence Tagging · arXiv.org

HighlightCurated by Aramai EditorialarXiv

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Introduces Tree of Thoughts, a framework letting LLMs explore and self-evaluate multiple reasoning paths with lookahead and backtracking.

LLMs are widely used for problem solving but remain confined to token-level, left-to-right inference, limiting tasks needing exploration or strategic lookahead. The authors introduce Tree of Thoughts (ToT), generalizing Chain-of-Thought prompting by letting models explore coherent text units ('thoughts') as intermediate steps. ToT considers multiple reasoning paths, self-evaluates choices, and looks ahead or backtracks for global decisions. On Game of 24, Creative Writing, and Mini Crosswords it sharply improves results—raising GPT-4's Game of 24 success from 4% to 74%.

Based on: Tree of Thoughts: Deliberate Problem Solving with Large Language Models · Neural Information Processing Systems

HighlightCurated by Aramai EditorialarXiv

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Proposes Transformer-XL, adding segment-level recurrence and a new positional encoding so language models learn dependency beyond a fixed-length context.

Transformers learn long-term dependency but are limited by a fixed-length context in language modeling. The authors propose Transformer-XL, combining segment-level recurrence with a novel positional encoding to learn dependencies beyond a fixed length without disrupting temporal coherence, also resolving context fragmentation. It captures dependency 80% longer than RNNs and 450% longer than vanilla Transformers, performs better on short and long sequences, is up to 1,800x faster at evaluation, and sets new state-of-the-art results on five benchmarks.

Based on: Transformer-XL: Attentive Language Models beyond a Fixed-Length Context · Annual Meeting of the Association for Computational Linguistics

HighlightCurated by Aramai EditorialarXiv

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Presents Transformers, an open-source library offering unified access to state-of-the-art Transformer architectures and curated pretrained models.

Recent NLP progress has been driven by advances in Transformer architectures, which enable higher-capacity models, and by pretraining, which lets that capacity be used across many tasks. Transformers is an open-source library that brings these advances to the broader machine learning community, providing state-of-the-art Transformer architectures under a unified API together with a curated collection of community-contributed pretrained models. It is designed to be extensible for researchers, simple for practitioners, and fast and robust in industrial deployment.

Based on: HuggingFace's Transformers: State-of-the-art Natural Language Processing · arXiv.org

HighlightCurated by Aramai EditorialAAAI Conference on Artificial Intelligence

Learning Entity and Relation Embeddings for Knowledge Graph Completion

Proposes TransR, a knowledge graph embedding model that learns entity and relation embeddings in separate spaces for link prediction.

This paper tackles knowledge graph completion via graph embeddings. Prior models like TransE and TransH treat a relation as a translation from head to tail entity but place entities and relations in one space. Since an entity has multiple aspects that different relations emphasize, the authors propose TransR to build entity and relation embeddings in separate spaces, projecting entities into a relation-specific space before translating. On link prediction, triple classification, and relational fact extraction, TransR gives consistent gains over TransE and TransH.

Based on: Learning Entity and Relation Embeddings for Knowledge Graph Completion · AAAI Conference on Artificial Intelligence

HighlightCurated by Aramai EditorialarXiv

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Presents CodeBERT, a bimodal Transformer pre-trained on natural language and programming language for code search and documentation.

CodeBERT is a bimodal pre-trained model for programming and natural language that learns general-purpose representations supporting tasks like code search and code documentation generation. Built on a Transformer architecture, it is trained with a hybrid objective including replaced token detection, letting it use both bimodal NL-PL pairs and unimodal data. Fine-tuned, CodeBERT achieves state-of-the-art results on natural language code search and code documentation generation, and it outperforms prior models on a zero-shot NL-PL probing dataset.

Based on: CodeBERT: A Pre-Trained Model for Programming and Natural Languages · Findings

HighlightCurated by Aramai EditorialarXiv

Finetuned Language Models Are Zero-Shot Learners

This paper shows instruction tuning, finetuning a large language model on many tasks phrased as instructions, greatly improves zero-shot generalization.

This paper studies instruction tuning as a simple way to improve the zero-shot abilities of large language models. A 137B-parameter pretrained model is finetuned on over 60 NLP tasks expressed via natural language instruction templates, producing a model called FLAN. On held-out task types, FLAN clearly beats its unmodified counterpart and outperforms zero-shot 175B GPT-3 on 20 of the 25 evaluated tasks, even topping few-shot GPT-3 on benchmarks like ANLI, RTE, and BoolQ. Ablations show the number of finetuning tasks, model scale, and instruction phrasing are all crucial.

Based on: Finetuned Language Models Are Zero-Shot Learners · International Conference on Learning Representations

HighlightCurated by Aramai EditorialarXiv

Survey of Hallucination in Natural Language Generation

Surveys hallucination in natural language generation: metrics, mitigation, and progress across summarization, dialogue, QA, and machine translation.

Natural language generation has advanced rapidly with sequence-to-sequence and Transformer-based language models, yielding more fluent output for tasks like summarization, dialogue, and data-to-text. However, such models often hallucinate unintended text, degrading performance and user trust. This survey comprehensively reviews hallucination in NLG in two parts: a general overview of metrics, mitigation methods, and future directions; and task-specific progress in abstractive summarization, dialogue generation, generative QA, data-to-text, and machine translation.

Based on: Survey of Hallucination in Natural Language Generation · ACM Computing Surveys

HighlightCurated by Aramai EditorialarXiv

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Presents GNMT, Google's deep LSTM neural machine translation system with attention, wordpieces, and low-precision inference for production use.

Neural machine translation (NMT) enables end-to-end translation but is costly to train and run and handles rare words poorly. GNMT uses a deep LSTM with 8 encoder and 8 decoder layers plus attention and residual connections, wiring the decoder's bottom layer to the encoder's top layer for parallelism. It uses low-precision inference for speed, sub-word 'wordpieces' for rare words, and length-normalized beam search with a coverage penalty. On WMT'14 benchmarks it rivals the state of the art, and human evaluation shows 60% fewer errors than Google's phrase-based system.

Based on: Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation · arXiv.org

HighlightCurated by Aramai EditorialNeural Computation

A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures

A review of LSTM cells and their variants, categorizing LSTM network architectures and surveying applications and future research directions.

Recurrent neural networks are widely used for sequential data such as text, audio, and video, but RNNs built from sigma or tanh cells cannot learn relevant information when the input gap is large. By introducing gate functions into the cell, the long short-term memory (LSTM) handles long-term dependencies well, and most notable RNN results have since relied on it. This review examines the LSTM cell and its variants, divides LSTM networks into LSTM-dominated and integrated categories, discusses applications, and outlines future research directions.

Based on: A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures · Neural Computation