Highlight

Improved Baselines with Momentum Contrastive Learning

Shows that adding an MLP head and stronger augmentation to MoCo yields improved contrastive learning baselines that surpass SimCLR.

Contrastive unsupervised learning has progressed with methods like MoCo and SimCLR. This note verifies two of SimCLR's design improvements by implementing them in the MoCo framework: an MLP projection head and stronger data augmentation. With these simple modifications, the authors establish stronger baselines that outperform SimCLR while not requiring large training batches, aiming to make state-of-the-art unsupervised learning research more accessible. Code will be released.

Based on: Improved Baselines with Momentum Contrastive Learning · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

Recurrent Models of Visual Attention

Introduces a recurrent visual attention model that adaptively selects image regions to process, trained end-to-end with reinforcement learning.

Applying convolutional networks to large images is costly because computation scales with pixel count. This paper presents a recurrent neural network that extracts information from an image or video by adaptively selecting a sequence of regions and processing only those at high resolution. It has built-in translation invariance, yet its computation is controlled independently of input size. Being non-differentiable, it is trained via reinforcement learning; it beats a CNN baseline on cluttered image classification and learns to track an object without explicit supervision.

Based on: Recurrent Models of Visual Attention · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Learning Entity and Relation Embeddings for Knowledge Graph Completion

Proposes TransR, a knowledge graph embedding model that learns entity and relation embeddings in separate spaces for link prediction.

This paper tackles knowledge graph completion via graph embeddings. Prior models like TransE and TransH treat a relation as a translation from head to tail entity but place entities and relations in one space. Since an entity has multiple aspects that different relations emphasize, the authors propose TransR to build entity and relation embeddings in separate spaces, projecting entities into a relation-specific space before translating. On link prediction, triple classification, and relational fact extraction, TransR gives consistent gains over TransE and TransH.

Based on: Learning Entity and Relation Embeddings for Knowledge Graph Completion · AAAI Conference on Artificial Intelligence

Curated by Aramai Editorial

Read summary →
Highlight

piecewiseSEM: Piecewise structural equation modelling in r for ecology, evolution, and systematics

Presents piecewiseSEM, an open-source R package implementing confirmatory path analysis for ecology, evolution, and systematics.

piecewiseSEM is an open-source R package implementing confirmatory path analysis, a form of structural equation modeling (SEM), for ecology, evolution, and systematics. Unlike covariance-based SEM, local estimation allows non-normal distributions, random effects, and varied correlation structures, but had not been automated. The package extends piecewise SEM to generalized linear, phylogenetic least-squares, and mixed effects models with familiar R syntax, and includes worked examples using random effects, temporal autocorrelation, and phylogenetic contrasts.

Based on: piecewiseSEM: Piecewise structural equation modelling in r for ecology, evolution, and systematics · arXiv

Curated by Aramai Editorial

Read summary →
Highlight

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Presents CodeBERT, a bimodal Transformer pre-trained on natural language and programming language for code search and documentation.

CodeBERT is a bimodal pre-trained model for programming and natural language that learns general-purpose representations supporting tasks like code search and code documentation generation. Built on a Transformer architecture, it is trained with a hybrid objective including replaced token detection, letting it use both bimodal NL-PL pairs and unimodal data. Fine-tuned, CodeBERT achieves state-of-the-art results on natural language code search and code documentation generation, and it outperforms prior models on a zero-shot NL-PL probing dataset.

Based on: CodeBERT: A Pre-Trained Model for Programming and Natural Languages · Findings

Curated by Aramai Editorial

Read summary →
Highlight

FiLM: Visual Reasoning with a General Conditioning Layer

Introduces FiLM, a feature-wise linear modulation layer that conditions neural network computation for visual reasoning tasks.

FiLM (Feature-wise Linear Modulation) is a general-purpose conditioning method that influences neural network computation through a simple feature-wise affine transformation driven by conditioning information. Applied to visual reasoning, where answering image questions requires multi-step high-level processing, FiLM layers halve the state-of-the-art error on the CLEVR benchmark. They modulate features coherently, remain robust to ablations and architectural changes, and generalize well to new data from few examples or even zero-shot.

Based on: FiLM: Visual Reasoning with a General Conditioning Layer · AAAI Conference on Artificial Intelligence

Curated by Aramai Editorial

Read summary →
Highlight

Program Synthesis with Large Language Models

Evaluates large language models on program synthesis using the new MBPP and MathQA-Python benchmarks across few-shot and fine-tuning regimes.

This paper probes how well current large language models (244M to 137B parameters) synthesize short Python programs from natural language, using two new benchmarks: MBPP with 974 entry-level tasks and MathQA-Python with 23,914 problems. Synthesis performance scales log-linearly with model size; the largest model solves 59.6% of MBPP via few-shot prompting and reaches 83.8% on MathQA-Python after fine-tuning. Human dialog feedback halves the error rate, but even the best models struggle to predict program execution outputs.

Based on: Program Synthesis with Large Language Models · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Introduces TriviaQA, a large-scale distantly supervised reading comprehension dataset of over 650K question-answer-evidence triples.

TriviaQA is a reading comprehension dataset of over 650K question-answer-evidence triples, including 95K QA pairs written by trivia enthusiasts plus independently gathered evidence documents averaging six per question for distant supervision. Its questions are more compositional, show greater syntactic and lexical variability, and demand more cross-sentence reasoning than prior datasets. Two baselines, a feature-based classifier and a strong neural network, reach only 23% and 40%, far below human performance of 80%.

Based on: TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension · Annual Meeting of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

Finetuned Language Models Are Zero-Shot Learners

This paper shows instruction tuning, finetuning a large language model on many tasks phrased as instructions, greatly improves zero-shot generalization.

This paper studies instruction tuning as a simple way to improve the zero-shot abilities of large language models. A 137B-parameter pretrained model is finetuned on over 60 NLP tasks expressed via natural language instruction templates, producing a model called FLAN. On held-out task types, FLAN clearly beats its unmodified counterpart and outperforms zero-shot 175B GPT-3 on 20 of the 25 evaluated tasks, even topping few-shot GPT-3 on benchmarks like ANLI, RTE, and BoolQ. Ablations show the number of finetuning tasks, model scale, and instruction phrasing are all crucial.

Based on: Finetuned Language Models Are Zero-Shot Learners · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

SRILM - an extensible language modeling toolkit

SRILM is an extensible C++ toolkit for building and experimenting with statistical N-gram language models for speech recognition and related tasks.

SRILM bundles C++ libraries, executable programs, and helper scripts for building and experimenting with statistical language models used in speech recognition and other applications. Freely available for noncommercial use, it supports creating and evaluating a variety of N-gram-based language model types, along with related tasks like statistical tagging and operations on N-best lists and word lattices. The paper summarizes the toolkit's functionality and discusses a design emphasizing rapid prototyping, reusability, and combinability of tools.

Based on: SRILM - an extensible language modeling toolkit · Interspeech

Curated by Aramai Editorial

Read summary →
Highlight

The R*-tree: an efficient and robust access method for points and rectangles

The R*-tree improves the R-tree by jointly optimizing area, margin, and overlap of bounding rectangles, giving a robust spatial access method.

The R*-tree is a spatial access method for points and rectangles that refines the classic R-tree. Whereas the R-tree only heuristically minimizes enclosing-rectangle area, the R*-tree jointly optimizes area, margin, and overlap of directory rectangles, tuned via extensive experiments over varied data and queries. In benchmarks it clearly outperforms existing R-tree variants, including Guttman's linear and quadratic versions and Greene's variant, for rectangles and multidimensional points. It supports point and spatial data together at slightly higher implementation cost.

Based on: The R*-tree: an efficient and robust access method for points and rectangles · ACM SIGMOD Conference

Curated by Aramai Editorial

Read summary →
Highlight

NLTK: The Natural Language Toolkit

NLTK is a Python, open-source suite of modules, datasets, and tutorials for research and teaching in computational linguistics and NLP.

The Natural Language Toolkit (NLTK) is a suite of program modules, data sets, and tutorials that support research and teaching in computational linguistics and natural language processing. Written in Python and distributed under the GPL open-source license, the toolkit was rewritten over the past year to simplify many of its linguistic data structures and to exploit recent enhancements in the Python language. This paper reports on the simplified toolkit and explains how it is used in teaching NLP.

Based on: NLTK: The Natural Language Toolkit · Annual Meeting of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →