Highlight

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Presents Swin Transformer, a hierarchical vision Transformer using shifted windows as a general-purpose vision backbone.

Adapting Transformers from language to vision is hard because visual entities vary greatly in scale and images have far higher pixel resolution than text has words. Swin Transformer addresses this with a hierarchical architecture using shifted windows, limiting self-attention to non-overlapping local windows while allowing cross-window connections, giving linear complexity in image size. As a general-purpose backbone it reaches 87.3% top-1 on ImageNet-1K, 58.7 box AP and 51.1 mask AP on COCO, and 53.5 mIoU on ADE20K, surpassing prior state of the art by wide margins.

Based on: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows · IEEE International Conference on Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

Bleu: a Method for Automatic Evaluation of Machine Translation

Proposes BLEU, a quick, inexpensive, language-independent automatic method for evaluating machine translation quality.

Human evaluation of machine translation is thorough but slow, costly, and its labor cannot be reused. The authors propose an automatic evaluation method that is fast, inexpensive, and language-independent, correlating highly with human judgments while adding little marginal cost per run. It is presented as an automated substitute for skilled human judges, useful whenever quick or frequent evaluation is needed.

Based on: Bleu: a Method for Automatic Evaluation of Machine Translation · Annual Meeting of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

Efficient Estimation of Word Representations in Vector Space

Proposes two new architectures for learning continuous word vector representations efficiently from very large datasets.

The paper proposes two novel architectures for computing continuous vector representations of words from very large datasets. Representation quality is measured on a word similarity task and compared against prior neural-network-based techniques. The authors report large accuracy gains at much lower computational cost, learning high-quality word vectors from a 1.6 billion word dataset in under a day. These vectors also achieve state-of-the-art results on a test set measuring syntactic and semantic word similarities.

Based on: Efficient Estimation of Word Representations in Vector Space · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

GloVe: Global Vectors for Word Representation

Introduces GloVe, a global log-bilinear regression model unifying matrix factorization and context-window methods for word vectors.

Prior word-vector methods captured semantic and syntactic regularities through vector arithmetic, but why these regularities arose was unclear. The authors make explicit the properties needed for such structure to emerge and propose GloVe, a global log-bilinear regression model combining matrix factorization with local context-window approaches. It trains only on nonzero entries of a word-word co-occurrence matrix. The resulting vectors score 75% on a word analogy task and outperform related models on similarity and named entity recognition.

Based on: GloVe: Global Vectors for Word Representation · Conference on Empirical Methods in Natural Language Processing

Curated by Aramai Editorial

Read summary →
Highlight

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin et al. present BERT, a bidirectional Transformer pretraining method that set new state-of-the-art results on eleven NLP tasks.

BERT pre-trains deep bidirectional representations by jointly conditioning on left and right context in every layer, unlike prior left-to-right language models. A single pretrained BERT model can be fine-tuned with one extra output layer for many tasks, pushing GLUE to 80.5, MultiNLI accuracy to 86.7%, and SQuAD v1.1 F1 to 93.2 — new state-of-the-art results across eleven NLP benchmarks.

Based on: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · North American Chapter of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

Attention is All you Need

Vaswani et al. propose the Transformer, an architecture built solely on attention, replacing recurrence and convolution.

The Transformer dispenses with recurrent and convolutional layers entirely, relying only on attention mechanisms. It is more parallelizable and faster to train than prior encoder-decoder models, reaching 28.4 BLEU on WMT 2014 English-to-German and a new state-of-the-art 41.8 BLEU on English-to-French after 3.5 days of training on eight GPUs.

Based on: Attention is All you Need · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Deep Residual Learning for Image Recognition

He et al. introduce residual learning, letting networks hundreds of layers deep train reliably and win ILSVRC/COCO 2015.

Very deep networks are hard to optimize directly. The authors reformulate layers to learn residual functions relative to their inputs, which makes substantially deeper networks (up to 152 layers) easier to train and more accurate. Residual nets took 1st place in ILSVRC 2015 classification (3.57% error) and drove a 28% relative improvement on COCO object detection.

Based on: Deep Residual Learning for Image Recognition · Computer Vision and Pattern Recognition

Curated by Aramai Editorial

Read summary →
Highlight

Foundation Capital on context graphs as a trillion-dollar opportunity

A short take on Foundation Capital characterising context graphs as the next trillion-dollar layer of AI infrastructure.

Macro framing for SCR: capital is converging on the meaning layer, not just the model. Foundation Capital named PlayerZero its first thesis investment with a $20M raise.

Based on: Context graphs: a trillion-dollar opportunity · Foundation Capital

Curated by Cruce Saunders

Read summary →