Highlight

Longformer: The Long-Document Transformer

Introduces Longformer, a transformer whose attention scales linearly with sequence length to process documents of thousands of tokens.

Standard transformers cannot process long sequences because self-attention scales quadratically with length. Longformer introduces attention that scales linearly, combining local windowed attention with task-motivated global attention as a drop-in replacement for self-attention. It reaches state-of-the-art results on character-level language modeling and, when pretrained and finetuned, consistently outperforms RoBERTa on long-document tasks, with new records on WikiHop and TriviaQA. A Longformer-Encoder-Decoder variant supports generative tasks like arXiv summarization.

Based on: Longformer: The Long-Document Transformer · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

Robust principal component analysis?

Proves that a matrix that is the sum of a low-rank and a sparse component can be exactly recovered by convex Principal Component Pursuit.

The paper asks whether a data matrix formed as the superposition of a low-rank component and a sparse component can be decomposed into its parts. Under suitable assumptions, it proves both can be recovered exactly by solving a convex program, Principal Component Pursuit, which minimizes a weighted combination of the nuclear norm and the l1 norm. This yields a principled robust PCA that recovers principal components even when a fraction of entries are arbitrarily corrupted or missing, with applications in video surveillance and face recognition.

Based on: Robust principal component analysis? · JACM

Curated by Aramai Editorial

Read summary →
Highlight

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

Introduces CycleGAN for unpaired image-to-image translation, using adversarial and cycle-consistency losses to map between domains without paired data.

Image-to-image translation usually learns a mapping from aligned image pairs, but paired data is often unavailable. This work translates images from a source to a target domain without paired examples, using an adversarial loss to make outputs indistinguishable from the target distribution. Because that mapping is under-constrained, it adds an inverse mapping and a cycle-consistency loss so translating back recovers the original. Results span style transfer, object transfiguration, season transfer, and photo enhancement, outperforming prior methods.

Based on: Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks · IEEE International Conference on Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

Large Language Models are Zero-Shot Reasoners

Shows that simply prepending 'Let's think step by step' turns large language models into strong zero-shot multi-step reasoners.

Large language models are known as strong few-shot learners, and chain-of-thought prompting with step-by-step exemplars reaches state-of-the-art on hard reasoning tasks. This paper shows LLMs are also capable zero-shot reasoners: simply adding 'Let's think step by step' before an answer elicits multi-step reasoning without hand-crafted examples. Using one prompt template, Zero-shot-CoT greatly outperforms standard zero-shot prompting on arithmetic, symbolic, and logical benchmarks, e.g. lifting MultiArith from 17.7% to 78.7% with a large InstructGPT model.

Based on: Large Language Models are Zero-Shot Reasoners · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks

Proposes MTCNN, a three-stage cascaded multitask CNN framework that jointly performs face detection and alignment in real time.

Face detection and alignment are challenging under varied poses, illumination, and occlusion. The paper proposes a deep cascaded multitask framework that exploits the correlation between the two tasks to improve both. Three stages of deep convolutional networks predict face and landmark locations in a coarse-to-fine manner, aided by a new online hard sample mining strategy. The method beats prior techniques on challenging face detection benchmarks including WIDER FACE and on a facial landmark alignment benchmark, while keeping real-time performance.

Based on: Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks · IEEE Signal Processing Letters

Curated by Aramai Editorial

Read summary →
Highlight

Attention U-Net: Learning Where to Look for the Pancreas

Attention U-Net adds trainable attention gates to CNNs so they focus on target structures, improving medical image segmentation efficiently.

The paper proposes attention gates (AGs) for medical imaging that automatically learn to focus on target structures of varying shapes and sizes, suppressing irrelevant regions while highlighting salient features. AGs remove the need for explicit external organ-localisation modules used in cascaded CNNs and integrate into standard architectures like U-Net with minimal overhead while boosting sensitivity and accuracy. Evaluated on two large CT abdominal datasets, Attention U-Net consistently improves on U-Net across datasets and training sizes.

Based on: Attention U-Net: Learning Where to Look for the Pancreas · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

LINE: Large-scale Information Network Embedding

Presents LINE, a scalable network embedding method that preserves local and global structure for graphs with millions of nodes.

LINE embeds very large information networks into low-dimensional vectors for visualization, node classification, and link prediction. Unlike prior methods that do not scale, it handles undirected, directed, and weighted networks via an objective preserving both local and global structure. An edge-sampling algorithm overcomes limitations of classical stochastic gradient descent, improving effectiveness and efficiency. Experiments on language, social, and citation networks embed millions of vertices and billions of edges in hours on one machine.

Based on: LINE: Large-scale Information Network Embedding · The Web Conference

Curated by Aramai Editorial

Read summary →
Highlight

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Analyzes encoder-decoder neural machine translation models, showing performance drops with longer sentences and more unknown words.

Neural machine translation is a then-new approach to statistical machine translation built purely from neural networks, using an encoder that maps a variable-length sentence to a fixed-length representation and a decoder that generates the translation. The paper analyzes two models: an RNN Encoder-Decoder and a newly proposed gated recursive convolutional neural network. Quality is good for short sentences without unknown words but degrades rapidly as sentence length and unknown words grow; the gated model learns grammatical structure automatically.

Based on: On the Properties of Neural Machine Translation: Encoder–Decoder Approaches · SSST@EMNLP

Curated by Aramai Editorial

Read summary →
Highlight

Neural Discrete Representation Learning

Introduces VQ-VAE, a generative model that learns discrete latent representations via vector quantisation, avoiding posterior collapse.

Learning useful representations without supervision remains a key challenge. The paper proposes the Vector Quantised-Variational AutoEncoder (VQ-VAE), a simple generative model whose encoder outputs discrete codes and whose prior is learned rather than static. By incorporating vector quantisation, the model avoids the posterior collapse that afflicts VAEs paired with powerful autoregressive decoders. Combined with an autoregressive prior, it generates high-quality images, videos, and speech, performs speaker conversion, and learns phonemes without supervision.

Based on: Neural Discrete Representation Learning · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Scales vision and vision-language pretraining with a dual-encoder contrastive model trained on over one billion noisy image alt-text pairs.

Visual and vision-language representation learning still depends on costly curated datasets, which limits scale. This work instead uses over one billion noisy image alt-text pairs collected without expensive filtering. A simple dual-encoder aligns image and text via a contrastive loss, and the corpus scale offsets its noise to yield state-of-the-art representations. The model transfers strongly to ImageNet and VTAB, enables zero-shot classification, and sets new records on Flickr30K and MSCOCO retrieval.

Based on: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision · International Conference on Machine Learning

Curated by Aramai Editorial

Read summary →
Highlight

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

BioBERT adapts BERT to biomedical text by pretraining on large-scale biomedical corpora, boosting NER, relation extraction, and question answering.

As biomedical literature grows, text mining becomes increasingly important, but applying general-domain NLP models directly to biomedical text suffers from a word-distribution shift. BioBERT adapts the BERT language model by pretraining it on large-scale biomedical corpora while keeping nearly the same architecture across tasks. It substantially outperforms BERT and prior state-of-the-art on biomedical named entity recognition, relation extraction, and question answering. The authors release pretrained weights and fine-tuning code freely.

Based on: BioBERT: a pre-trained biomedical language representation model for biomedical text mining · Bioinform.

Curated by Aramai Editorial

Read summary →
Highlight

Learning without Forgetting

Introduces Learning without Forgetting, which adds new tasks to a CNN using only new-task data while preserving performance on old tasks.

As tasks accumulate, storing and retraining on all prior data becomes infeasible, and old-task data may be unavailable when adding new CNN capabilities. Learning without Forgetting trains the network on only new-task data while preserving the original capabilities. It outperforms feature extraction and fine-tuning, and matches multitask learning that relies on the unavailable old data. Notably, it can even replace fine-tuning to improve new-task performance.

Based on: Learning without Forgetting · IEEE Transactions on Pattern Analysis and Machine Intelligence

Curated by Aramai Editorial

Read summary →