Highlight

Finetuned Language Models Are Zero-Shot Learners

This paper shows instruction tuning, finetuning a large language model on many tasks phrased as instructions, greatly improves zero-shot generalization.

This paper studies instruction tuning as a simple way to improve the zero-shot abilities of large language models. A 137B-parameter pretrained model is finetuned on over 60 NLP tasks expressed via natural language instruction templates, producing a model called FLAN. On held-out task types, FLAN clearly beats its unmodified counterpart and outperforms zero-shot 175B GPT-3 on 20 of the 25 evaluated tasks, even topping few-shot GPT-3 on benchmarks like ANLI, RTE, and BoolQ. Ablations show the number of finetuning tasks, model scale, and instruction phrasing are all crucial.

Based on: Finetuned Language Models Are Zero-Shot Learners · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

SRILM - an extensible language modeling toolkit

SRILM is an extensible C++ toolkit for building and experimenting with statistical N-gram language models for speech recognition and related tasks.

SRILM bundles C++ libraries, executable programs, and helper scripts for building and experimenting with statistical language models used in speech recognition and other applications. Freely available for noncommercial use, it supports creating and evaluating a variety of N-gram-based language model types, along with related tasks like statistical tagging and operations on N-best lists and word lattices. The paper summarizes the toolkit's functionality and discusses a design emphasizing rapid prototyping, reusability, and combinability of tools.

Based on: SRILM - an extensible language modeling toolkit · Interspeech

Curated by Aramai Editorial

Read summary →
Highlight

The R*-tree: an efficient and robust access method for points and rectangles

The R*-tree improves the R-tree by jointly optimizing area, margin, and overlap of bounding rectangles, giving a robust spatial access method.

The R*-tree is a spatial access method for points and rectangles that refines the classic R-tree. Whereas the R-tree only heuristically minimizes enclosing-rectangle area, the R*-tree jointly optimizes area, margin, and overlap of directory rectangles, tuned via extensive experiments over varied data and queries. In benchmarks it clearly outperforms existing R-tree variants, including Guttman's linear and quadratic versions and Greene's variant, for rectangles and multidimensional points. It supports point and spatial data together at slightly higher implementation cost.

Based on: The R*-tree: an efficient and robust access method for points and rectangles · ACM SIGMOD Conference

Curated by Aramai Editorial

Read summary →
Highlight

NLTK: The Natural Language Toolkit

NLTK is a Python, open-source suite of modules, datasets, and tutorials for research and teaching in computational linguistics and NLP.

The Natural Language Toolkit (NLTK) is a suite of program modules, data sets, and tutorials that support research and teaching in computational linguistics and natural language processing. Written in Python and distributed under the GPL open-source license, the toolkit was rewritten over the past year to simplify many of its linguistic data structures and to exploit recent enhancements in the Python language. This paper reports on the simplified toolkit and explains how it is used in teaching NLP.

Based on: NLTK: The Natural Language Toolkit · Annual Meeting of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

Adversarial Discriminative Domain Adaptation

ADDA unifies adversarial domain adaptation methods, combining discriminative modeling, untied weight sharing, and a GAN loss for unsupervised adaptation.

Adversarial Discriminative Domain Adaptation (ADDA) tackles unsupervised domain adaptation by aligning source and target feature distributions via adversarial learning. They propose a general framework casting prior adversarial adaptation methods as special cases differing in modeling, weight sharing, and loss. Within it they introduce ADDA, combining discriminative modeling, untied weight sharing, and a GAN-based loss. Simpler yet more effective than competing methods, it exceeds prior state-of-the-art on standard benchmarks and a hard cross-modality classification task.

Based on: Adversarial Discriminative Domain Adaptation · Computer Vision and Pattern Recognition

Curated by Aramai Editorial

Read summary →
Highlight

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

DeepLab combines deep convolutional networks with fully connected CRFs to sharpen boundary localization in semantic image segmentation.

This work unites deep convolutional neural networks (DCNNs) with probabilistic graphical models for semantic image segmentation. The invariance that aids DCNN classification leaves final-layer responses too poorly localized for segmentation, so the authors couple them with a fully connected Conditional Random Field (CRF). The resulting DeepLab system localizes boundaries more precisely than prior methods, setting a new state of the art on PASCAL VOC-2012 (71.6% IOU). Network re-purposing and the wavelet 'hole' algorithm enable dense computation at 8 fps on a GPU.

Based on: Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation

Swin-Unet is a pure Transformer with a U-shaped encoder-decoder and skip connections for medical image segmentation.

Swin-Unet is a pure Transformer with a U-shaped encoder-decoder architecture for medical image segmentation, addressing the limited global context of convolutional networks. Tokenized image patches pass through a hierarchical Swin Transformer encoder with shifted windows, and a symmetric decoder with patch-expanding layers restores resolution, linked by skip connections for local-global feature learning. On multi-organ and cardiac segmentation tasks, this convolution-free design outperforms methods based on full convolution or hybrid transformer-convolution architectures.

Based on: Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation · ECCV Workshops

Curated by Aramai Editorial

Read summary →
Highlight

Flow Matching for Generative Modeling

Flow Matching introduces a simulation-free method to train continuous normalizing flows by regressing vector fields of conditional probability paths.

Flow Matching is a simulation-free method for training Continuous Normalizing Flows (CNFs) by regressing vector fields along fixed conditional probability paths. It covers a general family of Gaussian paths subsuming diffusion paths, and using diffusion paths yields more robust, stable training. Using Optimal Transport displacement interpolation instead gives more efficient paths with faster training and sampling and better generalization. On ImageNet it beats diffusion baselines in likelihood and sample quality while enabling fast sampling with standard ODE solvers.

Based on: Flow Matching for Generative Modeling · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

In-datacenter performance analysis of a tensor processing unit

In-datacenter evaluation of Google's Tensor Processing Unit, a custom ASIC for neural network inference, versus contemporary CPUs and GPUs.

This paper evaluates the Tensor Processing Unit (TPU), a custom ASIC deployed in Google datacenters since 2015 to accelerate neural network inference. Built around a 65,536 8-bit MAC matrix-multiply unit delivering 92 TeraOps/s and 28 MiB of on-chip memory, its deterministic execution suits strict 99th-percentile latency needs better than CPUs or GPUs. Running production TensorFlow workloads (MLPs, CNNs, LSTMs), the TPU averaged 15-30X faster than a contemporary Haswell CPU or K80 GPU, with 30-80X better performance per watt.

Based on: In-datacenter performance analysis of a tensor processing unit · International Symposium on Computer Architecture

Curated by Aramai Editorial

Read summary →
Highlight

The I-TASSER Suite: protein structure and function prediction

The I-TASSER Suite predicts protein 3D structure and function via threading, structure clustering, and template-based ligand-binding annotation.

The I-TASSER Suite predicts protein structure and function through an integrated pipeline. Low-energy conformations are found by structure clustering, then refined via reassembly simulation and atomic-level energy minimization, with quality scored by confidence measures and the new ResQ method. Function is annotated by matching models to the BioLiP database to infer ligand-binding sites, EC numbers, and GO terms via COFACTOR, TM-SITE, and S-SITE. In CASP10 and CAMEO tests it produced correct folds and accurate ligand-binding predictions.

Based on: The I-TASSER Suite: protein structure and function prediction · Nature Methods

Curated by Aramai Editorial

Read summary →
Highlight

OPTICS: ordering points to identify the clustering structure

Introduces OPTICS, which builds an augmented ordering of a database representing its density-based clustering structure across a broad range of parameters.

Cluster analysis is a key database-mining method, but nearly all clustering algorithms need input parameters that are hard to set yet strongly affect results, with no global setting capturing intrinsic structure. Instead of an explicit clustering, OPTICS builds an augmented ordering of the database representing its density-based clustering structure, encoding density-based clusterings across a broad range of parameters. It supports automatic and interactive analysis, extracting representative points and arbitrary-shaped clusters, with visualizations for large datasets.

Based on: OPTICS: ordering points to identify the clustering structure · SIGMOD Conference

Curated by Aramai Editorial

Read summary →
Highlight

QLoRA: Efficient Finetuning of Quantized LLMs

Presents QLoRA, finetuning a 4-bit quantized LLM through LoRA adapters to train a 65B model on one 48GB GPU while preserving 16-bit performance.

QLoRA is an efficient finetuning method that cuts memory enough to tune a 65B model on one 48GB GPU while matching 16-bit performance. It backpropagates gradients through a frozen, 4-bit quantized model into Low-Rank Adapters (LoRA) via 4-bit NormalFloat (NF4), an information-theoretically optimal type for normal weights; double quantization; and paged optimizers for memory spikes. Guanaco models reach 99.3% of ChatGPT on Vicuna after 24 hours of tuning. Across 1,000+ models, data quality is shown to outweigh scale, and chatbot benchmarks prove unreliable.

Based on: QLoRA: Efficient Finetuning of Quantized LLMs · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →