Highlight

Evolutionary-scale prediction of atomic level protein structure with a language model

Uses a protein language model up to 15B parameters to predict atomic-level 3D structure directly from sequence, up to 60x faster than prior methods.

A large protein language model can infer three-dimensional structure directly from primary sequence, giving an order-of-magnitude speedup in high-resolution prediction. Trained up to 15B parameters, the models learn evolutionary patterns across millions of sequences that enable atom-level prediction up to 60x faster than state of the art while keeping accuracy. Building on this, the authors release the ESM Metagenomic Atlas, characterizing metagenomic proteins with over 617 million structures and more than 225 million high-confidence predictions, many structurally novel.

Based on: Evolutionary-scale prediction of atomic level protein structure with a language model · bioRxiv

Curated by Aramai Editorial

Read summary →
Highlight

Coordinate Attention for Efficient Mobile Network Design

Introduces coordinate attention, embedding positional information into channel attention for mobile networks with nearly no extra compute.

Channel attention such as Squeeze-and-Excitation lifts mobile network performance but ignores positional information important for spatially selective attention. The authors propose coordinate attention, factorizing channel attention into two 1D feature encodings that aggregate features along each spatial direction, capturing long-range dependencies while preserving precise positions. It plugs into MobileNetV2, MobileNeXt and EfficientNet with almost no overhead, and benefits ImageNet classification as well as object detection and semantic segmentation.

Based on: Coordinate Attention for Efficient Mobile Network Design · Computer Vision and Pattern Recognition

Curated by Aramai Editorial

Read summary →
Highlight

Graph Neural Networks: A Review of Methods and Applications

Surveys graph neural networks-models that capture graph dependencies via message passing-reviewing methods, applications, and open problems.

This survey reviews graph neural networks (GNNs), models capturing dependencies in graph data via message passing between nodes. Many tasks-modeling physical systems, learning molecular fingerprints, or reasoning over dependency trees and scene graphs-need graph inputs; unlike standard networks, GNNs retain neighborhood information at arbitrary depth. Early GNNs were hard to train, but advances in architecture and optimization enabled learning, with variants like GCN, GAT, and gated GNNs excelling. It reviews models, categorizes applications, and poses four open problems.

Based on: Graph Neural Networks: A Review of Methods and Applications · AI Open

Curated by Aramai Editorial

Read summary →
Highlight

CARLA: An Open Urban Driving Simulator

Introduces CARLA, an open-source simulator for autonomous urban driving research with open digital assets and flexible sensor and environment setups.

The authors introduce CARLA, an open-source simulator built to support development, training, and validation of autonomous urban driving systems. Beyond open code and protocols, CARLA provides freely usable digital assets such as urban layouts, buildings, and vehicles, and supports flexible specification of sensor suites and environmental conditions. Using CARLA, they study three approaches to driving: a classic modular pipeline, an end-to-end model trained by imitation learning, and one trained by reinforcement learning, evaluated in scenarios of increasing difficulty.

Based on: CARLA: An Open Urban Driving Simulator · Conference on Robot Learning

Curated by Aramai Editorial

Read summary →
Highlight

Neural Collaborative Filtering

Proposes NCF, a neural framework modeling user-item interactions from implicit feedback by replacing matrix factorization's inner product with a network.

Neural Collaborative Filtering (NCF) applies deep neural networks to collaborative filtering on implicit feedback, the core recommendation problem prior deep-learning work mostly left to matrix factorization. Earlier methods modeled user-item interaction with an inner product over latent features; NCF instead learns an arbitrary interaction function from data, using a multi-layer perceptron for non-linearities, and can generalize matrix factorization. On two real-world datasets it significantly outperforms state-of-the-art methods, and deeper networks improve performance.

Based on: Neural Collaborative Filtering · The Web Conference

Curated by Aramai Editorial

Read summary →
Highlight

Efficient Memory Management for Large Language Model Serving with PagedAttention

Introduces PagedAttention, an OS-paging-inspired attention algorithm, and vLLM, an LLM serving system that cuts KV cache waste and boosts throughput.

High-throughput LLM serving needs batching many requests, but each request's key-value (KV) cache is large and grows and shrinks dynamically, wasting memory via fragmentation and duplication. The authors propose PagedAttention, an attention algorithm inspired by virtual memory and paging in operating systems, and build vLLM on it, achieving near-zero KV cache waste and flexible cache sharing across requests. vLLM improves throughput of popular LLMs by 2-4x at similar latency versus systems like FasterTransformer and Orca, more so for longer sequences and bigger models.

Based on: Efficient Memory Management for Large Language Model Serving with PagedAttention · Symposium on Operating Systems Principles

Curated by Aramai Editorial

Read summary →
Highlight

Character-level Convolutional Networks for Text Classification

Empirically studies character-level convolutional networks for text classification, benchmarked against bag-of-words, n-grams, and word-based models.

This paper empirically explores using character-level convolutional networks (ConvNets) for text classification. The authors construct several large-scale datasets to demonstrate that character-level ConvNets can achieve state-of-the-art or competitive results. They compare against traditional models such as bag-of-words, n-grams, and their TF-IDF variants, as well as deep learning models including word-based ConvNets and recurrent neural networks.

Based on: Character-level Convolutional Networks for Text Classification · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

An overview of gradient descent optimization algorithms

Proposes a statistical test to compare ML models' metric values by splitting the test set into N parts and applying a modified Student's t-test.

The paper proposes a statistical test to compare the metric values of ML models on a test set. Because a metric depends on model and data, different models can win on different test sets, and cross-validation cannot guarantee independent measurements, precluding a t-test. For additive metrics, the author splits the test set into N parts; by the central limit theorem each part's metric is ~normal, so a modified t-test compares means, with normality tests and Q-Q plots sizing samples. A simpler confidence-interval variant proved rougher for CTR cross-entropy.

Based on: An overview of gradient descent optimization algorithms · Vestnik komp iuternykh i informatsionnykh tekhnologii

Curated by Aramai Editorial

Read summary →
Highlight

Adding Conditional Control to Text-to-Image Diffusion Models

Introduces ControlNet, an architecture that adds spatial conditioning (edges, depth, pose, etc.) to pretrained text-to-image diffusion models.

ControlNet is a neural architecture that adds spatial conditioning to large pretrained text-to-image diffusion models. It locks the pretrained model and reuses its encoding layers, learned from billions of images, linking trainable copies via 'zero convolutions'-zero-initialized layers that protect finetuning from noise. Tested controls include edges, depth, segmentation, and human pose on Stable Diffusion, with single or multiple conditions, with or without prompts. Training stays robust on small (<50k) and large (>1m) datasets, broadening control over image diffusion.

Based on: Adding Conditional Control to Text-to-Image Diffusion Models · IEEE International Conference on Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

Unsupervised Domain Adaptation by Backpropagation

A domain adaptation method learning features discriminative for the source task yet invariant to domain shift, using a simple gradient reversal layer.

Top deep models need massive labeled data, often unavailable for a target task; domain adaptation helps when labeled data from a similar but different domain exists. The authors propose a deep domain adaptation approach trained on labeled source data and unlabeled target data, needing none for the target. It yields features both discriminative for the source task and domain-invariant, via a few standard layers plus a simple gradient reversal layer trained by standard backpropagation. It outperforms prior state of the art on Office datasets under big domain shifts.

Based on: Unsupervised Domain Adaptation by Backpropagation · International Conference on Machine Learning

Curated by Aramai Editorial

Read summary →
Highlight

Gaussian Error Linear Units (GELUs)

Proposes the Gaussian Error Linear Unit (GELU), an activation xΦ(x) that weights inputs by their value rather than gating by sign as in ReLUs.

The authors propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function defined as xΦ(x), where Φ(x) is the standard Gaussian cumulative distribution function. Unlike ReLUs, which gate inputs by their sign, GELU weights inputs by their value. An empirical evaluation against the ReLU and ELU activations finds performance improvements across all considered computer vision, natural language processing, and speech tasks.

Based on: Gaussian Error Linear Units (GELUs) · arXiv

Curated by Aramai Editorial

Read summary →
Highlight

Neural Ordinary Differential Equations

Introduces neural ODEs, continuous-depth models that parameterize a hidden state's derivative with a network solved by a black-box ODE solver.

The paper introduces continuous-depth models that parameterize the hidden state's derivative with a neural network and compute outputs via a black-box ODE solver. These models have constant memory cost, adapt computation per input, and trade precision for speed. They demonstrate continuous-depth residual networks and continuous-time latent-variable models, and continuous normalizing flows for maximum-likelihood generative modeling without ordering data dimensions. Gradients backpropagate through any ODE solver without accessing its internals for end-to-end training.

Based on: Neural Ordinary Differential Equations · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →