Highlight

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

ALBERT introduces two parameter-reduction techniques and an inter-sentence coherence loss to scale BERT pretraining with less memory and faster training.

Scaling up model size in language representation pretraining tends to improve downstream performance but eventually runs into GPU/TPU memory limits and longer training times. ALBERT proposes two parameter-reduction techniques that cut memory use and speed up BERT training, allowing it to scale far better than the original. It also adds a self-supervised loss modeling inter-sentence coherence, which helps tasks with multi-sentence inputs. The best model sets new state-of-the-art results on GLUE, RACE, and SQuAD while using fewer parameters than BERT-large.

Based on: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

The Carbohydrate-Active EnZymes database (CAZy): an expert resource for Glycogenomics

The CAZy database is an expert, knowledge-based resource classifying enzymes that build and break down complex carbohydrates and glycoconjugates.

CAZy is a knowledge-based database specializing in enzymes that build and break down complex carbohydrates and glycoconjugates. As of September 2008 it described families of glycoside hydrolases, glycosyltransferases, polysaccharide lyases, carbohydrate esterases, and carbohydrate-binding modules, defined from characterized proteins and populated by similar sequences. Its classification reflects structural features, reveals evolutionary relationships, and frames mechanistic properties, aiding functional annotation in genome projects.

Based on: The Carbohydrate-Active EnZymes database (CAZy): an expert resource for Glycogenomics · Nucleic Acids Res.

Curated by Aramai Editorial

Read summary →
Highlight

The Stanford CoreNLP Natural Language Processing Toolkit

Describes Stanford CoreNLP, an extensible pipeline for core natural language analysis widely used in research, commercial, and government settings.

The paper presents the design and use of the Stanford CoreNLP toolkit, an extensible pipeline that provides core natural language analysis. It is widely adopted both in the research NLP community and among commercial and government users of open-source NLP technology. The authors attribute this uptake to a simple, approachable design, straightforward interfaces, robust and good-quality analysis components, and not requiring a large amount of associated baggage.

Based on: The Stanford CoreNLP Natural Language Processing Toolkit · Annual Meeting of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

ColabFold: making protein folding accessible to all

ColabFold combines MMseqs2's fast homology search with AlphaFold2 or RoseTTAFold to make accelerated protein structure prediction free and accessible.

ColabFold accelerates protein structure and complex prediction by pairing the fast homology search of MMseqs2 with AlphaFold2 or RoseTTAFold. Its 40-60-fold faster search and optimized model utilization enable prediction of close to 1,000 structures per day on a server with a single GPU. Delivered through Google Colaboratory, it is a free, accessible, open-source platform for protein folding, with novel environmental databases made available to the community.

Based on: ColabFold: making protein folding accessible to all · Nature Methods

Curated by Aramai Editorial

Read summary →
Highlight

The Cityscapes Dataset for Semantic Urban Scene Understanding

Introduces Cityscapes, a large-scale benchmark dataset for pixel- and instance-level semantic labeling of urban street scenes from 50 cities.

For semantic urban scene understanding, no prior dataset adequately captured real-world urban complexity. Cityscapes is a benchmark suite and large-scale dataset for pixel-level and instance-level semantic labeling, built from diverse stereo video sequences recorded in streets of 50 different cities. It provides 5000 images with high-quality pixel-level annotations and 20000 with coarse annotations, exceeding prior efforts in size, annotation richness, variability, and complexity. An accompanying study analyzes the dataset and evaluates state-of-the-art methods.

Based on: The Cityscapes Dataset for Semantic Urban Scene Understanding · Computer Vision and Pattern Recognition

Curated by Aramai Editorial

Read summary →
Highlight

Representation Learning with Contrastive Predictive Coding

Proposes Contrastive Predictive Coding, an unsupervised approach that learns representations by predicting future latents with a contrastive loss.

Unsupervised learning remains an important and challenging endeavor for AI. Contrastive Predictive Coding extracts useful representations from high-dimensional data by predicting the future in latent space using powerful autoregressive models. A probabilistic contrastive loss, kept tractable with negative sampling, induces the latent space to capture information maximally useful for predicting future samples. The approach achieves strong performance across four domains: speech, images, text, and reinforcement learning in 3D environments.

Based on: Representation Learning with Contrastive Predictive Coding · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

Playing Atari with Deep Reinforcement Learning

Presents the first deep learning model to learn control policies from raw pixels, via a Q-learning-trained convolutional network on Atari games.

This work presents the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network trained with a variant of Q-learning, taking raw pixels as input and outputting a value function that estimates future rewards. Applied to seven Atari 2600 games from the Arcade Learning Environment without adjusting the architecture or learning algorithm, it outperforms all previous approaches on six games and surpasses a human expert on three.

Based on: Playing Atari with Deep Reinforcement Learning · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

Representation Learning: A Review and New Perspectives

Reviews unsupervised feature learning and deep learning, covering probabilistic models, autoencoders, manifold learning, and deep networks.

Machine learning success generally depends on data representation, hypothesized to be because different representations can entangle or hide the explanatory factors of variation behind data. This review covers recent work in unsupervised feature learning and deep learning, spanning probabilistic models, autoencoders, manifold learning, and deep networks. It motivates open questions about objectives for learning good representations, computing representations (inference), and geometric links among representation learning, density estimation, and manifold learning.

Based on: Representation Learning: A Review and New Perspectives · IEEE Transactions on Pattern Analysis and Machine Intelligence

Curated by Aramai Editorial

Read summary →
Highlight

Segment Anything

Introduces the Segment Anything project: a promptable segmentation model (SAM) and SA-1B, a dataset of over 1 billion masks on 11M licensed images.

The Segment Anything (SA) project comprises a new task, model, and dataset for image segmentation. Using an efficient model in a data collection loop, the authors built the largest segmentation dataset to date, with over 1 billion masks on 11 million licensed, privacy-respecting images. The model is designed to be promptable, transferring zero-shot to new image distributions and tasks with performance often competitive with or superior to prior fully supervised results. SAM and the SA-1B dataset are released to foster research into foundation models for computer vision.

Based on: Segment Anything · IEEE International Conference on Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

Convolutional Neural Networks for Sentence Classification

Shows a simple CNN over pre-trained word vectors excels at sentence classification, improving the state of the art on 4 of 7 benchmark tasks.

The paper reports experiments with convolutional neural networks trained on top of pre-trained word vectors for sentence-level classification tasks. A simple CNN with little hyperparameter tuning and static vectors achieves excellent results across multiple benchmarks, and learning task-specific vectors through fine-tuning yields further gains. A simple architectural modification allows using both task-specific and static vectors. The models improve upon the state of the art on 4 out of 7 tasks, including sentiment analysis and question classification.

Based on: Convolutional Neural Networks for Sentence Classification · Conference on Empirical Methods in Natural Language Processing

Curated by Aramai Editorial

Read summary →
Highlight

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Introduces PointNet++, a hierarchical network applying PointNet recursively on nested partitions of point sets to learn multi-scale local features.

PointNet, a pioneering deep network for point sets, does not capture local structures induced by the metric space points live in, limiting fine-grained recognition and generalization to complex scenes. PointNet++ applies PointNet recursively on a nested partitioning of the input point set, learning local features with increasing contextual scales. Since point sets are often sampled with varying densities, novel set learning layers adaptively combine features from multiple scales. Results significantly surpass the state of the art on challenging 3D point cloud benchmarks.

Based on: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Proposes MAML, a model-agnostic meta-learning algorithm that trains models to adapt to new tasks from few samples via a few gradient steps.

The paper proposes a meta-learning algorithm that is model-agnostic: compatible with any model trained by gradient descent and applicable to classification, regression, and reinforcement learning. Model parameters are explicitly trained so that a small number of gradient steps on a small amount of data from a new task yields good generalization, in effect making the model easy to fine-tune. The approach achieves state-of-the-art results on two few-shot image classification benchmarks, performs well on few-shot regression, and accelerates fine-tuning for policy gradient RL.

Based on: Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks · International Conference on Machine Learning

Curated by Aramai Editorial

Read summary →