Highlight

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Empirically compares gated recurrent units (LSTM, GRU) against traditional tanh units in RNNs on polyphonic music and speech signal modeling tasks.

This paper compares different types of recurrent units in recurrent neural networks, focusing on sophisticated units with gating mechanisms such as the long short-term memory (LSTM) unit and the recently proposed gated recurrent unit (GRU). The units are evaluated on polyphonic music modeling and speech signal modeling tasks. Experiments show that the advanced gated units outperform traditional recurrent units such as tanh units, and that the GRU is comparable to the LSTM.

Based on: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

Caffe: Convolutional Architecture for Fast Feature Embedding

Presents Caffe, a BSD-licensed C++ deep learning framework with Python and MATLAB bindings for training and deploying CNNs on commodity hardware.

Caffe is a BSD-licensed C++ deep learning framework with Python and MATLAB bindings for training and deploying convolutional neural networks and other deep models on commodity architectures. Using CUDA GPU computation, it processes over 40 million images a day on a single K40 or Titan GPU, roughly 2 ms per image. By separating model representation from implementation, Caffe enables seamless switching among platforms from prototyping machines to cloud environments, and it is maintained by the Berkeley Vision and Learning Center with an active community of contributors.

Based on: Caffe: Convolutional Architecture for Fast Feature Embedding · ACM Multimedia

Curated by Aramai Editorial

Read summary →
Highlight

Inductive Representation Learning on Large Graphs

Presents GraphSAGE, an inductive framework that generates node embeddings by sampling and aggregating features from a node's local neighborhood.

Most node-embedding approaches are transductive, requiring all nodes present during training and failing to generalize to unseen nodes. GraphSAGE is a general inductive framework that leverages node feature information to generate embeddings for unseen data, learning a function that samples and aggregates features from a node's local neighborhood rather than training per-node embeddings. It outperforms strong baselines on three inductive node-classification benchmarks and generalizes to entirely unseen graphs in a protein-protein interaction dataset.

Based on: Inductive Representation Learning on Large Graphs · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

LoRA: Low-Rank Adaptation of Large Language Models

Proposes LoRA, which freezes pre-trained weights and injects trainable low-rank decomposition matrices into Transformer layers for efficient adaptation.

Full fine-tuning of large pre-trained language models becomes impractical at scale — deploying independent fine-tuned instances of GPT-3 175B is prohibitively expensive. LoRA freezes pre-trained weights and injects trainable rank decomposition matrices into each Transformer layer, cutting trainable parameters for downstream tasks by 10,000x and GPU memory by 3x versus fine-tuning GPT-3 with Adam. LoRA matches or beats fine-tuning quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, with higher training throughput and, unlike adapters, no added inference latency.

Based on: LoRA: Low-Rank Adaptation of Large Language Models · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

LLaMA: Open and Efficient Foundation Language Models

Introduces LLaMA, foundation language models (7B-65B) trained solely on publicly available data, with LLaMA-13B outperforming GPT-3 on most benchmarks.

LLaMA is a collection of foundation language models ranging from 7B to 65B parameters, trained on trillions of tokens. The work shows that state-of-the-art models can be trained using publicly available datasets exclusively, without proprietary or inaccessible data. LLaMA-13B outperforms the 175B GPT-3 on most benchmarks, LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B, and all models are released to the research community.

Based on: LLaMA: Open and Efficient Foundation Language Models · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Proposes the PReLU activation and a rectifier-aware initialization, achieving 4.94% top-5 ImageNet error — first to surpass human-level performance.

This work studies rectifier neural networks for image classification from two angles. It proposes the Parametric Rectified Linear Unit (PReLU), which generalizes the rectified unit and improves fitting at nearly zero extra cost with little overfitting risk, and derives a robust initialization method accounting for rectifier nonlinearities, enabling very deep rectified models to be trained from scratch. The models achieve 4.94% top-5 test error on ImageNet 2012 — a 26% relative improvement over GoogLeNet (6.66%) and the first result surpassing reported human-level performance (5.1%).

Based on: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification · IEEE International Conference on Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Introduces DeepLab for semantic segmentation, combining atrous convolution, atrous spatial pyramid pooling, and fully connected CRFs.

DeepLab addresses semantic image segmentation with three contributions. Atrous convolution controls the resolution of feature responses and enlarges filters' field of view without extra parameters or computation; atrous spatial pyramid pooling (ASPP) probes features at multiple sampling rates to capture objects and context at multiple scales; and a fully connected CRF on the final DCNN layer improves boundary localization. DeepLab set a new state of the art on PASCAL VOC-2012 (79.7% test mIOU) and advanced results on PASCAL-Context, PASCAL-Person-Part, and Cityscapes.

Based on: DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs · IEEE Transactions on Pattern Analysis and Machine Intelligence

Curated by Aramai Editorial

Read summary →
Highlight

Statistical Learning Theory

A chapter on statistical machine learning covering SVMs, k-nearest neighbor, Naive Bayes, artificial neural networks, and instance-based learning.

Statistical machine learning systems learn from data rather than the environment in general. This chapter presents techniques including Support Vector Machines for pattern recognition and classification, SVM prediction of structured objects, the k-nearest neighbor method, and Naive Bayes classifiers. It also introduces artificial neural networks — error-correction rules, Boltzmann learning, the Hebbian rule, competitive learning, deep learning — treats instance-based learning in detail, and concludes with a summary and practice exercises.

Based on: Statistical Learning Theory · Technometrics

Curated by Aramai Editorial

Read summary →
Highlight

Sequence to Sequence Learning with Neural Networks

Presents an end-to-end sequence-to-sequence learning approach using multilayered LSTMs to encode inputs to a fixed vector and decode target sequences.

Deep neural networks perform well on difficult tasks given large labeled training sets but cannot map sequences to sequences. This paper presents a general end-to-end sequence learning approach that uses a multilayered LSTM to encode the input sequence into a fixed-dimensional vector and a second deep LSTM to decode the target sequence. On WMT-14 English-to-French translation it reaches 34.8 BLEU versus 33.3 for a phrase-based SMT system, and 36.5 when reranking that system's 1000 hypotheses; reversing source word order markedly improved performance.

Based on: Sequence to Sequence Learning with Neural Networks · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Training language models to follow instructions with human feedback

Introduces InstructGPT: GPT-3 fine-tuned on demonstrations and human feedback rankings via RL to align language models with user intent.

Larger language models are not inherently better at following user intent and can produce untruthful, toxic, or unhelpful outputs. The authors align models by fine-tuning GPT-3 on labeler-written demonstrations, then further fine-tuning with reinforcement learning from human feedback using rankings of model outputs, producing InstructGPT. In human evaluations, outputs from the 1.3B-parameter InstructGPT are preferred over the 175B GPT-3 despite 100x fewer parameters, with improved truthfulness, less toxic generation, and minimal regressions on public NLP datasets.

Based on: Training language models to follow instructions with human feedback · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Explaining and Harnessing Adversarial Examples

Argues neural nets' adversarial vulnerability stems from their linear nature and derives a fast method to generate examples for adversarial training.

Machine learning models, including neural networks, consistently misclassify adversarial examples — inputs perturbed by small, worst-case changes that cause confident wrong answers. Countering earlier explanations based on nonlinearity and overfitting, the authors argue the primary cause is the models' linear nature, which explains why such examples generalize across architectures and training sets. This view yields a simple, fast method for generating adversarial examples; using them for adversarial training reduces the test error of a maxout network on MNIST.

Based on: Explaining and Harnessing Adversarial Examples · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies

Extends the Technology Acceptance Model (TAM2) with social influence and cognitive instrumental processes, validated in four longitudinal field studies.

Develops TAM2, an extension of the Technology Acceptance Model explaining perceived usefulness and usage intentions via social influence (subjective norm, voluntariness, image) and cognitive instrumental processes (job relevance, output quality, result demonstrability, ease of use). It was tested with longitudinal data on four systems at four organizations (N=156), spanning voluntary and mandatory usage at three measurement points. TAM2 was strongly supported, explaining 40-60% of the variance in usefulness perceptions and 34-52% of the variance in usage intentions.

Based on: A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies · Management Sciences

Curated by Aramai Editorial

Read summary →