Highlight

SciBERT: A Pretrained Language Model for Scientific Text

Releases SciBERT, a BERT-based language model pretrained on scientific text to improve downstream scientific NLP tasks.

Large-scale annotated data for scientific-domain NLP is expensive and hard to obtain. The authors release SciBERT, a pretrained language model based on BERT that leverages unsupervised pretraining on a large multi-domain corpus of scientific publications. SciBERT is evaluated on a suite of tasks including sequence tagging, sentence classification, and dependency parsing, using datasets from a variety of scientific domains. It shows statistically significant improvements over BERT and achieves new state-of-the-art results on several tasks, with code and pretrained models released publicly.

Based on: SciBERT: A Pretrained Language Model for Scientific Text · Conference on Empirical Methods in Natural Language Processing

Curated by Aramai Editorial

Read summary →
Highlight

Deep Image Prior

Shows that a generator network's structure alone captures image statistics, enabling restoration without any learning from data.

Deep convolutional networks are widely used for image generation and restoration, with performance usually attributed to priors learned from large datasets. This paper shows instead that a generator network structure alone captures much low-level image statistics prior to any learning. A randomly-initialized network can act as a handcrafted prior, giving excellent results on inverse problems such as denoising, super-resolution, and inpainting. The same prior can invert deep representations and restore images, highlighting the inductive bias of generator architectures.

Based on: Deep Image Prior · International Journal of Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

Deep Learning: Methods and Applications

A monograph surveying general deep learning methods and their applications across signal and information processing tasks.

This monograph provides an overview of general deep learning methodology and its applications to a variety of signal and information processing tasks. Application areas are chosen based on the authors' expertise, areas already transformed by deep learning such as speech recognition and computer vision, and areas with strong potential and research growth. The latter include natural language and text processing, information retrieval, and multimodal information processing empowered by multi-task deep learning.

Based on: Deep Learning: Methods and Applications · Foundations and Trends® in Signal Processing

Curated by Aramai Editorial

Read summary →
Highlight

End-to-End Training of Deep Visuomotor Policies

Presents a method to jointly train perception and control end-to-end, mapping raw robot images directly to motor torques.

This paper asks whether training perception and control jointly end-to-end outperforms training each separately for robotic policy search. The authors develop a method that learns policies mapping raw image observations directly to motor torques, represented by deep convolutional networks with 92,000 parameters. Training uses a partially observed guided policy search that transforms policy search into supervised learning, supervised by a trajectory-centric reinforcement learning method. It is evaluated on real-world manipulation tasks such as screwing a cap onto a bottle.

Based on: End-to-End Training of Deep Visuomotor Policies · Journal of machine learning research

Curated by Aramai Editorial

Read summary →
Highlight

Efficiently Modeling Long Sequences with Structured State Spaces

Introduces S4, a structured state space sequence model that efficiently captures very long-range dependencies across modalities.

S4 is a sequence model for long-range dependencies where RNNs, CNNs, and Transformers struggle at 10,000+ steps. It builds on the state space model x'=Ax+Bu, y=Cx+Du, and introduces a parameterization that conditions state matrix A with a low-rank correction, enabling stable diagonalization and reducing computation to a Cauchy kernel. This makes prior SSMs far more efficient while preserving their strengths. S4 reaches 91% on sequential CIFAR-10, narrows the gap to Transformers while generating 60x faster, and sets SOTA on Long Range Arena, including the 16k-length Path-X task.

Based on: Efficiently Modeling Long Sequences with Structured State Spaces · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Introduces PatchTST, a patch-based, channel-independent Transformer for multivariate time series forecasting and self-supervised learning.

PatchTST is an efficient Transformer design for multivariate time series forecasting and self-supervised representation learning. It rests on two ideas: segmenting series into subseries-level patches used as input tokens, and channel-independence where each univariate channel shares embeddings and Transformer weights. Patching retains local semantics, quadratically cuts attention cost, and lets the model attend to longer history. PatchTST substantially improves long-term forecasting over prior Transformers and yields strong self-supervised pre-training and transfer results.

Based on: A Time Series is Worth 64 Words: Long-term Forecasting with Transformers · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Introduces a convolutional neural network operating directly on graphs to learn differentiable molecular fingerprints, generalizing circular fingerprints.

This work introduces a convolutional neural network that operates directly on graphs, enabling end-to-end learning of prediction pipelines whose inputs are graphs of arbitrary size and shape. The architecture generalizes standard molecular feature extraction methods based on circular fingerprints. The authors show that these data-driven learned features are more interpretable and achieve better predictive performance than the fixed circular-fingerprint baselines across a variety of tasks.

Based on: Convolutional Networks on Graphs for Learning Molecular Fingerprints · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

A tutorial and survey of hardware and algorithmic techniques for processing deep neural networks efficiently to improve energy efficiency and throughput.

Deep neural networks power many AI applications, delivering top accuracy at high computational cost. Techniques for efficient DNN processing that improve energy efficiency and throughput without sacrificing accuracy or raising hardware cost are critical to deployment. This tutorial and survey overviews DNNs, discusses hardware platforms and architectures supporting them, and highlights trends in cutting computation cost via hardware design or joint hardware-algorithm codesign. It also summarizes benchmarking metrics and design considerations for evaluating DNN hardware.

Based on: Efficient Processing of Deep Neural Networks: A Tutorial and Survey · Proceedings of the IEEE

Curated by Aramai Editorial

Read summary →
Highlight

Understanding Black-box Predictions via Influence Functions

Uses influence functions from robust statistics to trace a model's prediction back to the training points most responsible for it.

To explain a black-box model's predictions, this work uses influence functions, a classic robust-statistics technique, to trace a prediction through the learning algorithm back to the training points most responsible for it. The authors develop an efficient implementation needing only oracle access to gradients and Hessian-vector products. Even on non-convex, non-differentiable models where the theory breaks down, approximations remain informative. On linear models and CNNs, they aid understanding behavior, debugging, detecting dataset errors, and crafting training-set attacks.

Based on: Understanding Black-box Predictions via Influence Functions · International Conference on Machine Learning

Curated by Aramai Editorial

Read summary →
Highlight

MLP-Mixer: An all-MLP Architecture for Vision

Introduces MLP-Mixer, an all-MLP vision architecture that mixes per-patch and cross-patch features without convolutions or attention.

CNNs are the go-to vision model, and attention-based Vision Transformers are popular, but this paper shows neither convolutions nor attention are necessary. MLP-Mixer is built exclusively from multi-layer perceptrons, with two layer types: MLPs applied per patch to mix channel features, and MLPs applied across patches to mix spatial information. Trained on large datasets or with modern regularization, it attains competitive image classification scores at pre-training and inference cost comparable to state-of-the-art models, and the authors hope it sparks research beyond CNNs and Transformers.

Based on: MLP-Mixer: An all-MLP Architecture for Vision · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Mapping Identifiers for the Integration of Genomic Datasets with the R/Bioconductor package biomaRt

Presents a protocol using the R/Bioconductor biomaRt package and BioMart web services to integrate and jointly analyze diverse genomic datasets.

Genomic experiments produce multiple views of biological systems, including DNA sequence, copy number variation, and mRNA and protein abundance, requiring integrated bioinformatic analysis. Public databases like Ensembl provide mappings between probe and target molecules, but these can be complex and dynamic. This protocol shows how to use R with BioMart web services, via the biomaRt package, to integrate and jointly analyze datasets. It discusses typical gene-to-transcript-to-protein mapping problems and offers a flexible, reproducible basis for data integration.

Based on: Mapping Identifiers for the Integration of Genomic Datasets with the R/Bioconductor package biomaRt · Nature Protocols

Curated by Aramai Editorial

Read summary →
Highlight

ClinVar: improving access to variant interpretations and supporting evidence

Describes updates to ClinVar, a public NIH archive of human genetic variants and their clinical significance, improving data access and search.

ClinVar is a freely available public NIH archive of human genetic variants and interpretations of their significance to disease. Interpretations are submitted by clinical and research labs, expert panels, and others, and data are aggregated by variant-disease pairs and by variant, accessible via the website, improved VCF files, and a new XML report. ClinVar recently began accepting phenotype-focused submissions, including provider interpretations and phenotyping-only data, and continues improving search with new indexed fields and filters.

Based on: ClinVar: improving access to variant interpretations and supporting evidence · Nucleic Acids Res.

Curated by Aramai Editorial

Read summary →