Normalization and variance stabilization of single-cell RNA-seq data using regularized negative binomial regression

The paper addresses a core problem in single-cell RNA-seq: significant cell-to-cell variation from technical factors, including the number of molecules detected per cell, which can confound biological heterogeneity with technical effects. The authors present a modeling framework for normalization and variance stabilization of molecular count data in which Pearson residuals from 'regularized negative binomial regression' are computed, using cellular sequencing depth as a covariate in a generalized linear model. Recognizing that an unconstrained negative binomial model can overfit scRNA-seq data, they pool information across genes with similar abundances to obtain stable parameter estimates.

The procedure removes the influence of technical characteristics from downstream analyses while preserving biological heterogeneity, and it omits heuristic steps such as pseudocount addition or log-transformation. As a result, it improves common downstream tasks including variable gene selection, dimensional reduction, and differential expression. The approach applies to any UMI-based scRNA-seq dataset and is freely available in the R package sctransform with a direct interface to the Seurat toolkit, which mattered for making principled normalization broadly usable.