An overview of gradient descent optimization algorithms

The paper addresses how to statistically compare the metric values of machine learning models on a test set. It notes that because metric values depend on both the model and the data, different models can turn out best on different test sets, so the traditional approach of simply comparing metric values is often not enough; and while cross-validation-based comparisons are sometimes used, they cannot guarantee independent measurements, which rules out a direct Student's t-test. For additive metrics, the author proposes dividing the test sample into N parts and computing the metric on each part.

Because each part's metric value is obtained as a sum of independent random variables, the central limit theorem implies the N values are realizations of a normally distributed variable, so a modification of the Student's t-test can compare the mean metric values; normality tests and quantile-quantile plots are used to estimate the required sample size. A simplified alternative instead builds confidence intervals for a base model and flags any model whose metric falls outside the interval as behaving differently, reducing computation, though experiments on the binary cross-entropy metric for click-through rate (CTR) prediction models showed it is rougher than the first approach.