A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning

This paper introduces the Scaled Conjugate Gradient (SCG) algorithm, a supervised learning method with a superlinear convergence rate built on the conjugate gradient methods well known in numerical optimization. SCG exploits second-order information from the neural network but requires only O(N) memory, where N is the number of weights. It is fully automated with no user-dependent parameters, and it avoids the time-consuming line search that conjugate gradient backpropagation and BFGS perform at every iteration to choose a step size.

Benchmarked against standard backpropagation, conjugate gradient backpropagation, and the one-step memoryless BFGS quasi-Newton algorithm, SCG yields a speed-up of at least an order of magnitude over backpropagation, with the advantage growing as the demanded error reduction increases. The author also shows that when a network's complexity is small relative to its problem domain, the weight space can contain long ravines with sharp curvature where backpropagation is inefficient, whereas SCG handles these ravine phenomena effectively.