Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
release_7zjdvv2xk5gfjbg2gp25iz7dli
by
Diego Granziol, Stefan Zohren, Stephen Roberts
2021
Abstract
We study the effect of mini-batching on the loss landscape of deep neural
networks using spiked, field-dependent random matrix theory. We demonstrate
that the magnitude of the extremal values of the batch Hessian are larger than
those of the empirical Hessian. We also derive similar results for the
Generalised Gauss-Newton matrix approximation of the Hessian. As a consequence
of our theorems we derive an analytical expressions for the maximal learning
rates as a function of batch size, informing practical training regimens for
both stochastic gradient descent (linear scaling) and adaptive algorithms, such
as Adam (square root scaling), for smooth, non-convex deep neural networks.
Whilst the linear scaling for stochastic gradient descent has been derived
under more restrictive conditions, which we generalise, the square root scaling
rule for adaptive optimisers is, to our knowledge, completely novel.
stochastic second-order methods and adaptive methods, we derive that the
minimal damping coefficient is proportional to the ratio of the learning rate
to batch size. We validate our claims on the VGG/WideResNet architectures on
the CIFAR-100 and ImageNet datasets. Based on our investigations of the
sub-sampled Hessian we develop a stochastic Lanczos quadrature based on the fly
learning rate and momentum learner, which avoids the need for expensive
multiple evaluations for these key hyper-parameters and shows good preliminary
results on the Pre-Residual Architecure for CIFAR-100.
In text/plain
format
Archived Files and Locations
application/pdf 2.9 MB
file_tztixt6hpvgrtffo2pd5dewqyi
|
arxiv.org (repository) web.archive.org (webarchive) |
2006.09092v4
access all versions, variants, and formats of this works (eg, pre-prints)