Guided evolutionary strategies: Augmenting random search with surrogate
gradients
release_ws5s7ciry5am3o6idymzhxx6ty
by
Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, Jascha
Sohl-Dickstein
2019
Abstract
Many applications in machine learning require optimizing a function whose
true gradient is unknown, but where surrogate gradient information (directions
that may be correlated with, but not necessarily identical to, the true
gradient) is available instead. This arises when an approximate gradient is
easier to compute than the full gradient (e.g. in meta-learning or unrolled
optimization), or when a true gradient is intractable and is replaced with a
surrogate (e.g. in certain reinforcement learning applications, or when using
synthetic gradients). We propose Guided Evolutionary Strategies, a method for
optimally using surrogate gradient directions along with random search. We
define a search distribution for evolutionary strategies that is elongated
along a guiding subspace spanned by the surrogate gradients. This allows us to
estimate a descent direction which can then be passed to a first-order
optimizer. We analytically and numerically characterize the tradeoffs that
result from tuning how strongly the search distribution is stretched along the
guiding subspace, and we use this to derive a setting of the hyperparameters
that works well across problems. Finally, we apply our method to example
problems, demonstrating an improvement over both standard evolutionary
strategies and first-order methods (that directly follow the surrogate
gradient). We provide a demo of Guided ES at
https://github.com/brain-research/guided-evolutionary-strategies
In text/plain
format
Archived Files and Locations
application/pdf 5.5 MB
file_uuwg4fyguvghxlcrt25jw6bg64
|
arxiv.org (repository) web.archive.org (webarchive) |
1806.10230v4
access all versions, variants, and formats of this works (eg, pre-prints)