NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search
release_auwetqgtdva5li5l2x5skuclka
by
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, Frank Hutter
2020
Abstract
The most significant barrier to the advancement of Neural Architecture Search
(NAS) is its demand for large computational resources, which hinders
scientifically sound empirical evaluations. As a remedy, several tabular NAS
benchmarks were proposed to simulate runs of NAS methods in seconds. However,
all existing tabular NAS benchmarks are limited to extremely small
architectural spaces since they rely on exhaustive evaluations of the space.
This leads to unrealistic results that do not transfer to larger search spaces.
To overcome this fundamental limitation, we propose NAS-Bench-301, the first
surrogate NAS benchmark, using a search space containing 10^18
architectures, many orders of magnitude larger than any previous tabular NAS
benchmark. After motivating the benefits of a surrogate benchmark over a
tabular one, we fit various regression models on our dataset, which consists of
∼60k architecture evaluations, and build surrogates via deep ensembles to
also model uncertainty. We benchmark a wide range of NAS algorithms using
NAS-Bench-301 and obtain comparable results to the true benchmark at a fraction
of the real cost. Finally, we show how NAS-Bench-301 can be used to generate
new scientific insights.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
2008.09777v3
access all versions, variants, and formats of this works (eg, pre-prints)