Oracle Complexity Separation in Convex Optimization
release_wjl4kv6jlfdd3jgx4x6nkoehey
by
Anastasiya Ivanova, Evgeniya Vorontsova, Dmitry Pasechnyuk, Alexander Gasnikov, Pavel Dvurechensky, Darina Dvinskikh, Alexander Tyurin
2022
Abstract
Many convex optimization problems have structured objective function written
as a sum of functions with different types of oracles (full gradient,
coordinate derivative, stochastic gradient) and different evaluation complexity
of these oracles. In the strongly convex case these functions also have
different condition numbers, which eventually define the iteration complexity
of first-order methods and the number of oracle calls required to achieve given
accuracy. Motivated by the desire to call more expensive oracle less number of
times, in this paper we consider minimization of a sum of two functions and
propose a generic algorithmic framework to separate oracle complexities for
each component in the sum. As a specific example, for the μ-strongly convex
problem min_x∈ℝ^n h(x) + g(x) with L_h-smooth function h
and L_g-smooth function g, a special case of our algorithm requires, up to
a logarithmic factor, O(√(L_h/μ)) first-order oracle calls for h and
O(√(L_g/μ)) first-order oracle calls for g. Our general framework
covers also the setting of strongly convex objectives, the setting when g is
given by coordinate derivative oracle, and the setting when g has a
finite-sum structure and is available through stochastic gradient oracle. In
the latter two cases we obtain respectively accelerated random coordinate
descent and accelerated variance reduction methods with oracle complexity
separation.
In text/plain
format
Archived Files and Locations
application/pdf 750.0 kB
file_sgm7wcz34zfltlbpxply6hndii
|
arxiv.org (repository) web.archive.org (webarchive) |
2002.02706v4
access all versions, variants, and formats of this works (eg, pre-prints)