Oracle Complexity Separation in Convex Optimization release_wjl4kv6jlfdd3jgx4x6nkoehey

by Anastasiya Ivanova, Evgeniya Vorontsova, Dmitry Pasechnyuk, Alexander Gasnikov, Pavel Dvurechensky, Darina Dvinskikh, Alexander Tyurin

Released as a article .

2022  

Abstract

Many convex optimization problems have structured objective function written as a sum of functions with different types of oracles (full gradient, coordinate derivative, stochastic gradient) and different evaluation complexity of these oracles. In the strongly convex case these functions also have different condition numbers, which eventually define the iteration complexity of first-order methods and the number of oracle calls required to achieve given accuracy. Motivated by the desire to call more expensive oracle less number of times, in this paper we consider minimization of a sum of two functions and propose a generic algorithmic framework to separate oracle complexities for each component in the sum. As a specific example, for the μ-strongly convex problem min_x∈ℝ^n h(x) + g(x) with L_h-smooth function h and L_g-smooth function g, a special case of our algorithm requires, up to a logarithmic factor, O(√(L_h/μ)) first-order oracle calls for h and O(√(L_g/μ)) first-order oracle calls for g. Our general framework covers also the setting of strongly convex objectives, the setting when g is given by coordinate derivative oracle, and the setting when g has a finite-sum structure and is available through stochastic gradient oracle. In the latter two cases we obtain respectively accelerated random coordinate descent and accelerated variance reduction methods with oracle complexity separation.
In text/plain format

Archived Files and Locations

application/pdf  750.0 kB
file_sgm7wcz34zfltlbpxply6hndii
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2022-03-11
Version   v4
Language   en ?
arXiv  2002.02706v4
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 00457429-e2a6-4887-b0ad-e6de3354809f
API URL: JSON