GIANT: Globally Improved Approximate Newton Method for Distributed
Optimization
release_bymn2vkpvrdulpntqa245dh6lm
by
Shusen Wang, Farbod Roosta-Khorasani, Peng Xu, Michael W. Mahoney
2018
Abstract
For distributed computing environment, we consider the empirical risk
minimization problem and propose a distributed and communication-efficient
Newton-type optimization method. At every iteration, each worker locally finds
an Approximate NewTon (ANT) direction, which is sent to the main driver. The
main driver, then, averages all the ANT directions received from workers to
form a Globally Improved ANT (GIANT) direction. GIANT is highly
communication efficient and naturally exploits the trade-offs between local
computations and global communications in that more local computations result
in fewer overall rounds of communications. Theoretically, we show that GIANT
enjoys an improved convergence rate as compared with first-order methods and
existing distributed Newton-type methods. Further, and in sharp contrast with
many existing distributed Newton-type methods, as well as popular first-order
methods, a highly advantageous practical feature of GIANT is that it only
involves one tuning parameter. We conduct large-scale experiments on a computer
cluster and, empirically, demonstrate the superior performance of GIANT.
In text/plain
format
Archived Files and Locations
application/pdf 2.3 MB
file_bdlpyznysjhuvdehy2znxwznfe
|
arxiv.org (repository) web.archive.org (webarchive) |
1709.03528v5
access all versions, variants, and formats of this works (eg, pre-prints)