Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations release_v5ew7elhxrh75ln6disnfqloli

by Debraj Basu, Deepesh Data, Can Karakus, Suhas Diggavi

Released as a article .



Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models. Recently, several approaches to mitigate this problem have been proposed, including different forms of gradient compression or computing local models and mixing them iteratively. In this paper, we propose Qsparse-local-SGD algorithm, which combines aggressive sparsification with quantization and local computation along with error compensation, by keeping track of the difference between the true and compressed gradients. We propose both synchronous and asynchronous implementations of Qsparse-local-SGD. We analyze convergence for Qsparse-local-SGD in the distributed setting for smooth non-convex and convex objective functions. We demonstrate that Qsparse-local-SGD converges at the same rate as vanilla distributed SGD for many important classes of sparsifiers and quantizers. We use Qsparse-local-SGD to train ResNet-50 on ImageNet and show that it results in significant savings over the state-of-the-art, in the number of bits transmitted to reach target accuracy.
In text/plain format

Archived Files and Locations

application/pdf  1.5 MB
web.archive.org (webarchive)
arxiv.org (repository)
Read Archived PDF
Type  article
Stage   submitted
Date   2019-06-06
Version   v1
Language   en ?
arXiv  1906.02367v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 5d659cb6-3e7d-4a38-bdd0-8f2fb23e5556