Faster Neural Network Training with Approximate Tensor Operations release_kwhjhtkwrvfzddpd3vvnjr4acq

by Menachem Adelman, Kfir Y. Levy, Ido Hakimi, Mark Silberstein

Released as a article .

2021  

Abstract

We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.
In text/plain format

Archived Files and Locations

application/pdf  4.0 MB
file_mvwzbqb32rczrlosftvwhct3au
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2021-10-25
Version   v3
Language   en ?
arXiv  1805.08079v3
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 0147e8d0-de6a-41b0-85a3-ce791e78c027
API URL: JSON