How Does Batch Normalization Help Binary Training? release_7a4t3ew5dvbsdifnajsm7rytby

by Eyyüb Sari, Mouloud Belbahri, Vahid Partovi Nia

Released as a article .

2019  

Abstract

Binary Neural Networks (BNNs) are difficult to train, and suffer from drop of accuracy. It appears in practice that BNNs fail to train in the absence of Batch Normalization (BatchNorm) layer. We find the main role of BatchNorm is to avoid exploding gradients in the case of BNNs. This finding suggests that the common initialization methods developed for full-precision networks are irrelevant to BNNs. We build a theoretical study on the role of BatchNorm in binary training, backed up by numerical experiments.
In text/plain format

Archived Files and Locations

application/pdf  116.7 kB
file_pk6mxs7e7vglrly3kc42h5znzi
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2019-09-18
Version   v1
Language   en ?
arXiv  1909.09139v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 4e37ba55-5c42-4b69-8c88-a77bcd4a9828
API URL: JSON