On the Expressive Power of Deep Neural Networks release_g5ntrs63djhl3dn73pzr77sagm

by Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl-Dickstein

Released as a article .

2016  

Abstract

We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings can be summarized as follows: (1) The complexity of the computed function grows exponentially with depth. (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights. (3) Regularizing on trajectory length (trajectory regularization) is a simpler alternative to batch normalization, with the same performance.
In text/plain format

Archived Files and Locations

application/pdf  1.2 MB
file_orokhsjrrjeqngadtjwj2c5jr4
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2016-06-24
Version   v2
Language   en ?
arXiv  1606.05336v2
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 2d8279d3-4610-47fd-b48f-ea31e574123f
API URL: JSON