On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels
release_qdr5a5hetbhyba27mpqjbsdnwe
by
Amnon Geifman, Meirav Galun, David Jacobs, Ronen Basri
2022
Abstract
We study the properties of various over-parametrized convolutional neural
architectures through their respective Gaussian process and neural tangent
kernels. We prove that, with normalized multi-channel input and ReLU
activation, the eigenfunctions of these kernels with the uniform measure are
formed by products of spherical harmonics, defined over the channels of the
different pixels. We next use hierarchical factorizable kernels to bound their
respective eigenvalues. We show that the eigenvalues decay polynomially,
quantify the rate of decay, and derive measures that reflect the composition of
hierarchical features in these networks. Our results provide concrete
quantitative characterization of over-parameterized convolutional network
architectures.
In text/plain
format
Archived Files and Locations
application/pdf 954.2 kB
file_nsfzy3xt5zdajn7motdp5rn6fa
|
arxiv.org (repository) web.archive.org (webarchive) |
2203.09255v1
access all versions, variants, and formats of this works (eg, pre-prints)