On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels release_qdr5a5hetbhyba27mpqjbsdnwe

by Amnon Geifman, Meirav Galun, David Jacobs, Ronen Basri

Released as a article .

2022  

Abstract

We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our results provide concrete quantitative characterization of over-parameterized convolutional network architectures.
In text/plain format

Archived Files and Locations

application/pdf  954.2 kB
file_nsfzy3xt5zdajn7motdp5rn6fa
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2022-03-17
Version   v1
Language   en ?
arXiv  2203.09255v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 8b48e350-0556-4fd8-8791-2de0cb074918
API URL: JSON