Rademacher Complexity for Adversarially Robust Generalization
release_m6lujazdo5dbpezrw63votp6yi
by
Dong Yin and Kannan Ramchandran and Peter Bartlett
2018
Abstract
Many machine learning models are vulnerable to adversarial attacks; for
example, adding adversarial perturbations that are imperceptible to humans can
often make machine learning models produce wrong predictions with high
confidence. Moreover, although we may obtain robust models on the training
dataset via adversarial training, in some problems the learned models cannot
generalize well to the test data. In this paper, we focus on ℓ_∞
attacks, and study the adversarially robust generalization problem through the
lens of Rademacher complexity. For binary linear classifiers, we prove tight
bounds for the adversarial Rademacher complexity, and show that the adversarial
Rademacher complexity is never smaller than its natural counterpart, and it has
an unavoidable dimension dependence, unless the weight vector has bounded
ℓ_1 norm. The results also extend to multi-class linear classifiers. For
(nonlinear) neural networks, we show that the dimension dependence in the
adversarial Rademacher complexity also exists. We further consider a surrogate
adversarial loss for one-hidden layer ReLU network and prove margin bounds for
this setting. Our results indicate that having ℓ_1 norm constraints on the
weight matrices might be a potential way to improve generalization in the
adversarial setting. We demonstrate experimental results that validate our
theoretical findings.
In text/plain
format
Archived Files and Locations
application/pdf 310.9 kB
file_g7npex5zlbhidmxz5c2qzpzdwi
|
arxiv.org (repository) web.archive.org (webarchive) |
1810.11914v1
access all versions, variants, and formats of this works (eg, pre-prints)