The Limitations of Deep Learning in Adversarial Settings

This work studies how deep neural networks can be attacked, showing that imperfections in the training phase leave them vulnerable to adversarial samples, inputs deliberately crafted to be misclassified. The authors formalize the space of adversaries against DNNs and introduce a novel class of algorithms that construct adversarial examples based on a precise understanding of the mapping between a network's inputs and outputs.

Applied to computer vision, the attack reliably generates samples that human subjects still classify correctly but that the network misclassifies into attacker-specified targets, achieving a 97% adversarial success rate while modifying on average only about 4.02% of the input features per sample. The authors further define a hardness measure to compare how vulnerable different sample classes are and sketch preliminary defenses based on a predictive distance between a benign input and a target classification, highlighting concrete security limitations of deep learning.