Practical Black-Box Attacks against Machine Learning

This paper introduces the first practical black-box attack against machine learning models such as deep neural networks, which are vulnerable to adversarial examples, malicious inputs modified to produce erroneous outputs while looking unmodified to humans. Unlike all prior adversarial attacks, which require knowledge of the target's internals or its training data, the only capability assumed here is observing the labels the target assigns to inputs the attacker chooses. The strategy trains a local substitute model to imitate the target, using inputs synthetically generated by the adversary and labeled by querying the target, then uses that substitute to craft adversarial examples.

In a properly blinded, real-world evaluation, adversarial examples crafted against the substitute successfully transferred to the target models. A DNN hosted by MetaMind's online API misclassified 84.24% of the crafted examples, while logistic-regression substitutes against Amazon and Google models yielded misclassification rates of 96.19% and 88.94%, and the approach also evaded defenses previously shown to make adversarial crafting harder. By showing that mere query access suffices to attack deployed models, the work exposed a serious and general security risk for machine learning systems.