Are adversarial examples inevitable?

A number of adversarial attacks on neural networks have been recently proposed. To counter these attacks, a number of authors have proposed a range of defenses. However, these defenses are often quickly broken by new and revised attacks. Given the lack of success at generating robust defenses, we are led to ask a fundamental question: Are adversarial attacks inevitable?

In this paper, we identify a broad class of problems for which adversarial examples cannot be avoided. We also derive fundamental limits on the susceptibility of a classifier to adversarial attacks that depend on properties of the data distribution as well as the dimensionality of the dataset.

Adversarial examples occur when a small perturbation to an image changes its class label. There are different ways of measuring what it means for a perturbation to be “small”; as such, our analysis considers a range of different norms. While the $\ell_{\infty}$-norm is commonly used, adversarial examples can be crafted in any $\ell_{p}$-norm (see Figure 1). We will see that the choice of norm can have a dramatic effect on the strength of theoretical guarantees for the existence of adversarial examples. Our analysis also extends to the $\ell_{0}$-norm, which yields “sparse” adversarial examples that only perturb a small subset of image pixels (Figure 2).

As a simple example result, consider a classification problem with $n$-dimensional images with pixels scaled between 0 and 1 (in this case images live inside the unit hypercube). If the image classes each occupy a fraction of the cube greater than $\frac{1}{2}\exp(-\pi\epsilon^{2})$, then images exist that are susceptible to adversarial perturbations of $\ell_{2}$-norm at most $\epsilon$. Note that $\epsilon=10$ was used in Figure 1, and larger values are typical for larger images.

Finally, in Section 8, we explore the causes of adversarial susceptibility in real datasets, and the effect of dimensionality. We present an example image class for which there is no fundamental link between dimensionality and robustness, and argue that the data distribution, and not dimensionality, is the primary cause of adversarial susceptibility.

We consider the problem of classifying data points that lie in a space $\Omega$ (either a sphere or a hypercube) into $m$ different object classes. The $m$ object classes are defined by probability density functions $\{\rho_{c}\}_{c=1}^{m},$ where $\rho_{c}:\Omega\to\mathbb{R}$. A “random” point from class $c$ is a random variable with density $\rho_{c}.$ We assume $\rho_{c}$ to be bounded (i.e., we don’t allow delta functions or other generalized functions), and denote its upper bound by $U_{c}=\sup_{x}\rho_{c}(x).$

We also consider a “classifier” function $\mathcal{C}:\Omega\to\{1,2,\ldots,m\}$ that partitions $\Omega$ into disjoint measurable subsets, one for each class label. The classifier we consider is discrete valued – it provides a label for each data point but not a confidence level.

With this setup, we can give a formal definition of an adversarial example.

In plain words, a point has an $\epsilon$-adversarial example if we can sneak it into a different class by moving it at most $\epsilon$ units in the distance $d$.

We consider adversarial examples with respect to different $\ell_{p}$-norm metrics. These metrics are written $d_{p}(x,\hat{x})=\|x-\hat{x}\|_{p}.$ A common choice is $p=\infty,$ which limits the absolute change that can be made to any one pixel. However, $\ell_{2}$-norm and $\ell_{1}$-norm adversarial examples are also used, as it is frequently easier to create adversarial examples in these less restrictive metrics.

We also consider sparse adversarial examples in which only a small subset of pixels are manipulated. This corresponds to the metric $d_{0},$ in which case the constraint $\|x-\hat{x}\|_{0}\leq\epsilon$ means that an adversarial example was crafted by changing at most $\epsilon$ pixels, and leaving the others alone.

We begin by looking at the case of classifiers for data on the sphere. While this data model may be less relevant than the other models studied below, it provides a straightforward case where results can be proven using simple, geometric lemmas. The more realistic case of images with pixels in $[0,1]$ will be studied in Section 4.

The idea is to show that, provided a class of data points takes up enough space, nearly every point in the class lies close to the class boundary. To show this, we begin with a simple definition.

Our result provides bounds on the probability of adversarial examples that are independent of the shape of the class boundary. This independence is a simple consequence of an isoperimetric inequality. The classical isoperimetric inequality states that, of all closed surfaces that enclose a unit volume, the sphere has the smallest surface area. This simple fact is intuitive but famously difficult to prove. For a historical review of the isoperimetric inequality and its variants, see Osserman et al. 1978. We will use a special variant of the isoperimetric inequality first proved by Lévy & Pellegrino 1951 and simplified by Talagrand 1995.

The classical isoperimetric inequality is a simple geometric statement, and frequently appears without absolute bounds on the size of the $\epsilon$-expansion of a half-sphere, or with bounds that involve unspecified constants (Vershynin 2017). A tight bound derived by Milman & Schechtman 1986 is given below. The asymptotic blow-up of the $\epsilon$-expansion of a half sphere predicted by this bound is shown in Figure 3.

Lemmas 1 and 2 together can be taken to mean that, if a set is not too small, then in high dimensions almost all points on the sphere are reachable within a short $\epsilon$ jump from that set. These lemmas have immediate implications for adversarial examples, which are formed by mapping one class into another using small perturbations. Despite its complex appearance, the result below is a consequence of the (relatively simple) isoperimetric inequality.

In the above result, we measure the size of adversarial perturbations using the geodesic distance. Most studies of adversarial examples measure the size of perturbation in either the $\ell_{2}$ (Euclidean) norm or the $\ell_{\infty}$ (max) norm, and so it is natural to wonder whether Theorem 1 depends strongly on the distance metric. Fortunately (or, rather unfortunately) it does not.

It is easily observed that, for any two points $x$ and $y$ on a sphere,

where $d_{\infty}(x,y)$, $d_{2}(x,y)$, and $d_{g}(x,y)$ denote the $l_{\infty}$, Euclidean, and geodesic distance, respectively. From this, we see that Theorem 1 is actually fairly conservative; any $\epsilon$-adversarial example in the geodesic metric would also be adversarial in the other two metrics, and the bound in Theorem 1 holds regardless of which of the three metrics we choose (although different values of $\epsilon$ will be appropriate depending on the norm).

The above result about the sphere is simple and easy to prove using classical results. However, real world images do not lie on the sphere. In a more typical situation, images will be scaled so that their pixels lie in $[0,1]$, and data lies inside a high-dimensional hypercube (but, unlike the sphere, data is not confined to its surface). The proof of Theorem 1 makes extensive use of properties that are exclusive to the sphere, and is not applicable to this more realistic setting. Are there still problem classes on the cube where adversarial examples are inevitable?

This question is complicated by the fact that geometric isoperimetric inequalities do not exist for the cube, as the shapes that achieve minimal $\epsilon$-expansion (if they exist) depend on the volume they enclose and the choice of $\epsilon$ (Ros 2001). Fortunately, researchers have been able to derive “algebraic” isoperimetric inequalities that provide lower bounds on the size of the $\epsilon$-expansion of sets without identifying the shape that achieves this minimum (Talagrand 1996; Milman & Schechtman 1986). The result below about the unit cube is analogous to Proposition 2.8 in Ledoux 2001, except with tighter constants. For completeness, a proof (which utilizes methods from Ledoux) is provided in Appendix A.

Using this result, we can show that most data samples in a cube admit adversarial examples, provided the data distribution is not excessively concentrated.

When adversarial examples are defined in the $\ell_{2}$-norm (or for any $p\geq 2$), the bound in equation 4 becomes

Provided the class distribution is not overly concentrated, equation 5 guarantees adversarial examples with relatively “small” $\epsilon$ relative to a typical vector. In $n$ dimensions, the $\ell_{2}$ diameter of the cube is $\sqrt{n},$ and so it is reasonable to choose $\epsilon=O(\sqrt{n})$ in equation 5. In Figure 1, we chose $\epsilon=10.$ A similarly strong bound of $1-U_{c}\sqrt{n}\exp(-2\pi\epsilon^{2}/n)/(2\pi\epsilon)$ holds for the case of the $\ell_{1}$-norm, in which case the diameter is $n.$

Oddly, equation 4 seems particularly weak when the $\ell_{\infty}$ norm is used. In this case, the bound on the right side of equation 4 becomes equation 5 just like in the $\ell_{2}$ case. However, $\ell_{\infty}$ adversarial examples are only interesting if we take $\epsilon<1$, in which case equation 5 becomes vacuous for large $n$. This bound can be tightened up in certain situations. If we prove Theorem 2 using the tighter (but messier) bound of equation 2 instead of equation 3, we can replace equation 4 with

for $p\geq 2,$ where $\hat{\Phi}(z)=\frac{1}{\sqrt{2\pi}}\int_{z}^{\infty}e^{-t^{2}/2}dt\geq\frac{1}{\sqrt{2\pi}z}e^{-z^{2}/2}$ (for $z>0$), and $\alpha=\Phi^{-1}(1-f_{c})$. For this bound to be meaningful with $\epsilon<1$, we need $f_{c}$ to be relatively small, and $\epsilon$ to be roughly $f_{c}$ or smaller. This is realistic for some problems; ImageNet has 1000 classes, and so $f_{c}<10^{-3}$ for at least one class.

Interestingly, under $\ell_{\infty}$-norm attacks, guarantees of adversarial examples are much stronger on the sphere (Section 3) than on the cube. One might wonder whether the weakness of Theorem 4 in the $\ell_{\infty}$ case is fundamental, or if this is a failure of our approach. One can construct examples of sets with $\ell_{\infty}$ expansions that nearly match the behavior of equation 5, and so our theorems in this case are actually quite tight. It seems to be inherently more difficult to prove the existence of adversarial examples in the cube using the $\ell_{\infty}$-norm.

A number of papers have looked at sparse adversarial examples, in which a small number of image pixels, in some cases only one (Su et al. 2017), are changed to manipulate the class label. To study this case, we would like to investigate adversarial examples under the $\ell_{0}$ metric. The $\ell_{0}$ distance is defined as

If a point $x$ has an $\epsilon$-adversarial example in this norm, then it can be perturbed into a different class by modifying at most $\epsilon$ pixels (in this case $\epsilon$ is taken to be a positive integer).

Theorem 2 is fairly tight for $p=1$ or $2$. However, the bound becomes quite loose for small $p,$ and in particular it fails completely for the important case of $p=0.$ For this reason, we present a different bound that is considerably tighter for small $p$ (although slightly looser for large $p$).

The case $p=0$ was studied by Milman & Schechtman 1986 (Section 6.2) and McDiarmid 1989, and later by Talagrand 1995; Talagrand 1996. The proof of the following theorem (appendix B) follows the method used in Section 5 of Talagrand 1996, with modifications made to extend the proof to arbitrary $p$.

Using this result, we can prove a statement analogous to Theorem 2, but for sparse adversarial examples. We present only the case of $p=0,$ but the generalization to the case of other small $p$ using Lemma 4 is straightforward.

Tighter bounds can be obtained if we only guarantee that adversarial examples exist for some data points in a class, without bounding the probability of this event.

It is interesting to consider when Theorem 4 produces non-vacuous bounds. When the $\ell_{2}$-norm is used, the bound becomes $\vol[\supp(\rho_{c})]\geq\exp(-\pi\epsilon^{2})/2.$ The diameter of the cube is $\sqrt{n},$ and so the bound becomes active for $\epsilon=\sqrt{n}.$ Plugging this in, we see that the bound is active whenever the size of the support satisfies $\vol[\supp(\rho_{c})]>\frac{1}{2e^{\pi n}}.$ Remarkably, this holds for large $n$ whenever the support of class $c$ is larger than (or contains) a hypercube of side length at least $e^{-\pi}\approx 0.043.$ It is important note, however, that the bound being “active” does not mean there are adversarial examples with a “small” $\epsilon.$

There are a number of ways to escape the guarantees of adversarial examples made by Theorems 1-4. One potential escape is for the class density functions to take on extremely large values (i.e., exponentially large $U_{c}$); the dependence of $U_{c}$ on $n$ is addressed separately in Section 8.

In this section, we discuss the relationship between dimensionality and adversarial robustness, and explore how the predictions made by the theorems above are reflected in experiments.

It is commonly thought that high-dimensional classifiers are more susceptible to adversarial examples than low-dimensional classifiers. This perception is partially motivated by the observation that classifiers on high-resolution image distributions like ImageNet are more easily fooled than low resolution classifiers on MNIST (Tramèr et al. 2017a). Indeed, Theorem 2 predicts that high-dimensional classifiers should be much easier to fool than low-dimensional classifiers, assuming the datasets they classify have comparable probability density limits $U_{c}.$ However, this is not a reasonable assumption; we will see below that high dimensional distributions may be more concentrated than their low-dimensional counterparts.

We study the effects of dimensionality with a thought experiment involving a “big MNIST” image distribution. Given an integer expansion factor $b,$ we can make a big MNIST distribution, denoted $b$-MNIST, by replacing each pixel in an MNIST image with a $b\times b$ array of identical pixels. This expands an original $28\times 28$ image into a $28b\times 28b$ image. Figure 4a shows that, without adversarial training, a classifier on big MNIST is far more susceptible to attacks than a classifier trained on the original MNIST¹¹ 1 Only the fully-connected layer is modified to handle the difference in dimensionality between datasets..

However, each curve in Figure 4a only shows the attack susceptibility of one particular classifier. In contrast, Theorems 1-4 describe the fundamental limits of susceptibility for all classifiers. These limits are an inherent property of the data distribution. The theorem below shows that these fundamental limits do not depend in a non-trivial way on the dimensionality of the images in big MNIST, and so the relationship between dimensionality and susceptibility in Figure 4a results from the weakness of the training process.

Theorem 5 predicts that the perturbation needed to fool all $56\times 56$ classifiers is twice that needed to fool all $28\times 28$ classifiers. This is reasonable since the $\ell_{2}$-norm of a $56\times 56$ image is twice that of its $28\times 28$ counterpart. Put simply, fooling big MNIST is just as hard/easy as fooling the original MNIST regardless of resolution. This also shows that for big MNIST, as the expansion factor $b$ gets larger and $\epsilon$ is expanded to match, the concentration bound $U_{c}$ grows at exactly the same rate as the exponential term in equation 2 shrinks, and there is no net effect on fundamental susceptibility. Also note that an analogous result could be based on any image classification problem (we chose MNIST only for illustration), and any $p\geq 0.$

We get a better picture of the fundamental limits of MNIST by considering classifiers that are hardened by adversarial training²² 2 Adversarial examples for MNIST/CIFAR-10 were produced as in Madry et al. 2017 using 100-step/20-step PGD. (Figure 4b). These curves display several properties of fundamental limits predicted by our theorems. As predicted by Theorem 5, the $112\times 112$ classifer curve is twice as wide as the $56\times 56$ curve, which in turn is twice as wide as the $28\times 28$ curve. In addition, we see the kind of “phase transition” behavior predicted by Theorem 2, in which the classifier suddenly changes from being highly robust to being highly susceptible as $\epsilon$ passes a critical threshold. For these reasons, it is reasonable to suspect that the adversarially trained classifiers in Figure 4b are operating near the fundamental limit predicted by Theorem 2.

Theorem 5 shows that increased dimensionality does not increase adversarial susceptibility in a fundamental way. But then why are high-dimensional classifiers so easy to fool? To answer this question, we look at the concentration bound $U_{c}$ for object classes. The smallest possible value of $U_{c}$ is 1, which only occurs when images are “spread out” with uniform, uncorrelated pixels. In contrast, adjacent pixels in MNIST (and especially big MNIST) are very highly correlated, and images are concentrated near simple, low-dimensional manifolds, resulting in highly concentrated image classes with large $U_{c}$. Theory predicts that such highly concentrated datasets can be relatively safe from adversarial examples.

We can reduce $U_{c}$ and dramatically increase susceptibility by choosing a more “spread out” dataset, like CIFAR-10, in which adjacent pixels are less strongly correlated and images appear to concentrate near complex, higher-dimensional manifolds. We observe the effect of decreasing $U_{c}$ by plotting the susceptibility of a $56\times 56$ MNIST classifier against a classifier for CIFAR-10 (Figure 4, right). The former problem lives in $3136$ dimensions, while the latter lives in $3072,$ and both have 10 classes. Despite the structural similarities between these problems, the decreased concentration of CIFAR-10 results in vastly more susceptibility to attacks, regardless of whether adversarial training is used. The theory above suggests that this increased susceptibility is caused at least in part by a shift in the fundamental limits for CIFAR-10, rather than the weakness of the particular classifiers we chose.

Informally, the concentration limit $U_{c}$ can be interpreted as a measure of image complexity. Image classes with smaller $U_{c}$ are likely concentrated near high-dimensional complex manifolds, have more intra-class variation, and thus more apparent complexity. An informal interpretation of Theorem 2 is that “high complexity” image classes are fundamentally more susceptible to adversarial examples, and Figure 4 suggests that complexity (rather than dimensionality) is largely responsible for differences we observe in the effectiveness of adversarial training for different datasets.

The question of whether adversarial examples are inevitable is an ill-posed one. Clearly, any classification problem has a fundamental limit on robustness to adversarial attacks that cannot be escaped by any classifier. However, we have seen that these limits depend not only on fundamental properties of the dataset, but also on the strength of the adversary and the metric used to measure perturbations. This paper provides a characterization of these limits and how they depend on properties of the data distribution. Unfortunately, it is impossible to know the exact properties of real-world image distributions or the resulting fundamental limits of adversarial training for specific datasets. However, the analysis and experiments in this paper suggest that, especially for complex image classes in high-dimensional spaces, these limits may be far worse than our intuition tells us.

T. Goldstein and his students were generously supported by DARPA Lifelong Learning Machines (FA8650-18-2-7833), the Office of Naval Research (N00014-17-1-2078), the DARPA Young Faculty Award program (D18AP00055), and the Sloan Foundation. The work of C. Studer was supported in part by Xilinx Inc., and by the US NSF under grants ECCS-1408006, CCF-1535897, CCF-1652065, and CNS-1717559.