MixUp as Locally Linear Out-Of-Manifold Regularization

We note that this hypothesis only assumes a fold-2 local linearity and a global choice of $\Lambda$. If Hypothesis 2 holds and one is given $\Lambda$, then one can implementing the hypothesis as a constraint on ${\cal H}$, by imposing on the optimization problem the following constraint.

Standard MixUp Constraint: Every $H\in{\cal H}$ is $\Lambda$-linear w.r.t. every ${\bf X}\in{\cal D}^{(2)}$,

This gives rise a regularization scheme, which can be implemented by repeating the following process: draw a random pair data points¹¹ 1 Although we have required in the definition of ${\cal D}^{(2)}$ that two points in matrix ${\bf X}$ should have different labels, this needs not to be rigorously followed in implementation. Relaxing this condition in fact is expected to decrease the chance of manifold intrusion. Similar comments apply to the AdaMixUp later. in ${\cal D}$ to form ${\bf X}$, and draw a random $\lambda$ from $\Lambda$; take ${\bf X}\lambda$ as an additional training example and $G({\bf X})\lambda$ as its training target.

This scheme, generating more training examples, is essentially the standard MixUp scheme [\citeauthoryearZhang et al.2017].

In this paper, we however point out that one needs to caution with such a regularization scheme.

To induce a contradiction, suppose that Hypothesis 2 holds for $\Lambda$, namely, $G$ is $\Lambda$-linear with respect to every ${\bf X}\in{\cal M}^{(2)}$. Then ${\cal G}({\bf X}\lambda)={\cal G}({\bf X})\lambda$ for every $\lambda\in\Lambda^{\prime}\subseteq\Lambda$. Since the two columns (points) in ${\bf X}$ have different labels, the two columns of $G({\bf X})$ must be two different one-hot vectors (degenerate distributions). But every $\lambda\in\Lambda^{\prime}$ is non-degenerate; as a consequence, every $G({\bf X})\lambda$ must be non-degenerate. This contradicts the previous argument that ${\cal G}({\bf X}\lambda)$ is a single-point-mass. Therefore Hypothesis 2 fails on a region in $\Lambda$ with non-zero measure. $\Box$

Geometrically, the failure of Hypothesis 2 in this lemma is due to that the mixing $x_{1}$ and $x_{2}$ by $\lambda$ causes the mixed point ${\bf X}\lambda$ to “intrude” into ${\cal M}$ and “collide” with a point, say, $x^{\prime}$ in ${\cal M}$. We call such a phenomenon manifold intrusion. Note that in this case, ${\bf X}\lambda$ and $x^{\prime}$ are in fact the same point, but they have different labels. Obviously, when such intrusion occurs, regularization using ${\bf X}$ and $\lambda$ will contradict with the original training data. This essentially induces a form of under-fitting.

Figure 2 depicts the effect of MixUp on constraining the space ${\cal H}$, where the region ${\cal G}$ denotes the space of functions in ${\cal F}({\cal X},{\cal Y})$ satisfying (4).

When the space ${\cal H}$ is large, there is a large region of ${\cal H}$ satisfying (4). This gives a large intersection of ${\cal H}$ and ${\cal G}$, each member of which is a perfect classifier on the training set but performing questionably on the testing set. Thus the model without MixUp tends to result in large prediction variances or over-fitting (left figure).

When MixUp is applied and manifold intrusion does not occur (middle figure), the MixUp constraint effectively reduces the space ${\cal H}$. Since the MixUp constraint does not conflict (4) (at least explicitly), the reduced ${\cal H}$ would contain members compatible with (4). Thus the intersection of ${\cal G}$ and ${\cal H}$ reduces while remaining non-empty. Each member in the intersection is a perfect classifier on the training set and the mixed examples. But the significantly reduced intersection gives rise to significantly lower prediction variances on the testing set, thereby serving to regularize the model and reduce over-fitting.

When the application of MixUp results in manifold intrusion (right figure), there is no member in ${\cal H}$ that satisfies both the MixUp constraint and the condition (4). The intersection of ${\cal H}$ and ${\cal G}$ then becomes empty. In this case, no member in ${\cal H}$ can fit both the original training data and the (intruding) mixed examples. This gives rise to under-fitting and high prediction biases.

From the view point of restricting the space ${\cal H}$, one wishes to make $\Lambda$ as large as possible. But this lemma sets a limit on the possible $\Lambda$ that makes Hypothesis 2 hold. Due to this lemma, we need to assure that the $\Lambda$ we choose does not cause such an “intrusion”.

Let $\Lambda^{*}$ denote the largest policy region in ${\mathbb{S}}_{2}$ such that manifold intrusion does not occur. In the original MixUp, recall that inappropriate hyper-parameter result in no gain and only negative impact. We believe that much of this is due to manifold intrusion. On the other hand, the search of good hyper-parameter $\alpha$ in the original MixUp can be seen as searching for $\Lambda^{*}$ by trial and error.

In this paper, we generalize the standard MixUp to regularization with higher-fold local linearities, where $\Lambda^{*}$ (more precisely its generalization) is learned from the data.

If this hypothesis holds, and if for every $k\in\{2,3,\ldots,k_{\rm max}\}$ and every ${\bf X}\in{\cal M}^{(k)}$, $\Lambda({\bf X})$ in the hypothesis is given, then the hypothesis can be implemented as the following constraint on ${\cal H}$.

Generalized MixUp Constraint: For every $k\in\{2,3,\ldots,k_{\rm max}\}$, every ${\bf X}\in{\cal D}^{(k)}$ and every $\Lambda({\bf X})$, every $H\in{\cal H}$ is $\Lambda({\bf X})$-linear w.r.t. ${\bf X}$.

Imposing such a constraint in training provides another regularization scheme, which will be one ingredient of the proposed AdaMixUp scheme. We will present AdaMixUp in the next section, after developing its other ingredients.

This lemma parallels Lemma 1 and can be proved similarly. Lemma 2 similarly suggests that when the imposed constraint results in intrusion into the manifold ${\cal M}$, Hypothesis 3 necessarily fails. The lemma also sets a limit for each $\Lambda({\bf X})$. For each ${\bf X}$, we will denote the largest $\Lambda({\bf X})$ by $\Lambda^{*}({\bf X})$. The following result immediately follows.

In reality, one expects that $\Lambda^{*}$ is strictly contained in $\Lambda^{*}({\bf X})$. That is, even when we take $k_{\rm max}=2$, the Generalized MixUp Hypothesis imposes a stronger constraint than the Standard MixUp Hypothesis, and hence can provide a stronger regularization without causing under-fitting. When $k_{\rm max}>2$, the Generalized MixUp Hypothesis imposes additional constraints, which further regularizes the model and improves generalization.

Instead of constructing $k_{\rm max}-1$ networks $\pi_{k}$’s, we in fact implement these networks recursively using a single network $\pi_{2}$. For this purpose, we recursively parametrize $\Lambda({\bf X})\in{\mathbb{S}}_{k}$, which we will rewrite as $\Lambda_{k}({\bf X})$ for clarity.

For an ${\bf X}\in{\cal X}^{(k)}$, we may express it as $[{\bf X}^{(k-1)},x_{k}]$, where ${\bf X}^{(k-1)}$ is the sub-matrix of ${\bf X}$ containing the first $k-1$ columns. Then， for $k>2$, we parametrize

This then allows the use of single network $\pi_{2}$ to express $\Lambda_{k}({\bf X})$ for all ${\bf X}$ and all $k$.

Specifically, a Fold-2 mixing policy is parameterized by a single number $\gamma$. For each input ${\bf X}\in{\cal X}^{(2)}$, $\pi_{2}$ returns two positive values $\alpha$ and $\Delta$ with $\alpha+\Delta\leq 1$, which specifies $\Lambda({\bf X})$ as the interval $(\alpha,\alpha+\Delta)$ as the range for $\gamma$. To this end, the last layer of the $\pi_{2}$ network is implemented as a softmax function, which generates a triplet of values $(\alpha,\Delta,\alpha^{\prime})$ with sum of one, where the third element $\alpha^{\prime}$ is discarded. Using network $\pi_{2}$, we generate a set of Fold-2 mixed examples by repeatedly drawing a pair of samples in ${\cal D}$ and mixing them using a random policy drawn from policy region generated by $\pi_{2}$ on the pair of samples.

To generate mixed samples with an increased maximum fold number, we consider the original examples and previously generated mixed examples as the new training set ${\cal D}_{\rm new}$. We then repeat the following process: draw a pair of samples, one from ${\cal D}_{\rm new}$ and the other from ${\cal D}$; submit the pair to network $\pi_{2}$ to generate a policy region; draw a random policy from the region and obtain a new sample by mixing the pair of samples using the drawn policy.

To allow the gradient signal to back-propagate through the policy sampler, a reparametrization trick similar to that of [\citeauthoryearKingma and Welling2013] is used. Specifically, drawing $\gamma$ from $(\alpha,\alpha+\Delta)$ is implemented by drawing $\epsilon$ from the uniform distribution over $(0,1)$, then let $\gamma:=\Delta\cdot\epsilon+\alpha$.

The network $\varphi$ is a neural network that classifies original examples from the mixed examples. In our implementation, $\varphi$ shares with the classifier network all but the final layers. The final layer of $\varphi$ is a simple logistic regression binary classifier.

The network $\varphi$ and $\pi_{2}$ are trained jointly, where we iterate over minimizing ${\cal L}_{\cal D}+{\cal L}_{{\cal D}^{\prime}}$ and minimizing ${\cal L}_{\rm intr}$ in (6) via mini-batched SGD. A Fold-2 AdaMixUp is illustrated in Figure 3.

We evaluate AdaMixUp on eight benchmarks. MNIST is the popular digit (1-10) recognition dataset with 60,000 training and 10,000 test gray-level, 784-dimensional images. Fashion is an image recognition dataset having the same scale as MNIST, containing 10 classes of fashion product pictures. SVHN is the Google street view house numbers recognition data set with 73,257 digits (1-10) 32x32 color images for training, 26,032 for testing, and 531,131 additional, easier samples. We did not use the additional images. Cifar10 is an image classification dataset with 10 classes, 50,000 training and 10,000 test samples. Cifar100 is similar to CIFAR10 but with 100 classes and 600 images each. Cifar10-S and Cifar100-S are respectively Cifar10 and Cifar100 reduced to containing only 20% of the training samples. ImageNet-R is the ImageNet-2012 classification dataset [\citeauthoryearRussakovsky et al.2014] with 1.3 million training images, 50,000 validation images, and 1,000 classes. We follow the data processing approaches used in Mixup [\citeauthoryearZhang et al.2017], except that the crop size is 100x100 instead of 224x224 due to our limited computational resources. We report both top-1 and top-5 error rates.

In our experiments, the Intrusion Discriminator, Classifier, and Policy Region Generator have the same network architecture, and the first two share the same set of parameters. We test AdaMixUp on two types of baseline networks: a three layers CNN as implemented in [\citeauthoryearWu and others2016] as the baseline for easier tasks MNIST and Fashion, and a ResNet-18 as implemented in [\citeauthoryearZagoruyko and Komodakis2016] for the other six more difficult tasks. All models examined are trained using mini-batched backprop, as specified in [\citeauthoryearWu and others2016] and [\citeauthoryearZagoruyko and Komodakis2016], for 400 epochs. Each reported performance value (accuracy or error rate) is the median of the performance values obtained in the final 10 epochs.

Our first experiment compares Fold-2 AdaMixUp (i.e., mixing image pairs) to two networks: the baseline networks (i.e., 3-layer CNN for MNIST and Fashion and ResNet-18 for the rest datasets) and MixUp on the baseline networks. We present the error rates obtained by the Baseline, MixUp, and AdaMixUp, along with the relative error reduction of AdaMixUp over the Baseline, in Table 1.

Table 1 indicates that the AdaMixUp outperforms both the Baseline and MixUp on all the eight testing datasets. Also, the relative error reduction of AdaMixUp over the Baseline is at least 5.7%, and with a large margin in some cases. For example, for the SVHN and Cifar10 datasets, the relative error reduction achieved by AdaMixUp are over 30%.

Table 1 also suggests that, with the default $\alpha$ value, MixUp failed to improve the baseline systems’ accuracy on four out of the eight datasets, namely the MNIST, Cifar10-S, Cifar100-S, and Imagenet-R datasets, as underlined in Table 1, suggesting over-regularization or under-fitting.

In Table 2, we also present the $\alpha$ and $\Delta$ values learned by the Policy Region Generator, along with the losses of both the Intrusion Discriminator and the Classifier of the AdaMixUp method. Results in Table 2 indicate that the values of $\alpha$ and $\Delta$ vary for different datasets; the former ranges from 0.4 to 0.9 but the range for $\Delta$ is much smaller, with values between 0.010 and 0.035. Noting that the intrusion losses are fairly close to 0 on all the datasets, suggesting that these datasets well support Fold-2 AdaMixUp under the proposed structure of Policy Region Generator.

Figure 4 depicts the behavior of Policy Region Generator in Fold-2 AdaMixUp over training iterations. It appears that the Policy Region Generator initially explores a wide range of policy space before settling down at around 40K iterations (when the means of $\alpha$ and $\Delta$ both stabilize, at 0.48 and 0.03 respectively).

Figure 5 shows the average of the drawn mixing policy $\gamma$ in Fold-2 AdaMixUp during training iterations and the corresponding classification performance of the model. The range of $\gamma$ appears to stabilize at around 40K iterations (left plot), consistent with the observation in Figure 4. Interestingly, also at the same time, the classification error drops and begins to stabilize (right plot).

Figures 6 shows some typical mixed images in Fold-2 AdaMixUp and their corresponding original images in MNIST. We note that these mixed images obviously distinguish themselves from the original images. Thus they do not intrude into the data manifold.

Table 3 presents the error rates obtained by Fold-3 and Fold-4 AdaMixUp on the seven testing datasets (excluding ImageNet due to limited computational resources). We see that increasing the mixing fold from 2 to higher values, the performance of AdaMixUp improves. This is due to stronger constraints are imposed to the model and hence results in stronger regularization. One however expects that imposing stronger constraints makes the model more difficult to fit and further increasing mixing fold may eventually lead to diminishing gain and even under-fitting. This is hinted by Table 4, where the averages of learned mixing parameters and the training losses of the Fold-3 AdaMixUp are presented.

In Table 4, we see that the intrusion losses of Fold-3 AdaMixUp are much higher than those of Fold-2 AdaMixUp (Table 2). This implies that in higher-fold AdaMixUp, it is more difficult for the Policy Region Generator to carve a good policy region that is not regarded by the Intrusion Discriminator as causing intrusion. This difficulty is also suggested by the high $\alpha_{2}$ values in Table 4 (often close to 1). They indicate that in these cases, mixing only occurs “slightly”.

We vary the number of filters on each layer of the ResNet-18 with half quarter, quarter, half, and three-quarter of the original number of filters (denoted as base filter), and present the test error rates on Cifar100 in Figure 7 (green curves).

The green curves in Figure 7 indicate that the Fold-2 AdaMixUp benefits from large number of filters deployed: the accuracy keeps improving while enlarging the number of filters. On the contrary, the ResNet-18 received dramatically accuracy improvement before the number of filters is half of the base but obtained no further improvement after.

We down sample the Cifar100 data with 20%, 40%, 60% and 80% of the training samples, and present the test error rates obtained by Fold-2 AdaMixUp in Figure 7 (red curves).

The red curves in Figure 7 indicate that more training samples help with both the ResNet-18 and AdaMixUp, but the accuracy gap between the two methods are widening while increasing the training sample size.

Table 5 lists the test error rates obtained by the Fold-2 AdaMixUp with ResNet-18 on the Cifar10 and Cifar100 data, but without the Intrusion Discriminator.

Table 5 shows that excluding the Intrusion Discriminator component, the AdaMixUp method was still able to improve the accuracy over the baseline, but obtained much lower accuracy than that of including the Intrusion Discriminator unit.

We also evaluate an alternative structure for AdaMixUp, where mixing happens on the layer before the final softmax layer in ResNet-18. Our results suggest that the test errors of the Fold-2 AdaMixUp increased dramatically due to the high cost of the Intrusion Discriminator. The error rates increased from 20.97% to 22.21% and from 3.52% to 4.94%, respectively for Cifar100 and Cifar10. Notably, both have large intrusion loss of around 0.49, which may due to the fact that perceptual similarity for images in the hidden embedding space may not as distinguishable as that in the input space. As a result, the Policy Generator failed to find good mixing policies to avoid colliding into the data manifold.