Mixup and CutMix

Mixup and CutMix are data augmentation methods that create new training examples by combining two examples and their labels. They regularize the model by discouraging overly sharp decision boundaries.

Both methods are usually used for classification. They replace hard one-example training with interpolated training examples.

Motivation

A classifier trained only on ordinary examples may learn decision boundaries that are too sharp. It may assign extreme confidence to regions of input space that are poorly supported by data.

Mixup and CutMix reduce this problem. They train the model on synthetic examples that lie between training samples.

For two examples $(x_i, y_i)$ and $(x_j, y_j)$, both methods construct a mixed input $\tilde{x}$ and a mixed target $\tilde{y}$. The target is no longer a single class. It becomes a soft label.

The general form is

where $\lambda\in[0,1]$ controls how much each example contributes.

Mixup

Mixup combines two inputs by linear interpolation:

The labels are combined with the same coefficient:

If $x_i$ is an image of a cat, $x_j$ is an image of a dog, and $\lambda=0.7$, the mixed image is 70 percent cat image and 30 percent dog image. The target is also 70 percent cat and 30 percent dog.

For one-hot labels, this produces a soft target distribution.

Sampling the Mixing Coefficient

The mixing coefficient $\lambda$ is commonly sampled from a Beta distribution:

The parameter $\alpha$ controls the strength of mixing.

Small values such as $0.2$ and $0.4$ are common starting points.

Mixup Loss

For classification, the loss is computed against both labels:

In PyTorch:

import torch
import torch.nn.functional as F

def mixup_batch(x, y, alpha=0.2):
    batch_size = x.size(0)

    dist = torch.distributions.Beta(alpha, alpha)
    lam = dist.sample().to(x.device)

    index = torch.randperm(batch_size, device=x.device)

    mixed_x = lam * x + (1.0 - lam) * x[index]
    y_a = y
    y_b = y[index]

    return mixed_x, y_a, y_b, lam

def mixup_cross_entropy(logits, y_a, y_b, lam):
    loss_a = F.cross_entropy(logits, y_a)
    loss_b = F.cross_entropy(logits, y_b)
    return lam * loss_a + (1.0 - lam) * loss_b

Used in a training step:

model.train()

for x, y in train_loader:
    x = x.to(device)
    y = y.to(device)

    x, y_a, y_b, lam = mixup_batch(x, y, alpha=0.2)

    optimizer.zero_grad()

    logits = model(x)
    loss = mixup_cross_entropy(logits, y_a, y_b, lam)

    loss.backward()
    optimizer.step()

Why Mixup Regularizes

Mixup imposes a smoothness assumption. If two inputs are mixed, the model should produce a corresponding mixture of their labels.

This discourages sharp changes between training examples. It encourages the model to behave more linearly in regions between samples.

In classification, this often improves:

Mixup can be viewed as a form of vicinal risk minimization. The model trains not only on the empirical examples, but also on a neighborhood around them.

Limitations of Mixup

Mixup can produce unrealistic inputs. A linear blend of two images may look like a transparent overlay, which rarely occurs in real data.

For many image classification tasks, this still works well because the method regularizes the decision function. But for tasks where pixel-level realism matters, such as segmentation or detection, naive mixup may need modification.

Mixup may also hurt when examples should not be linearly interpolated. Some structured inputs do not have meaningful averages.

Examples:

For text models, mixup is usually applied in embedding space rather than token ID space.

CutMix

CutMix combines images by cutting a patch from one image and pasting it into another.

Let $M$ be a binary mask with the same spatial dimensions as the image. Then

The mixed label is

Here $\lambda$ is the fraction of the image area taken from $x_i$. If 80 percent of the image comes from $x_i$ and 20 percent comes from $x_j$, the label uses the same proportions.

CutMix Implementation

A simple CutMix implementation for image tensors of shape [B, C, H, W]:

import torch
import torch.nn.functional as F

def rand_bbox(height, width, lam):
    cut_ratio = torch.sqrt(1.0 - lam)
    cut_h = int(height * cut_ratio)
    cut_w = int(width * cut_ratio)

    cy = torch.randint(0, height, size=(1,)).item()
    cx = torch.randint(0, width, size=(1,)).item()

    y1 = max(cy - cut_h // 2, 0)
    y2 = min(cy + cut_h // 2, height)
    x1 = max(cx - cut_w // 2, 0)
    x2 = min(cx + cut_w // 2, width)

    return y1, y2, x1, x2

def cutmix_batch(x, y, alpha=1.0):
    batch_size, channels, height, width = x.shape

    dist = torch.distributions.Beta(alpha, alpha)
    lam = dist.sample().to(x.device)

    index = torch.randperm(batch_size, device=x.device)

    y_a = y
    y_b = y[index]

    y1, y2, x1, x2 = rand_bbox(height, width, lam)

    mixed_x = x.clone()
    mixed_x[:, :, y1:y2, x1:x2] = x[index, :, y1:y2, x1:x2]

    patch_area = (y2 - y1) * (x2 - x1)
    image_area = height * width
    lam_adjusted = 1.0 - patch_area / image_area

    return mixed_x, y_a, y_b, lam_adjusted

def cutmix_cross_entropy(logits, y_a, y_b, lam):
    loss_a = F.cross_entropy(logits, y_a)
    loss_b = F.cross_entropy(logits, y_b)
    return lam * loss_a + (1.0 - lam) * loss_b

The adjusted $\lambda$ is computed from the actual patch area, because clipping at image boundaries may change the patch size.

Why CutMix Works

CutMix preserves local image structure better than mixup. Instead of blending two entire images, it inserts a real patch from another image.

This has several effects.

The model must recognize objects from partial evidence. It cannot rely on a single discriminative region. It also learns that multiple objects or object parts may appear in the same input.

CutMix often improves localization behavior, even when trained only for image classification. Because the label depends on patch area, the model is encouraged to connect spatial evidence with class probability.

Mixup Versus CutMix

Mixup is more general because it can be applied to vectors and embeddings. CutMix is especially natural for images.

Combining with Label Smoothing

Mixup and CutMix already create soft labels. Label smoothing also creates soft labels. Combining them can work, but the total amount of target smoothing may become too strong.

If using Mixup or CutMix, use smaller label smoothing or disable it initially. Then tune both together.

A heavily regularized image classifier might use:

The exact values should be selected by validation performance.

Scheduling Mixup and CutMix

Some training recipes apply Mixup or CutMix throughout all training. Others disable them near the end.

Disabling strong augmentation during the final epochs may let the model adapt to the true data distribution. This is sometimes called augmentation decay or mixup off.

A simple schedule:

def mixup_alpha_for_epoch(epoch, total_epochs):
    if epoch > 0.8 * total_epochs:
        return 0.0
    return 0.2

When alpha becomes zero, ordinary training resumes.

Mixup for Embeddings

For text and some structured data, raw input mixing may be meaningless. But hidden representations or embeddings can often be mixed.

Suppose token embeddings produce a representation $h$. We can mix two hidden representations:

The rest of the model receives $\tilde{h}$, and the loss uses mixed labels.

This approach is sometimes called manifold mixup when applied inside hidden layers.

Manifold Mixup

Manifold mixup generalizes mixup by applying interpolation to hidden representations instead of only raw inputs.

If $h_l(x)$ is the representation at layer $l$, manifold mixup constructs

The model then continues forward from that layer.

This encourages smoother internal representations. It can make class representations more compact and separated.

Implementation requires model code that can expose intermediate layers or split the forward pass.

Practical Guidelines

For image classification, use Mixup and CutMix after building a baseline with ordinary augmentation. Start with modest values such as mixup_alpha=0.2 and cutmix_alpha=1.0.

For small datasets, these methods may help but can also underfit if too strong. For large datasets, they are often part of high-performing training recipes.

For text, do not mix integer token IDs. Use embedding-space or hidden-state mixing if using mixup-style regularization.

For medical, scientific, or safety-critical datasets, inspect mixed examples and validate assumptions carefully. Artificial mixtures may violate domain semantics.

Summary

Mixup and CutMix are regularization methods that combine examples and labels.

Mixup linearly interpolates two inputs. CutMix replaces an image patch with a patch from another image. Both train the model with soft targets and reduce overconfident, overly sharp decision boundaries.

In PyTorch, they can be implemented by shuffling a batch, mixing inputs, and computing a weighted loss against two target labels.