acceptodds
Under review as a conference paper at ICLR 2027

Beyond Input Interpolation: Isolating Mixup's First-Order Regularization Kernel

Abstract

Mixup couples input interpolation with stochastic-gradient regularization. We isolate a first-order regularization kernel through deliberate purification: retaining a specified gradient structure while removing the input-dependent interpolation residual. For target-affine prediction gradients, target interpolation is exact at source inputs; the removed response captures input nonlinearity at the common mixed target. Sombrero Loss realizes this kernel exactly on original inputs with paired targets, then extends to independent targets. For targets of uniform mean, its expected gradient matches label smoothing at an analytically determined strength, while intrinsic noise has an explicit, data-dependent covariance. Under a prediction model, the average conditional per-example covariance and model-averaged generalized Gauss–Newton matrix share two components, specifying curvature-matching conditions and a local mechanism that can favor flatter regions. Mean-matched ablations and spectral measurements support regularization beyond average smoothing. CIFAR and ImageNet experiments show competitive generalization without input interpolation and improved stability over tested mixing strengths. Detection, translation, and class-imbalanced classification demonstrate the construction's applicability and target-prior flexibility.

Primary area:
optimization
Keywords:
Mixup, regularization, gradient dynamics, stochastic optimization, structured gradient noise, label smoothing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XeQcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Sombrero Loss, a regularization method inspired by Mixup that operates on the original inputs without performing input interpolation. The authors analyze its expected gradient and noise covariance, establish a connection with label smoothing and curvature, and evaluate the method on several classification and structured prediction tasks.

Critical strengths

I think the paper addresses an interesting problem by investigating whether the regularization benefits of Mixup can be obtained without mixing the inputs. The proposed method is relatively simple, and the theoretical analysis provides some useful insights into its expected gradient and noise structure. Another positive point is that the authors evaluate the method on several real-world datasets, including CIFAR and ImageNet. Some of the experiments also include multiple runs and show promising improvements over Mixup and standard training.

Critical weaknesses

My main concern is about the novelty of the proposed method. Since Sombrero Loss has the same expected gradient as label smoothing and uses randomly sampled targets, I am not fully convinced that the practical method is sufficiently different from existing stochastic label-smoothing approaches. The paper would benefit from stronger comparisons with these methods.

Another concern is the connection between the theoretical analysis and the experimental results. The authors provide a detailed analysis of gradient noise and its relationship with curvature, but this connection relies on several assumptions. I think the paper needs stronger empirical evidence showing that the proposed mechanism actually explains the improvements during training.

I also have some concerns about the experimental evaluation. Although the main classification results are promising, several important ablation experiments are reported using only single runs. This makes it difficult to determine whether the relatively small improvements are consistent. The detection and translation results also show only modest gains without repeated-run statistics.

Another point is the connection with Mixup. The appendix shows that the retained gradient can differ significantly from the actual Mixup gradient, especially at larger mixing strengths. While this does not invalidate the proposed method, I think the authors should clarify what is actually being preserved from Mixup and why this particular component is important.

Finally, I could not find implementation code in the submitted materials so I was not able to reproduce it.

Questions for authors
  1. Since the proposed Sombrero Loss has the same expected gradient as label smoothing and uses stochastic targets, could the authors explain more clearly what distinguishes it from existing randomized label-smoothing methods? It would be useful to include a direct comparison with stochastic label smoothing under matched mean and variance.

  2. Proposition 4 connects the intrinsic gradient covariance with a modeled GGN matrix under several assumptions. Could the authors provide empirical measurements showing how closely the actual gradient-noise covariance aligns with the actual GGN or Hessian during training?

  3. Table 1 provides mean-matched comparisons to support the contribution of stochastic regularization beyond label smoothing, but most results are from single runs. Could the authors repeat these experiments across multiple seeds and report the variation?

  4. Appendix D.6 shows that the retained gradient can differ significantly from the actual Mixup gradient. Could the authors explain how this affects the interpretation of the proposed first-order purification and why the retained component is preferable to other possible decompositions?

  5. The improvements in detection and translation are relatively small and reported as point estimates. Could the authors provide repeated-run results to establish whether these improvements are consistent?

  6. Could the authors provide the implementation code and more complete reproduction instructions, especially for the centered and decoupled variants?

Note: I might not be an expert in this field so I might have some misunderstanding of the method, and I could not review the appendix deeply.

Reviewer_8qDqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper takes the mixup gradient, drops the input interpolation term, and keeps only the first order kernel. It realizes this on original inputs as the Sombrero loss, with paired or independently unifrom sampled targets y_r.

Critical strengths
  • Decomposition is clean, and expectation equivalence to label smoothing is exact and useful.
  • Broad evaluation.
Critical weaknesses
  • Stripped of the formalism, Uniform Sombrero is label smoothing with a stochastic target, i.e., random label replacement, which is DisturbLabel (Xie et al., CVPR 2016) and closely related to Mixup-TO (Wang et al. 2023); neither is positioned against properly (DisturbLabel is not cited).
  • central claim 'regularization beyond mean smmoothing' rests on tab 1, where LS is a single run and the gap (0.9) is about two ERM SDs, tab 2 is one checkpoint. No ablation with isotropic noise of matched trace tests whether the structure of the noise (props 3-4) matters at all.
  • Gains are small and mostly point estimates: ImageNet R-18 +0.7 over Mixup (3 seeds), R-50 and DeiT-S below CutMix/DeiT-recipe references, detection +0.4 AP single run.
  • The curvature-alignment section (Props. 3–4) yields no tested prediction. It is presented as mechanism but functions as motivation.
Questions for authors
  • How does Uniform Sombrero differ from DisturbLabel, empirically and conceptually?
  • Can you report LS over 5 seeds and a sweep at matched mean (Table 1), and an isotropic-noise control with the same gradient covariance trace?
  • Does any prediction of Props. 3–4 change accuracy when violated?
  • Why is ImageNet R-50/DeiT-S Sombrero not compared against Mixup/CutMix under the same recipe in-house?
Reviewer_TWnpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper decomposes the Mixup gradient into a retained first-order expression and a residual input-interpolation term, then discards the residual and realizes the retained expression directly on original (unmixed) inputs. This yields Sombrero Loss: a per-example convex combination of the loss under the observed label and the loss under a perturbation label, with retention rate λ_s = λ² + (1−λ)² ∈ [1/2, 1]. Paired bidirectional permutation (f, f⁻¹) makes the batch gradient exactly equal to the retained Mixup expression (Prop. 10). Independent uniform perturbation targets extend the construction and give an exact mean-gradient identity with Label Smoothing (Prop. 2), plus a closed-form conditional noise covariance decomposing into a "base" and a "class-specific" PSD structure (Prop. 3). Under an ε-contamination prediction model, the GGN shares the same two structural matrices (Prop. 4), motivating coefficient calibration (Dec-Sombrero). Experiments cover CIFAR-100/10, ImageNet-1K (ShuffleNetV2, ResNet-18/50, DeiT-S), COCO detection, WMT translation, and long-tailed classification, with Hessian spectra, loss landscapes, mean-matched controls, and a gradient probe.

Contributions: (i) a first-order decomposition/purification of Mixup with exact paired realization; (ii) mean-regularization and structured-noise characterization with a conditional curvature link; (iii) broad empirical evaluation including non-classification tasks and prior flexibility.

Critical strengths
  1. Exactness of the paired realization (Prop. 10) plus a candid residual probe. The identity is proved under stated conditions and, importantly, the authors measure the gap to full Mixup (Table 14, RelGap up to 1.94) rather than claiming approximation. This is a rare and credibility-enhancing combination: a clean construction plus an honest account of what it is not. It is consequential because it makes "we study a different objective, deliberately" a defensible scientific position rather than a hidden approximation.

  2. Mean-matched experimental design (LS vs. Sombrero, ERM vs. Cen-Sombrero). Because Prop. 2 gives an exact mean-gradient match with LS, the accuracy and spectrum comparisons isolate the stochastic component. This directly addresses the obvious reviewer objection ("isn't this just label smoothing?") with a controlled intervention, and it is the strongest evidence in the paper for a mechanism beyond mean regularization.

Critical weaknesses
  1. The headline accuracy claim is only weakly supported relative to Mixup, and the ablations are entangled. The CIFAR-100 repeated-run gap over Mixup is 78.59±0.30 vs. 77.94±0.56 (5 seeds) and over Manifold Mixup 78.59 vs. 78.34±0.35 — within ~1 SD, i.e., not clearly separated. On ImageNet-1K the ResNet-18 gap is 71.12±0.23 vs. 70.41±0.28 (3 seeds), also modest. Meanwhile, Table 4 mixes local runs with cited external recipes (CutMix, DeiT, external ShuffleNet baseline with LS), and the Sombrero/Dec-Sombrero entries there are single-run point estimates under a different recipe; the "Bold: best result from this work" presentation invites over-reading. The Mixup-TO ablation (Table 9) changes coefficient law and target sampling simultaneously, so it cannot attribute the gain. Since the paper's practical pitch is "competitive with Mixup without touching inputs," this evidential weakness is consequential.

  2. The curvature-alignment story rests on an auxiliary model and an approximation, and the calibration is not validated as improving anything. Prop. 4's GGN structure holds under an ε-contamination prediction model with moment-factorization assumptions; γ_H ≈ 1−ε is an empirical approximation (Fig. 8 shows visible residuals), and the calibration ε* = 1 − γ_in is described as "a geometric design principle rather than a guarantee." Table 8's estimates are five-seed moments at converged minima — they characterize, but do not test, the model. Crucially, nowhere is Dec-Sombrero shown to outperform Sombrero because of coefficient matching (the gains in Table 4 are single-run and within noise of each other). The mechanism claim is thus suggestive but not causally established. Since the alignment argument is the paper's main theoretical differentiator from plain LS, this limits the strength of the contribution.

Questions for authors
  1. Can you report repeated runs (≥3 seeds) for the ImageNet-1K Sombrero/Dec-Sombrero entries in Table 4, and clarify which entries are single-run? Given the CIFAR-100 SDs (~0.3), what is the minimum detectable difference for your protocol, and do you claim a significant improvement over Mixup or "competitive"?
  2. Mixup-TO differs from Sombrero in both coefficient law and target sampling. Can you add a variant with Mixup-TO's paired batch-shuffled targets under Sombrero's λ_s law (i.e., hold the prior fixed, vary only pairing), to attribute the 77.10 → 78.49 gap?
  3. Is there a setting where Dec-Sombrero's γ_in/γ_H matching yields a measurable benefit over Sombrero at matched mean smoothing — e.g., a spectrum or landscape statistic, not just accuracy? If not, should the calibration be presented as an analysis tool rather than a method?
  4. The mean-gradient identity requires ∇_h C affine in y. For detection (Hungarian assignment) and translation (token-level CE), does the LS equivalence still hold example-wise, or only approximately? Please state explicitly which theoretical claims transfer to Sec. 5.2.
  5. Fig. 7 shows mass concentrating at λ_s = 0.5 as α grows, i.e., the perturbation term dominates in expectation for large α — yet the accuracy sweep degrades gracefully up to α = 32 (77.3–77.6%). Can you explain mechanistically why heavy perturbation is so benign here, and whether this is consistent with the "noise magnitude vanishes as α→0" discussion in App. C.5?
  6. CutMix and SAM are discussed but not compared. A single CIFAR-100 comparison against SAM at matched compute would materially strengthen the "implicit sharpness-aware behavior" claim.

Comments on the paper

1 comment

Thank you for your reply. I hope your health is fine now.

I am still not convinced about the novelty since for me it looks like a combination of several known ideas.

Also, without the code, I cannot check the reproducibility. It would be good if you send the code, at least for CIFAR10 to confirm your results.

For now, I am maintaining my score 2 (weak reject) but it is not my last rating, and I am open to change my assessment since the work seems useful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.