acceptodds
Desk rejected at ICLR 2027

Mixup Mitigates Instance-Level Overfitting under Structural Overparameterization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Overparameterization is commonly measured by the number of fitted model parameters relative to the training sample size. In this paper, we examine mixup augmentation through the complementary notion of *structural overparameterization*: a nonlinear student contains directions that are unnecessary for a linear teacher but can fit observation noise. For Gaussian covariates, we derive precise high-dimensional characterizations of the prediction risk under full-pair mixup. Our analysis combines a decomposition of pairwise interactions with leave-many-out arguments to handle the complex dependence among mixup samples generated from training data. The obtained risk formulas separate background prediction error from label-noise sensitivity, revealing how mixup changes their tradeoff under *homogeneous* and *heterogeneous* noise. In particular, under explicit conditions, we show that mixup achieves *strictly lower risk* than ordinary nonlinear regressor, with the ridge penalty optimized separately for each method. This advantage consequently extends to mixup jointly tuned over its mixing weight and ridge strength. Numerical experiments support the predicted risk comparisons. Motivated by this analysis, we propose a selective mixup rule based on training-time fitting progress in neural networks with learned features on three real-world regression datasets. Under training-label perturbations, selected mixup reduces test error relative to empirical risk minimization and separately tuned weight decay.

Primary area:
learning theory
Keywords:
Mixup, Random feature regression, Structural overparameterization, High-dimensional generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RfNBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies full-pair mixup in nonlinear random-feature regression with a linear teacher and Gaussian covariates. It derives high-dimensional prediction-risk formulas that separate signal reconstruction, residual fitting, and irreducible error. It also gives conditions under which midpoint mixup improves on ordinary random-feature regression after separate ridge tuning. Numerical studies examine these predictions, and experiments on three regression datasets evaluate selective mixup based on training-time fitting progress.

Critical strengths
  1. The comparison against separately optimized ridge addresses a meaningful question. The risk decomposition explains how mixup changes signal reconstruction and noise sensitivity while accounting for dependence among mixed pairs.
  2. Numerical risk calculations and several stress tests support the theoretical findings. The supplementary package provides detailed protocols and per-split results, making the empirical claims reasonably inspectable.
Critical weaknesses

The selective-network experiments do not separate interpolation effects from reduced exposure to selected labels. Appendix S.7.2 acknowledges that selected labels have smaller total expected coefficients because partners come from the unselected set. A control that downweights the same selected observations without mixing covariates is missing. The random-set and fixed-q comparisons do not resolve this issue.

The practical comparison lacks regression-specific alternatives such as C-Mixup and RC-Mixup. This limits the evidence for the competitiveness of the selective rule, although it does not undermine the theoretical comparison with ridge.

Questions for authors

Using the same pilot ranking and validation protocol, how does selective mixup compare with simply downweighting or trimming the selected observations, without mixing covariates?

How does the selective rule compare with C-Mixup or RC-Mixup under matched perturbations, data splits, and tuning budgets?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer RfNB - Part 1/2

Thank you for your comments on the comparison with separately optimized ridge, the risk decomposition, and the numerical results. Below, we report comparisons with downweighting and trimming control. We also compare selected Mixup with C-Mixup and RC-Mixup and report the search budgets. We discuss how the methods' relative performance varies across the tested settings.

R3.1. Using the same pilot ranking and validation protocol, how does selective mixup compare with simply downweighting or trimming the selected observations, without mixing covariates?

We ran two sets of paired experiments to address this concern. Both use Concrete, Energy, and Airfoil, the original tanh network, and the same ten data splits. Within each condition, all methods share a newly fitted pilot ranking and the same network initialization. For a nominal selection fraction qq, SqS_q contains the ⌈qn⌉\lceil qn\rceil highest-ranked observations among the nn training samples. The selected set is therefore identical across methods at fixed qq. Each training trajectory uses AdamW with learning rate 0.001, batch size 96, and 1,500 optimizer updates. We select weight decay and checkpoint separately for each method using the same validation grids.

The matched downweighting control retains the original, unmixed inputs and observed labels. Its data loss is

Ldw(θ)=1n∑i=1nwi(fθ(xi)−yi)2.L_{\mathrm{dw}}(\theta)=\frac{1}{n}\sum_{i=1}^{n}w_i\bigl(f_\theta(x_i)-y_i\bigr)^2.

Writing qeff=∣Sq∣/nq_{\mathrm{eff}}=|S_q|/n for the actual selected fraction, we use

\begin{cases} 1/2, & i\in S_q,\\\\ (2-q_{\mathrm{eff}})/\{2(1-q_{\mathrm{eff}})\}, & i\notin S_q. \end{cases}$$ These weights have mean one and match the relative expected contribution of each label to the Mixup targets. We draw each minibatch uniformly with replacement from the full training set. The batch loss is the sum of weighted squared residuals divided by the batch size. Trimming excludes $S_q$ from fitting and uses the unweighted data loss $$L_{\mathrm{trim}}(\theta)=\frac{1}{n-|S_q|}\sum_{i\notin S_q}\bigl(f_\theta(x_i)-y_i\bigr)^2.$$ The first experiment fixes the perturbed fraction at 1% and varies the perturbation strength. The figure below reports the fixed-$q$ comparison at $q=0.10$ across perturbation strengths; the following paragraph focuses on $a=6$. Lines show mean test RMSE, and bands show one sample standard deviation over ten paired splits. ![Fixed-q comparison at q=0.10: selected Mixup, matched downweighting, and trimming on Concrete, Energy, and Airfoil.](https://anonymous.4open.science/api/repo/rebuttal-figures-C501/file/r23_fixed_q010.png) Mixup has lower RMSE than matched downweighting on Concrete and Energy in all ten splits. Downweighting performs better on Airfoil, with lower RMSE in nine of ten splits.
AuthorsOpenReview ↗

Official Response to Reviewer RfNB - Part 2/2

The second experiment compares Mixup with trimming as perturbations are spread across more observations. We fix the mean squared injected perturbation, normalized by the original training-response variance, at E∈{0.04,0.16,0.36}E\in\{0.04,0.16,0.36\}. For each energy level, we vary the perturbed fraction over ρ∈{0.01,0.05,0.10,0.20}\rho\in\{0.01,0.05,0.10,0.20\} and reduce the per-observation amplitude as this fraction increases. Each method independently selects q∈{0,0.01,0.025,0.05,0.10,0.20}q\in\{0,0.01,0.025,0.05,0.10,0.20\} using validation data. Setting q=0q=0 gives ordinary training without selecting any observations.

The figure below compares sparse and dispersed perturbations at the same energy, E=0.36E=0.36. Lines show mean test RMSE, and bands show one sample standard deviation over ten paired splits, with qq selected independently for each method.

Density comparison at E=0.36 with independently tuned q: selected Mixup, matched downweighting, and trimming on Concrete, Energy, and Airfoil.

At a perturbed fraction of 1%, trimming has lower mean RMSE on all three datasets. Here, the same perturbation energy is concentrated on fewer observations, so each perturbation is larger. At 20%, Mixup has lower mean RMSE on all three datasets. Its advantage is most consistent on Energy, where it has lower RMSE in eight of ten splits.

Across all nine dataset-by-energy combinations, the mean RMSE difference (trimming minus Mixup) is larger at a perturbed fraction of 20% than at 1%. At 20%, Mixup has lower mean RMSE in six of these nine combinations. This pattern is consistent with a tradeoff between removing severely perturbed labels and the possible loss of useful training information through trimming.

The two experiments provide complementary evidence. Matching expected label exposure does not reproduce Mixup's gains on Concrete and Energy. The comparison with trimming depends on the perturbation pattern: trimming performs well for sparse, large perturbations, whereas Mixup can achieve lower RMSE than validation-tuned trimming when perturbations are more dispersed. We have added the experimental settings and results of these comparisons to the appendix of the revised manuscript.

R3.2. How does the selective rule compare with C-Mixup or RC-Mixup under matched perturbations, data splits, and tuning budgets?

We have added C-Mixup and RC-Mixup to the sparse-perturbation experiment on Concrete, Energy, and Airfoil. We compare the methods over ten paired splits and all six strengths a∈{0,1,2,4,6,8}a\in\{0,1,2,4,6,8\}. We reuse the selected-Mixup models trained in the earlier downweighting and trimming control experiment. For a>0a>0, the nominal perturbed fraction is 1%. All methods share the data splits, observed training labels, preprocessing, network architecture, and initializations.

The figure shows mean test RMSE (lines) and one sample SD (bands) over the ten paired splits. The selected-Mixup results use tuned qq.

Selected Mixup with tuned q, C-Mixup, and RC-Mixup on Concrete, Energy, and Airfoil.

Selected Mixup has lower mean RMSE than C-Mixup in 16 of the 18 conditions. RC-Mixup has lower mean RMSE than selected Mixup in 14 conditions, including all three datasets at a≥2a\ge2. Selected Mixup has lower mean RMSE than RC-Mixup on Concrete and Airfoil at a=0,1a=0,1. The results support competitiveness with C-Mixup in this setting, while RC-Mixup performs better under stronger perturbations.

Selected Mixup uses pilot fitting progress to decide which observations to mix. RC-Mixup combines iterative sample cleaning with label-based pairing and adaptive bandwidth selection. These mechanisms could be combined by applying our selection rule within RC-Mixup's retained sample pool and using its pairing strategy. We leave the evaluation of this combination to future work.

The full hyperparameter searches for C-Mixup and RC-Mixup use larger budgets than selected Mixup. Per split and strength, selected Mixup uses 21 candidates and 32,100 optimizer updates, including its 600-update pilot. C-Mixup uses 70 candidates and 105,000 updates on Concrete and Energy, and 105 candidates and 157,500 updates on Airfoil. The corresponding RC-Mixup searches use 70/105 outer configurations and 245,000/367,500 updates when run separately.

We have added a more detailed discussion of these experiments to the appendix of the revised manuscript.

Dear Reviewer RfNB,

Thank you again for your thoughtful feedback. Your suggestions prompted us to add comparisons with downweighting, trimming, C-Mixup, and RC-Mixup. Since the discussion period is about to close, we would appreciate any thoughts on these additional results and our responses. We would be happy to clarify anything further.

Best, Authors

Reviewer_8ZRnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when mixup helps beyond ordinary ridge regularization, in scalar regression with noisy labels. The theoretical setting is deliberately controlled: isotropic Gaussian covariates, a linear teacher, and a fixed nonlinear student with orthonormal random features, with the feature-to-sample ratio held at a constant order.

Full-pair mixup with squared loss is analyzed in a high-dimensional limit. Because the linear teacher satisfies the mixing identity exactly while nonlinear predictions generally do not, mixup imposes chord constraints rather than only fitting at the observed points.

The empirical part adds a selector: rank training observations by fitting progress and mix the top-ranked ones with unselected partners.

Critical strengths

The comparison is against a strong, optimized baseline.

Dependence across mixed pairs is handled explicitly.

The risk decomposition has explanatory value.

Critical weaknesses

Linear teacher, isotropic Gaussian design, fixed orthonormal features, full-pair squared-loss training. Theorem 4.3 says "there exist"; it is not a general superiority claim. The anisotropy experiment exists but shows the gain vanishing as signal aligns with the leading covariance direction.

For exogenous noise profiles the risk depends on the average variance, not on whether corruption is concentrated or dispersed. The single-spike example is an interpretable construction, not evidence that sparse corruption uniquely produces the benefit; the non-vanishing construction also scales the spike variance with n.

Only unselected labels serve as partners, so selected labels receive lower total expected weight in the training targets — the authors state this themselves. Random-set and fixed-q controls test ranking quality, but a score-matched downweighting or trimming control is missing, so "interpolation constraint" is not isolated from "downweight suspected bad labels".

C-Mixup and RC-Mixup are discussed but not evaluated; robust-loss baselines are absent; selected procedures use more training trajectories than the baselines. Without injected corruption, the selected variant is worse on Energy and Airfoil, and random selection wins on Airfoil at one setting.

Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer 8ZRn - Part 1/4

Thank you for your comments on the comparison with separately optimized ridge, our treatment of pair dependence, and the risk decomposition. Below, we clarify the theoretical conditions and the role of the noise model. We present the additional comparisons with downweighting, trimming, C-Mixup, and RC-Mixup and report the search budgets. We also discuss how relative performance varies across settings.

R2.1. Linear teacher, isotropic Gaussian design, fixed orthonormal features, full-pair squared-loss training. Theorem 4.3 says "there exist"; it is not a general superiority claim. The anisotropy experiment exists but shows the gain vanishing as signal aligns with the leading covariance direction.

We would like to clarify that Appendix S.4.4, particularly Corollary S.4.14, gives explicit sufficient conditions for the comparison in Theorem 4.3. For the linear teacher, mixing two clean labels gives the correct label at the mixed input. A nonlinear student's predictions do not satisfy this identity. Training on mixed pairs therefore changes how the student fits the signal and residuals. The theorem identifies when this change improves the tradeoff between signal reconstruction and residual fitting.

Under the assumptions of Corollary S.4.14, the two sufficient conditions are

c\frac{\mathcal E_{\rm eff}}{\mathcal B}>\frac{\kappa(1/2)}{a_V},$$ where $c=\lim p/n$, $c_{\rm sep}$ is the threshold given in the appendix, and $\mathcal B$ is the effective signal energy captured by the ordinary random features. The background effective residual energy $\mathcal E_{\rm eff}$ includes observation noise, signal outside the feature span, and approximation error from representing a linear teacher with nonlinear features. The first condition concerns the tradeoff between fitting the signal and fitting residual variation. Nonlinear features can fit patterns absent from the linear teacher's signal. Training on mixed pairs constrains predictions between observations, while the clean teacher labels remain consistent with interpolation. The condition sets a threshold for the feature-to-sample ratio, at or above which this tradeoff favors Mixup. The second condition concerns the regularizing effect of Mixup. It ensures that this effect is compatible with the regularization level preferred by ordinary ridge. For example, take the tanh activation, $p/n=1.25$, a unit-norm teacher in the feature span, and no observation noise ($\sigma_\varepsilon^2=\tau^2=0$). Numerical evaluation gives $c_{\rm sep}\approx0.6711$ and $\mathcal E_{\rm eff}/\mathcal B\approx0.07473>0.02300\approx\kappa(1/2)/(a_Vc)$. Both conditions therefore hold. At a positive Mixup ridge strength ($\gamma_{\rm mix}\approx0.01720$), the asymptotic risk is $0.33939$, compared with $0.38876$ for optimally tuned ordinary ridge. Regarding the modeling assumptions, similar teacher-student settings appear in other random-feature analyses. [Hu and Lu (2023)](https://arxiv.org/pdf/2009.07669) study fixed nonlinear random features under isotropic Gaussian inputs, including linear-teacher ridge regression. [Gerace et al. (2020, Section 3.2 and Figure 5)](https://proceedings.mlr.press/v119/gerace20a/gerace20a.pdf) study ridge regression with a linear teacher. They compare Gaussian and orthogonal feature projections, including at separately optimized regularization strengths. Theorem 4.3 now states the comparison for the Gaussian contamination model in Example 2 and refers directly to the conditions in Appendix S.4.4. These conditions are unchanged: the contamination and original single-observation models have the same noise-averaged limiting risk when their background parameters and excess variance match.
AuthorsOpenReview ↗

Official Response to Reviewer 8ZRn - Part 2/4

R2.2. For exogenous noise profiles the risk depends on the average variance, not on whether corruption is concentrated or dispersed. The single-spike example is an interpretable construction, not evidence that sparse corruption uniquely produces the benefit; the non-vanishing construction also scales the spike variance with n.

We agree that, under the theorem's assumptions, the noise-averaged asymptotic risk depends on the average exogenous training-noise variance, rather than on whether that variance is concentrated or dispersed. We have replaced Example 2 with a Gaussian special case of Huber's (1964) contamination model. Each training error independently follows N(0,σε2)\mathcal N(0,\sigma_\varepsilon^2) with probability 1−η1-\eta and N(0,σout2)\mathcal N(0,\sigma_{\rm out}^2) with probability η\eta. The errors are independent of the covariates and feature. Both variances remain fixed as the sample size grows, with 0<η<10<\eta<1 and 0≤σε2≤σout2<∞0\leq\sigma_\varepsilon^2\leq\sigma_{\rm out}^2<\infty.

With τ2=η(σout2−σε2)\tau^2=\eta(\sigma_{\rm out}^2-\sigma_\varepsilon^2), the average training-noise variance is σε2+τ2\sigma_\varepsilon^2+\tau^2, while the test noise variance remains σε2\sigma_\varepsilon^2. Applying Theorem 3.1 gives the revised Corollary 4.2:

R‾cont(λ,γ;τ2)=R‾hom(λ,γ)+τ2χ(λ,γ),\overline R_{\rm cont}(\lambda,\gamma;\tau^2)=\overline R_{\rm hom}(\lambda,\gamma)+\tau^2\chi(\lambda,\gamma),

where R‾hom\overline R_{\rm hom} is the background risk with training and test noise variance σε2\sigma_\varepsilon^2. The contamination risk averages over both the mixture indicators and the errors. It depends on the contamination parameters only through τ2\tau^2.

We have recomputed the finite-design curve in panel (b) of the theory figure using η=0.05\eta=0.05, zero background and test noise, and σout2=τ2/η\sigma_{\rm out}^2=\tau^2/\eta. The calculation averages over both the mixture indicators and noise amplitudes, using the same Gaussian design and ridge-search protocol. The limiting curve is unchanged. The finite-design calculation changes because the two models have different training-noise covariance profiles.

AuthorsOpenReview ↗

Official Response to Reviewer 8ZRn - Part 3/4

R2.3. Only unselected labels serve as partners, so selected labels receive lower total expected weight in the training targets — the authors state this themselves. Random-set and fixed-q controls test ranking quality, but a score-matched downweighting or trimming control is missing, so "interpolation constraint" is not isolated from "downweight suspected bad labels".

We ran two sets of paired experiments to address this concern. Both use Concrete, Energy, and Airfoil, the original tanh network, and the same ten data splits. Within each condition, all methods share a newly fitted pilot ranking and the same network initialization. For a nominal selection fraction qq, SqS_q contains the ⌈qn⌉\lceil qn\rceil highest-ranked observations among the nn training samples. The selected set is therefore identical across methods at fixed qq. Each training trajectory uses AdamW with learning rate 0.001, batch size 96, and 1,500 optimizer updates. We select weight decay and checkpoint separately for each method using the same validation grids.

The matched downweighting control retains the original, unmixed inputs and observed labels. Its data loss is

Ldw(θ)=1n∑i=1nwi(fθ(xi)−yi)2.L_{\mathrm{dw}}(\theta)=\frac{1}{n}\sum_{i=1}^{n}w_i\bigl(f_\theta(x_i)-y_i\bigr)^2.

Writing qeff=∣Sq∣/nq_{\mathrm{eff}}=|S_q|/n for the actual selected fraction, we use

\begin{cases} 1/2, & i\in S_q,\\\\ (2-q_{\mathrm{eff}})/\{2(1-q_{\mathrm{eff}})\}, & i\notin S_q. \end{cases}$$ These weights have mean one and match the relative expected contribution of each label to the Mixup targets. We draw each minibatch uniformly with replacement from the full training set. The batch loss is the sum of weighted squared residuals divided by the batch size. Trimming excludes $S_q$ from fitting and uses the unweighted data loss $$L_{\mathrm{trim}}(\theta)=\frac{1}{n-|S_q|}\sum_{i\notin S_q}\bigl(f_\theta(x_i)-y_i\bigr)^2.$$ The first experiment fixes the perturbed fraction at 1% and varies the perturbation strength. The figure below reports the fixed-$q$ comparison at $q=0.10$ across perturbation strengths; the following paragraph focuses on $a=6$. Lines show mean test RMSE, and bands show one sample standard deviation over ten paired splits. ![Fixed-q comparison at q=0.10: selected Mixup, matched downweighting, and trimming on Concrete, Energy, and Airfoil.](https://anonymous.4open.science/api/repo/rebuttal-figures-C501/file/r23_fixed_q010.png) Mixup has lower RMSE than matched downweighting on Concrete and Energy in all ten splits. Downweighting performs better on Airfoil, with lower RMSE in nine of ten splits. The second experiment compares Mixup with trimming as perturbations are spread across more observations. We fix the mean squared injected perturbation, normalized by the original training-response variance, at $E\in\{0.04,0.16,0.36\}$. For each energy level, we vary the perturbed fraction over $\rho\in\{0.01,0.05,0.10,0.20\}$ and reduce the per-observation amplitude as this fraction increases. Each method independently selects $q\in\{0,0.01,0.025,0.05,0.10,0.20\}$ using validation data. Setting $q=0$ gives ordinary training without selecting any observations. The figure below compares sparse and dispersed perturbations at the same energy, $E=0.36$. Lines show mean test RMSE, and bands show one sample standard deviation over ten paired splits, with $q$ selected independently for each method. ![Density comparison at E=0.36 with independently tuned q: selected Mixup, matched downweighting, and trimming on Concrete, Energy, and Airfoil.](https://anonymous.4open.science/api/repo/rebuttal-figures-C501/file/r23_density_E036.png) At a perturbed fraction of 1%, trimming has lower mean RMSE on all three datasets. Here, the same perturbation energy is concentrated on fewer observations, so each perturbation is larger. At 20%, Mixup has lower mean RMSE on all three datasets. Its advantage is most consistent on Energy, where it has lower RMSE in eight of ten splits. Across all nine dataset-by-energy combinations, the mean RMSE difference (trimming minus Mixup) is larger at a perturbed fraction of 20% than at 1%. At 20%, Mixup has lower mean RMSE in six of these nine combinations. This pattern is consistent with a tradeoff between removing severely perturbed labels and the possible loss of useful training information through trimming. The two experiments provide complementary evidence. Matching expected label exposure does not reproduce Mixup's gains on Concrete and Energy. The comparison with trimming depends on the perturbation pattern: trimming performs well for sparse, large perturbations, whereas Mixup can achieve lower RMSE than validation-tuned trimming when perturbations are more dispersed. We have added the experimental settings and results of these comparisons to the appendix of the revised manuscript.
Reviewer_wf5zOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Mixup through the lens of structural overparameterization, using a linear model as the teacher and a nonlinear random-feature model as the student. It derives the high-dimensional prediction risk and characterizes the tradeoff between signal reconstruction and residual fitting. Based on these results, it establishes sufficient conditions under which full-pair Mixup outperforms ordinary random-feature ridge regression.

Critical strengths

The theoretical results are clearly presented. Figure 1 is particularly helpful for understanding the key theoretical insights, offering intuition for how Mixup constrains instance-level overfitting.

Critical weaknesses
  • The paper studies Mixup in a regression setting, whereas Mixup is traditionally associated with classification and the interpolation of one-hot labels. Although regression-specific approaches such as C-Mixup exist, it would still be helpful to acknowledge this distinction and discuss the motivation for studying Mixup in regression before introducing the formal problem setup.

  • I believe the data model could also be better motivated. In particular, Example 2 assigns a noise variance that grows with the sample size, creating an increasingly extreme disparity across instances, which seems somewhat artificial as a model of real data. Discussing whether similar phenomena arise under i.i.d. heavy-tailed noise, where unusually large errors occur, or providing a concrete real-world example that motivates such substantial instance-wise noise heterogeneity would strengthen the justification for this setting.

Questions for authors
  • In Section 1 (Our Contributions), the paper highlights that dependencies among mixed samples are handled using new RMT techniques, but the main text does not mention them. Could the authors briefly describe these techniques in the main text?
  • The high-level intuition of this paper seems similar to that underlying the results of [1]. Could the authors discuss this connection?

Reference

[1] Oh and Yun (2023). Provable Benefit of Mixup for Finding Optimal Decision Boundaries.

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer wf5z - Part 1/3

Thank you for your comments on the theoretical presentation and Figure 1. Below we clarify the regression setting, noise model, proof techniques, and connection to prior work.

R1.1. The paper studies Mixup in a regression setting, whereas Mixup is traditionally associated with classification and the interpolation of one-hot labels. Although regression-specific approaches such as C-Mixup exist, it would still be helpful to acknowledge this distinction and discuss the motivation for studying Mixup in regression before introducing the formal problem setup.

Thanks for this suggestion, and we agree that the motivation for studying Mixup in regression should be stated before the formal setup. Several studies beyond C-Mixup and RC-Mixup examine Mixup for continuous prediction. RegMix [A] studies data augmentation for regression with limited training data. Zhou et al. [B] evaluate Mixup for neural time-series forecasting, motivated by the limited availability of historical data. SK Mixup [C] uses response distances to adjust interpolation in regression and evaluates both prediction and calibration. DE-Mixup [D] addresses errors that can arise in synthetic regression targets under linear interpolation. ProbMix [E] reformulates Mixup for conditional density estimation by fusing conditional predictive densities. These studies motivate questions about how mixing affects generalization in regression and when mixed targets are consistent with the underlying input-response relationship.

We have added a discussion of these studies to the introduction to explain the motivation for Mixup in regression.

References for this response:

  • [A] Seong-Hyeon Hwang and Steven Euijong Whang. RegMix: Data Mixing Augmentation for Regression. arXiv preprint, 2021; revised 2022. Paper.
  • [B] Yun Zhou, Liwen You, Wenzhen Zhu, and Panpan Xu. Improving Time Series Forecasting with Mixup Data Augmentation. ECML PKDD International Workshop on Machine Learning for Irregular Time Series, 2023. Paper.
  • [C] Quentin Bouniot, Pavlo Mozharovskyi, and Florence d'Alché-Buc. Tailoring Mixup to Data for Calibration. ICLR 2025. Paper.
  • [D] Zhengzhang Hou, Zhanshan Li, Yanbo Liu, Geoff Nitschke, You Lu, and Ximing Li. Denoising Mixup for Regression. AAAI 2026, 40(26):21805–21813. Paper.
  • [E] Yousef El-Laham, Niccolo Dalmasso, Svitlana Vyetrenko, Vamsi K. Potluru, and Manuela Veloso. Mixup Regularization: A Probabilistic Perspective. UAI 2025, PMLR 286:1184–1219. Paper.
AuthorsOpenReview ↗

Official Response to Reviewer wf5z - Part 2/3

R1.2. I believe the data model could also be better motivated. In particular, Example 2 assigns a noise variance that grows with the sample size, creating an increasingly extreme disparity across instances, which seems somewhat artificial as a model of real data. Discussing whether similar phenomena arise under i.i.d. heavy-tailed noise, where unusually large errors occur, or providing a concrete real-world example that motivates such substantial instance-wise noise heterogeneity would strengthen the justification for this setting.

We have replaced the single-observation construction in Example 2 with a Gaussian special case of Huber's (1964) contamination model. The training errors are independent of the covariates and feature and satisfy

(1-\eta)\mathcal N(0,\sigma_\varepsilon^2) +\eta\mathcal N(0,\sigma_{\rm out}^2), \qquad 0<\eta<1,\quad 0\leq\sigma_\varepsilon^2\leq\sigma_{\rm out}^2<\infty.$$ Both variances remain fixed as the sample size grows. For small $\eta$, the higher-variance component models occasional large label errors among mostly reliable observations. The test noise variance remains $\sigma_{\rm test}^2=\sigma_\varepsilon^2$. The fixed mixture has zero mean and a finite eighth moment, so it satisfies the noise assumptions of Theorem 3.1. With $\tau^2=\eta(\sigma_{\rm out}^2-\sigma_\varepsilon^2)$, the average training-noise variance is $\sigma_\varepsilon^2+\tau^2$. Under the theorem's remaining assumptions, the revised Corollary 4.2 gives the additional risk $\tau^2\chi(\lambda,\gamma)$. This result does not require any individual noise variance to grow with the sample size. The i.i.d. heavy-tailed noise suggested in the review can produce unusually large errors even when all observations have the same noise variance. Theorem 3.1 covers this case for a fixed, centered noise distribution with a finite eighth moment. At fixed variance and other model parameters, tail shape does not enter the noise-averaged asymptotic risk formula. Under the conditions of Theorem 4.3, Mixup can improve on separately optimized ridge even in the homoscedastic case $\tau^2=0$, without any individual noise variance growing with the sample size. We have updated Example 2, Corollary 4.2, and Theorem 4.3 to use this contamination model. The appendix retains the original single-observation model and its influence results as a separate illustration of the local overfitting in Figure 1. We have also recomputed the finite-design curve in panel (b) of the theory figure for the contamination model. The limiting curve is unchanged.
AuthorsOpenReview ↗

Official Response to Reviewer wf5z - Part 3/3

R1.3. In Section 1 (Our Contributions), the paper highlights that dependencies among mixed samples are handled using new RMT techniques, but the main text does not mention them. Could the authors briefly describe these techniques in the main text?

Full-pair Mixup forms n2n^2 mixed samples by reusing the same nn observations. These samples are dependent because different pairs can share an observation. For distinct i,j,ki,j,k, our isotropic Gaussian design gives

\qquad \operatorname{Cov}\left(x_{ij}^{\mathrm{mix}},x_{ik}^{\mathrm{mix}}\right)=\lambda^2 I_d.$$ Each observation appears in $2n-1$ ordered pairs. Ordinary random-feature analyses such as [Mei and Montanari (2022)](https://arxiv.org/abs/1908.05355) start from independent training observations. [Morisset et al. (2026)](https://arxiv.org/html/2605.10290v1) allow dependent representations within independent source blocks. These assumptions do not cover the overlapping pairs in full-pair Mixup. For the odd activations in Theorem 3.1, we use a Hoeffding decomposition. Conditional on $W$, write $g_i=W^\top x_i$ and $z_{ij}=\phi_W(x_{ij}^{\mathrm{mix}})$. For $i\ne j$, $$z_{ij}=f_1(g_i)+f_2(g_j)+\eta_{ij}, \qquad \mathbb E[\eta_{ij}\mid g_i]=\mathbb E[\eta_{ij}\mid g_j]=0,$$ where $f_1,f_2$ are the centered endpoint conditional means. For clean labels, Equation (S.3.41) in the proof of Lemma S.2.1 gives an $L^4$ error bound of $Cn^{-1/2}\log^C n$. This bound applies to the Gram-matrix approximation in operator norm and to the score approximation. The approximations retain the isotropic interaction term and empirical-mean corrections. Conditional on $W$, the endpoint blocks $[f_1(g_i),f_2(g_i)]$ are independent across original observations. In Appendix S.3.5, the leave-one-out estimate in Equation (S.3.31) and the leave-two-out covariance calculations control the required resolvent terms. Together with control of the centering corrections, these estimates yield the clean-label equivalents in Lemma S.2.1. We treat observation noise separately using Lemma S.4.3. Its label-susceptibility estimate has an $L^2$ error bound uniform over observation indices. Under the theorem's assumptions, this allows the noise contribution to be expressed through the average training-noise variance. We have added a proof overview in Section 3 of the revised manuscript to explain how these steps handle dependence among mixed samples. ### R1.4. The high-level intuition of this paper seems similar to that underlying the results of [1]. Could the authors discuss this connection? Both studies analyze the benefits of Mixup: Oh and Yun (2023) focus on classification, while we study regression. Oh and Yun consider binary linear classification with logistic loss and two symmetric Gaussian class distributions. For approximating the Bayes-optimal boundary, they give a Mixup sample-complexity upper bound quadratic in the separability parameter, up to logarithmic factors. Ordinary empirical risk minimization can require exponentially many samples as this parameter increases. Their intuition is that mixing opposite-class observations can supply examples near the boundary, where original observations may be scarce. We study full-pair Mixup in nonlinear random feature regression with a linear teacher and noisy labels. Our high-dimensional risk analysis describes the tradeoff between signal reconstruction and residual fitting. We give conditions under which Mixup improves on ordinary random feature ridge regression, with the ridge strength optimized separately for each method. We have added a brief description of Oh and Yun (2023) to the related work section of the revised manuscript to clarify the distinction between their classification result and our regression analysis. Junsoo Oh and Chulhee Yun. *Provable Benefit of Mixup for Finding Optimal Decision Boundaries*. ICML 2023, Proceedings of Machine Learning Research 202:26403–26450. [Official paper page](https://proceedings.mlr.press/v202/oh23a.html).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.