- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies full-pair mixup in nonlinear random-feature regression with a linear teacher and Gaussian covariates. It derives high-dimensional prediction-risk formulas that separate signal reconstruction, residual fitting, and irreducible error. It also gives conditions under which midpoint mixup improves on ordinary random-feature regression after separate ridge tuning. Numerical studies examine these predictions, and experiments on three regression datasets evaluate selective mixup based on training-time fitting progress.
Critical strengths
- The comparison against separately optimized ridge addresses a meaningful question. The risk decomposition explains how mixup changes signal reconstruction and noise sensitivity while accounting for dependence among mixed pairs.
- Numerical risk calculations and several stress tests support the theoretical findings. The supplementary package provides detailed protocols and per-split results, making the empirical claims reasonably inspectable.
Critical weaknesses
The selective-network experiments do not separate interpolation effects from reduced exposure to selected labels. Appendix S.7.2 acknowledges that selected labels have smaller total expected coefficients because partners come from the unselected set. A control that downweights the same selected observations without mixing covariates is missing. The random-set and fixed-q comparisons do not resolve this issue.
The practical comparison lacks regression-specific alternatives such as C-Mixup and RC-Mixup. This limits the evidence for the competitiveness of the selective rule, although it does not undermine the theoretical comparison with ridge.
Questions for authors
Using the same pilot ranking and validation protocol, how does selective mixup compare with simply downweighting or trimming the selected observations, without mixing covariates?
How does the selective rule compare with C-Mixup or RC-Mixup under matched perturbations, data splits, and tuning budgets?
3 comments
Official Response to Reviewer RfNB - Part 2/2
The second experiment compares Mixup with trimming as perturbations are spread across more observations. We fix the mean squared injected perturbation, normalized by the original training-response variance, at . For each energy level, we vary the perturbed fraction over and reduce the per-observation amplitude as this fraction increases. Each method independently selects using validation data. Setting gives ordinary training without selecting any observations.
The figure below compares sparse and dispersed perturbations at the same energy, . Lines show mean test RMSE, and bands show one sample standard deviation over ten paired splits, with selected independently for each method.
At a perturbed fraction of 1%, trimming has lower mean RMSE on all three datasets. Here, the same perturbation energy is concentrated on fewer observations, so each perturbation is larger. At 20%, Mixup has lower mean RMSE on all three datasets. Its advantage is most consistent on Energy, where it has lower RMSE in eight of ten splits.
Across all nine dataset-by-energy combinations, the mean RMSE difference (trimming minus Mixup) is larger at a perturbed fraction of 20% than at 1%. At 20%, Mixup has lower mean RMSE in six of these nine combinations. This pattern is consistent with a tradeoff between removing severely perturbed labels and the possible loss of useful training information through trimming.
The two experiments provide complementary evidence. Matching expected label exposure does not reproduce Mixup's gains on Concrete and Energy. The comparison with trimming depends on the perturbation pattern: trimming performs well for sparse, large perturbations, whereas Mixup can achieve lower RMSE than validation-tuned trimming when perturbations are more dispersed. We have added the experimental settings and results of these comparisons to the appendix of the revised manuscript.
R3.2. How does the selective rule compare with C-Mixup or RC-Mixup under matched perturbations, data splits, and tuning budgets?
We have added C-Mixup and RC-Mixup to the sparse-perturbation experiment on Concrete, Energy, and Airfoil. We compare the methods over ten paired splits and all six strengths . We reuse the selected-Mixup models trained in the earlier downweighting and trimming control experiment. For , the nominal perturbed fraction is 1%. All methods share the data splits, observed training labels, preprocessing, network architecture, and initializations.
The figure shows mean test RMSE (lines) and one sample SD (bands) over the ten paired splits. The selected-Mixup results use tuned .
Selected Mixup with tuned q, C-Mixup, and RC-Mixup on Concrete, Energy, and Airfoil.
Selected Mixup has lower mean RMSE than C-Mixup in 16 of the 18 conditions. RC-Mixup has lower mean RMSE than selected Mixup in 14 conditions, including all three datasets at . Selected Mixup has lower mean RMSE than RC-Mixup on Concrete and Airfoil at . The results support competitiveness with C-Mixup in this setting, while RC-Mixup performs better under stronger perturbations.
Selected Mixup uses pilot fitting progress to decide which observations to mix. RC-Mixup combines iterative sample cleaning with label-based pairing and adaptive bandwidth selection. These mechanisms could be combined by applying our selection rule within RC-Mixup's retained sample pool and using its pairing strategy. We leave the evaluation of this combination to future work.
The full hyperparameter searches for C-Mixup and RC-Mixup use larger budgets than selected Mixup. Per split and strength, selected Mixup uses 21 candidates and 32,100 optimizer updates, including its 600-update pilot. C-Mixup uses 70 candidates and 105,000 updates on Concrete and Energy, and 105 candidates and 157,500 updates on Airfoil. The corresponding RC-Mixup searches use 70/105 outer configurations and 245,000/367,500 updates when run separately.
We have added a more detailed discussion of these experiments to the appendix of the revised manuscript.
Dear Reviewer RfNB,
Thank you again for your thoughtful feedback. Your suggestions prompted us to add comparisons with downweighting, trimming, C-Mixup, and RC-Mixup. Since the discussion period is about to close, we would appreciate any thoughts on these additional results and our responses. We would be happy to clarify anything further.
Best, Authors
Official Response to Reviewer RfNB - Part 1/2
Thank you for your comments on the comparison with separately optimized ridge, the risk decomposition, and the numerical results. Below, we report comparisons with downweighting and trimming control. We also compare selected Mixup with C-Mixup and RC-Mixup and report the search budgets. We discuss how the methods' relative performance varies across the tested settings.
R3.1. Using the same pilot ranking and validation protocol, how does selective mixup compare with simply downweighting or trimming the selected observations, without mixing covariates?
We ran two sets of paired experiments to address this concern. Both use Concrete, Energy, and Airfoil, the original tanh network, and the same ten data splits. Within each condition, all methods share a newly fitted pilot ranking and the same network initialization. For a nominal selection fraction q, Sq contains the ⌈qn⌉ highest-ranked observations among the n training samples. The selected set is therefore identical across methods at fixed q. Each training trajectory uses AdamW with learning rate 0.001, batch size 96, and 1,500 optimizer updates. We select weight decay and checkpoint separately for each method using the same validation grids.
The matched downweighting control retains the original, unmixed inputs and observed labels. Its data loss is
Ldw(θ)=n1∑i=1nwi(fθ(xi)−yi)2.
Writing qeff=∣Sq∣/n for the actual selected fraction, we use
\begin{cases} 1/2, & i\in S_q,\\\\ (2-q_{\mathrm{eff}})/\{2(1-q_{\mathrm{eff}})\}, & i\notin S_q. \end{cases}$$ These weights have mean one and match the relative expected contribution of each label to the Mixup targets. We draw each minibatch uniformly with replacement from the full training set. The batch loss is the sum of weighted squared residuals divided by the batch size. Trimming excludes $S_q$ from fitting and uses the unweighted data loss $$L_{\mathrm{trim}}(\theta)=\frac{1}{n-|S_q|}\sum_{i\notin S_q}\bigl(f_\theta(x_i)-y_i\bigr)^2.$$ The first experiment fixes the perturbed fraction at 1% and varies the perturbation strength. The figure below reports the fixed-$q$ comparison at $q=0.10$ across perturbation strengths; the following paragraph focuses on $a=6$. Lines show mean test RMSE, and bands show one sample standard deviation over ten paired splits.  Mixup has lower RMSE than matched downweighting on Concrete and Energy in all ten splits. Downweighting performs better on Airfoil, with lower RMSE in nine of ten splits.