- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies minimum-norm interpolation in overparameterized linear regression with independent sub-Gaussian input coordinates and independent sub-Gaussian observation noise. Theorem 1 gives sufficient conditions for benign overfitting for norms with , and Theorem 3 formulates an extension to general norms under assumptions on the expected dual geometry and its concentration. The positive argument approximates the residual interpolation cost by a Euclidean norm, using a quantitative central limit theorem on flat dual vectors together with truncation and uniform concentration. Theorem 2 gives a scaled-Rademacher example in which every minimum- interpolator has excess risk with constant probability. The question is relevant, but the contribution needs to be distinguished more carefully from existing geometric analyses. In particular, the positive result does not recover known isotropic consistency regimes, and the negative result is a specific cube-based construction rather than a general characterization of non-Gaussian minimum- interpolation.
Critical strengths
The paper provides an identifiable route beyond Gaussian designs: the dual optimizer need not be flat, so a fixed-direction central limit theorem alone is insufficient. Combining the expected-norm truncation argument in Lemma 3 with flat-vector approximation and uniform concentration addresses this obstacle. This is the most interesting methodological ingredient to me. The treatment of independent sub-Gaussian observation noise, anisotropic designs, and , including , extends the scope of earlier results in useful directions. These extensions should not be confused with sharper statistical guarantees, but they are genuine strengths. The counterexample has a transparent mechanism and applies to every minimum-norm solution, rather than relying on an unfavorable choice among multiple minimizers. The proof sketches in Section 4 are also helpful for understanding the positive argument.
Critical weaknesses
1. The main text does not adequately distinguish the contribution from existing non-Gaussian and geometric results
The abstract describes existing results for non-Euclidean norms as “essentially limited to Gaussian distributions.” However, the published ICML 2024 paper by Kur, Abdalla, Bizeul, and Yang [1] already develops a Banach-space geometric framework and establishes sharp bounds for the noise-effect term in regression under sub-Gaussian covariates. The subsequent Kur–Bizeul manuscript [2] provides a more extensive treatment, including localized bounds for the structural-error term.
The submission acknowledges both papers and explicitly describes their bounds as sharp in Appendix A, page 16. My concern is therefore not the absence of a citation, but the lack of a substantive comparison in the main text. The introduction should distinguish what was already established from what is new here: the present work allows broader distributional assumptions and covers , but its sufficient conditions do not recover some existing guarantees under common assumptions. This tradeoff is important for assessing the contribution and should not be left implicit.
2. The positive bound fails to recover consistency, not merely sharp rates, in an important common setting
Let , and consider , , isotropic covariates with unit coordinate variances, and a fixed positive observation-noise variance. For , the dual exponent satisfies . Specializing equation (1) to gives the incompatible requirements
Thus, the stated sufficient conditions do not recover isotropic benign overfitting even in this basic one-sparse-signal setting.
For a concrete non-Gaussian comparison, take
with independent of , and choose , , and . Specializing Corollary 1 and Theorem 4 of [2] to this setting gives
and therefore
In contrast, setting in the present Theorem 1 yields the term
Consequently, the theorem's smallness prerequisite fails. This is a limitation of the guarantee, not evidence that the estimator itself is inconsistent. Nevertheless, the gap is more substantial than a nonoptimal rate: the bound does not establish consistency in a setting where consistency is already known.
One source of this loss is visible in Section 4.1, page 6. For , the proof controls the noise-fitting coefficients through the global inequality
leading to a noise-block risk estimate of order . Although the inequality is correct, it does not exploit sufficient structure of the selected interpolator to recover the sharper behavior. The paper should explain this limitation through a direct comparison under common assumptions; listing differences in the admissible distribution classes does not address it.
3. The relationship to the existing quotient-body framework should be made explicit
For a noise-block unit ball , define
For a feasible residual , standard convex duality gives
The dual feasible set in Section 4 is therefore exactly the polar of the projected norm ball. The dual formulation and the quotient-norm formulation describe the same residual interpolation cost, rather than different underlying geometric objects.
The connection also extends to the variational comparison between fitting the signal and fitting the residual. In the case, with finite , , , and , write
The interpolation constraint becomes , and the optimization reduces exactly to
\min_{\delta\in\mathbb{R}} \left\\{ |1-\delta|^p + J_N(\xi+\delta X_1)^p \right\\}.
This is the same scalar comparison between signal shrinkage and shifted-noise interpolation cost that appears in the localized quotient-norm formulation of [2, Section 6.1.2].
These identities are standard, and their equivalence does not imply that the complete proofs are identical. In particular, the flat-vector central limit theorem and truncation argument are meaningful distinguishing ingredients. The paper should explicitly identify those ingredients and explain their relationship to the existing geometric framework, rather than leaving the methodological comparison implicit.
4. The counterexample has a transparent but elementary geometric mechanism, and its scope should be stated carefully
On the event that every sign pattern appears among the noise-block columns, the construction gives the exact identities
The full image is consequently a cube with an additional pair of antipodal vertices:
Since , the first coefficient minimizes
For , the elementary midrange calculation in Appendix E yields
The remaining statistical step is to choose an -dependent noise distribution with a rare magnitude of order occurring with probability . With constant probability, exactly one observation has this larger magnitude, producing the stated risk lower bound.
This is a valid and transparent counterexample, and simplicity is not itself a weakness. However, the argument does not require a new random-polytope estimate beyond ensuring that all cube vertices are sampled. Moreover, the cube identity alone does not imply the stated divergence: the distinguished higher-variance coordinate and the particular noise distribution are also essential to this proof.
The statistical interpretation should reflect these restrictions. The construction requires exponentially many coordinates and does not establish the same conclusion for polynomial-dimensional Rademacher designs or for standard Gaussian observation noise. It demonstrates a failure of uniform extension to the specified sub-Gaussian class, rather than a general characterization of non-Gaussian interpolation.
Relatedly, the statement after equation (1) that the worsening dimension condition as is not a proof artifact needs qualification. A counterexample at does not establish the necessity or sharpness of the particular sufficient dimension condition for finite .
5. The and general-norm extensions need a more explicit interpretation
For a one-dimensional, unit-variance signal block and fixed remaining parameters, the sufficient conditions for fixed include
These conditions require the variance of each noise-fitting coordinate to vanish relative to the signal-coordinate variance. They do not provide an isotropic consistency guarantee.
The conditions are nevertheless nonvacuous. For example,
satisfy them while keeping
Thus, the relevant limitation is not that the result is empty, but that it establishes consistency in an anisotropic regime where the covariance favors the signal coordinates. The paper should describe this contribution as a non-Gaussian extension in that regime, rather than leave its relationship to isotropic guarantees or sharp characterizations unclear.
Similarly, Theorem 3 provides an abstract sufficient-condition framework for general norms. Verifying its assumptions on the expected dual geometry and uniform concentration constitutes a substantial part of applying the theorem. The paper carries out this verification quantitatively for norms, but should distinguish the abstract generality of the theorem from explicit statistical guarantees for additional norm families.
Questions for authors
I would be happy if the authors would address the bullets in the weaknesses, after addressing those and the ones of the other reviewers. I would gladly consider the increase the score of this work.
3 comments
Thank you for the detailed response. I appreciate your acknowledgment of the concerns raised in the reviews. However, I remain concerned that the current manuscript overstates the scope of the results and does not clearly communicate the tradeoffs with prior work. I would therefore like to see a revised manuscript during the discussion period, with the substantive changes clearly indicated. I find it a bit surprising that with LLMs those overlaps did not appear before the submission of this manuscript and were ``hidden'' in the appendix.
-
Positioning and prior work. Please revise the abstract and introduction to acknowledge the existing non-Gaussian results for and explain precisely what your work adds. The current characterization of prior results as essentially limited to Gaussian distributions is misleading. The main text should clearly state the assumptions and regimes in which your guarantees establish benign overfitting.
-
Loss of sharpness and distributional assumptions. Please explain in the main text where the analysis loses sharpness relative to existing results. As you acknowledge, the issue extends beyond suboptimal rates: your sufficient conditions do not recover known consistency guarantees for isotropic designs when .
-
Relationship to existing proof approaches. Please make the connection to the quotient-body framework and the geometric ideas underlying earlier Gaussian analyses, including the work of Koehler et al. (2021), more explicit. The CLT, truncation, and uniform concentration arguments are meaningful technical contributions. Their novelty should be explained in the context of these existing approaches.
-
Scope of the counterexample. I understand the conceptual value of separating Gaussian and non-Gaussian behavior. Nevertheless, I still find the relevance of this example to standard statistical settings limited, given the exponentially large dimension and the particular sample-size-dependent noise distribution. Please state these restrictions prominently and make clear which conclusions the construction supports.
-
Interpretation of the results. I agree that the extension to non-Gaussian designs for is novel. However, the manuscript should state more prominently that the sufficient conditions require a strong variance separation between the signal and noise-fitting coordinates. Please discuss whether some degree of anisotropy is intrinsic to the problem, particularly for , and distinguish this from the necessity of the specific quantitative conditions imposed by your analysis.
I still believe the paper has sufficient merit for ICLR, provided that the presentation accurately reflects these qualifications. I would be happy to reconsider my score after reviewing a revised manuscript that addresses these points and the other reviewers.
Thanks for the careful follow-up. We uploaded a revised manuscript in which all substantive changes are marked in red. We agree that the comparison with prior non-Gaussian work belonged in the main text rather than the appendix. Below we summarize how each point is addressed.
1. Positioning and prior work. The abstract now states that non-Gaussian results are known for , while results for remain largely limited to Gaussian designs. The related-work discussion has moved from the appendix into the introduction, where we state that Kur et al. (2024) and Kur–Bizeul (2026) give sharper guarantees for under isotropic sub-Gaussian designs and Gaussian noise. What our work adds is: (i) , including , for non-Gaussian designs; (ii) anisotropic designs and sub-Gaussian label noise; and (iii) a general-norm framework, which is conditional on Assumptions 3–4 and verified here for the family (as now stated after Corollary 2). The first contribution bullet now states explicitly that our guarantees concern anisotropic regimes.
2. Loss of sharpness. Sec. 3 now states that for our sufficient conditions do not recover known isotropic consistency regimes, and Sec. 4.1 identifies where this happens. With , the two conditions in (1) become , which are incompatible, although consistency is known there (e.g., your example with and ). The loss occurs in the noise-block step: we bound , which is tight only for sparse vectors, whereas for delocalized vectors . If were delocalized, the noise-block term would be of order rather than , and the two conditions would become compatible. Our dual argument controls but not the delocalization of , so it cannot recover these regimes.
3. Relationship to existing proof approaches. Sec. 4.1 now states that the dual feasible set is the polar of the quotient body , and that our decomposition (2) is the same comparison between signal shrinkage and residual-interpolation cost as in Koehler et al. (2021) and the localized quotient-norm analysis of Kur–Bizeul (2026). What differs is how the residual cost is approximated. For Gaussian designs, rotational invariance gives for every . For non-Gaussian designs this holds only approximately and for flat , via a quantitative CLT, while the dual optimizer need not be flat. The revision now explains the truncation lemma and uniform concentration bound that bridge this gap in this context.
4. Scope of the counterexample. The abstract, introduction, and Sec. 5 now state these restrictions explicitly. The construction uses exponentially many coordinates and an -dependent noise law (with uniformly bounded sub-Gaussian norm). It does not establish failure at polynomial dimension or for a fixed noise distribution such as Gaussian noise. Its conclusion is that Gaussian-design guarantees do not extend uniformly to sub-Gaussians, not a characterization of non-Gaussian interpolation. We also removed the claim that our dimension condition is "not an artifact", and now state that the example does not establish its necessity for any finite .
5. Interpretation of the results. The introduction and Sec. 3 now state that our guarantees require the noise-fitting variances to be much smaller than the signal variances. Sec. 3 also explains why some anisotropy is intrinsic for , including , and separates this from our specific quantitative conditions. Write . Informally, in the regime where our dual approximation holds, the minimum--norm problem reduces to Unless is large, the minimizer is shrunk away from by a non-vanishing amount, so the bias does not vanish. Thus , the first condition in (1), is needed up to constants. For we have , so with this forces . For this matches the exact isotropic Gaussian bias . For with isotropic coordinates, , so the interpolator fits even the signal through the noise-fitting coordinates and . Some variance separation is therefore intrinsic for . By contrast, we do not claim that the second condition, , or the logarithmic factors are necessary; for the second condition is precisely where our analysis is loose (point 2).
We hope these revisions address your concerns.
Rebuttal to Reviewer 7Zqj
We thank the reviewer for the constructive and insightful feedback. Below, we address the reviewer’s concerns. If any concerns remain that prevent the reviewer from raising their score, please let us know.
-
-
-
-
-
=\left(1-\frac nd\right)\lVert w^\star\rVert_2^2 +\frac{\sigma^2 n}{d-n-1}.$$ Thus, the bias approaches $\lVert w^\star\rVert_2^2$, so the risk does not approach $0$. Regarding general norms, Theorem 3 indeed provides a conditional framework whose assumptions we explicitly verify for the $\ell_p$ family.Relation to prior work: We agree that Kur et al. (2024) and Kur–Bizeul (2026) should be discussed more prominently in the introduction. Their work provides sharper guarantees for 1<p≤2 in important isotropic sub-Gaussian settings, but they do not cover the regime p>2 as we do in our paper. Our results are in some sense complementary: their bounds are based on geometric properties that do not hold for many common norms (e.g. ℓp with p>2) but do cover isotropic designs when 1<p<2, whereas ours are intentionally based on more general uniform concentration and CLT arguments, making them applicable to more general norms under suitable assumptions. We will clarify these differences and the tradeoffs in the paper.
Isotropic consistency and quantitative sharpness: Indeed our sufficient conditions do not recover known consistency guarantees for isotropic designs when 1<p<2. The inequality ∥wN∥2≤∥wN∥p that the reviewer mentioned is indeed loose in this regime. Our aim was to provide a framework that would be generally applicable, but case by case improvements are reserved for future work.
Relationship to quotient-body geometry: Indeed there is a strong relationship with the quotient-body framework, and we are not the first to analyze linear regression through the dual objective. The main technical contribution is the quantitative approximation of the dual objective by Euclidean geometry under non-Gaussian inputs, including the challenge that the dual optimizer need not be flat. Our CLT, truncation, and uniform concentration arguments address this challenge. We will explain the relationship to the existing geometric framework more clearly.
The ℓ1 counterexample: We agree that the construction exploits a particularly tractable cube geometry, and that our counterexample does not establish the necessity of the dimensional requirements in our positive results as p→1, and will clarify this distinction. The main point of the result is the statistical separation from p>1: even under assumptions that suffice to prove benign overfitting when p>1, every minimum-ℓ1 interpolator can have diverging risk. The example is also chosen to cleanly illustrate the geometric differences. Although our construction requires exponentially many coordinates, this is also a regime where Gaussian-design minimum-ℓ1 interpolation is known to benignly overfit (Koehler et al., 2021, e.g., d=⌈en2⌉ with the same covariance structure), demonstrating that the exponential dimensionality alone does not explain the failure. We will clarify the relevant discussions in the paper and the relation to the dimensional dependencies in the positive results, as you suggested.
The p>2 and general-norm results: We agree that our explicit p>2 consistency guarantees concern anisotropic distributions, and do not establish isotropic consistency. Nevertheless, as you mentioned, this is a meaningful extension beyond existing work, which does not establish benign overfitting for p>2 under non-Gaussian distributions. Moreover, anisotropy is a natural condition in this context: even for isotropic Gaussian inputs, minimum-ℓ2-norm interpolation is inconsistent for fixed nonzero targets when d/n→∞, since