acceptodds
Under review as a conference paper at ICLR 2027

Benign Misfitting Beyond One Spike

Abstract

Benign misfitting occurs when accurate prediction requires larger empirical training loss than the zero predictor's. We show this phenomenon occurs for noiseless overparameterized Gaussian regression with signal spikes of unequal strengths and independent training and test pairs from one distribution. Specifically, we show that there exists a range of sample sizes in which every sufficiently accurate linear predictor in the span of the training data must misfit. So good generalization here cannot be explained by good training fit. A reflection argument extends this limitation to all learned linear predictors when no signal spike orientation is preferred in advance. We then characterize when a single epoch of zero-initialized stochastic gradient descent can reach predictors that generalize well, despite this requiring actually ascending in empirical training loss. An explicit pair of examples is given to separate the geometry forcing misfitting from whether such misfitting predictors are learnable by a single epoch of constant-rate SGD.

Primary area:
learning theory
Keywords:
overparameterized, misfitting, generalization, single-epoch training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NZ1wOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between training loss and generalization in high-dimensional linear regression. The authors show that, under specific spectral and dimensional conditions, well-generalizing predictors must incur large training losses. They further establish conditions under which one-pass SGD achieves vanishing test error despite increasing training loss, and demonstrate how label-irrelevant signal directions can affect SGD's generalization performance.

Critical strengths
  1. The paper provides formal guarantees showing that well-generalizing predictors necessarily incur large training losses under the specified assumptions.

  2. The authors derive necessary and sufficient conditions for the consistency of one-pass SGD in the considered setting.

  3. The paper illustrates how label-irrelevant high-variance features can affect SGD's generalization performance.

Critical weaknesses
  1. The theoretical results rely on a highly specialized setting, including noiseless Gaussian linear regression, specific covariance structures, and restrictive dimensional scaling conditions. It is unclear whether the observed phenomenon persists under more general or realistic settings.

  2. The relationship between overfitting and generalization in high-dimensional regression has been extensively studied. Although the paper establishes a stronger theoretical characterization, it remains unclear what fundamentally new insights it provides beyond existing understandings of overfitting and regularization.

  3. The analysis focuses on one-pass SGD with a batch size of one and a constant learning rate. Given the extensive literature on online SGD for linear regression, the authors should better clarify the novelty of their results and whether the conclusions extend to more common optimization settings.

  4. The experiments primarily focus on synthetic settings closely aligned with the theoretical assumptions. Additional experiments examining the sensitivity to sample size, covariance structure, and optimization settings would help establish the broader relevance of the findings.

Questions for authors
  1. How do the main findings differ fundamentally from existing results on overfitting and regularization in high-dimensional linear regression? What new insights does the misfitting phenomenon provide beyond the established trade-off between training overfitting and generalization?

  2. How essential are the specific spectral and dimensional scaling assumptions? Does the misfitting phenomenon persist when these assumptions are relaxed, particularly when the sample size increases relative to the feature dimension?

  3. How do the results differ from existing analyses of online SGD for linear regression? Do similar conclusions hold for multi-pass SGD, mini-batch SGD, or decaying learning rates?

  4. Can the authors provide empirical evidence of the misfitting phenomenon under more realistic data distributions or learning settings, beyond the synthetic constructions used in the paper?

1 comment
AuthorsOpenReview ↗

Misfitting isn't the same as benign overfitting or where traditional regularization would take us. And misfitting happens quite generally, not just when our theorem's assumptions hold.

Thank you for your review and for asking how misfitting differs from ordinary regularization.

1. What does misfitting add to the usual trade-off between training fit and generalization?

Regularization has always given up some training fit for better generalization.

But how much training fit can we give up with positive ridge? Positive ridge's training loss can never exceed that of the zero predictor. In the regime of out Theorem 1, a predictor whose coefficients are a weighted sum of training inputs (a span predictor) must incur many times that loss to beat the zero predictor's test error by a fixed fraction. This rules out positive ridge, however its penalty is chosen, even from the data. Our submission shows that this effect arises even with noiseless labels and fresh inputs from the same distribution.

What does work is moving the other way. Guarded negative ridge learns both example problems of our Corollary 4. In the easier one, a single large-step SGD pass also learns the target while ending with higher training loss than it started with.

Why is the fit so much worse on the training points? A weighted sum of training inputs picks up their nuisance features (weak features unrelated to the label). On a training point those features line up with themselves. On a fresh point they are independent and contribute much less error. Ranade and Sahai showed this mechanism with one signal direction. Our Theorem 1 extends it to random inputs with several signal directions.

2. How essential are the assumptions, particularly as sample size increases?

More data can remove the need to misfit. Suppose the sample size nn far exceeds the number of signal directions and is far smaller than the number of nuisance directions. With common signal variance γ\gamma and d0d_0 unit-variance nuisance directions, the misfitting range is d0/γ2≪n≪d0/γd_0/\gamma^2\ll n\ll d_0/\gamma. Far below it even the best span predictor learns little. Far above it the interpolator predicts well (Appendix G). For unequal strengths, the weakest and strongest signals set the lower and upper ends of Theorem 1's guarantee, respectively.

We use Gaussian sampling to control inner products in the training-loss proof and to obtain the exact SGD recurrence. The experiment below shows the effect with non-Gaussian inputs, correlated features and a decaying background spectrum. The misfitting phenomenon is far more general than the proof here.

3. What is new about online SGD? What extends to mini-batches, extra passes and decaying rates?

The upper bounds of Varre et al. and Zou et al. use steps too small to learn even the strongest signal in one pass here. Could a larger step work? In the misfitting regime, one constant rate learns every target precisely when sample size grows faster than total signal variance divided by the weakest signal variance (our Theorem 3).

Mini-batches and extra passes can change which predictor SGD finds. Starting from zero, linear SGD stays in the training span even with minibatches and learning-rate schedules, so any accurate (i.e. good generalization) iterate must fit poorly. If no single-example step enlarges the error on the example it uses, test error stays near that of predicting zero for any number of passes (Proposition E.1).

In our Corollary 4's easier example, we find that a late rate cut preserves vanishing expected error. Raising only variances in directions the label ignores gives the harder example "B". There, no nonnegative schedule fixed before training makes one-pass expected error tend to zero as the problem grows. Data-dependent (but not precient) rates remain open.

4. What happens with less restrictive data and different optimizers?

We tested a Fourier kernel on a phase xx drawn unevenly around a circle. The signal is proportional to sin⁡(2πx)+sin⁡(4πx)\sin(2\pi x)+\sin(4\pi x). Uneven sampling gives neighboring sine features a correlation of 0.25. The 32,000-plus background frequencies have variances that decrease from 13.5 to 0.5.

How well could we fit the data if we required test error at most one quarter of the zero predictor's? The known covariance lets us compute the minimum training loss in the span. With 64 examples, even that minimum exceeded the zero predictor's training loss on all 64 draws (median ratio 2.33). With 128 examples, it fell below that baseline on every draw (median ratio 0.28). Including a background frequency correlated with the signal preserved the 64-example result on every draw.

With 64 examples the minimum-norm interpolator fit every label and had median test error 0.632 (predicting zero gives 1). One pass of SGD gave 0.081 despite median training loss 13.8 times that of predicting zero. Adding momentum of 0.1 gave 0.077. Adam scales each coordinate's update separately. At the same step size its median test error went from 0.863 in the sine coordinates to 0.143 after a random rotation. We will add this experiment and the sample-size comparison to the revision.

Reviewer_cyYFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies benign misfitting in noiseless i.i.d. Gaussian regression with unequal signal spikes. It identifies a regime where accurate span predictors must incur large training loss. It also characterizes when one-pass constant-step SGD learns and constructs two problem families with similar optimal span performance but different SGD learnability. Synthetic experiments support the theoretical results.

Critical strengths
  1. A rigorous extension to an i.i.d. multispike setting. Theorem 1 shows that benign misfitting persists when training and test examples come from the same distribution. The guarantee applies simultaneously to all sufficiently accurate span predictors, making it a structural statement about the learning problem. The reflection argument further clarifies the role of the span restriction.

  2. A precise characterization of constant-step SGD. Theorem 3 identifies an additional obstruction involving the total signal strength relative to the weakest signal direction. The A/B construction illustrates that similar optimal span performance does not imply similar SGD performance. The proofs are detailed, and the numerical results help explain the distinction between the geometric and algorithmic conclusions.

Critical weaknesses

Moderate conceptual novelty. The central benign-misfitting phenomenon and its attainability by one-pass SGD were already established in the single-spike model of [1]. Theorem 1 extends the same nuisance-amplification mechanism to random Gaussian data, using concentration of Gram matrices in place of exact orthogonality. The SGD analysis also builds on the existing Gaussian second-moment recurrence in [2]. Thus, the contribution primarily develops an existing framework. I find the multispike characterization and A/B separation meaningful, but the extent of the new technical ideas could be explained more explicitly, particularly in relation to classical least-squares SGD analyses.

Questions for authors
  1. Technical contribution. Could the authors clarify the main technical advances in Theorems 1 and 3 beyond the geometric argument in [1] and the Gaussian SGD recurrence in [2]? In particular, how does the condition qn→0q_n \to 0 relate to classical condition-number-based analyses of least-squares SGD, such as [3], when restricted to the signal block? What additional arguments are needed to establish necessity and sufficiency in the presence of the nuisance directions?

  2. Scope of the A/B separation. Appendix H does not extend the separation to varying step sizes. Is there evidence that family B remains difficult under a simple decreasing-step schedule, or could such a schedule remove the separation? This would help clarify whether the obstruction is specific to constant step sizes or reflects a broader difficulty for one-pass SGD.

[1] Gireeja Ranade and Anant Sahai. The Fourth Quadrant: A Stylized View of Benign Misfitting. arXiv:2608.01032v1, 2026.

[2] Blake Bordelon and Cengiz Pehle­van. Learning Curves for SGD on Structured Features. ICLR, 2022.

[3] Prateek Jain, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Accelerating Stochastic Gradient Descent for Least Squares Regression. COLT, 2018.

1 comment

Thank you for your review and for asking what changes between the single-spike problem and ours.

1a. What changes in Theorem 1 when there are several signal directions?

In the single-spike example of Ranade and Sahai [1], multiplying the minimum-norm interpolator's predictions by a constant recovers the signal. What if we try the same fix here? In the examples of our submission's Corollary 4, the label depends on two directions that need different corrections. Even the best multiplier leaves limiting test error 1/2, where predicting zero gives 1. We can still learn an accurate linear predictor in both examples with guarded negative ridge (Appendix F.2) or SGD.

The geometric argument in [1] uses training inputs with exactly orthogonal nuisance components (the many weak features unrelated to the label). Our Theorem 1 controls their inner products when we draw the examples independently. With high probability, every accurate predictor whose coefficient vector is a weighted sum of the training inputs has training loss many times that of predicting zero. The same draw gives this guarantee for every unit target (the "unit" nature is just setting units so that the regression targets themselves are neither exploding nor vanishing in asymptotic families). We also give a formula for the best test risk in the span whose relative error tends to zero uniformly over such unit targets.

1b. How does Theorem 3 relate to Jain et al. [3], and why are its two conditions necessary and sufficient?

Stepping back, why should a direction the label ignores matter to SGD? Each update uses that feature too. Going from family A to B in our Corollary 4, we increased only the variances of such label-irrelevant directions. For that label, the best test risk in the training span stayed the same to leading order. One pass of constant-rate SGD can learn A, while in B no constant rate can give vanishing expected error.

On the signal coordinates, your intuition about Jain et al. is right. Our trace condition qn→0q_n\to0 says that the sample size grows faster than their distributional condition number. Write Γ\Gamma for the signal covariance, with largest and smallest variances γ1\gamma_1 and γp\gamma_p. Then qn=tr⁡(Γ)/(nγp)q_n=\operatorname{tr}(\Gamma)/(n\gamma_p). For Gaussian inputs, κ(Γ)=(tr⁡Γ+2γ1)/γp\kappa(\Gamma)=(\operatorname{tr}\Gamma+2\gamma_1)/\gamma_p, giving qn<κ(Γ)/n≤3qnq_n<\kappa(\Gamma)/n\leq3q_n.

For tail-averaged accelerated SGD, the sample requirement in [3] on the full covariance already exceeds d0d_0, the number of unit-variance nuisance directions. Here n≪d0n\ll d_0. For plain SGD, steps of order 1/d01/d_0 are too small even to learn the strongest signal in one pass. Bordelon and Pehlevan [2] give the expected-error recurrence at a specified step size. Theorem 3 uses it to decide whether any constant step can make the last iterate accurate for every unit target.

A successful rate has to do two things. It must be large enough to learn the weakest signal in nn updates and small enough to control the error those updates add in the signal and nuisance directions. Even a step size that works can amplify existing nuisance error. We can afford this because nuisance error starts at zero and we only have to bound what accumulates in one pass. The conditions qn→0q_n\to0 and nγp2/d0→∞n\gamma_p^2/d_0\to\infty let us choose such a step.

To see why both conditions are needed, take the target in the weakest direction. At a step size of 2/(tr⁡Γ+2γp)2/(\operatorname{tr}\Gamma+2\gamma_p) or larger, the expected squared error in the original signal coefficients cannot decrease. Below that threshold, SGD learns the weak signal too slowly if qnq_n stays bounded away from zero. If nγp2/d0n\gamma_p^2/d_0 stays bounded, any step fast enough to learn the weak signal creates nuisance error early in the pass that persists to the end. This gives a lower bound even for the best constant step size chosen for the problem.

2. Can a decreasing learning rate remove the A/B separation?

The separation persists. Your question led us to go beyond Appendix H of our original submission for schedules fixed before training. A late tenfold rate cut preserves vanishing expected error in A. In B, no nonnegative schedule fixed before training makes expected error tend to zero as dimension and sample size grow.

Even so, the cut did help B at the size we tested. With 4,096 examples, cutting the rate four-fifths of the way through the pass reduced B's expected test error from 0.148 to 0.080. In A it went from 0.015 to 0.018. We tuned the starting rates separately. B remains hard because controlling error in its strong directions limits both the progress we can make with small steps and the number of large steps we can take. Small steps leave some weak signal unlearned. Making up the difference with large steps leaves nuisance error at the end of the pass. We will include the proof in the camera-ready. Whether rates adaptively chosen from earlier observations can help in B remains open.

Reviewer_uDVWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends the benign misfitting phenomenon previously established by Ranade and Sahai (2026), where accurate generalization can require training loss much larger than that of the zero predictor. While the prior work considered a stylized single-spike model with fixed training inputs, this paper studies noiseless overparameterized linear regression with multiple signal directions of unequal strengths and i.i.d. Gaussian training and test data.

The paper makes three main contributions. First, it proves that, in a particular high-dimensional regime, every sufficiently accurate predictor in the training-data span must have large training loss, despite the existence of accurate predictors (Theorem 1). Second, it extends this necessity beyond span predictors under unknown, rotationally symmetric signal orientations (Theorem 2). Third, it provides necessary and sufficient conditions for one-pass constant-learning-rate SGD to achieve vanishing expected test risk (Theorem 3). Notably, it shows that strong directions irrelevant to the labels can prevent SGD from learning, even when accurate predictors exist.

The theoretical results are supported by proofs, exact Gaussian SGD dynamics, and synthetic experiments, including matched examples separating statistical learnability from the success of one-pass SGD. The main novelty lies in extending benign misfitting to a more general Gaussian multispike setting and characterizing the additional optimization challenges introduced by heterogeneous signal strengths, rather than discovering the misfitting phenomenon itself.

Critical strengths

1. Extension of benign misfitting to a more general setting.

The paper extends previous results on benign misfitting from a stylized single-spike model to i.i.d. Gaussian data with multiple signal directions of unequal strengths. Importantly, it proves that poor training fit is necessary for accurate span predictors, rather than merely exhibiting an example where it occurs. This strengthens the theoretical understanding of the phenomenon and shows that it is not simply an artifact of the earlier fixed training design.

2. Necessary and sufficient conditions for one-pass SGD consistency.

The paper derives an exact asymptotic criterion for when one-pass constant-learning-rate SGD achieves vanishing expected test risk uniformly over all targets. The result is particularly interesting because it identifies how strong directions irrelevant to the labels can prevent SGD from learning, even when accurate predictors exist. The constructed examples illustrate this separation between statistical learnability and the limitations of constant-rate SGD.

Critical weaknesses

1. Limited understanding beyond linear models.

The paper focuses on linear regression, and its SGD analysis is restricted to a single pass with a constant learning rate. It is unclear whether the misfitting phenomenon persists when training more expressive models such as MLPs or nonlinear kernel methods on the same data. Can these methods achieve both low training and test errors, or do they exhibit similar misfitting or overfitting behavior? Understanding whether the phenomenon extends beyond linear predictors would substantially strengthen its significance.

2. Unclear practical relevance of the spectral regime.

The main results rely on a specific high-dimensional regime with many nuisance features and carefully separated signal strengths. Although the authors provide finite-dimensional synthetic examples, it remains unclear whether this regime arises in realistic learning problems. Can the authors identify practical settings or applications in which these spectral conditions naturally occur? Establishing such a connection would help clarify the broader relevance of the theoretical findings.

Questions for authors

Please see weaknesses

1 comment

Thank you for your review and thoughtful questions.

1a. Can an MLP or a kernel method fit the same data and still predict well?

Such predictors definitely exist on the multispike examples in our submission. We can build them by hand. Can ordinary training find them from a random initialization? That second question is open (but looks dubious). Start with the accurate linear predictor learned by guarded negative ridge (Appendix F.2). A ReLU network can correct its prediction at each training point in a region so narrow that a fresh input almost never lands there. This fits every label exactly with arbitrarily little change in test error. Adding narrow Gaussian kernels centered on the training points to the linear predictor works for the same reason.

We tried training a ReLU network on a simpler problem with one signal direction. Following Ranade and Sahai (2026, Appendix E), the idea was to take the median of three linear predictions. Imagine training each predictor on a different third of the data. A point on which one predictor makes a large training error is fresh for the other two. If both predict it accurately, so does the median.

In our preliminary experiment, every network saw all the data and all layers were trainable. We used the same architecture in two groups. In one, we set the upper-layer weights by hand to compute the median. In the other, those weights started at random. We tried several learning rates and schedules on pilot data and tuned the two groups separately. Each data set had 96 examples, one signal direction of variance 100 and 48,000 noise directions of variance 1. On 30 new data sets, both groups reduced training loss below 1% of the zero predictor's. On fresh data, median test error was 0.13 with the median initialization and 0.73 with random upper layers (predicting zero gives 1). A directly trained linear model also gave 0.73. Starting from random weights, we could fit the labels closely. None of our 60 runs reached our test-error target of 0.15. (We could, of course, find misfitting predictors by SGD that did.)

We also tested a fixed Fourier kernel on a separate problem. Its features are nonlinear functions of one scalar input, and the predictor is a linear combination of those features. The signal was a sum of two slow sine waves, and the kernel allowed thousands of weaker, higher frequencies. With 64 examples, the minimum-norm kernel interpolator had median test error 0.632. One pass of SGD gave 0.081, with median training loss 13.8 times that of predicting zero. So SGD found a misfitting predictor (as our results would have suggested would happen). Could a different choice of coefficients avoid the trade-off? On all 64 data sets, every predictor in the span of the observed feature vectors with test error at most 0.25 had training loss at least 1.40 times that of predicting zero.

1b. Does the restriction to one pass and a constant rate matter?

The training-loss conclusions carry over. For linear SGD started at zero, extra passes, mini-batches and changes of rate keep the coefficient vector in the span of the training inputs. Any accurate iterate must therefore fit poorly in the regime of our submission's Theorem 1. With single-example steps small enough never to increase the error on the example just used, test error stays near that of predicting zero however many passes we make (Proposition E.1). We also extended the one-pass analysis for the paper's example with strong directions unused by the label. There, no nonnegative step schedule fixed before training makes expected test error tend to zero as the problem grows. SGD can't work at all for that example even with a learning rate schedule.

2. Where would a few strong directions and many weak ones come from?

A noisy sensor array gives a classic physical example. Imagine many sensors measuring the same temperature, each with independent noise of equal variance. A temperature change moves all the readings together. Differences between sensors contain only noise. Adding sensors makes the common direction stronger while each noise direction retains the variance of a single sensor. Ranade and Sahai, §2.1

If the sensors measure distinct linear combinations of a few underlying quantities, the clean readings lie in a low-dimensional subspace. (Think latent space) Each well-measured combination gives a spike in the covariance. This is a linear version of the noisy manifold hypothesis. Measurement noise spreads the observations around that subspace. Whether misfitting occurs also depends on the sample size. There must be enough independent recordings to estimate the shared signal while the many noise directions together still dominate the fit to the training points. The spikes can be equally strong for our result. We use unequal spike strengths to show how a high-variance direction the label ignores can prevent one-pass SGD from learning a weaker direction it needs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.