acceptodds
Under review as a conference paper at ICLR 2027

A Data-Centric Approach to Improving LLM Generalization

Abstract

Simplicity bias (SB)—the tendency of gradient-based methods to prioritize dominant features—shapes how deep networks learn and generalize. Despite its demonstrated impact on generalization in CNNs, the role of SB in transformer generalization remains largely unexplored. We theoretically analyze a multi-head self-attention model and prove that Sharpness-Aware Minimization (SAM) reduces simplicity bias in transformers. Together with the empirical observation that SAM improves generalization, this suggests that reducing SB may contribute to improved generalization. Our theory further reveals that examples associated with dominant features exhibit rapid early loss drops, while examples associated with less-represented features lag behind. This makes SB observable through training dynamics and motivates a simple data-centric approach: we partition data based on early loss dynamics and amplify less-represented examples through upsampling or synthetic generation. For synthetic generation, we find that generated examples should remain sufficiently similar to their corresponding real data, achievable with diffusion LLMs. Across Llama2 (7B), Phi2 (2.7B), Llama3.2 (1B), Gemma3 (1B), and Qwen3 (0.6B), our approach consistently improves performance with AdamW and Muon, achieving a 6x wall-clock speedup and 2x memory reduction over SAM. Improvements reach 8.3% on mathematical reasoning, 26% on code generation, and 4.1% on instruction following.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sharpness-Aware Minimization (SAM), Simplicity Bias, LLM Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Gp9ROpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to understand simplicity bias in transformers and its implications on generalization empirically and theoretically using SAM. It then introduces an upsampling mitigation strategy to mitigate simplicity bias and improve ID and OOD generalization in transformers.

Critical strengths

Theory regarding SAM reducing simplicity bias in transformers is novel and not previously articulated. Note that "articulated" means not explicitly discussed because [1] studied SAM impact on simplicity bias theoretically in CNNs. Since, this paper extended the theory to transformers, it still counts as a contribution. But note that I have not fully gone through the exact proof of the proposed theorems and I would depend on my fellow reviewers for their expertise on the same. However, refer to Point 2 in Critical Weakness for further discussion on this.

Critical weaknesses

This paper has severe critical weaknesses regarding novelty. I state them as follows:

  1. Authors portray their paper as if they are the first to suggest reducing simplicity bias to improve generalization in transformers. However, we already know from prior literature that "simplicity bias" causes "spurious correlations" which essentially decreases the generalization performance [2,3], and it has also already been extensively empirically shown in transformers [4,5]. This essentially means simplicity bias degrades generalization even for transformers because spurious correlations degrade generalization and spurious correlations are caused by SB (a simple transitive logic). The authors have completely removed any discussion about this analysis which in my opinion is extremely important to properly be thought about before claiming anything. Hence, the authors claim on "the role of SB in transformers remain largely underexplored (line 14)" is false. I agree no one has previously shown theoretically that SAM reduces SB in transformers, however this being the only novel contribution of this paper is not publication worthy at ICLR. The authors must clearly clarify how their work really differs from prior works on SB causing spurious correlations which reduces generalization.

  2. Apart from not discussing works on spurious correlations, the authors fail to discuss [6] which already shows that SAM increases balanced feature learning, i.e., it adaptively suppresses well-representated features which gives remaining less-representative features opportunity to be learned. The authors theory merely provides further support to this and there is nothing new from what is already known.

  3. The authors claim that "examples associated with dominant features exhibit rapid early loss drops, while examples associated with less-represented features lag behind (line 18-20)" is already well established from spurious correlations papers and there is nothing new about this. It is merely stating an already known facts. The authors again need to clarify what exactly is new in their current paper which is not already known.

  4. The authors only focus on generative tasks for showing reducing SB improves generalization. However, SB causes spurious correlations and there are ample amount of papers which provide mitigation strategies to reduce "SB/spurious correlations" in tasks like WaterBirds, MNLI, CelebA, MultiNLI. The authors must show whether their proposed strategy outperforms the mitigation strategies in these papers or not and how expensive/cheap their method is over these. I expect them to atleast show performance and timing comparison against 3-4 well-known strategies, especially - DFR, JTT, CNC, for the above listed tasks. The authors can easily extrapolate their proposed method to these new datasets as their method is just clustering followed by upsampling, which is topic-agnostic.

  5. In Table 5 and Fig 6, CL has higher generalization performance than FT+UP (proposed method by the authors) for 6 out of 7 datasets. This means the proposed method by itself is not really performing well. This is another major issue since their original method cannot outperform well known strategies. The authors need to provide a mechanistic analysis where their method is performing poorly via experiments and why. Furthermore, no timing comparison has been provided to compare UP with CL and OHEM (the authors have only compared with SAM which we already know is very expensive). Also, regarding UP - why have the authors only used Pythia-70M for clustering process? Why no other model of similar size from a different model family was used?

  6. The authors should also provide clear examples of well-represented and less-represented samples that they detect via their clustering operation. It is highly informative to know what kind of commonality well-represented samples have and how it differs from less represented samples.

Questions for authors

Please see Critical Weaknesses.

[1] Changing the training data distribution to reduce simplicity bias improves in-distribution generalization, Arxiv 2026

[2] Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness, TMLR 2024

[3] The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning, TMLR 2026

[4] Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training, Arxiv 2025

[5] Trapped by simplicity: When Transformers fail to learn from noisy features, ICLR 2026

[6] Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning, ICLR 2024

Overall, this paper is just merely stating already known things even in transformers (as discussed in Critical Weaknesses) and does not offer any new novel insights other than theoretically relating SAM reducing SB (something which I hope other reviewers can confirm is correct or not based on their theoretical expertise). If the authors wish to submit a rebuttal, I hope they could properly clarify all the points raised by me in the Critical Weaknesses section. (PS: I have not used any AI tool to review your paper).

4 comments
AuthorsOpenReview ↗

Rebuttal response part-1 (of 2)

We thank the reviewer for the comments. We respectfully disagree with the assessment of novelty, and several statements in the review do not match the paper or the cited works.

1. Novelty relative to [1–5].

The reviewer’s argument rests on an invalid inference. Simplicity bias can lead to reliance on spurious features when such features are present, but simplicity bias itself is a broader and separate concept. This distinction is fundamental to our setting.

The spurious-correlation literature [2, 3] studies a different problem: robustness under distribution shift, where a spurious attribute correlates with the label and the metric is worst-group or OOD accuracy. [2] proposes a regularizer evaluated on such benchmarks, and [3] is a survey of that area. Our theoretical setting contains no spurious features at all: every eigen-direction of the data covariance is predictive of the label. The same holds in our experiments, which involve no spurious attribute or group structure. The issue we identify is independent of spurious correlations: certain predictive features are learned late.

The reviewer’s own references contradict the transitive argument in the review:

  • [2] notes that simplicity bias is regarded as beneficial for in-distribution generalization in several cases, and harmful under distribution shift.
  • [1] states that prior simplicity-bias mitigation had been shown effective for OOD settings, and that showing a benefit without spurious correlations or distribution shift was new. It proves this for a two-layer CNN and evaluates image classifiers.

That reducing simplicity bias improves generalization when nothing spurious is present therefore does not follow from the spurious-correlation literature, and it has not been shown for LLMs.

Neither [4] nor [5] shows what the review attributes to them:

  • [4] removes entity and event biases acquired in pre-training, evaluated under OOD perturbation on two binary reasoning benchmarks. It does not study simplicity bias or training dynamics.
  • [5] trains small transformers on Boolean functions with noisy features and characterizes simplicity by the sensitivity of the learned function. It contains no SAM, no analysis of training dynamics and no LLM fine-tuning, and lists natural language as future work.

Finally, results established for CNN classifiers, such as [1] and [6], cannot be assumed to extend to transformers. Our analysis considers attention heads with trainable key, query and value parameters, which enter the output multiplicatively. Even in its reduced form (Lemma 1), the model is a three-layer network with cubic interactions between its weights, and its dynamics pass through 2ᵈ fixed-point manifolds saddle to saddle (Appendix A.3). Analyses of two-layer CNNs do not cover these dynamics, which is why the transformer setting has required separate technical treatment (Zhang et al., 2025).

Our contributions, which appear in none of these works, are:

  • to our knowledge, the first analysis of SAM’s training dynamics in multi-head attention, proving that SAM reduces simplicity bias (Theorems 3 and 5);
  • a data-centric method derived from it, improving five LLMs across three domains at 6× lower wall-clock cost than SAM;
  • the finding that synthetic augmentation helps only when it stays close to the source.

The review treats the theorem as our only contribution and does not engage with the method or experiments. In the revised manuscript, we have added a discussion of these works in §2.

2. Relation to [6].

The claim that our theory “merely provides further support” to [6] is incorrect.

  • Setting: [6] studies image classifiers (ResNet-18, small MLP and CNN models), with no transformers or language models.
  • Question: [6] explicitly targets out-of-distribution and transfer settings, measured by linear-probe feature quality.
  • Result: [6] is empirical, using a 4-dimensional toy problem and a last-layer-only variant of SAM. It has no theorem and no analysis of training dynamics.
  • Method: [6] proposes none.

A compatible empirical finding on image classifiers corroborates our result; it does not anticipate it.

3. Early loss as a signal.

In the spurious-correlation literature, early-training signals are used in classifiers to identify groups defined by a spurious attribute. We derive the signal from the dynamics of attention and apply it to generative fine-tuning, where there are no such groups and our clusters cover 28–63% of the data (§7). Our contribution is also not simply the use of early loss, but its theoretical motivation and application to feature learning in LLMs.

AuthorsOpenReview ↗

Rebuttal response part-2 (of 2)

4. Comparison with DFR, JTT and CNC on Waterbirds, CelebA and MultiNLI.

These are outside the scope of our paper. The benchmarks are image and text classification tasks with annotated spurious attributes, evaluated by worst-group accuracy. The baselines are group-robustness methods for classifiers (last-layer retraining, upweighting misclassified examples, contrastive learning on classifier representations) and are not defined for generative fine-tuning. The closest applicable baseline, loss-based upweighting of hard examples, is OHEM, which we compare against: it underperforms standard fine-tuning (27.7 vs 28.2) while our method reaches 30.0 (Table 5). We make no claim about group robustness, so this comparison would not test any claim in the paper.

5. Table 5 and curriculum learning.

The statement that CL beats our method on “6 out of 7 datasets” is incorrect. Table 5 has six benchmarks; CL is higher on four and ours on two, including MMLU-Math, where CL falls to 13.9, well below standard fine-tuning (27.2). More importantly, we do not claim to replace CL. We state in §2 and §7 that the methods are complementary, and our upsampling improves every baseline it is added to: standard fine-tuning (28.2 to 30.0), CL (31.3 to 32.4) and OHEM (27.7 to 30.9).

On cost (updated in §7 and Table 5), our overhead is a 1.38× increase in steps plus ~0.3% for the proxy, corresponding to 1.42× standard FT wall-clock time end-to-end on Llama3.2-1B MathInstruct. By comparison, CL runs in 0.98× (within run-to-run variance of FT), while OHEM requires 1.93× (and OHEM+UP 2.55×) due to per-step forward passes over its 2,048-example pool. Appendix E.1 analyzes when and why the method helps less.

6. Proxy model. The statement that only Pythia-70M was used is incorrect. Fig. 7(a) and Table 8 also use Phi-2 (2.7B, a different family) as the proxy, and both yield gains (57.3 and 58.0 vs 56.5).

7. Cluster examples. These are already in the paper: Appendix H and Table 14, referenced from §7.

Thanks for the responses. My concerns are resolved. I have raised my score.

AuthorsOpenReview ↗

Thank you and brief follow-up on remaining concerns

Dear Reviewer,

Thank you for reading our responses, confirming that your concerns are resolved, and raising your score. All corresponding updates, including the related work discussion of [1–6] in §2 and the wall-clock comparisons with curriculum learning and OHEM in §7 and Table 5, are included in the revised manuscript. We further provide a brief summary of the problem and our contributions here.

We note that the updated review still carries a weak-reject rating (2) and a soundness score of 1. Please let us know if any concern remains regarding the paper’s soundness or evaluation, and we will clarify it before the discussion closes. If none remains, we would be grateful if you could consider updating the rating and the soundness score to reflect your current assessment.

Best regards,

The Authors

Reviewer_aWUYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Key insight: Examples containing less-represented features tend to exhibit slower loss reduction during early training. Their loss trajectories can therefore serve as a proxy for identifying underrepresented features.
  • Theoretical contribution: The paper analyzes multi-head linear self-attention and theoretically shows that Sharpness-Aware Minimization (SAM) reduces simplicity bias by making feature learning more uniform.
  • Proposed method: Train a small proxy model briefly, cluster examples based on their early loss trajectories, and amplify the high-loss cluster through targeted upsampling or synthetic data generation.
  • Synthetic augmentation: The authors use diffusion LLMs to generate examples that preserve the original features while introducing variations, achieving better results than simple duplication.
  • Experimental results: Across five LLM families (0.6B–7B), the method improves mathematical reasoning (up to 8.3%), code generation (up to 26%), and instruction following (up to 4.1%). Improvements are observed with both AdamW and Muon.
Critical strengths
  1. Strong connection between theory and practice: The paper theoretically analyzes how SAM reduces simplicity bias in transformers and translates this insight into a practical data-centric method. This provides a meaningful motivation for the proposed approach.
  2. Practical effectiveness and efficiency: Experiments across five model families, multiple tasks, and two optimizers show consistent improvements. The method avoids SAM's expensive per-step optimization overhead, making it attractive for practical LLM fine-tuning.
Critical weaknesses
  1. Limited theoretical generalizability: The theoretical results rely on a simplified multi-head linear attention model and early-training assumptions. It remains unclear how well these results explain simplicity bias in modern LLMs, weakening the theoretical justification for the proposed method.
  2. Limited validation at scale: Experiments focus on LoRA fine-tuning models up to 7B parameters on a limited set of tasks. It is unclear whether the gains generalize to full-parameter training or large-scale pretraining, which limits the strength of the paper's broader claims.
Questions for authors
  1. Clarification: How robust are the improvements across random seeds and evaluation benchmarks, particularly given the use of a proxy model to identify underrepresented examples?
  2. Clarification: Can the authors provide stronger evidence that the proposed method improves generalization specifically by reducing simplicity bias, rather than through generic data reweighting or increased training exposure?
  3. Future work: Can the approach scale to full-parameter training or large-scale pretraining, beyond the current LoRA fine-tuning experiments?
4 comments
AuthorsOpenReview ↗

Rebuttal response part-1 (of 2)

We thank the reviewer for the comments and address each point below.

W1. Theoretical generalizability.

Multi-head linear self-attention on in-context regression is the standard tractable setting for analyzing transformer training dynamics (Garg et al., 2022; Von Oswald et al., 2023; Ahn et al., 2023; Zhang et al., 2025). Rank-one key and query matrices capture most of the behavior of the general rank-R case (Zhang et al., 2025). A complete theory of training dynamics for nonlinear multi-layer LLMs does not exist, so this criticism applies equally to the entire literature we build on.

More importantly, the theory is not our evidence that the method works on LLMs. Its role is to identify a mechanism and produce testable predictions, which we then test directly on pretrained nonlinear LLMs:

  • Prediction 1: early loss separates examples by feature representation. Clustering early loss trajectories yields groups that differ by format and reasoning modality (Appendix H), and not by length (Appendix E.2) or label noise (§7).
  • Prediction 2: amplifying the lagging group helps, amplifying the fast group hurts. This is what we observe (Table 10).
  • Prediction 3: the loss shifts away from dominant features. Loss on well-represented examples rises by 25% and loss on less-represented examples falls by 78% (§7).

As stated in our limitations (Appendix I), the method itself does not rely on any of the theoretical idealizations.

W2. Validation at scale.

Our claims concern fine-tuning, and within that scope the evaluation is broad: five model families from 0.6B to 7B parameters, three domains, eleven benchmarks and two optimizers. Average performance improves in every configuration we tested.

We use LoRA, following the established setup of Nguyen et al. (2024b), because full-parameter fine-tuning of all five models across three domains, two optimizers and our ablations exceeds our compute budget (4× A6000); LoRA is also the dominant fine-tuning practice at this scale. Our math experiments use a high adapter rank (r = 256, Table 3), so the adapters are far from a minimal-capacity setting.

Two further points:

  • Nothing in the method is specific to LoRA. It is a one-time preprocessing step on the data, costing about 0.3% of baseline compute, after which any standard training procedure is used unchanged. Our theory analyzes full-parameter training of the attention weights, not low-rank adapters.
  • Scale is the motivation for the method. SAM, the optimizer-level alternative, did not finish fine-tuning a 2.7B model in six days on 4 GPUs (Fig. 2). A data-centric approach is what makes reducing simplicity bias feasible at LLM scale.

In the revised manuscript, we state the scope (supervised finetuning) explicitly in the abstract, introduction, and conclusion. Full-parameter finetuning and pretraining are natural next steps, which we now highlight alongside RL as future work (Appendix I).

AuthorsOpenReview ↗

Rebuttal response part-2 (of 2)

Q1. Robustness across seeds, benchmarks and the proxy.

  • Seeds. On code, gains for Qwen3 hold across seeds: the average rises from 29.42 ± 2.28 to 34.30 ± 1.00, and HumanEval from 17.10 ± 1.69 to 26.83 ± 6.04 (Table 6). On math with Phi-2, per-benchmark standard deviations under upsampling are 0.1–1.6 points (Table 8).
  • Benchmarks. Average accuracy improves for every model on math (Table 4), code (Table 6) and instruction following (Table 7), under both AdamW and Muon (Table 8), and under 10% and 20% label noise (Fig. 5). Individual benchmarks vary, as expected, but the direction of the average is uniform.
  • Proxy. The 70M proxy and the 2.7B target model used as its own proxy both yield gains (57.3 and 58.0 vs 56.5, Table 8), although the smaller proxy selects 28% fewer examples (§7.1). Gains are also stable across 5, 10 and 15 proxy checkpoints (Fig. 7c).

Q2. Evidence that the gains come from reducing simplicity bias.

Both alternatives the reviewer names are tested directly in the paper.

  • Increased training exposure. Compute-matched random upsampling, with the same number of added examples and steps, gives no gain (54.2 vs 54.5 on Qwen3; 28.3 vs 28.2 on Llama3.2, Table 10).
  • Generic reweighting. The effect depends on which examples are amplified, in the direction the theory predicts: less-represented examples help (55.7; 30.0), well-represented examples hurt (53.0; 26.6) (Table 10).
  • Hard-example reweighting. OHEM underperforms standard fine-tuning (27.7 vs 28.2), while our method reaches 30.0 (Table 5).
  • Features, not specific examples. Replacing duplicates with synthetic variants that stay close to the source improves further (up to 8.3%, Fig. 4 in the revised PDF; Fig. 3 originally). Synthetic variants that drift from the source fall below standard fine-tuning (55.7 vs 56.8, Table 11). The benefit tracks whether the underlying features are preserved.
  • Direct measurement. The loss shift reported under W1 (Prediction 3) is the signature of reduced reliance on dominant features.

Q3. Scaling to full-parameter training and pretraining.

Please see W2. The method is independent of how the target model is trained, and its overhead is a brief run of a 70M proxy, so we see no obstacle in principle. We leave the empirical study to future work and have scoped our claims explicitly to supervised finetuning.

AuthorsOpenReview ↗

Follow-up on our rebuttal and revised manuscript

Dear Reviewer,

Thank you again for your positive feedback on our work. We have posted a detailed response addressing your points, including the theoretical generalizability, validation at scale, robustness of the improvements, and evidence that gains come from reduced SB. We further provide a brief summary of the problem significance and our contributions here.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we kindly request an updated score.

Best regards,

The Authors

I thank the authors for the response. They have addressed my concerns and I would keep my positive rating.

Reviewer_ozMhOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper notes that sharpness-aware minimization (SAM) can lead to improved generalization in transformers. The authors derive the continuous-time limit of SAM dynamics for single-layer, multi-head linear attention transformers on an in-context linear regression task, and by comparing these dynamics to existing results on GD for such transformers, they show that SAM allocates time more evenly across learning different features of the data, as opposed to GD which spends more time learning more important features (directions with larger eigenvalues in the data covariance matrix) vs. less important ones. They define a notion of simplicity bias based on the time these optimization algorithms allocate to learning different features; by the definition of their notion, SAM has a lower simplicity bias to which they attribute its improved generalization performance.

Based on this theoretical result, the authors propose a data-centric algorithm for reducing simplicity bias in transformer training. By upweighting examples which correspond to the less dominant features, we can effectively increase the time that GD spends learning these features, achieving the same simplicity bias reduction as SAM but without the additional computational or memory costs. Since there is not a direct analog of eigenvectors/eigenvalues for language data, the authors instead stratify samples by their loss value early in training on a smaller model as a proxy, since existing results show that dominant directions are learned first in the in-context linear regression setting. Applying their method to LoRA finetuning of open-weight models ranging from 0.6B to 7B parameters, they show that their method gives consistent improvement on math, code, and instruction following tasks, and can applies to both AdamW and Muon optimizers.

Critical strengths

The theoretical results are generally clear, and the connection between the theoretical results and the proposed algorithm is natural. The empirical results are also consistently positive and show that the intuition derived from the theory can be practically relevant.

Critical weaknesses

I don't believe the paper substantiates its explanation of why the proposed method (1) should be considered a type of simplicity bias, or (2) that reducing simplicity bias according to this metric causally improves generalization. For point (1), it seems that in terms of generalization, only the simplicity of the final model should matter, rather than the dynamics of intermediate checkpoints. Why does evening out the time spent learning different features matter for generalization? For point (2), the connection between the theoretical results and upsampling points which correspond to "non-dominant features" is clear. However, this algorithm--upweighting samples with high loss earlier in training, which might just correspond to more challenging (and therefore potentially more useful) examples--could be very naturally motivated by other means. I didn't see empirical verification that this upweighting causes something similar to the more evenly-spaced plateaus which are observed in the synthetic setting. Even if this could be verified, it's not clear to me how to show that allocating training time more uniformly to different features is the causal mechanism behind improved generalization.

Questions for authors
  1. One can infer that the perturbation matrix P=ρ∥∥∇w(t)L(w(t))∥∥2H(t)P = \frac{\rho}{\|\|\nabla_{w^{(t)}} \mathcal{L}(w^{(t)})\|\|_2} H^{(t)}, but since this is a central quantity it would help to state it explicitly and also note that it is time dependent.

  2. The final expression in equation (6) makes sense as it follows directly from the final expression in (5). What is the derivation of the first equality?

  3. Is there any empirical validation of Assumption 4? Also, how critical is it to the results that the same constant cc hold across different features?

  4. If I understand correctly, Theorem 5 equates the time to learning the ii-th feature to at least some constant factor cc with the dips in the loss curve. Can the authors provide empirical evidence that these two times do indeed coincide?

  5. Also regarding the "time to learning" of each feature: the argument that SAM reduces simplicity bias by making the time spent learning each feature more uniform makes me think that the tit_is be understood as the time between the loss plateaus in Fig. 1 (since maximal entropy is obtained when all the tit_i are equal). However, the definitions given in Theorem 5 seem more like the total time required to learn each feature. Can the authors clarify this?

  6. With regard to the definition of "removing simplicity bias" itself, Lemma 2 shows that the saddle points of GD and SAM are the same, so in principle I would expect the set of final models learned by either procedure to be similar as well (assuming both optimization algorithms take us towards a local min or saddle point). In principle, the amount of time spent in each phase during the optimization could have no impact on the final output, and it seems that the simplicity bias of the final model is the more important quantity rather than the simplicity bias of intermediate checkpoints. Nevertheless, the empirical results in the paper show different test loss for transformers with and without the proposed notion of simplicity bias. Can the authors provide more intuition for why the distribution of learning times during optimization should have any impact on the final model that gets learned, given that the set of saddle points is the same?

Since the empirical results are strong and the proposed algorithm has a very natural connection to the theory, I would happily raise my score if the authors can clarify these questions.

4 comments
AuthorsOpenReview ↗

Rebuttal response part-1 (of 2)

We thank the reviewer for the careful reading.

We address each point below.

W1. Why is this simplicity bias, and why does it matter for generalization?

We respectfully disagree that simplicity bias should be assessed only on the final model. Simplicity bias is defined in the literature we build on as the tendency of gradient-based training to learn dominant features before weaker ones (Kalimeris et al., 2019; Hu et al., 2020; Rende et al., 2024 for transformers). This is the definition in the first sentence of our abstract. It is a property of training dynamics, and our entropy measure (Eq. 8) quantifies exactly this: the nonuniformity of the time required to learn different features. A larger entropy means that these times are more homogeneous, so a method that causes weaker features to be learned earlier reduces simplicity bias by definition. We do not claim that this measure captures every notion of simplicity bias.

Our claim about generalization is likewise stated in terms of dynamics, and has four steps:

  1. SAM consistently improves generalization.
  2. Theorem 5 proves that SAM reduces simplicity bias in our setting, making feature learning more uniform.
  3. We reproduce this property through the data alone, and observe gains across five models, three domains and two optimizers.
  4. The opposite intervention, upsampling well-represented examples, degrades performance, and compute-matched random upsampling gives no gain (Table 10).

We do not claim to characterize the simplicity of the final solution, and our argument does not depend on it. This is also the state of the art for SAM itself: properties of its solutions such as flatness have been shown, but their connection to generalization remains unproven and debated (Dinh et al., 2017 [1]; Andriushchenko et al., 2023 [2]; Wen et al., 2023 [3]). Requiring such a characterization here would hold our work to a standard that the SAM literature does not meet.

[1] Dinh et al., Sharp Minima Can Generalize For Deep Nets, ICML 2017.

[2] Andriushchenko et al., A Modern Look at the Relationship between Sharpness and Generalization, ICML 2023.

[3] Wen et al., Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization, NeurIPS 2023.

W2. Alternative explanations and empirical verification.

Our method is not motivated by the heuristic that difficult examples are more useful. It follows from a specific prediction of our analysis of training dynamics under SAM: examples carrying less-represented features show slower early loss reduction. It therefore targets under-represented features, not difficulty. The alternative explanation, that we upweight “more challenging and therefore more useful” examples, is tested directly in the paper and does not hold.

  • Hard-example upweighting does not reproduce the gains. OHEM implements exactly this hypothesis and underperforms standard fine-tuning (27.7 vs 28.2, Table 5). Our method reaches 30.0 and improves OHEM when combined with it (30.9).
  • We do not select hard examples. We amplify 28–63% of the dataset (§7), identified once by a 70M proxy in the first third of an epoch. Appendix H shows the clusters separate by format and reasoning modality: elementary arithmetic with Python rationales falls in the less-represented cluster.
  • Not noise or length. Mislabeled examples are not concentrated in the selected cluster (§7), and cluster membership explains more than twice the variance that sequence length does (Appendix E.2).
  • The effect is specific and directional. Random upsampling is neutral (54.2 vs 54.5 on Qwen3; 28.3 vs 28.2 on Llama3.2), upsampling less-represented examples helps (55.7; 30.0), and upsampling well-represented examples hurts (53.0; 26.6) (Table 10).

On empirical verification: evenly spaced plateaus are not the right observable in real LLMs. With higher-rank key-query matrices, the stage-wise loss drops smooth into a monotone decrease while the feature-learning sequence is preserved. This is established in the theory we build on (Zhang et al., 2025, Appendix F.4) and stated in our §3 and the Fig. 1 caption. The observable prediction of our theory is at the level of per-example loss, and we verify it directly (§7, “Evidence of Reduced SB”; Appendix E.2):

  • Well-represented examples: loss rises by 25%, although they are seen exactly as often as in standard fine-tuning. This is the signature of reduced reliance on dominant features.
  • Less-represented examples: loss falls by 78%, and it falls within every length bin.
AuthorsOpenReview ↗

Rebuttal response part-2 (of 2)

Q1. P(t) = ρ · H(t) / ‖∇L(w(t))‖₂, where H(t) is the Hessian at w(t). It is time-dependent through both factors and is detached from the gradient computation. We have stated this explicitly right after Eq. 5 in the revised manuscript.

Q2. The first equality is the definition of the SAM gradient flow. It is the continuous-time limit (η → 0, ρ fixed) of update (4), in the same convention as the GD flow τẆ = −½ ∂L/∂W. The gradient is evaluated at the perturbed point W + ρ∇L/‖∇L‖, and there is no chain-rule term because the perturbation is detached. We have reworded this as a definition in §4.1.

Q3. The assumption follows from the dynamics themselves. The early-training ODE in Theorem 3 is obtained by dropping the higher-order term in the full dynamics, τv̇ᵢ = λᵢ²vᵢ² − λᵢaᵢvᵢ⁵ (Appendix C.2). The dropped term is a fraction aᵢvᵢ³/λᵢ of the retained one. The early phase therefore ends when this fraction reaches a fixed tolerance δ, that is, at

vᵢ = (δλᵢ/aᵢ)^(1/3) = δ^(1/3) · (λᵢ + (λᵢ + Tr(Σ))/N)^(−1/3),

which is exactly a fixed fraction δ^(1/3) of the final weight in Eq. 2, for every feature. For large N this is cλᵢ^(−1/3) with c = δ^(1/3), which is Assumption 4.

The identical constant c specifies the same relative threshold for every feature; it does not assume identical learning dynamics. It is sufficient for the majorization argument in Theorem 5 but not necessary: the proof needs only that the endpoints increase as λᵢ decreases, and that GD and SAM share the same endpoint, which is natural since both share the same fixed points (Lemma 2). What matters is the λᵢ^(−1/3) scaling, which is the mechanism through which SAM’s slowdown acts non-uniformly across features. We clarified this in Assumption 4 and added a new subsection in the appendix to discuss the relaxed assumption.

Q4. In the setting of Theorem 3, the two times coincide up to a small correction. Because v̇ ∝ v², almost all of the time is spent near initialization: for GD, tᵢ = (τ/λᵢ²)(1/ε − 1/uᵢ), while the time remaining after the early phase is at most τ/(λᵢ²uᵢ). The loss drop for feature i therefore occurs at tᵢ(1 + O(ε/uᵢ)), and tᵢ is insensitive to c.

We verified the closed forms numerically in this setting. For GD, the predicted tᵢ match the simulated times to within 0.1% for all features. For SAM, the first-order formula matches to within 1% at ρ = ε/4, with the error growing as ρ approaches the bound in Theorem 5, as expected of a first-order expansion.

Q5. tᵢ is the time for the weight of feature i to grow from its initialization ε to the end of its early phase, cλᵢ^(−1/3). It is defined per feature, as illustrated by the green segments in Fig. 8. Under the ansatz (2)–(3), where only the active head evolves, this is the early part of the i-th plateau, and by the argument in Q4 it accounts for almost all of that plateau. The entropy is maximal when all features take equal time, which is the sense in which SAM makes feature learning more uniform. We clarified this in the statement of Theorem 5.

Q6. Lemma 2 states that GD and SAM share the same set of fixed-point manifolds in the stylized model. Its purpose is to show that the difference in feature learning comes from the optimizer’s dynamics and not from a changed landscape. It makes no statement about which fixed point is reached, and we make no claim that the two optimizers reach the same or different solutions. Our theoretical claim is about the dynamics (Theorem 5), and the effect on generalization is measured empirically. Characterizing the solutions reached under reduced simplicity bias is an interesting direction that we leave to future work. We also note that our data-centric method changes the training distribution, so Lemma 2, which compares optimizers on the same data, does not apply to it.

AuthorsOpenReview ↗

Follow-up on our rebuttal and revised manuscript

Dear Reviewer,

Thank you again for your careful reading and constructive questions. We have posted a detailed response addressing your points, including clarification on the definition of SB and its role in improving generalization, alternative explanations of improved generalization and empirical verification that it’s coming from reduced SB, and detailed answer to your questions. All corresponding clarifications are included in the revised manuscript with blue text. We further provide a brief summary of the problem significance and our contributions here.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we kindly request an updated score.

Best regards,

The Authors

Thanks to the authors for their thorough response to my review. My questions have been answered and I have raised my score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our contribution

Dear Area Chair and Reviewers,

Thank you for the thoughtful feedback, which helped sharpen our theoretical presentation and empirical positioning. Since our detailed responses span multiple threads, we summarize our core contribution below.

Summary. Gradient-based optimizers exhibit a simplicity bias (SB), tending to learn dominant, easy-to-capture features first while potentially overlooking other important patterns. In practice, excessive SB can limit models’ ability to generalize to diverse and challenging tasks. While SB has been well studied in CNNs, how optimizers and data shape it in the training dynamics of transformers has remained theoretically unexplored. We show that SAM counteracts SB in transformers by promoting more uniform feature learning, but its computational cost (6× the wall-clock time and 2× the memory of AdamW in our experiments) makes it impractical for LLMs. Our work bridges this gap between theoretical understanding and practical scalability. On the theoretical side, we provide, to our knowledge, the first analysis of SAM’s training dynamics in multi-head attention, establishing how it leads to more uniform feature learning. On the practical side, we translate this insight into a simple, data-centric method that captures this effect using standard optimizers, with only a one-time preprocessing cost. Evaluated on five LLMs for mathematical reasoning and two each for code generation and instruction following, our method yields gains of up to 8.3%, 26%, and 4.1%, respectively. Controlled ablations further establish that these gains come from better coverage of less-represented features, rather than simply additional training. Together, our results show how theoretical insights into optimization can lead to more effective and computationally efficient LLM training, without the overhead of SAM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.