- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to understand simplicity bias in transformers and its implications on generalization empirically and theoretically using SAM. It then introduces an upsampling mitigation strategy to mitigate simplicity bias and improve ID and OOD generalization in transformers.
Critical strengths
Theory regarding SAM reducing simplicity bias in transformers is novel and not previously articulated. Note that "articulated" means not explicitly discussed because [1] studied SAM impact on simplicity bias theoretically in CNNs. Since, this paper extended the theory to transformers, it still counts as a contribution. But note that I have not fully gone through the exact proof of the proposed theorems and I would depend on my fellow reviewers for their expertise on the same. However, refer to Point 2 in Critical Weakness for further discussion on this.
Critical weaknesses
This paper has severe critical weaknesses regarding novelty. I state them as follows:
-
Authors portray their paper as if they are the first to suggest reducing simplicity bias to improve generalization in transformers. However, we already know from prior literature that "simplicity bias" causes "spurious correlations" which essentially decreases the generalization performance [2,3], and it has also already been extensively empirically shown in transformers [4,5]. This essentially means simplicity bias degrades generalization even for transformers because spurious correlations degrade generalization and spurious correlations are caused by SB (a simple transitive logic). The authors have completely removed any discussion about this analysis which in my opinion is extremely important to properly be thought about before claiming anything. Hence, the authors claim on "the role of SB in transformers remain largely underexplored (line 14)" is false. I agree no one has previously shown theoretically that SAM reduces SB in transformers, however this being the only novel contribution of this paper is not publication worthy at ICLR. The authors must clearly clarify how their work really differs from prior works on SB causing spurious correlations which reduces generalization.
-
Apart from not discussing works on spurious correlations, the authors fail to discuss [6] which already shows that SAM increases balanced feature learning, i.e., it adaptively suppresses well-representated features which gives remaining less-representative features opportunity to be learned. The authors theory merely provides further support to this and there is nothing new from what is already known.
-
The authors claim that "examples associated with dominant features exhibit rapid early loss drops, while examples associated with less-represented features lag behind (line 18-20)" is already well established from spurious correlations papers and there is nothing new about this. It is merely stating an already known facts. The authors again need to clarify what exactly is new in their current paper which is not already known.
-
The authors only focus on generative tasks for showing reducing SB improves generalization. However, SB causes spurious correlations and there are ample amount of papers which provide mitigation strategies to reduce "SB/spurious correlations" in tasks like WaterBirds, MNLI, CelebA, MultiNLI. The authors must show whether their proposed strategy outperforms the mitigation strategies in these papers or not and how expensive/cheap their method is over these. I expect them to atleast show performance and timing comparison against 3-4 well-known strategies, especially - DFR, JTT, CNC, for the above listed tasks. The authors can easily extrapolate their proposed method to these new datasets as their method is just clustering followed by upsampling, which is topic-agnostic.
-
In Table 5 and Fig 6, CL has higher generalization performance than FT+UP (proposed method by the authors) for 6 out of 7 datasets. This means the proposed method by itself is not really performing well. This is another major issue since their original method cannot outperform well known strategies. The authors need to provide a mechanistic analysis where their method is performing poorly via experiments and why. Furthermore, no timing comparison has been provided to compare UP with CL and OHEM (the authors have only compared with SAM which we already know is very expensive). Also, regarding UP - why have the authors only used Pythia-70M for clustering process? Why no other model of similar size from a different model family was used?
-
The authors should also provide clear examples of well-represented and less-represented samples that they detect via their clustering operation. It is highly informative to know what kind of commonality well-represented samples have and how it differs from less represented samples.
Questions for authors
Please see Critical Weaknesses.
[1] Changing the training data distribution to reduce simplicity bias improves in-distribution generalization, Arxiv 2026
[2] Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness, TMLR 2024
[3] The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning, TMLR 2026
[4] Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training, Arxiv 2025
[5] Trapped by simplicity: When Transformers fail to learn from noisy features, ICLR 2026
[6] Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning, ICLR 2024
Overall, this paper is just merely stating already known things even in transformers (as discussed in Critical Weaknesses) and does not offer any new novel insights other than theoretically relating SAM reducing SB (something which I hope other reviewers can confirm is correct or not based on their theoretical expertise). If the authors wish to submit a rebuttal, I hope they could properly clarify all the points raised by me in the Critical Weaknesses section. (PS: I have not used any AI tool to review your paper).
4 comments
Rebuttal response part-2 (of 2)
4. Comparison with DFR, JTT and CNC on Waterbirds, CelebA and MultiNLI.
These are outside the scope of our paper. The benchmarks are image and text classification tasks with annotated spurious attributes, evaluated by worst-group accuracy. The baselines are group-robustness methods for classifiers (last-layer retraining, upweighting misclassified examples, contrastive learning on classifier representations) and are not defined for generative fine-tuning. The closest applicable baseline, loss-based upweighting of hard examples, is OHEM, which we compare against: it underperforms standard fine-tuning (27.7 vs 28.2) while our method reaches 30.0 (Table 5). We make no claim about group robustness, so this comparison would not test any claim in the paper.
5. Table 5 and curriculum learning.
The statement that CL beats our method on “6 out of 7 datasets” is incorrect. Table 5 has six benchmarks; CL is higher on four and ours on two, including MMLU-Math, where CL falls to 13.9, well below standard fine-tuning (27.2). More importantly, we do not claim to replace CL. We state in §2 and §7 that the methods are complementary, and our upsampling improves every baseline it is added to: standard fine-tuning (28.2 to 30.0), CL (31.3 to 32.4) and OHEM (27.7 to 30.9).
On cost (updated in §7 and Table 5), our overhead is a 1.38× increase in steps plus ~0.3% for the proxy, corresponding to 1.42× standard FT wall-clock time end-to-end on Llama3.2-1B MathInstruct. By comparison, CL runs in 0.98× (within run-to-run variance of FT), while OHEM requires 1.93× (and OHEM+UP 2.55×) due to per-step forward passes over its 2,048-example pool. Appendix E.1 analyzes when and why the method helps less.
6. Proxy model. The statement that only Pythia-70M was used is incorrect. Fig. 7(a) and Table 8 also use Phi-2 (2.7B, a different family) as the proxy, and both yield gains (57.3 and 58.0 vs 56.5).
7. Cluster examples. These are already in the paper: Appendix H and Table 14, referenced from §7.
Thanks for the responses. My concerns are resolved. I have raised my score.
Thank you and brief follow-up on remaining concerns
Dear Reviewer,
Thank you for reading our responses, confirming that your concerns are resolved, and raising your score. All corresponding updates, including the related work discussion of [1–6] in §2 and the wall-clock comparisons with curriculum learning and OHEM in §7 and Table 5, are included in the revised manuscript. We further provide a brief summary of the problem and our contributions here.
We note that the updated review still carries a weak-reject rating (2) and a soundness score of 1. Please let us know if any concern remains regarding the paper’s soundness or evaluation, and we will clarify it before the discussion closes. If none remains, we would be grateful if you could consider updating the rating and the soundness score to reflect your current assessment.
Best regards,
The Authors
Rebuttal response part-1 (of 2)
We thank the reviewer for the comments. We respectfully disagree with the assessment of novelty, and several statements in the review do not match the paper or the cited works.
1. Novelty relative to [1–5].
The reviewer’s argument rests on an invalid inference. Simplicity bias can lead to reliance on spurious features when such features are present, but simplicity bias itself is a broader and separate concept. This distinction is fundamental to our setting.
The spurious-correlation literature [2, 3] studies a different problem: robustness under distribution shift, where a spurious attribute correlates with the label and the metric is worst-group or OOD accuracy. [2] proposes a regularizer evaluated on such benchmarks, and [3] is a survey of that area. Our theoretical setting contains no spurious features at all: every eigen-direction of the data covariance is predictive of the label. The same holds in our experiments, which involve no spurious attribute or group structure. The issue we identify is independent of spurious correlations: certain predictive features are learned late.
The reviewer’s own references contradict the transitive argument in the review:
That reducing simplicity bias improves generalization when nothing spurious is present therefore does not follow from the spurious-correlation literature, and it has not been shown for LLMs.
Neither [4] nor [5] shows what the review attributes to them:
Finally, results established for CNN classifiers, such as [1] and [6], cannot be assumed to extend to transformers. Our analysis considers attention heads with trainable key, query and value parameters, which enter the output multiplicatively. Even in its reduced form (Lemma 1), the model is a three-layer network with cubic interactions between its weights, and its dynamics pass through 2ᵈ fixed-point manifolds saddle to saddle (Appendix A.3). Analyses of two-layer CNNs do not cover these dynamics, which is why the transformer setting has required separate technical treatment (Zhang et al., 2025).
Our contributions, which appear in none of these works, are:
The review treats the theorem as our only contribution and does not engage with the method or experiments. In the revised manuscript, we have added a discussion of these works in §2.
2. Relation to [6].
The claim that our theory “merely provides further support” to [6] is incorrect.
A compatible empirical finding on image classifiers corroborates our result; it does not anticipate it.
3. Early loss as a signal.
In the spurious-correlation literature, early-training signals are used in classifiers to identify groups defined by a spurious attribute. We derive the signal from the dynamics of attention and apply it to generative fine-tuning, where there are no such groups and our clusters cover 28–63% of the data (§7). Our contribution is also not simply the use of early loss, but its theoretical motivation and application to feature learning in LLMs.