acceptodds
Withdrawn from ICLR 2027

When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A common response to spurious correlations is to weaken them, for instance by balancing the training data so that the shortcut feature no longer predicts the label. We find that in small transformers this can make the robust rule harder to learn. In our tasks, a shortcut feature agrees with the label on a fraction of training examples and contradicts it on a held-out adversarial split. When the label is the sum parity of an integer sequence and the shortcut is the parity of its largest element, the fraction of two-layer seeds that reach adversarial accuracy grows from 0% at to 53% at , whereas for one-layer models it drops from 33% to 0%. The benefit depends on how far is from chance rather than on the direction in which it deviates. On a ternary task, two-layer models generalize in 80–93% of seeds both when the shortcut is correlated and when it is anti-correlated with the label, and in no seeds at chance. In that setting they memorize the training set, a failure that disappears with 16 more data. All models learn the shortcut-only predictor first, and in the runs that succeed the robust rule emerges while attention to the shortcut token remains high. Finally, models trained briefly with an informative shortcut and then moved to balanced data still generalize in most seeds, which suggests that the shortcut serves as an optimization scaffold rather than working by amplifying the gradient of the misclassified minority.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Interpretability, Data attribution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FP9cOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the role of spurious correlations during model training, demonstrating that in small Transformer models, the presence of spurious correlations can actually improve generalization on bias-conflicting samples. The authors argue that the utility of spurious correlations depends on the deviation of the bias ratio from the random-chance baseline. Empirical evaluations across four synthetic tasks (e.g., Max-Equal-Sum-Parity) and various bias-ratio sweeps highlight this phenomenon and prompt questions regarding when, how, and what training signals contribute during optimization.

Critical strengths
  1. The conceptualization of the uninformative shortcut (where the bias-aligned sample ratio matches the random-chance baseline) offers a fresh angle on understanding shortcuts.

  2. The experiments conducted on small Transformers across different bias-ratio settings provide evidence supporting the proposed claims.

Critical weaknesses
  1. Limited practical significance and scope. My primary concern lies in the practical implications of this study. The finding that spurious correlations benefit model training is currently validated only on small Transformers and synthetic tasks, while reducing the bias ratio to random chance is shown to impair generalization. However, the paper lacks discussion on whether these conclusions hold for larger, real-world models (such as pre-trained Language Models) or image classification models where spurious correlations are extensively studied. The broader practical value of these insights remains unclear.

  2. Missing connection to existing debiasing literature. In addressing spurious correlations, the paper focuses primarily on adjusting the bias ratio of bias-aligned samples. However, existing debiasing research relies on countermeasures such as sample re-weighting, data augmentation, and advanced training algorithms, which have proven effective at mitigating shortcut overfitting and improving generalization without explicitly introducing uninformative shortcuts. The submission lacks a discussion or comparison with these advanced debiasing techniques and should broaden its scope to contextualize its findings within the existing literature.

  3. Clarity of mechanistic analysis. Section 5 offers a mechanistic interpretability analysis, including gradient dynamics, circuit evolution, and QK/OV fingerprints. However, this analysis is restricted to a single synthetic task and is currently hard to follow. A broader, more structured presentation is needed to substantiate these mechanistic claims.

  4. Lack of vision generalization. The evaluation of the submission is limited to auto-regressive generation text tasks. Given that spurious correlations are heavily studied in image classification (e.g., CNNs or CLIP models) with explicit shortcut signals like backgrounds, it remains uncertain whether vision models behave similarly. Expanding the study to image datasets or vision-language models would significantly enhance the real-world impact and applicability of the paper.

  5. Minor issues.

  • I recommend that full names are provided upon first introduction (e.g., line 047 MESP), otherwise it is hard to follow the submission.
  • Broken sentence: line 048, There and on two…
  • As noted in the limitations (Section 6), the paper would benefit from broader studies or ablation experiments across different transformer architectures, learning rates, optimizers, training epochs, batch sizes, and tasks to confirm the robustness of the proposed insights.
Questions for authors

Please refer to the Weaknesses.

Reviewer_2PpfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies counter-intuitive results in a toy setting: using one or two layer transformers, authors demonstrate that spurious correlations may help generalization in arithmetic tasks such as Max-Equal-Sum-Parity.

Authors vary the ratio of the training set samples where the labels have spurious correlations to the inputs. Presumably, increasing this ratio would result in shortcut learning but authors demonstrate that this is not always the case. Authors show that distance to the random default spurious correlation ratio is more important than the ratio itself. The supporting analysis using robust circuit ablations is provided.

Critical strengths

Clearly written. Authors demonstrate counter-intuitive generalization evidence. This by itself should be interesting to the community. Authors provide analysis and discussion for the provided evidence.

Critical weaknesses
  • This is a toy task. Since this is not an application paper, the toy settings are not a crucial weakness in itself. Authors already clearly state this limitation.
  • The limited experimental settings weaken the conclusions. I believe, in this instance, for this paper, asking for more experiments is warranted. For the conclusions to be better supported I would want to see at least two more settings with more layers and heads. I believe the computational costs of the experiments are acceptable for most institutions and would not take too long to run in these toy settings.
  • Reasoning behind the choice of spurious ratios that are swept are unclear. Being limited to only 3 ratios weakens the main conclusion, i.e., distance to the random default is more important than the ratio itself. Again, more experiments are needed.
  • Discussion section of the paper is its weakest. It is very difficult to tie the discussion section to the analyses the authors provided earlier in section 5. In its current form, the discussion is basically a bullet list of reiterating earlier analyses.

Minor:

  • Various LLM-prose throughout the text, e.g. Since the model is already perfect when the head withdraws is a meaningless sentence. A head does not "withdraw". This is simply an ablation where the authors null some activations. This type of seemingly-meaningful LLM-writing only makes reading more difficult.
Questions for authors

Major:

  • In line 128, spurious ratio is defined as the fraction of shortcut-consistent training examples and for binary task the ratios are all above the random default of 0.5, as opposed to the ternary where the ratios are below, above and at random default. I do not understand why the binary setting's spurious ratios are all above.
  • Discussion section weaknesses I listed above should be revised. I very strongly urge the authors to write this chapter without any LLM support.

Minor:

  • Line 166: Authors state because three residue classes did not fit reliably at the base width. This is unclear. What exactly does "did not fit" mean here?
  • Line 167: depth and head count are swept identically in both cases Unclear. It seems to me that binary and ternary tasks have only one set of depth and width value each so I do not follow how they are swept. -Line 177: Choice of using AdamW. For this toy setting, wouldn't SGD be a better (i.e. easier to analyze gradient statistics and probably easier to interpret) optimizer? Did the authors have a reasoning behind choosing AdamW -- or does it matter at all?
  • Line 355: Attention to the shortcut token does not indicate shortcut use either, because the max element and seq[0] are summands of the true rule. Maybe the word choice here should not be indicate since surely it indicates it because the token is by definition weighted highly if attention value is high. Authors then state Reliance on the shortcut lies in the readout, which attention weights cannot show. which I am not sure I understand. What does readout mean here? Why can't attention show the reliance?
  • What is θ\theta? It should be clearly defined in the main text.
  • What is amplification? What is scaffolding? These should be clearly defined in the main text.
Reviewer_yRGXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies tiny transformers (1–2 layers, d=64/128, no LayerNorm) on tasks such as sum parity of five integers. A shortcut feature, the parity of the maximum or first element, agrees with the label on a fraction r of training examples. The “adversarial” split is all anti-shortcut. At the chance ratio (r=0.5) no two-layer configuration generalizes much, while at r=0.9 the best one (2 layers, 2 heads) reaches 100% adversarial accuracy in 53% (MESP) or 77% (FESP) of 30 seeds. One-layer models get trapped by the shortcut. On a ternary task, generalization is high on both sides of the chance ratio (80–93%) and zero at chance. At chance, two-layer models memorize, and 16× more data removes the failure. Transplanting r=0.9 checkpoints into balanced training and minority upweighting lead the authors to favor an “optimization scaffold” over gradient amplification.

Critical strengths

The empirical hygiene is good for a toy study. It uses 30 seeds, Wilson intervals and Fisher tests, a threshold-robustness check, a Bonferroni correction on the exploratory marker, and calibration of the circuit criterion. Counterexamples are reported (n11, where the null is not worst; ME-Mod3, where depth inverts), and one of the authors’ own hypotheses is reported as refuted. The depth × correlation interaction and the “distance from chance” pattern are new observations.

The potential impact is a useful caution: removing a shortcut is not neutral for learnability. The small setting allows mechanistic analysis (QK/OV, gradients, transplant) that would be hard at scale.

Critical weaknesses
  1. Applicability and novelty are overstated. In all four tasks the shortcut is a summand of the true rule (the max or first element’s parity). By construction, shortcut-consistent examples then force the other tokens’ parity to be even (90% of the data at r=0.9). So r changes the input distribution, and the shortcut acts as a partial solution in a staircase, which Daniely & Malach and Abbe et al. already link to learnability. It is not an independent spurious feature like a background or a marker token, and there is no such control. At r=0.5 there is no spurious correlation at all. The models fail in-distribution (held-out shortcut-consistent accuracy 0.50, training accuracy 0.96), and 16× more data fixes it. This looks like a small-data learnability failure rather than a robustness failure. The title and abstract (“spurious correlations facilitate robust generalization”) go beyond what the tasks show. The framing is against group-robustness methods (Group DRO, subsampling, JTT), but none is evaluated.

  2. Scalability and evidence for the claims are weak. The setting is very small, with one learning rate and no width sweep, and the effect is inconsistent. In the abstract’s MESP result (Table 1), only 2L,2H shows the clear rise (0% to 53%). 2L,1H goes from 10% to 23%, and the “one-layer drop from 33% to 0%” is 1L,1H only (1L,2H is 3/30 at every ratio). n11 and ME-Mod3 contradict the claim. The decisive practical question, whether informative ratios still help at 16× data, was not run, although it takes about 23 minutes per condition. The scaffold conclusion rests on a transplant of 6/10 vs. 0/30 across hardware with an optimizer reset (same-hardware 4/10 vs. 0/10, p=0.087). Four of the six transplant generalizers also generalized at r=0.9. Upweighting is compared across hardware too and does not discriminate between the explanations. The abstract’s “suggests ... scaffold rather than gradient amplification” is stronger than this evidence. Impact beyond toy tasks is untested.

Questions for authors
  1. Can you add a control with an independent shortcut (e.g., an extra flag token that agrees with the label with probability r) to show the benefit does not depend on the shortcut being part of the rule? If it does, how should the title and claims change?

  2. Because the shortcut is a summand, how do you separate label–shortcut correlation from the change in the other tokens’ parity distribution?

  3. Can you report in-distribution held-out accuracy at every r, and justify calling the r=0.5 failure a robustness failure?

  4. Does r=0.9 still beat r=0.5 at N=262,144 (and at one intermediate N)? If not, what is the practical message?

  5. Can you run the reset-only and readout-refit transplant controls on the same hardware, with seeds disjoint from the r=0.9 generalizers, and a Group DRO, subsampling or JTT baseline at r=0.9?

  6. How sensitive are the results to the learning rate, LayerNorm and width? What predicts when the null fails, given n11 and ME-Mod3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.