acceptodds
Under review as a conference paper at ICLR 2027

Sparks of Generalization: Capacity Windows and Steering During Transformer Pretraining

Abstract

Can a transformer learn rules while retaining facts under the same next-token objective? We study Qwen3 and Gemma3 trained from scratch on controlled corpora with native tokenizers, one head per layer, and one or two layers in both architectures (three in Qwen3). Single-digit addition and subtraction probe in-distribution generalization, requiring all 196 training and four withheld equations correct at the final evaluation. With facts included, success also requires recall of at least 48 of 50 facts at that evaluation (joint success). In one-layer arithmetic-only baselines of both architectures, generalization is most frequent within a limited hidden-width range. At the widest tested width, all 10 seeds per architecture fit the training equations and none generalizes. To steer generalization, we introduce answer consistency, an auxiliary loss on arithmetic representations grouped by operator and training answer. Across thirteen steering blocks (three arms on shared seeds) using arithmetic alone or adding facts and approximately 1,400 prose tokens, it raises success over the no-penalty baseline in nine and over a per-step shuffled-label control in five (nominal, unadjusted paired , all assigned seeds). In one-layer arithmetic-only models at the widest tested width, steering yields no detectable gain in success. With facts and this prose corpus, steered one-layer width-14 models reach joint success in 34/60 Qwen3 and 46/60 Gemma3 seeds, versus 20/60 and 32/60 for baseline and 17/60 and 33/60 for shuffled labels. Steering also tightens same-answer representations at the regularized site across the architectures and depths studied.

Primary area:
foundation or frontier models, including LLMs
Keywords:
transformer pretraining, memorization, in-distribution generalization, model capacity, training-time steering, arithmetic learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_e1AWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the learning of factual knowledge (which signifies memorization) and the rules (which signifies generalaization) in Transformers. The authors conduct experiments that train tiny transformers from scratch on single-digit arithmetic (and evaluate on a few held-out equations) and sometimes with factual knowledge of capital city. The main conclusion is that the best generalization is often achieved when at the most narrow models. The authors also propose an auxiliary loss that clusters hidden states by answer and shwo that it achieves joint success in generalization over rules and memorizing facts.

Critical strengths
  • The perspective that models trained with next-token prediction are required to learn generalizable rules and remembering fact is interesting, the tension between generlaization and memorization worths studying.
Critical weaknesses
  • The test is quite narrow (for example the main task is 196 single-dgit addition and substractions as training and 5+7, 5-7, 7+5, 7-5 as test) and the model is tiny (1 or 2 layers transformer with hidden dim smaller than 50), so I am not sure how significant the results are.

  • The above might be fine if a mechnistic understanding or theoreitical analysis is done properly. However, most of the results stays empirical.

  • The writing and clarity need improvement. For example, it's hard to parse useful information from the abstract. It feels like LLMs are intensively used in writing.

Questions for authors
  • L16, 17 in the abstract, Why 'success requires recall of least 48 of 50 facts at that evaluation', how is the number determined?

  • Steering benefits (the auxiliary loss) are shown only for the (5, 7) held-out pair, and the one other pair tested, (4, 9) shows no benefit. Can you show results on some additional held-out pairs?

1 comment

Thank you for the review.

Narrow task and small models:
The narrow scope is by design, which the paper should have stated up front (the abstract and introduction now do). A small corpus gives us a controlled setting in which we can account for every training token, and small models let us train each configuration on up to 60 seeds, which is needed because outcomes vary widely between seeds. We do not claim that the results transfer to large models.

Mechanistic understanding:
The paper offers no theory, but Section 6 is a mechanistic analysis. It measures what the loss changes inside the model (same-answer representations move closer together in both architectures) with controls matched for the size of that change. It does not explain why generalization improves (stated in the paper).

Writing:
Your point about clarity is fair. We rewrote the paper for clarity, starting with the abstract, which now gives the problem and the setup before any result (the numbers you reviewed are unchanged). Our use of AI tools is covered in the paper's AI-use statement.

The 48-of-50 threshold:
The threshold is a convention for near-perfect recall that allows two misses, not a derived value. Appendix C.3 now repeats every test under every threshold from none to all 50. Only the Qwen3 gain over the baseline in row 6 depends on the threshold (p < .05 at 48 or 49 and between .058 and .090 otherwise), while its gain over the shuffled-label control holds at every threshold.

Other withheld pairs:
The (4,9) pair had too few successes in any arm to show a benefit or its absence (1, 2 and 1 of 20). We have now withheld six other pairs, with 12 seeds per pair at two widths. Answer consistency raises pooled success from 8 to 22 of 72 models at width 14 (p = .0066) and from 29 to 50 of 72 at width 8 (p = .0001). No single pair is significant on its own, so the claim is limited to the pooled result (§5.4, Table 15).

Reviewer_oFsKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether small transformers trained from scratch can learn arithmetic rules while retaining factual knowledge. Experiments with reduced Qwen3 and Gemma3 models suggest that arithmetic generalization is most likely within a limited range of model widths. The proposed answer consistency loss brings representations of equations with the same operator and answer closer together and improves joint arithmetic-and-fact success in several tested settings. Further analysis shows that most of the apparent swap-loss advantage disappears after matching the intervention size.

Critical strengths

1. The main comparisons are easy to interpret

The methods are compared using the same initialization and data-order seeds, and all assigned runs are included. In the main mixed-corpus setting, answer consistency raises joint success from 20/60 to 34/60 for one-layer Qwen3 and from 32/60 to 46/60 for one-layer Gemma3. The shuffled-label control reaches 17/60 and 33/60. These results show a clear advantage in the main tested settings. (Section 3; Table 1.)

2. The representation analysis is informative

After the intervention size is matched, most of the swap-loss advantage disappears. This shows that the lower swap loss may mainly reflect a smaller change in the representation, rather than stronger functional interchangeability. The paper treats this result cautiously and does not present representation contraction as a complete explanation of task success. (Section 6.)

Critical weaknesses

1. Generalization is shown on only one main split

The main positive results all use the same held-out operand pair, (5,7)(5,7). Using more seeds makes the estimate for this split more reliable, but it does not show that the benefit transfers to other operand pairs. A small second-split check uses only one steering setting and has too few successful runs to settle this question. Results on a few prespecified operand pairs would make the generalization claim much stronger. (Sections 3 and 5.5.)

2. The control does not isolate the role of answer semantics

The shuffled-label control changes both the meaning and the stability of the groups: answer-based groups are meaningful and fixed, while shuffled groups change at every training step. The comparison therefore cannot show whether the gain comes from grouping by answer, keeping groups stable, or a more general regularization effect. A more direct control would assign equations to random groups once and keep those groups fixed throughout training, while matching group sizes and relevant token structure. This matters because Table 2 shows that representations also move closer for equations with different answers. The current control is useful, but it does not fully establish that answer-based grouping is the key factor. (Table 2.)

Questions for authors
  1. Could you repeat the experiment on a few prespecified held-out operand pairs and report the main outcomes over all seeds?

  2. Could you add a control with fixed random groups, matched as closely as possible in group size and token structure, to separate answer meaning from group stability?

  3. Please clarify what the intervals in Figure 1 represent. If they are intended to estimate the success-rate difference across new seeds, they should include uncertainty in how often the paired outcomes disagree.

  4. How were the loss weights selected? For one or two main settings, could you compare the method with another representation loss or a standard regularization baseline under a similar tuning budget?

2 comments

The author has not replied, so I will keep the current score.

Thank you for the review, and apologies for the late reply. We were working diligently to collect as many results as possible for the experiments you requested (at a scope that could be done within the available time), and the results are mixed.

Other withheld pairs:
We withheld six other pairs (three drawn at random) and trained 12 seeds per pair at widths 14 and 8. Answer consistency raises pooled success from 8 to 22 of 72 models at width 14 (p = .0066) and from 29 to 50 of 72 at width 8 (p = .0001). It is ahead on five of the six pairs at width 14 and on all six at width 8. Twelve seeds are too few to support a claim about any single pair, so the claim is limited to the pooled result (§5.4, Table 15).

Fixed random groups:
We built the control as you proposed (groups assigned once per seed, matched to the answer groups in size, operator and token), and it changed our claim.

Successful modelsBaselineShuffledFixed randomAnswer consistency
Arithmetic alone (of 40)not run51625
Facts and 1× prose, row 6 (of 60)20173034

Fixed random groups improve on the shuffled control in both settings (p = .019 each), and answer consistency is not significantly better than fixed random groups (p = .078 and .52). We no longer claim that the groups must be derived from the answers (abstract, discussion and conclusion updated accordingly). This is consistent with your observation about Table 2, since both persistent losses pull different-answer equations together (Appendix C.6). The swap and distance measurements of Section 6 give similar results for the fixed-groups models and the steered models (Appendix D.7).

Some pairs in a random group share an answer by chance (about one in nine), so we ran two further controls on arithmetic alone. Fixed groups with no such pair succeed in 17 of 40 models and groups that keep only those pairs in 16 of 40, both above the shuffled control's 5 (p = .0075 and .013; Appendix C.6). Both kinds of fixed grouping therefore improve on the shuffled control. The second control does not show that the chance pairs produced the random groups' gain, because the loss pulls each pair harder in such small groups.

A limitation of all these controls is that the fixed groups stay within one operator and one answer range, whereas the shuffled control mixes them. An improvement over the shuffled control therefore does not separate the persistence of the groups from that confinement (stated in Appendix C.6).

Intervals in Figure 1:
Your read is correct, they are conditional on the number of discordant seeds (caption has been updated accordingly). Appendix E now reports an unconditional interval for all thirteen comparisons (Tango's score interval, +.23 [.04, .41] against the baseline for row 6). The score interval is slightly more generous than the exact tests, so significance is still assessed with the exact tests.

Loss weights and other baselines:
λ was selected from training-side outcomes only (the number of models that fit and the final auxiliary loss), never from the withheld equations. Appendix A.5 now states the rule for each experiment. The fixed random groups serve as the comparison with another representation loss (same loss, weight and site). We also added a weight-decay baseline, training the 60 arithmetic-only seeds at width 14 with weight decay 0.1 and no auxiliary loss (0.3 was tried first, and none of six models fit the training equations). Weight decay does not reproduce the gain in this configuration, with 11 of 60 models generalizing against 38 with answer consistency (p < .0001) and 18 for the baseline (p = .25; Appendix A.3). This is one strength in one configuration, and dropout was not tested.

Reviewer_MtdGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether language models can learn generalizable rules during pretraining while retaining factual knowledge. The authors introduce "Answer Consistency," an auxiliary objective that encourages arithmetic expressions sharing the same operator and answer to have similar hidden representations at the units-prediction position. Results show that this method improves in distribution generalization in small-scale settings with minimal loss of factual recall. However, these findings remain preliminary, as the experiments rely on tiny model architectures, narrow task domains, and limited held-out examples.

Critical strengths
  1. It investigates whether language models can acquire generalizable rules during pretraining while retaining factual knowledge, which is relevant to understanding the relationship between generalization and memorization.

  2. The design makes it easier to distinguish training-set memorization, generalization to unseen equations, and factual retention.

Critical weaknesses
  1. Most experiments are conducted with extremely small one- to three-layer variants of Qwen3 and Gemma3. It remains unclear whether the observed effects would persist in standard-sized language models with substantially greater capacity. The authors acknowledge this in the limitations, but the acknowledgment does not reduce how much it limits the contribution.

  2. The generalization experiments focus almost entirely on single-digit addition and subtraction. Consequently, the paper does not demonstrate that Answer Consistency can generalize to other datasets, more complex arithmetic problems, or non-arithmetic reasoning tasks.

Questions for authors
  1. Why did you choose such simple model configurations (one head per layer, one to three layers, hidden width up to 48)? Is there a specific reason beyond experimental control, and do you expect the findings to hold for more conventional configurations?

  2. The generalization task is limited to single-digit addition and subtraction. Did you try any harder tasks, such as multi-digit arithmetic or a non-arithmetic rule-learning task?

  3. Have you considered how Answer Consistency would apply to general tasks? For arithmetic the grouping follows directly from discrete training labels, but how would it be defined for tasks where the target is not discrete or has no natural answer class?

1 comment
AuthorsOpenReview ↗

Response to Reviewer MtdG: model scale, harder tasks, tasks without answer classes

Thank you for the review.

Small/Simple configurations:
We chose small configurations primarily for control and for the number of seeds. Training from scratch on a fully specified corpus gives us a controlled setting in which we can account for every training token, and small models let us train up to 60 seeds per configuration. The seeds are needed because outcomes vary widely between them (20 of 60 baselines succeed in row 6). Small widths are also where the task is informative, since success at one layer is concentrated in a narrow range of widths (at width 48 all ten seeds of each architecture fit the training equations and none generalizes). We did not test conventional configurations (and do not claim the findings hold there), although the gain over the baseline appears at 1-3 layers in Qwen3 (but only one- layer in Gemma3).

Harder tasks:
Every experiment in this paper uses single-digit addition and subtraction, and we have no result on other rules to report. We instead added breadth within the task by withholding six other pairs. Answer consistency raises pooled success from 8 to 22 of 72 models at width 14 (p = .0066) and from 29 to 50 of 72 at width 8 (p = .0001; §5.4). The result still concerns one small task (noted in the new Limitations paragraph).

Tasks without a natural answer class:
The loss requires only a partition of the flagged training examples, so it applies unchanged wherever several inputs share a discrete target. Continuous or unique targets would need groups from another source (binned targets or a clustering), which we have not tested. A new control bears on this question: Fixed random groups improve on the shuffled-label control in both one-layer settings where they were tested (p = .019 in both), and answer consistency is not significantly better than them. The improvement remains with arithmetic alone when no two grouped equations share an answer (§5.4). The random groups were drawn within classes that share an operator and the answer's sign and tens digit, so this is not yet a test without any answer information.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Thank you all for the reviews. We have uploaded a revised PDF with the following changes:

  • Fixed random groups (Reviewer oFsK): they improve on the shuffled-label control, and answer consistency is not significantly better than them. We no longer claim that the groups must be derived from the answers (abstract, discussion and conclusion updated accordingly).
  • Other withheld pairs (Reviewers oFsK and e1AW): answer consistency raises success from 8 to 22 of 72 models at width 14 (p = .0066) and from 29 to 50 of 72 at width 8 (p = .0001), pooled over six new pairs.
  • Weight decay (Reviewer oFsK): at 0.1 it does not reproduce the gain at width 14 with arithmetic alone (11 of 60 models generalize, against 38 with answer consistency and 18 for the baseline; Appendix A.3).
  • Statistics: Appendix E adds unconditional intervals for all thirteen comparisons, A.5 states how each loss weight was chosen, and C.3 repeats every test under every recall threshold.
  • Writing: the abstract, introduction and main sections were rewritten for clarity and a Limitations paragraph was added. The numbers you reviewed are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.