- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the learning of factual knowledge (which signifies memorization) and the rules (which signifies generalaization) in Transformers. The authors conduct experiments that train tiny transformers from scratch on single-digit arithmetic (and evaluate on a few held-out equations) and sometimes with factual knowledge of capital city. The main conclusion is that the best generalization is often achieved when at the most narrow models. The authors also propose an auxiliary loss that clusters hidden states by answer and shwo that it achieves joint success in generalization over rules and memorizing facts.
Critical strengths
- The perspective that models trained with next-token prediction are required to learn generalizable rules and remembering fact is interesting, the tension between generlaization and memorization worths studying.
Critical weaknesses
-
The test is quite narrow (for example the main task is 196 single-dgit addition and substractions as training and 5+7, 5-7, 7+5, 7-5 as test) and the model is tiny (1 or 2 layers transformer with hidden dim smaller than 50), so I am not sure how significant the results are.
-
The above might be fine if a mechnistic understanding or theoreitical analysis is done properly. However, most of the results stays empirical.
-
The writing and clarity need improvement. For example, it's hard to parse useful information from the abstract. It feels like LLMs are intensively used in writing.
Questions for authors
-
L16, 17 in the abstract, Why 'success requires recall of least 48 of 50 facts at that evaluation', how is the number determined?
-
Steering benefits (the auxiliary loss) are shown only for the (5, 7) held-out pair, and the one other pair tested, (4, 9) shows no benefit. Can you show results on some additional held-out pairs?
Thank you for the review.
Narrow task and small models:
The narrow scope is by design, which the paper should have stated up front (the abstract and introduction now do). A small corpus gives us a controlled setting in which we can account for every training token, and small models let us train each configuration on up to 60 seeds, which is needed because outcomes vary widely between seeds. We do not claim that the results transfer to large models.
Mechanistic understanding:
The paper offers no theory, but Section 6 is a mechanistic analysis. It measures what the loss changes inside the model (same-answer representations move closer together in both architectures) with controls matched for the size of that change. It does not explain why generalization improves (stated in the paper).
Writing:
Your point about clarity is fair. We rewrote the paper for clarity, starting with the abstract, which now gives the problem and the setup before any result (the numbers you reviewed are unchanged). Our use of AI tools is covered in the paper's AI-use statement.
The 48-of-50 threshold:
The threshold is a convention for near-perfect recall that allows two misses, not a derived value. Appendix C.3 now repeats every test under every threshold from none to all 50. Only the Qwen3 gain over the baseline in row 6 depends on the threshold (p < .05 at 48 or 49 and between .058 and .090 otherwise), while its gain over the shuffled-label control holds at every threshold.
Other withheld pairs:
The (4,9) pair had too few successes in any arm to show a benefit or its absence (1, 2 and 1 of 20). We have now withheld six other pairs, with 12 seeds per pair at two widths. Answer consistency raises pooled success from 8 to 22 of 72 models at width 14 (p = .0066) and from 29 to 50 of 72 at width 8 (p = .0001). No single pair is significant on its own, so the claim is limited to the pooled result (§5.4, Table 15).