- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how neural networks learn to combine known facts, why they can lose this ability during training, and whether occasional practice can preserve it. It measures how learning and forgetting depend on supervision of intermediate steps, weight decay, and optimizer settings. Experiments use small neural networks and transformers to study fact retention and two-step reasoning, including whether practicing individual facts in the format needed for the second reasoning step improves performance.
Critical strengths
I think the controlled forgetting experiments in the small looped network is the paper’s strongest contribution. By varying learning rate and weight decay separately, the authors find that the time to lose compositional ability is approximately inversely proportional to their product. This goes beyond showing that forgetting occurs and establishes a measurable relationship between training settings and the persistence of compositional ability.
Critical weaknesses
-
The connection between the experiments is insufficiently established. The toy study maintains composition through an intermediate-state penalty, whereas the transformer studies examine fact rehearsal and additional examples that practice the second reasoning step. The paper does not convincingly show how the toy measurements explain the transformer composition results or guide these interventions. This limits the broader significance of the proposed common explanation and leaves its practical implications unclear.
-
The evidence for transferring the rehearsal scaling relationship to transformers is fragile. The fact-retention experiment uses only two weight-decay values and two seeds per condition, with substantial training instability. Appendix A acknowledges that using the last rather than the first crossing of the retention threshold changes the reported exponent from 2.34 to 0.7. The main scaling plots also lack uncertainty estimates across seeds, and no confidence intervals are provided for the fitted exponents. These limitations make the claimed transfer difficult to assess and insufficiently supported.
Questions for authors
Clarification questions:
- What do the toy scaling results explain or predict about the transformer composition experiment?
- Is there a reason why the first 50% accuracy crossing used to define the rehearsal threshold, given that using the last crossing changes the transformer exponent from 2.34 to 0.7?
- Can you report per-seed rehearsal thresholds and uncertainty in the fitted exponents, particularly for the transformer fact-retention experiment?
3 comments
I appreciate the authors' response. The changes address some of my concerns.
The authors withdrew the toy model’s rehearsal law prediction for the transformer. But this leaves a disconnection in the paper regarding the purpose of the toy model and the practical implications of its lessons. The remaining connection is mainly about the additional forgetting with the added weight-decay. The toy model looks like a specific example where weight decay can destroy composition, while the explanations for the transformer results (e.g., the positional encoding explanation, repair without weight decay) remain separate from the toy. This weakens the coherence of the paper.
I would expect the toy model to first capture the practical phenomenon, including the forgetting observed without weight decay, and to derive lessons using the simplicity of the toy model. Then the lessons learned in the simplified setting should lead to predictions about how the practical setting should behave, which the authors can test. Otherwise, the toy model’s explanatory value remains unclear. I suggest the authors develop the toy model further for this purpose.
Given these remaining concerns, I’m keeping my score unchanged.
Thank you; we agree the toy should carry the argument, and the revision now makes it do so (Section 3). The toy captures the phenomenon, including forgetting without weight decay. In the looped network the account's forces can be seen alone. Under AdamW (decay alone) it keeps composition without weight decay and loses it at a rate proportional to (ηλ)^1.04. Under Lion, the sign step alone more than halves composition within 100 steps without weight decay and removes it within about 800, its facts at 1.00. In a new registered test with a never-composed control trained the same way: while it is partly lost, 100 steps after the signal stops, a short repair adds 0.45–0.55 more composition than it builds in the control; once lost, from 2,000 steps on, at most 0.07 more (4 of 4 seeds; Appendix C, OPT-R). Lessons, and their tests in the transformer. Decay acts on ηλ: in the 58M transformer the part of the loss that weight decay adds follows (ηλ)^1.24 (Section 5.1). Read as a dose, this places erasure before a run: for a learning rate and weight decay never used, we registered the 58M crossing at a dose of 1 within a factor of two (0.80 in the integrated dose, window 0.30–1.80); it landed at 0.72 and 0.84 (Section 5). The toy suggests why the position encoding matters. Under AdamW without weight decay, composition survived facts-only training in our runs where the facts loss exercises the computation it needs. In the toy the second hop is the atomic step applied twice, so it is kept, even with the two-hop rows dropped (Appendix C, U1). In the 58M transformer the second hop reads facts at positions whose learned embeddings no single-fact row trains: there composition is lost without weight decay yet a short repair restores it (0.91–0.99), and with rotary positions, which carry no position-specific weights, it is kept (two bases; Section 5.1). In our from-scratch runs, every run that erased moved every weight (decay, or a sign step); drift under AdamW at most hid. We believe this resolves the coherence concern you raised; if it does, we would be grateful if you would reconsider your score.
Response to Reviewer K2qP
Thank you for naming the controlled forgetting result, a time to lose composition proportional to 1/(lr × wd), as the paper's strongest contribution. The revision is built around it, now measured in transformers at 58M and 0.3B.
W1 / Q. The toy and transformer studies are not connected: what do the toy's scaling results predict about the transformer? Section 3 now states the toy's single role. In the looped network, facts-only training still trains everything composition needs, so there is no drift and only decay can act. It therefore tests the account's two predictions directly: (a) nothing is lost without weight decay (12 of 12 seeds keep composition at 1.00), and (b) the time to lose composition scales as 1/(ηλ), measured as (ηλ)^1.04 (CI [0.97, 1.09]). In the 58M transformer, (b) carries over to the part of the loss that weight decay adds, (ηλ)^1.24 (CI [1.04, 1.57]) (Section 5.1, Figure 4). (a) fails with learned positions, where the transformer loses composition without decay to drift, and largely holds with rotary positions, which remove most of the drift (0.87 and 1.00 after 60k facts-only steps). What the toy does not predict, the dose at which a short repair stops working, is measured in the transformer (Section 5). The account also guides the two interventions in Section 6: weight decay erases composition because it pulls every weight toward zero, including those the loss no longer uses, so decaying toward the starting weights instead keeps it; and the account says which rows must be replayed. The claim that the toy's rehearsal law predicts the transformer's is withdrawn.
W2 / Q. Why the first 50% crossing, given the instability? It was the rule we used, and on a non-monotone curve it is not a safe one: the last crossing gives 0.7, and Appendix C reports both. The instability is real in the follow-up too (101 of 120 runs have a transient spike, so the registered rule that excludes every spiking run is inconclusive). The registered follow-up removes the choice instead of defending it: each run is read in an endpoint window that excludes transient spikes (fixed before any rehearsed run existed), and the registered cycle-average endpoint gives the same exponent.
Per-seed thresholds and uncertainty in the fitted exponents. All are in Appendix D (Table 10). Per-seed p50, four seeds: λ = 0.3: 5.42, 5.57, 5.63, 5.54 × 10^-4; λ = 0.5: 5.53, 5.61, 5.56, 5.59 × 10^-4; λ = 1: 1.60, 1.49, 1.56, 1.69 × 10^-3; at λ = 2 and 3 the facts were kept at every rate tested. The exponent is 0.91 (95% CI [0.86, 0.96]; per-seed slopes 0.85-0.97), outside the predicted [1.4, 2.8]. The interval covers seed noise only, and we report 0.91 as the registered fit that rejected the transfer, not as a law. The other two exponents are 1.04 [0.97, 1.09] (looped network) and 1.24 [1.04, 1.57] (transformer decay part; the loosest, as seven of its eleven cells have two bases). For the new 0.3B results we give three repair seeds per state and a never-composed control per base; one of the two pairs is clean, and we report it as one pair (Sections 5 and 8).
The submission's exponent of 2.34 is withdrawn, and nothing in the revision rests on it. If the revision gives what the submission lacked, one clear result with a transformer at its centre and uncertainty stated where it matters, we would be grateful if you would reconsider your score.