- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors studies whether circuits learned earlier affect a model’s ability to learn from later data. The authors suppress induction-related heads during target-task training, keep the data fixed, and restore the heads for evaluation. In a two-layer transformer, suppressing a previous-token head blocks long-range copying, while supplying its output restores learning. In early Pythia checkpoints, induction-head suppression prevents or delays learning of a mapped-copy task. Longer runs show recovery through other heads, investigated through ablations and adaptive suppression. These results demonstrate that the selected interventions affect learning, but the evidence that recovery requires rebuilding the same prerequisite computation is less conclusive: models can learn the target while ordinary induction remains suppressed. The restriction to two copy-based tasks and early checkpoints from one pretrained model family also limits the broader conclusions about when data can teach a model.
Critical strengths
-
Training-only suppression addresses a question that evaluation-time ablation cannot answer. Keeping the target batches identical and restoring heads for evaluation is a useful design for studying how internal computations affect learning.
-
The central result is supported by repeated runs and several controls. The sixteen Pythia-160M pretraining runs, loss-matched controls, and key-dependence tests make the observed learning difference difficult to dismiss as an isolated failure or a simple positional shortcut.
-
The longer training runs qualify the initial blocking result. Showing that models can eventually recover, and examining the heads involved in recovery, adds useful evidence about the difference between preventing learning and delaying it.
Critical weaknesses
-
The prerequisite-rebuilding interpretation is not sufficiently established. Target-task learning precedes recovery on the ordinary-induction probe at several model sizes. More importantly, adaptive suppression triggered by ordinary induction keeps that score near zero while the model still learns a key-dependent solution. The subsequent key-triggered intervention suppresses heads based on retrieval performance on the target itself. This shows that disrupting emerging target-relevant computation can prevent learning, but does not establish that the original induction prerequisite must be rebuilt. The paper needs a clearer account of what computation is necessary and how the replacement is shown to implement it.
-
Whole-head suppression leaves uncertainty about the cause of the learning failure. Removing a head deletes all of its contributions, not just induction, and changes the representations used by downstream layers. The loss-matched controls are helpful, but matching damage to natural-text loss does not match disruption on the target examples. In Pythia, the suppressed heads also continue to change through other training examples, so enabling them at evaluation does not restore their original function. The evidence supports a causal effect of the selected intervention, but its specificity to induction remains less clear. Finer interventions would strengthen the central interpretation.
-
The experimental scope is too narrow for the broader claims. There are many repetitions and intervention variants, but they all concern one prerequisite and two closely related copy-based tasks. The pretrained experiments use early checkpoints from a single model family. These are reasonable settings for a case study, but they provide limited evidence about other capabilities or learning from natural data. No experiment tests whether circuit information improves curriculum design or predicts data usefulness outside these tasks. This limits the significance of the paper’s broader framing.
Questions for authors
refer to the weakness, if you can solve my questions, I will raise my score
Thank you for stating precisely what would establish the interpretation. We answer the three weaknesses in order.
1. Must recovery rebuild the original prerequisite? No. We agree and corrected the claim in the abstract, Section 5 and the title of Figure 3. Recovery can use a specialized route, and the acquisition result is unchanged: removing selected computations changes what identical subsequent data teach within the tested budget.
The shared causal contribution we identify is transmission of source-position information through the selected heads. We test it with the same operation in the original and the replacement heads. In the original heads, replacing only the source-position value prevents learning, and keeping only that value preserves it (point 2). In the replacement heads of a recovering Pythia-160M model, replacing that value lowers target accuracy from 84.4% to 1.2%; removing the four heads lowers it to 3.9%; removing four matched heads leaves 85.5% (Appendix U).
How the replacement implements it. With the original heads suppressed, the model outputs the mapped partner on 80.5% of marker-source probes and the marker itself on 9.8%, while ordinary induction is zero. Key relocation gives 73.6% paired accuracy and deleting the key lowers accuracy to 2.7%, so the solution uses the key. The model uses the replacement heads for source-dependent, task-specific recovery; the mapping may also involve other components. In the frozen-lower-network 1B experiment, the upper layers reach 98% target accuracy and pass the key tests while ordinary induction stays at or below 2/256, all with suppression in place (Appendix N). The key-triggered adaptive intervention disrupts the emerging target route; it does not show that this route is the original induction algorithm.
2. Does whole-head damage explain the effect? We edited one vector per selected head: the value it reads from the source position. The operation preserves queries, keys, attention and other value entries at that layer. Earlier edits can change later-layer inputs; other heads retain source access (Appendix R). Replacing only this vector with a distractor's value prevents learning (at most 6.6% over 500 updates). Keeping only this vector, with the head's other values set to zero, leaves learning unchanged (crossing at update 200, as without intervention; 99.3% of key pairs). The source-only arm retains its source-value gradient. Comparing detached correct and distractor values then matches the local gradient cut while varying content. In three pretraining runs × two streams, correct content permits learning in all six cells (88.3–94.9%, all passing the key tests), while the distractor's content stays below 50% (Appendix S). Natural-text-loss controls are complementary; we do not claim they match damage on the target examples.
Does enabling a head restore its original function? In the controlled model, yes: its weights and inputs are fixed. In pretrained models, enabling heads only removes the intervention. Holding the selected heads' weights at their initial values still prevents learning (at most 2.7% over 500 updates) while matched frozen controls cross at update 200 (Appendix R). New experiments report evaluation with the intervention removed and retained. Within 500 updates no suppressed model in these experiments reaches 50% in either mode; in longer runs two GPT-2 models learn under the mask about 1.5-fold later than their controls (General response 2).
3. Scope. Five independently pretrained GPT-2 models, three final GPT-2 checkpoints and final Pythia-160M reproduce the effect (Appendix T). The tasks remain copying-based, and the paper does not demonstrate a readiness predictor or a curriculum benefit (Section 8).
Summary. (1) The rebuilding claim is corrected; source-value interventions identify a causal contribution in both the original and replacement heads. (2) Edits confined to one source-position vector per head reproduce the effect, and the acquisition contrast persists with matched local gradient cuts and with head weights fixed. (3) Coverage now includes a second model family and final checkpoints; task generality remains a stated limitation.