acceptodds
Under review as a conference paper at ICLR 2027

Downshift: Induction Circuits Recruit the Fastest Surviving RoPE Band

Abstract

In pretrained Pythia-70M, disabling the fastest rotary position embedding (RoPE) band in one previous-token head reproduces 87% of the induction-accuracy drop caused by disabling it model-wide. Yet induction forms and recruits band 1 with this rotation disabled throughout training, across three data-order seeds from one public 70M initialization. Head-set knockouts at 70M and 160M verify the recruited band's contribution to synthetic induction and held-out text prediction. Separate early-checkpoint tests at 70M support fastest-surviving-frequency allocation over fixed-index selection. This *band downshift* links frequency-based recruitment to acquired functional dependence, illustrating why strong endpoint dependence need not imply developmental necessity.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
mechanistic interpretability, transformer circuits, rotary position embeddings (RoPE), induction heads and circuits, in-context learning, circuit formation, training dynamics, causal interventions

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_gCzhOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines how RoPE frequency bands support positional routing and content matching in induction circuits. In Pythia-70M, it shows that induction can form without the fastest rotation and acquire functional dependence on the next-fastest band.

Critical strengths
  1. The paper provides an interesting perspective on the familiar two-stage induction circuit through RoPE frequency bands. It concretely links previous-token routing to dependence on fast rotations, and content matching to slowly rotating or unrotated channels.

  2. The experiments provide complementary evidence through score decomposition, training-time and post-training interventions, frequency permutations, natural-text evaluation, and path patching. I particularly appreciate the distinction between attribution to a band and functional dependence established through interventions.

Critical weaknesses

As acknowledged in the paper, the experiments use different starting checkpoints, training budgets, and optimization schedules. Although the controls within each protocol are matched, these differences limit direct comparisons between formation and post-formation repair, as well as comparisons across model sizes.

Questions for authors
  1. Which findings do you expect to be specific to RoPE’s rotary structure, and which might generalize to other positional encodings? Please explain the reasoning behind these expectations.

  2. How dependent is the fastest-surviving-frequency preference on the one-token offset of previous-token routing? If the positional head instead retrieved information from a fixed integer offset k≥2k\ge2 tokens back, would you expect the same selection rule, or would the preferred band depend on the offset?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gCzh

Thank you for valuing the distinction between attribution to a band and dependence established by intervention.

[W. Different checkpoints, budgets and schedules across experiments.]

We agree that the protocols support different comparisons. Our central causal claims rest on within-protocol comparisons: formation compares matched E1 runs, and dependence compares knockouts within an endpoint. Sec. 5 reports allocation under the step-512 protocol. Two additions clarify these comparisons. (1) We retrained the ladder and permutation runs under the step-512 schedule using a fixed data stream that differs from the original. For runs with original counterparts, recruited-band margin shares differ by less than 0.04; knockouts are evaluated within each new endpoint. (2) We added controls to check whether the lower top-3/global knockout ratio at 160M persists under both E1 and E2: 70M under E2 from step 512, and 160M under E1 from the public initialization (3 data-order seeds; learning rate 6 × 10⁻⁴, otherwise the E1 protocol). The ratio is lower at 160M under both E1 and E2, ranging across the tested controls from 0.92–0.97 at 70M and 0.66–0.92 at 160M. The revision will state which within-protocol comparisons support each claim.

[Q1. Which findings are RoPE-specific, and which should generalize.]

We separate what we have evidence for from what we expect.

General principles and expectations. Dependence, preference and necessity are distinct regardless of the positional encoding. We also expect the division of labour between previous-token routing and the induction head's content lookup to generalize: content lookup must work across different offsets. In our measurements, the induction head's matching margin is concentrated in unrotated dimensions in Pythia and quasi-static bands in the 11 full-rotary survey models.

Encoding-dependent details. The fastest-surviving-frequency rule requires identifiable frequency bands. A fixed frequency basis also exists in sinusoidal absolute encodings. Our intervention exploits RoPE's multiplicative structure: setting a rotation to identity removes direct phase change while retaining the Q/K dimensions and an unrotated inner-product term. Concentration also varies across RoPE configurations. The 8-band vs. 32-band comparison changes both frequency spacing and rotary dimensionality, so their effects are not isolated (reply to Reviewer 3qwg, [W2]).

Expectations for other encodings (untested here): sinusoidal absolute encodings may show analogous reallocation across their fixed frequency bands, though learned projections can mix frequencies and couple position with content; learned absolute embeddings have no architecturally defined bands corresponding to our knockout; NoPE has no explicit positional embeddings, so our band intervention has no direct counterpart.

[Q2. Dependence on the one-token offset.]

We expect the same fastest-surviving-frequency preference for fixed-offset routing: in an idealized positional score, adjusting the phase can align the score peak with offset k; within Pythia's frequency grid, faster rotation increases the score gap between offset k and k ± 1 at equal amplitude. A one-layer k-back toy on Pythia's eight-band grid isolates the routing stage; for k ∈ {2, 3, 5, 8} (five seeds per cell, trained with and without band 0), single-band knockouts rank band 0 first in every control and band 1 in every band-0-off run (5/5 at each k; band-0 drops 0.67-0.92, band-1 drops 0.75-0.98), and the same band has the largest margin share in every run (0.57-0.64 and 0.76-0.97). These tests support the same band preference at the tested offsets in this toy.

Thank you again; we are glad to answer further questions.

Thank you again for your feedback. We have uploaded the revised manuscript with the following clarifications:

Experimental protocols and the claims they support: Table 1 and Appendix A.1 (Table 4).

Additional E1/E2 control comparisons for head-set concentration: Appendix A.2.

The distinction between formation and post-formation repair: Sec. 6 and Appendix A.9.

Reviewer_gCzhOpenReview ↗

Reply to authors

Thank you for the clarifications and additional experiments. I am satisfied with the response and will maintain my rating.

Reviewer_EjyHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces how RoPE frequencies support induction circuits. Disabling the fastest frequency after training harms induction, but a 70M model can learn induction with that frequency disabled from initialization. It instead recruits band 1. Functional knockouts at 70M and 160M show that this band contributes to synthetic induction and natural-text prediction. Separate removal and frequency-permutation experiments suggest that recruitment follows the fastest available frequency rather than a fixed dimension index.

Critical strengths
  1. The training intervention directly tests whether band 0 is needed for learning. At 70M, induction develops with band 0 disabled from initialization. This clearly separates learned dependence from developmental necessity.

  2. The band-1 result has strong functional support. Head-level knockouts sharply reduce induction accuracy after band-0-off training. Random-head controls have much smaller effects. The natural-text results also show that the recruited band matters beyond the synthetic task.

Critical weaknesses
  1. The main frequency-selection claim is only partly tested. Functional knockouts support band-1 recruitment, but the deeper removal and frequency-permutation experiments rely on attribution scores. The reported results already show that high attribution can accompany little effect on accuracy.

  2. The key findings have limited coverage. Training from initialization uses one 70M weight initialization. The 160M experiments start from an existing checkpoint, and the larger-model survey is descriptive. OLMo accuracy declines in both conditions. These results offer limited support for treating the observed shift as a general pattern.

Questions for authors
  1. Are output-level knockout results available for the recruited bands in the deeper removal and frequency-permutation experiments? These results would help assess whether the fastest surviving band carries the function or merely receives the largest attribution score.

  2. Are there results from independent weight initializations or another architecture that reproduce both successful induction learning and dependence on the recruited band? The current data-order seeds share starting weights, while the OLMo runs lose accuracy.

4 comments
AuthorsOpenReview ↗

Response to Reviewer EjyH (1/3)

Thank you for the careful assessment. We address W1/Q1 and W2/Q2 together.

[W1/Q1. Output-level knockouts for the recruited bands in the deeper-removal and permutation runs.]

Yes. We have now added these tests (Global Response 1.1). We retrained models for the ladder and permutation experiments under the original step-512 schedule, using a fixed data stream that differs from the original, with three seeds per condition. At each new endpoint we applied rotation knockouts globally and to the top-k previous-token heads for every surviving band, measured held-out-text costs, and tested 10 layer-matched random head sets for the predicted band.

On the 64-token probe, the predicted band has the largest global knockout cost and its top-3 knockout cost exceeds that of every random head set at all 18 endpoints. This includes causal support for the deeper-rung replacements (bands 2 and 3) and the permuted replacement. For the latter (index 6, θ = 0.316), global cost is 0.59 vs. < 0.12 for any other surviving band, and top-3 cost is 0.54–0.56 vs. ≤ 0.04 for random sets. These interventions establish functional dependence on the recruited rotation.

Large margin shares do not necessarily imply large knockout effects, as the new results also illustrate (Global Response 1.2). On held-out text the recruited band dominates in the tested conditions whose fastest surviving band has a wavelength of approximately 6 or 20 tokens; at rungs 2 and 3 several slower bands cost as much or more. The revised Sec. 5.1 will state this scope.

AuthorsOpenReview ↗

Response to Reviewer EjyH (2/3)

[W2/Q2. Coverage: initializations, the 160M start, the survey, OLMo.]

Yes. We reproduce both formation without band 0 and dependence on band 1 across all three independent 70M weight initializations. The experiments below distinguish this result from the two kinds of 160M evidence. Each pairs control training with band-0-off training.

  1. 70M: independent initializations, step 0. We ran E1 on pythia-70m-seed{1,2,3}, with data order fixed (train seed 0). The band-0-off models learn induction (accuracy 0.77-0.79) and depend on band 1: top-3 rotation knockouts reduce accuracy by 0.570-0.719 vs. ≤ 0.017 for layer-matched random sets of three heads. The table reports each seed's results.
  2. 160M: independent initializations, continuation from step 512. We ran E2 on the step-512 checkpoints of pythia-160m-weight-seed{1,2,3}, again with data order fixed (train seed 0). The band-0-off models reach accuracy 0.76-0.79 and depend on band 1 (top-3 drops 0.459-0.473 vs. ≤ 0.021 for random sets of three heads). This tests dependence after continuation across independent initializations.
  3. 160M: one public initialization, step 0. We ran E1 with three data-order seeds from the same public step-0 weights (learning rate 6 × 10⁻⁴, otherwise the E1 protocol). Controls learn induction (0.82–0.84) and depend on band 0. Band-0-off models also learn induction (0.81-0.84) and depend on band 1 (top-3 drops 0.56-0.64 vs. < 0.004 for random sets of three heads).

Another architecture and RoPE configurations. A 70M-scale Llama-style model trained from scratch learns induction and depends on θ = 0.316 when the four fastest rotations are disabled throughout training (3 of 3 seeds; global knockout cost 0.73-0.77). Under the same four-band removal, the 32-band Pythia configuration recruits θ = 0.316 and the base-500,000 configuration recruits θ = 0.194 (3 of 3 each). Full results are in our reply to Reviewer 3qwg, [W2]. The original 11-model survey remains descriptive (A.7).

Additional checks. On all 18 public PolyPythias final checkpoints, band-0 rotation knockouts reduce accuracy (global cost 0.47-0.67; top-3 cost 0.36-0.59 vs. < 0.014 for random sets of three heads). Head-level concentration varies across initializations; the revision will identify Sec. 3's single-head result as a property of the public checkpoint and report head-set effects throughout. Protecting the top-3 previous-token heads during position interpolation recovers 83-93% of the PI-induced accuracy loss across the 12 endpoints of the 70M and 160M weight-seed experiments (reply to Reviewer 3qwg, [W3]).

OLMo. Accuracy declines in both training conditions for an unresolved reason. We discuss this limitation in Limitations and report the detailed results in A.8; these runs do not establish successful induction learning or dependence on a recruited band.

AuthorsOpenReview ↗

Response to Reviewer EjyH (3/3) per-initialization table for [W2, Q2]

Costs are accuracy drops on the 64-token probe from setting the predicted band's rotation to identity globally or in the top-3 previous-token heads selected by previous-token attention on an independent calibration probe. Max random: largest drop over 10 layer-matched head sets of the same size. Original E1/E2 results are in Tables 2–3. Semicolon-separated values follow the seed order shown for each table.

(1) E1 at 70M from step 0: three independent PolyPythias initializations (data order fixed; the paper's E1 protocol)

Weight-seed order: 1; 2; 3.

Condition (band)Native acc.Global costTop-3 cost (max random)
control (band 0)0.752; 0.768; 0.7600.676; 0.636; 0.6960.645 (0.017); 0.595 (0.005); 0.661 (0.047)
band 0 off (band 1)0.787; 0.773; 0.7680.725; 0.683; 0.7500.646 (0.017); 0.570 (0.011); 0.719 (0.010)

(2) E2 at 160M: step-512 continuation across three weight seeds (independent initializations, continuation data order fixed; the paper's E2 protocol)

Weight-seed order: 1; 2; 3.

Condition (band)Native acc.Global costTop-3 cost (max random)
control (band 0)0.774; 0.779; 0.7690.661; 0.674; 0.6440.544 (0.004); 0.605 (0.011); 0.564 (0.010)
band 0 off (band 1)0.787; 0.776; 0.7640.603; 0.643; 0.6340.472 (0.019); 0.459 (0.014); 0.473 (0.021)

(3) E1 at 160M from one public step-0 initialization (lr 6 × 10⁻⁴, otherwise the E1 protocol; three data-order seeds)

Data-order seed order: 0; 1; 2.

Condition (band)Native acc.Global costTop-3 cost (max random)
control (band 0)0.841; 0.831; 0.8170.768; 0.761; 0.7690.510 (0.001); 0.636 (0.006); 0.621 (0.031)
band 0 off (band 1)0.843; 0.821; 0.8140.795; 0.764; 0.7500.642 (0.003); 0.622 (0.002); 0.562 (0.002)

Every band-0-off run attains high induction accuracy with band-1 dependence, while controls depend on band 0; all 18 top-3 knockout costs exceed their maximum random-control cost.

Timing, margin shares, natural-text results, and additional diagnostics will be included in the revised appendix.

Thank you again for your feedback. We have uploaded the revised manuscript incorporating the results and clarifications presented in our responses:

Output-level knockouts for the deeper-rung and permuted replacements, including context dependence: Sec. 5.1 and Appendix A.11 (Tables 15-17).

Expanded initialization and continuation experiments: Secs. 4.1-4.3 and Appendices A.1-A.2. Tables 4-5 specify the starting checkpoints and seed settings and report the causal results.

Llama-style replication: Sec. 4.4 and Appendix A.13. The OLMo results and their scope are clarified in Appendix A.8 and Limitations.

Reviewer_3qwgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that previous-token heads in Pythia-70M rely primarily on the highest-frequency component of RoPE. However, when this frequency is disabled during training, the model recruits the next-highest frequency to implement the same function, which also shows a similar sensitivity to ablation. This suggests that a functional dependency observed in a trained model does not necessarily imply developmental necessity.

Critical strengths
  1. I find this phenomenon quite interesting. It encourages further investigation into which architectural components are truly necessary for developing certain capabilities and which are simply preferred by the training process.

  2. The counterfactual retraining experiments, along with additional controls such as frequency permutations, make the findings reasonably convincing.

Critical weaknesses
  1. Limited causal evidence for the ablation ladder. Only the band-0-to-band-1 transition is supported by causal knockout experiments, while the other transitions in the ablation ladder rely solely on margin attribution. Since margin attribution does not necessarily reflect causal importance, the lack of causal validation weakens the evidence for the proposed fastest-surviving-frequency rule.

  2. The experiments are limited to the Pythia family. It is unclear whether the findings generalize to other architectures, particularly those with different implementations or configurations of RoPE.

  3. The practical implications of the findings remain unclear. While the phenomenon itself is interesting, I struggle to see how this insight could be directly useful or lead to potential applications.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3qwg (1/3)

Thank you for the careful reading and for framing the question we care most about: which components are necessary and which only preferred.

[W1. Causal evidence for the ablation ladder.]

We agree that margin shares alone do not establish dependence on a band's rotation (Sec. 4.2; Table 4). We retrained models for the ladder and permutation experiments under the original step-512 schedule, using a fixed data stream that differs from the original, with three seeds per condition. We then tested every surviving band using rotation knockouts applied globally and to the top-k previous-token heads, with layer-matched random-head controls for the predicted band (Global Response 1.1).

On the 64-token probe, the predicted band has the largest global knockout cost, and its top-3 knockout cost exceeds that of every random head set, at all 18 endpoints. This provides causal evidence for bands 2 and 3 and the permuted replacement.

These qualifications will be reported in Sec. 5.1 and A.11. At rung 2, rotation knockouts of band 3 alone reduce accuracy by 0.33–0.38, band 2 alone by 0.43–0.46, and both together by 0.59–0.60, while band 3's margin share is 0.19–0.20. On held-out text, the recruited band dominates in the tested conditions whose fastest surviving band has a wavelength of approximately 6 or 20 tokens, not at the deeper rungs.

[W2. Generalization beyond Pythia.]

We added training and rotation-knockout tests for different RoPE configurations and a Llama-style architecture. The original 11-model survey remains descriptive (A.7).

RoPE configuration. In full-rotary Pythia-block models (32 bands instead of 8), we tested whether disabling the four fastest rotations throughout training would shift dependence to bands 4–7, led by band 4 (θ = 0.316), which has the same frequency as band 1 in the original eight-band Pythia configuration. This holds in 3 of 3 seeds: the group knockout reduces accuracy by 0.74–0.76, and band 4 alone by 0.52–0.61 vs. ≤ 0.072 for any other surviving band. With bands 0–3 disabled throughout training, raising the RoPE base to 500,000 instead recruits θ = 0.194, absent from Pythia's original grid, in 3 of 3 seeds (global cost 0.30–0.56 vs. < 0.075 for any other surviving band). Concentration on a single band varies across these configurations; the control and group results are in the per-run table for [W2].

Architecture. A 70M-scale Llama-style decoder (RMSNorm, SwiGLU, no biases, Llama initialization, full rotary) was trained from scratch with the same tokenizer, data and schedule. Its controls depend on the fastest band. With the four fastest rotations disabled throughout training, all three seeds learn induction and depend on θ = 0.316: top-3 rotation knockouts reduce accuracy by 0.71–0.75 vs. ≤ 0.09 for layer-matched random sets. All 18 endpoints across the three configurations form induction; the per-run table gives the global, group and head-set results.

Independent-initialization and 160M from-initialization results are reported separately in Global Response 1.3. The OLMo attempt still loses induction accuracy in both training conditions for an unresolved reason (A.8); the new architecture result concerns the controlled 70M-scale Llama-style model described above.

AuthorsOpenReview ↗

Response to Reviewer 3qwg (2/3) per-run table for [W2]: RoPE configuration and architecture

All 18 endpoints: three configurations at 70M scale, trained from step 0 for 32,000 updates with the E1 schedule. Seeds vary data order from public step-0 weights in the two full-rotary Pythia-block configurations; the Llama block varies both weight initialization and data order. Controls rotate all 32 bands (predicted index 0, θ = 1). Bands-0–3-off runs set those rotations to identity throughout training (predicted index 4: θ = 0.316 at base 10,000; θ = 0.194 at base 500,000).

Costs are accuracy drops on the 64-token probe after setting selected rotations to identity. Predicted-band and group costs are global. Max other: largest single-band cost among other surviving bands. Group: predicted four bands / next four in frequency order. Top-3: three previous-token heads selected by previous-token attention on an independent calibration probe; parentheses give the maximum over 10 layer-matched random sets for the same predicted band.

32-band Pythia block, base 10,000

RunNative acc.Global cost (max other)Group cost: predicted / next 4Top-3 (max random)
seed 0, control0.7690.391 (0.038)0.74 / 0.050.315 (0.003)
seed 1, control0.7570.473 (0.047)0.75 / 0.060.440 (0.002)
seed 2, control0.7540.389 (0.048)0.74 / 0.120.342 (0.008)
seed 0, bands 0–3 off0.7640.607 (0.030)0.76 / 0.070.531 (0.007)
seed 1, bands 0–3 off0.7480.515 (0.032)0.74 / 0.070.435 (0.007)
seed 2, bands 0–3 off0.7640.544 (0.072)0.76 / 0.150.462 (0.004)

Llama block, base 10,000

RunNative acc.Global cost (max other)Group cost: predicted / next 4Top-3 (max random)
seed 0, control0.8080.577 (0.019)0.78 / 0.020.565 (0.012)
seed 1, control0.8230.525 (0.036)0.79 / 0.030.440 (0.005)
seed 2, control0.7940.557 (0.021)0.78 / 0.030.491 (0.001)
seed 0, bands 0–3 off0.8130.766 (0.035)0.81 / 0.010.737 (0.055)
seed 1, bands 0–3 off0.8100.729 (0.049)0.80 / 0.020.709 (0.010)
seed 2, bands 0–3 off0.8070.774 (0.028)0.80 / 0.020.752 (0.088)

Pythia block, base 500,000

RunNative acc.Global cost (max other)Group cost: predicted / next 4Top-3 (max random)
seed 0, control0.7500.524 (0.034)0.74 / 0.130.497 (0.002)
seed 1, control0.7490.562 (0.055)0.74 / 0.110.550 (0.004)
seed 2, control0.7650.564 (0.045)0.76 / 0.230.532 (0.006)
seed 0, bands 0–3 off0.7590.366 (0.074)0.74 / 0.180.291 (0.004)
seed 1, bands 0–3 off0.7730.297 (0.064)0.75 / 0.180.234 (0.006)
seed 2, bands 0–3 off0.7820.560 (0.067)0.77 / 0.210.523 (0.003)

In base-10,000 Pythia with bands 0–3 disabled throughout training, knocking out bands 4–7 together reduces accuracy more than knocking out bands 8–11 together. Across all 18 endpoints, the predicted band has the largest global knockout cost, and its top-3 knockout cost exceeds that of every random head set.

AuthorsOpenReview ↗

Response to Reviewer 3qwg (3/3)

[W3. Practical implications.]

A practical use of our analysis is to identify and protect previous-token heads that are sensitive to position interpolation (PI; Chen et al., 2023), reducing degradation within the training context. Keeping three such heads at their native frequencies recovers 79–95% of the PI-induced loss in induction accuracy, compared with 4% or less on average for layer-matched random sets of three heads. We selected the heads using the paper's attention-based head-selection criterion and applied PI to the remaining heads.

These zero-shot tests used pure PI without fine-tuning, at s = 2, on the 12 E1/E2 endpoints, the public 70M and 160M, and the 12 independent-initialization models. PI on every head lowers accuracy from 0.75–0.82 to 0.01–0.11 (0.16–0.22 in the band-0-off 160M models). Including the six permuted models, recovery ranges from 77–95% of the lost accuracy.

The band most sensitive to PI follows training history and frequency assignment. Applying PI only to one band in the top-3 heads gives:

Training conditionBand receiving PI (s = 2) only in top-3 headsInduction accuracy
All bandsBand 00.06–0.25
All bandsBand 10.73–0.82
Band 0 offBand 10.09–0.38

In the permuted models, the sensitive band follows the fastest surviving frequency to index 7, or index 6 when index 7 is disabled throughout training.

The same head protection also reduces held-out natural-text loss under PI at s = 4. Entries below are NLL increases over the corresponding native model, in nats/token:

ModelsPI on every headPI with top-3 heads kept at native frequencies
12 E1/E2 endpoints+1.0 to +1.2+0.30 to +0.44
Public 70M+1.24+0.49
Public 160M+1.55+0.84

Appendix A.10 contrasts PI with changes to the RoPE base: PI slows every frequency, whereas changing the base leaves the original band-0 frequency unchanged.

All evaluations are within the training context; long-context quality was not evaluated.

Thank you again for your feedback. We have uploaded the revised manuscript incorporating the results and clarifications presented in our responses:

Causal evidence for the ladder and frequency permutation, including context dependence: Sec. 5.1 and Appendix A.11 (Tables 15-17).

RoPE configuration and Llama-style experiments: Sec. 4.4 and Appendix A.13 (Tables 18-19).

Previous-token head protection under position interpolation within the training context: Sec. 6 and Appendix A.14 (Tables 20-22).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers. We added causal tests, replications and practical evaluations.

1. New evidence

[1.1 The ladder and frequency permutation (3qwg-W1, EjyH-W1/Q1).]

After retraining under six conditions with three seeds each, the predicted band has the largest global knockout cost and its top-3 knockout cost exceeds that of every random head set at all 18 endpoints on the 64-token probe.

Knockouts set rotations to identity; costs are accuracy drops. Ranges span three seeds. Top-3 denotes previous-token heads. Max other: largest cost among other surviving bands; max random: maximum over 10 layer-matched head sets.

Endpoint (predicted band, θ)Native acc.Global cost: predicted / max otherTop-3: predicted / max random
control (band 0, θ = 1)0.67–0.680.57–0.58 / 0.06–0.080.54–0.55 / 0.04–0.06
ablate {0} (band 1, θ = 0.316)0.66–0.700.58–0.60 / 0.06–0.080.54–0.56 / 0.01–0.03
ablate {0,1} (band 2, θ = 0.1)0.64–0.670.43–0.46 / 0.33–0.380.37–0.38 / 0.01–0.06
ablate {0,1,2} (band 3, θ = 0.032)0.63–0.670.41–0.44 / 0.04–0.060.29–0.31 / 0.02–0.05
perm, no ablation (idx 7, θ = 1)0.65–0.690.56–0.59 / 0.06–0.090.53–0.56 / 0.07–0.09
perm, idx 7 ablated (idx 6, θ = 0.316)0.67–0.680.59 / 0.06–0.110.54–0.56 / 0.01–0.04

[1.2 Margin share and context dependence.]

Margin-share magnitude does not establish knockout effect size. On the 64-token probe, band 3 at rung 2 has share 0.19–0.20 and global knockout cost 0.33–0.38; band 4 at rung 3 has share 0.20 and cost 0.04–0.06. With global knockouts on held-out text, the recruited band dominates in the tested conditions whose fastest surviving band has a wavelength of approximately 6 or 20 tokens. At rungs 2–3, its knockout still increases NLL, but slower bands have larger effects. In longer-context probes, recruited-band costs are comparatively stable, while the largest slower-band costs generally increase. Slower bands cause larger global accuracy drops in tested blocks or prefixes from 128 tokens at rungs 2–3, and at prefix 384 at every 70M rung. Functional dependence does not imply dominance in every context.

[1.3 Initializations and protocols (EjyH-W2/Q2).]

Independent 70M initializations reproduce formation without band 0 and dependence on band 1. At 160M, step-0 runs share one initialization and learn induction without band 0; independent-weight runs continue from step 512. Both show band-1 dependence after band-0-off training. Additional E1/E2 controls retain lower top-3/global ratios at 160M (gCzh-W).

[1.4 Architecture, practical use and offset.]

Different RoPE configurations and a Llama-style model also show recruitment and dependence (3qwg-W2). Head protection reduces accuracy loss under zero-shot position interpolation (PI) within the training context (3qwg-W3). A one-layer routing toy tests fixed offsets (gCzh-Q2).

2. Revision (by Nov 16, AoE): Sec. 5.1 will include ladder/permutation causal tests and context dependence, with detailed margin-share comparisons, knockout and text results in A.11. Sec. 3 will distinguish the public checkpoint's single-head result from head-set evidence across initializations. Sec. 4 will incorporate the expanded initialization and RoPE/architecture experiments, with details in A.1–A.2 and A.13. Sec. 6 will summarize PI head protection; training-time scaling and detailed PI evaluations will remain in A.12 and A.14, respectively. A.8 will clarify the unsuccessful OLMo results, and Limitations will state the scope of the expanded evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.