- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper examines how RoPE frequency bands support positional routing and content matching in induction circuits. In Pythia-70M, it shows that induction can form without the fastest rotation and acquire functional dependence on the next-fastest band.
Critical strengths
-
The paper provides an interesting perspective on the familiar two-stage induction circuit through RoPE frequency bands. It concretely links previous-token routing to dependence on fast rotations, and content matching to slowly rotating or unrotated channels.
-
The experiments provide complementary evidence through score decomposition, training-time and post-training interventions, frequency permutations, natural-text evaluation, and path patching. I particularly appreciate the distinction between attribution to a band and functional dependence established through interventions.
Critical weaknesses
As acknowledged in the paper, the experiments use different starting checkpoints, training budgets, and optimization schedules. Although the controls within each protocol are matched, these differences limit direct comparisons between formation and post-formation repair, as well as comparisons across model sizes.
Questions for authors
-
Which findings do you expect to be specific to RoPE’s rotary structure, and which might generalize to other positional encodings? Please explain the reasoning behind these expectations.
-
How dependent is the fastest-surviving-frequency preference on the one-token offset of previous-token routing? If the positional head instead retrieved information from a fixed integer offset tokens back, would you expect the same selection rule, or would the preferred band depend on the offset?
3 comments
Thank you again for your feedback. We have uploaded the revised manuscript with the following clarifications:
Experimental protocols and the claims they support: Table 1 and Appendix A.1 (Table 4).
Additional E1/E2 control comparisons for head-set concentration: Appendix A.2.
The distinction between formation and post-formation repair: Sec. 6 and Appendix A.9.
Reply to authors
Thank you for the clarifications and additional experiments. I am satisfied with the response and will maintain my rating.
Response to Reviewer gCzh
Thank you for valuing the distinction between attribution to a band and dependence established by intervention.
[W. Different checkpoints, budgets and schedules across experiments.]
We agree that the protocols support different comparisons. Our central causal claims rest on within-protocol comparisons: formation compares matched E1 runs, and dependence compares knockouts within an endpoint. Sec. 5 reports allocation under the step-512 protocol. Two additions clarify these comparisons. (1) We retrained the ladder and permutation runs under the step-512 schedule using a fixed data stream that differs from the original. For runs with original counterparts, recruited-band margin shares differ by less than 0.04; knockouts are evaluated within each new endpoint. (2) We added controls to check whether the lower top-3/global knockout ratio at 160M persists under both E1 and E2: 70M under E2 from step 512, and 160M under E1 from the public initialization (3 data-order seeds; learning rate 6 × 10⁻⁴, otherwise the E1 protocol). The ratio is lower at 160M under both E1 and E2, ranging across the tested controls from 0.92–0.97 at 70M and 0.66–0.92 at 160M. The revision will state which within-protocol comparisons support each claim.
[Q1. Which findings are RoPE-specific, and which should generalize.]
We separate what we have evidence for from what we expect.
General principles and expectations. Dependence, preference and necessity are distinct regardless of the positional encoding. We also expect the division of labour between previous-token routing and the induction head's content lookup to generalize: content lookup must work across different offsets. In our measurements, the induction head's matching margin is concentrated in unrotated dimensions in Pythia and quasi-static bands in the 11 full-rotary survey models.
Encoding-dependent details. The fastest-surviving-frequency rule requires identifiable frequency bands. A fixed frequency basis also exists in sinusoidal absolute encodings. Our intervention exploits RoPE's multiplicative structure: setting a rotation to identity removes direct phase change while retaining the Q/K dimensions and an unrotated inner-product term. Concentration also varies across RoPE configurations. The 8-band vs. 32-band comparison changes both frequency spacing and rotary dimensionality, so their effects are not isolated (reply to Reviewer 3qwg, [W2]).
Expectations for other encodings (untested here): sinusoidal absolute encodings may show analogous reallocation across their fixed frequency bands, though learned projections can mix frequencies and couple position with content; learned absolute embeddings have no architecturally defined bands corresponding to our knockout; NoPE has no explicit positional embeddings, so our band intervention has no direct counterpart.
[Q2. Dependence on the one-token offset.]
We expect the same fastest-surviving-frequency preference for fixed-offset routing: in an idealized positional score, adjusting the phase can align the score peak with offset k; within Pythia's frequency grid, faster rotation increases the score gap between offset k and k ± 1 at equal amplitude. A one-layer k-back toy on Pythia's eight-band grid isolates the routing stage; for k ∈ {2, 3, 5, 8} (five seeds per cell, trained with and without band 0), single-band knockouts rank band 0 first in every control and band 1 in every band-0-off run (5/5 at each k; band-0 drops 0.67-0.92, band-1 drops 0.75-0.98), and the same band has the largest margin share in every run (0.57-0.64 and 0.76-0.97). These tests support the same band preference at the tested offsets in this toy.
Thank you again; we are glad to answer further questions.