acceptodds
Under review as a conference paper at ICLR 2027

Downshift: Induction Circuits Recruit the Fastest Surviving RoPE Band

Abstract

In pretrained Pythia-70M, disabling the fastest rotary position embedding (RoPE) band in one previous-token head reproduces 87% of the induction-accuracy drop caused by disabling it model-wide. Yet induction forms and recruits band 1 with this rotation disabled throughout training, across three data-order seeds from one public 70M initialization. Head-set knockouts at 70M and 160M verify the recruited band's contribution to synthetic induction and held-out text prediction. Separate early-checkpoint tests at 70M support fastest-surviving-frequency allocation over fixed-index selection. This *band downshift* links frequency-based recruitment to acquired functional dependence, illustrating why strong endpoint dependence need not imply developmental necessity.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.