acceptodds
Under review as a conference paper at ICLR 2027

When Native Fit Misleads: Evaluating Attention Surrogates under Positional Interventions

Abstract

Attention surrogates offer interpretable approximations of attention mechanisms, but strong reconstruction of native attention may not imply faithful prediction under positional interventions. We study this gap using controlled position-ID shifts that modify the positional information of a sequence suffix within a single attention head while keeping tokens, causal masks, and upstream representations fixed. We compare two parameter-matched surrogate families for RoPE attention in Qwen3-1.7B-Base: a smooth positional-field surrogate (Cubic Field) and a learned-frequency rotary surrogate (Rotary Field). Although native-validation KL consistently selects Rotary Field, we identify reproducible heads on OpenWebText and WikiText-103 for which it is less faithful to positional interventions than both Cubic Field and a no-change baseline. The misranking persists in model-output evaluations, although the output-level differences are small. An exploratory extension to T5 suggests that this phenomenon is not limited to RoPE. These results demonstrate that native reconstruction fidelity alone does not guarantee positional-intervention fidelity and highlight the need to evaluate attention surrogates directly on the mechanistic responses they aim to explain.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, attention surrogates, positional encoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NLsdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether attention surrogates that fit native attention can also predict responses to positional interventions. It compares Cubic Field and Rotary Field while keeping tokens and upstream states fixed. On several Qwen heads, Rotary Field fits native attention better but predicts intervention effects less accurately. The main findings are replicated on another dataset, with additional experiments examining the scope of the results.

Critical strengths

Freezing upstream states and anchoring predictions to native teacher scores cleanly separates positional-response error from native replacement error. Null gives the normalized risk a meaningful reference. Meanwhile, the main Qwen reversals persist across seeds, disjoint corpus evaluations, and gap geometries. Paired document inference and the disclosure of negative controls, small output effects, and unestablished Cubic-over-Null benefits make the bounded empirical claim credible.

Critical weaknesses
  1. The contribution is limited. Reconstruction does not ensure interventional fidelity is already established by Dyer et al. [1], and attention proxies already undergo causal replacement checks in Hayes et al. [2]. The controlled positional-response protocol and replicated Qwen counterexamples are useful additions, but the paper does not establish which conclusions about existing attention-explanation methods should change as a result. Its main comparison concerns two purpose-built surrogate families. Thus, the demonstrated contribution is largely a specialized empirical case study rather than a substantially new understanding of surrogate faithfulness.
  2. The reasons for the failures remain unclear. The paper shows that better native fit can lead to worse intervention predictions, but does not clearly explain why this happens or which heads are likely to fail. The frequency analyses do not resolve this issue, limiting the additional insight beyond demonstrating the mismatch.

[1] Interventionally Consistent Surrogates for Complex Simulation Models. NeurIPS 2024.

[2] Explaining Attention with Program Synthesis. arXiv 2026.

Questions for authors
  1. Does the ranking reversal persist when both native fit and intervention fidelity are evaluated using attention KL?
  2. Is there evidence explaining why the identified Qwen heads fail while the other tested heads do not?
  3. Could simply scaling down Rotary Field’s predicted changes match Cubic Field’s performance, with the scale chosen independently of the test data?
  4. What do the Table 5 results establish about the claim that the mismatch is “not confined to RoPE”, given that no target meets the combined selection-failure criterion?
2 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer NLsd

We thank Reviewer NLsd for the thoughtful feedback. This comment addresses the first three questions. We address the T5 results and the comparison with prior work in a second, continued comment.

Does the reversal persist under attention KL at both stages?

Yes, with simultaneous interval support on L21H13 and L25H12. We reanalyzed 96 archived head–method records from 32 Qwen heads on the same 512 WikiText-103 documents (cutoff 128, gap 64). Cubic (C) and Rotary (R) are fixed three-seed ensembles. For shifted KL, we add each predicted change to the teacher’s native scores. A zero change (0) preserves the native attention.

Table 29 in Appendix M gives the differences in 10−310^{-3} nats per valid query row. Positive native C−R favors Rotary. Positive shifted R−C and R−0 favor Cubic and no change, respectively, because lower KL is better. Brackets are simultaneous 95% intervals for all nine contrasts, based on 10,000 paired document-bootstrap draws.

  Head  Native KL (Cubic − Rotary)  Shifted KL (Rotary − Cubic)  Shifted KL (Rotary − 0)
L21H1315.50 [13.66, 17.34]2.51 [2.37, 2.65]2.08 [1.89, 2.27]
L24H100.88 [−1.23, 2.99]1.72 [1.58, 1.85]2.21 [2.08, 2.33]
L25H126.92 [4.71, 9.12]2.81 [2.65, 2.96]3.17 [3.02, 3.32]

Only the native interval for L24H10 crosses zero. Shifted KL favors Cubic and no change over Rotary on all three heads. By point estimate, 8 of 32 heads reverse under matched KL and 3 of 32 under matched MSE (Appendix M, Tables 29–31, pp. 55–57).

Why do these heads fail?

Figure 3 and Appendix I.1–I.2 (pp. 35–36) identify one source of the squared error. On the three historical failure heads, Rotary predicted changes 1.15–1.25 times the teacher’s response norm, with directional cosines of 0.371–0.475. On three selected success controls, its cosines were 0.789–0.883. Expanding the squared error in Eq. 6 shows why this matters. A prediction does worse than zero change when its squared norm exceeds twice its inner product with the teacher’s change. All three failure heads satisfy this condition (Table 16).

We also tested a possible frequency-based explanation. On L24H10, fixing the frequencies to the teacher’s in a separately fitted surrogate reduced the risk from 1.632 to 1.412. This is still above 1, the value for no change (Table 16). Across 32 OpenWebText heads, the native frequency-discrepancy statistic had a rank correlation of −0.107 with Rotary response risk. It also fell below our prespecified threshold for separating the failure heads (AUC 0.759 against a threshold of 0.80; Appendix F, Table 10).

Could scaling down Rotary explain the Cubic advantage?

It could. A scale of zero gives the no-change baseline. At cutoff 128 and gap 64, zero had lower point risk than Cubic on two of the three historical heads in the separate WikiText cohort. Cubic had lower point risk than zero on all three original OpenWebText heads (Tables 5 and 16, pp. 21 and 30).

To test a nonzero scale independently, we examined the disjoint 128-document calibration archive. It retains only the squared prediction error and the teacher energy. These two quantities do not determine the prediction norm and the teacher–prediction dot product that a scale fit needs. The independent comparison we can report is therefore the zero-scale endpoint.

AuthorsOpenReview ↗

Authors’ Response to Reviewer NLsd (continued)

How far does the T5 result go?

No T5 target met the combined selection-failure rule (0/15). At cutoff 64 and gap 128, the mean native-policy risk exceeded no change by 0.211 and exceeded the bucket-aware surrogate by 0.418. Only seven material heads supported both comparisons, below the required 16. We read this as a localized discrepancy in a model with additive positional bias.

The revised abstract states that none of the 15 T5 targets met the combined criterion (p. 1). The Introduction distinguishes this local T5 discrepancy from the replicated Qwen finding (p. 2). See Section 7, Fig. 4 and Appendix K, Tables 23–24 (pp. 49–50).

What is new relative to Dyer and Hayes?

Dyer et al. show that observationally fitted simulator surrogates can misjudge policy interventions. Hayes et al. select executable attention programs by agreement with held-out attention maps, and they evaluate head replacement through perplexity and task accuracy. Neither study tests whether selecting a surrogate by native attention fit preserves the ranking of predictions for a position shift isolated to one frozen head.

As described in Sections 2–3, we hold tokens, causal masks and upstream states fixed. We add each predicted change to the same native teacher scores (Section 3.2, p. 4), so the zero-change baseline preserves the head’s native attention. On the same three Qwen heads across two corpora, with fresh fits, native KL selected the surrogate family whose predicted score change had a higher error than the no-change baseline (Table 1 and Appendix B). For a method used to explain positional sensitivity, the natural test is whether its predicted change improves on leaving the head at its native attention.

Reference

  • Joel Dyer, Nicholas Bishop, Yorgos Felekis, Fabio Massimo Zennaro, Anisoara Calinescu, Theodoros Damoulas, Michael Wooldridge. "Interventionally Consistent Surrogates for Complex Simulation Models." NeurIPS (2024).
  • Amiri Hayes, Belinda Z Li, Jacob Andreas. "Explaining Attention with Program Synthesis." arXiv (2026).
Reviewer_AQ6MOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper motivates the problem of how attention surrogates cannot natively reconstruct changes to the positional embeddings of an attention head, and presents several nuggets of evidence to support this claim.

Critical strengths

Interesting problem statement creating a framework for attention surrogates when interventions are a change in the positional embeddings.

Critical weaknesses
  • Sloppy writing with heavy AI usage (randomly selected paragraph, lines 58-68 is 100% AI generated according to Pangram. Authors say in teh AI use statement that they only used AI for manuscript polishing, unsure if this is polishing or more that was undeclared). Please re-write your paper to be clear, explaining the first occurrence of jargon. For e.g. from the abstract:

    • "strong reconstruction of native attention may not imply faithful prediction under positional interventions" -- predictions of what exactly?
    • native validation KL -- what does this even mean?
    • The misranking persists in model output evaluations, although the output-level differences are small -- mis-ranking of what? output-level differences between what?
    • Citation for claim in sentences 43-45, 45-46?
  • So.Much.Jargon. It's unclear whether this is from a sub-field that is niche to interpretability, but the text is full of jargon and makes the paper extremely hard to read. The terms teacher, surrogate etc. are new and should be defined carefully. This paper is very very hard to read.

Questions for authors

Qwen is evaluated on 32 prespecified stress-test heads pre-specified where?

1 comment
AuthorsOpenReview ↗

Authors’ Response to Reviewer AQ6M

We thank Reviewer AQ6M for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.

How the 32 Qwen heads were fixed.

Appendix A of the submitted paper (p. 17, lines 864–879) lists all 32 layer/head identifiers. After excluding the earlier discovery heads, we crossed four depth bins with four head-index bins and drew two heads from each of the 16 cells. Within this 32-head panel, we designated a six-head subset that had shown reversals in earlier work. We made this designation before we evaluated the fresh confirmation documents. We then reused the panel on the second corpus, with fresh fits and new document assignments.

This sequence is what “prespecified” means in the paper. It supports confirmation on a defined stress-test panel. It does not estimate how often the effect occurs across a whole model. Section 4.1 of the revised paper (p. 5) explains the selection procedure and points readers to the roster on p. 17.

What the paper predicts.

In the revised Introduction (p. 1), we define an attention surrogate as a fitted approximation of selected computations or behaviors of an attention module. In our experiments, the teacher is the original frozen head, and the surrogate approximates its scores. For the same text, we shift the position IDs of a suffix inside that one head. Tokens, causal mask and upstream representations stay fixed. The surrogate then predicts the change in the head’s pre-softmax attention scores. We compare this prediction with the change the original head actually makes.

Native-validation KL, misranking and output differences.

Native-validation KL measures the gap between the original and fitted attention probabilities on held-out text with ordinary position IDs. Lower is better. We rank the same surrogate families twice: first by native fit, then by error in predicting the positional change.

On OpenWebText head L24H10, native-validation KL favors Rotary (0.0770 versus 0.0820). After the shift, the normalized response risk favors Cubic (0.9823) over Rotary (3.2591). Predicting no change gives a risk of 1.

The separate output test follows the effect through the rest of the network and compares next-token probabilities. On the three qualifying OpenWebText heads, Rotary’s mean excess output KL over Cubic is 1.709×10−51.709 \times 10^{-5} nats/token. The revised abstract (p. 1) distinguishes the score change within a head from the next-token comparison averaged over qualifying heads. Section 4.2 (p. 7) states the L24H10 reversal in one sentence, with the values in Table 5 (p. 21).

Support for lines 43–46.

The first sentence defines our choice of validation metric. The second explains why a good fit on ordinary observations cannot, by itself, establish accuracy under an intervention. We now cite Dyer et al. (NeurIPS 2024), already in Related Work, at the point where this distinction first appears (p. 1). Equation 4 in Section 3.2 gives the attention-specific reason: the response error is the difference between the reconstruction errors at the two position maps.

AI use and human review.

We wrote the prose of the submitted manuscript and used AI to polish it. The AI Use Statement (p. 10) separately discloses AI assistance with literature search, code implementation, execution and debugging, and figures. The submission form also discloses research assistance. A detector score for a single paragraph cannot establish who wrote or reviewed it.

Reference

  • Joel Dyer, Nicholas Bishop, Yorgos Felekis, Fabio Massimo Zennaro, Anisoara Calinescu, Theodoros Damoulas, Michael Wooldridge. "Interventionally Consistent Surrogates for Complex Simulation Models." NeurIPS (2024).
Reviewer_ddjvOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper examines whether attention surrogates that better reconstruct native attention also predict positional interventions more faithfully. It introduces a controlled, head-local position-ID intervention and compares parameter-matched Cubic Field and Rotary Field surrogates. Despite better native fit, Rotary Field performs worse than Cubic Field and a no-change baseline on several Qwen heads, with replication across corpora and gap configurations. The misranking persists at the model-output level, although absolute effects are small. Pythia does not exhibit the same failure.

Critical strengths

Controlled evaluation design - Holding tokens and upstream representations fixed, and adding predicted changes to the teacher’s native scores, isolates intervention-response error from native reconstruction error. This makes the comparison interpretable.

Reproducible counterexamples. The principal Qwen failures replicate across fresh corpus-specific fits and multiple intervention configurations, with seedwise checks and simultaneous confidence bounds. This supports their reliability rather than a chance ranking reversal.

Critical weaknesses

Limited demonstrated scope and practical significance. The strongest evidence concerns a few heads in a selected Qwen panel. Output differences are approximately 10−510^{-5} nats/token, Cubic Field does not reliably outperform the no-change baseline at the output level, and end-to-end token-insertion experiments do not establish the same failure. These limitations reduce the result’s broader significance.

Limited explanatory insight. The study establishes that misranking occurs but does not identify its cause. The spectral diagnostic is unsuccessful, and Cubic Field’s lower response error does not consistently indicate better directional recovery. Consequently, the findings provide limited guidance on constructing more faithful surrogates.

Questions for authors

For the Qwen counterexamples, does the ranking reversal persist when native selection and intervention evaluation use matched metrics—for example, centered-score MSE at both stages? This would help distinguish intervention-specific failure from sensitivity to the evaluation metric.

1 comment
AuthorsOpenReview ↗

Authors’ Response to Reviewer ddjv

We thank Reviewer ddjv for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.

Does the ranking reverse under centered-score MSE at both stages?

Yes, by point estimate, on all three historically replicated heads with the ensembles held fixed. For this rebuttal, we compared saved measurements at native and shifted positions on a separate 512-document Qwen/WikiText-103 cohort. We did not reselect checkpoints by validation MSE. Rotary has lower native centered-score MSE on all three heads. Cubic has lower MSE for the change after the position shift on all three. The table gives the errors in squared-score units. Lower is better.

HeadNative MSE, Cubic / RotaryResponse MSE, Cubic / Rotary
L21H130.5503 / 0.40290.0638 / 0.0828
L24H101.0268 / 0.70300.0549 / 0.0795
L25H120.8917 / 0.64810.0564 / 0.0755

Across the 32-head panel, these three are the only MSE reversals. Native error uses all valid causal entries. For response error, we center each row over all valid keys and then score the pairs whose relative positions changed. Appendix M, Table 31 (p. 57) reports all 32 heads.

How much of this could be a metric effect?

We also ran a matched attention-KL check that uses the same valid query rows at both stages. On L21H13 and L25H12, the simultaneous 95% intervals support both a native advantage for Rotary and a shifted advantage for Cubic. On L24H10, the native Cubic-minus-Rotary interval crosses zero. After the shift, however, Rotary’s KL on this head is higher than that of both Cubic and the no-change baseline.

Practical significance and explanation.

The surrogate that best reproduces a head’s attention on ordinary text can still misrepresent how that head responds when positions change. When choosing a surrogate to explain positional sensitivity, we therefore ask whether it predicts the change in attention scores more accurately than a baseline that predicts no change.

Table 1 shows where native selection failed this test. On three of 32 Qwen heads on OpenWebText, native-validation KL favored Rotary, but its predicted score changes were less accurate than both Cubic’s predictions and the no-change baseline. The same three heads qualified again on WikiText-103, along with L18H02 (4/32). None of the 12 Pythia heads qualified on either corpus.

Across the three qualifying OpenWebText heads, Rotary’s output KL exceeded Cubic’s by 1.709×10−51.709 \times 10^{-5} nats/token and exceeded no change by 1.658×10−51.658 \times 10^{-5} nats/token. Cubic’s output advantage over no change was inconclusive (Appendix C, Eqs. 16–18). End-to-end token insertion did not meet the combined failure criterion (Section 5).

Section 5 (p. 8), Figure 3 and Appendix I.1 explain why Cubic can have lower error even when it predicts the direction less accurately. At the WikiText anchor, Rotary on L24H10 predicted a change 1.248 times the teacher’s response norm, with a directional cosine of 0.371. Cubic predicted a much smaller change (0.158 times the teacher norm) with poorer alignment (cosine −0.318). The resulting risks were 1.632 for Rotary, 1.126 for Cubic and 1 for no change (Table 16, p. 30). On this head, Cubic’s advantage over Rotary came from predicting a smaller change. For this reason, we report the magnitude and alignment of the response alongside its squared error.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of author responses and manuscript revisions

We thank all three reviewers for their careful reading and feedback. Reviewer AQ6M highlighted the question of how to evaluate attention surrogates under changes in position. Reviewers ddjv and NLsd noted that holding the tokens and upstream representations fixed lets us isolate each head’s response to the intervention.

In response to a question raised by both ddjv and NLsd, we added a comparison using the same metric before and after the position shift. Using the 512 WikiText-103 documents from Appendix G, two of the three previously identified Qwen heads showed a ranking reversal under attention KL: Rotary fit native attention more closely, while Cubic predicted the shifted attention more accurately. On the third head, Cubic also had lower shifted KL, but the native ordering was uncertain. Appendix M reports the estimates and simultaneous confidence intervals.

Following NLsd’s questions about novelty and head-specific failures, we added a prospective study of 24 additional Qwen heads with 144 fresh native-only fits. The study identifies susceptibility associated with weak local positional responses that Rotary reproduces poorly. On five additional heads, native fit favored Rotary across all three seeds, yet its averaged prediction was worse than Cubic and no change under a one-position shift within the fitted positional range. Appendix N and Figure 14 present the design, full-panel results and uncertainty. Section 6 now references these findings.

We also clarified the prediction target in the abstract and Introduction, explained the 32-head panel selection, and narrowed the T5 claim to reflect that none of its 15 targets met the combined failure criterion. Our updated NLsd response addresses novelty and head-specific insights separately. The manuscript additions are marked in the highlighted revision.

AuthorsOpenReview ↗

Thanks for your reply

Thank you for clarifying your remaining concerns. We address novelty and head-specific insight separately.

Further clarification of our novelty

Our contribution is a controlled test of native-fit selection for explanations of positional sensitivity. Dyer et al. establish the broader distinction between observational and interventional fidelity in simulator surrogates. Hayes et al. select executable attention programs by attention agreement and evaluate causal replacement effects. We ask whether native fit selects the surrogate that best predicts how the original head responds to a specified positional change.

We hold tokens, causal masks and upstream representations fixed, and add each predicted change to the same native teacher scores. This isolates response-prediction error from native replacement error. Equation 4 explains why native agreement is insufficient: response error depends on the difference between reconstruction errors at the original and shifted positions. The no-change baseline tests whether an explanation improves on assuming no response.

The resulting selection failure replicates on the same three Qwen heads across two corpora with fresh fits. Matched-KL analysis also supports reversals on L21H13 and L25H12 using the same metric before and after intervention. Together, the controlled evaluation and replicated counterexamples establish a concrete validation requirement for attention surrogates used as positional explanations.

Insights into why certain Qwen heads fail

One hypothesis is that these failures concentrate in heads whose true positional responses are weak and poorly reproduced by the surrogate. To test this hypothesis, we conducted the following experiment.

We fitted Cubic and Rotary on 24 additional Qwen heads with three seeds each, giving 144 fresh native-only fits. On 128 separate audit articles, we compared the local positional derivatives of the teacher and Rotary. We flagged heads where Rotary's derivative prediction was less accurate than predicting a zero derivative. We fixed these flags before testing one-position and 64-position shifts on 512 new articles.

In the one-position score-response test, all eight flagged heads were worse than both Cubic and no change. The other 16 heads were better than both. Both results have simultaneous 95% interval support. All evaluated positions and offsets stayed within 0–255. For five of the eight flagged heads, native validation and held-out native KL favored Rotary in every seed. The table below shows these five heads. Values use three-seed averaged predictions. Native KL is measured in nats per valid query row. The no-change baseline has a normalized response risk of 1. Lower values are better for both metrics.

  Head  Native KL: Cubic  Native KL: Rotary  Gap-1 risk: Cubic  Gap-1 risk: Rotary
L22H010.084970.078341.0002.174
L23H150.068400.057760.9981.611
L24H050.055830.051021.0011.967
L26H040.077400.063921.0001.934
L26H050.076010.063511.0001.519

At gap 64, five of the eight flagged heads met the prespecified ensemble-failure criterion. None of the 16 unflagged heads did. Across the six layers containing both groups, the equal-layer mean difference in Rotary gap-64 response risk (flagged minus unflagged) was 1.407, with a simultaneous 95% interval of [1.364, 1.450]. The matched weak-teacher-sensitivity comparison selected the same eight heads. Figure 14 in Appendix N shows the full panel.

The local audit distinguishes the two groups. In the eight flagged heads, the true local responses are weak and Rotary's errors are large relative to them. In the other 16 heads, the responses are stronger and the relative errors are smaller. The one-position test shows the same contrast: Rotary loses to no change in the first group and beats it in the second.

Reference

  • Joel Dyer, Nicholas Bishop, Yorgos Felekis, Fabio Massimo Zennaro, Anisoara Calinescu, Theodoros Damoulas, Michael Wooldridge. "Interventionally Consistent Surrogates for Complex Simulation Models." NeurIPS (2024).
  • Amiri Hayes, Belinda Z Li, Jacob Andreas. "Explaining Attention with Program Synthesis." arXiv (2026).

Thank you for your additional responses. I appreciate your efforts, but I remain not fully convinced about the novelty and explanatory insight of the work. For example, the new experiments provide useful observations about which heads are prone to failure, but the underlying mechanisms remain insufficiently explained.

After carefully considering your responses and the other reviewers' comments, I have decided to maintain my original rating of 2. I encourage you to further strengthen the work and submit the revised version to a future conference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.