acceptodds
Withdrawn from ICLR 2027

Sparse Attention-Head Patching for Cross-Notation Numerical Comparison: Donor-Position Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prompt-final attention-head patches can restore mixed-notation answers by transferring a donor-selected option position. In stored Llama and Qwen confirmations, aligned donors improve held-out pair consistency from 0/16 to 14/16 and 1/16 to 15/16; summary-level swapped-donor counts are 61/64 and 30/32 presentations, respectively. A relabeled-option intervention shows that, in this Llama setup, target choices follow the donor-selected option position even when donor and target use different labels. This is an output-level transfer result in one model and prompt protocol, not evidence for a portable latent router. A separately preregistered, outcome-informed Llama follow-up finds that knockout of the frozen six-head set reduces signed margins beyond all 30 matched nulls and that operand-span K- and V-vector donor patches shift margins toward a counterfactual answer. The aggregate knockout loss is 0.880 (95% pair-bootstrap CI [0.805,0.962]), but is concentrated in digits/digits prompts (2.390 vs. 0.127 and 0.123 in reciprocal mixed-notation conditions). A later held-out screen across scientific, comma, zero-padded, and word formats met its 80% intact-accuracy floor in only 2/8 format-template cells and stopped before causal tests. A same-cohort chat-format sensitivity run failed donor competence (43/64), so transfer under the original double-BOS rendering remains untested. The evidence supports cross-label donor-position transfer in the standalone prompt setup and a bounded operand-linked effect in one Llama task, not a broad sparse numerical circuit. Unsupported older results are omitted.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, numerical comparison, cross-notation transfer, attention heads, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ie5oOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether sparse attention-head patching can repair numerical comparison failures when numbers are written in different formats. The authors study Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, transferring prompt-final attention-head outputs from a correct donor prompt into a failing target prompt.

Aligned-donor patching improves consistency in both models. However, when the donor operands are swapped, the target tends to follow the donor's answer rather than its own correct answer. A follow-up experiment in Llama, using different answer labels for donor and target, suggests that the transferred information is associated with the selected option position instead of the literal answer token.

The paper also studies the native role of these heads through knockout and operand-level interventions. These experiments find some causal effects on answer margins but the effects are substantially stronger for same-format comparisons. Additional experiments across numerical formats fail their predefined competence criteria.

Overall, the results support a limited answer-position transfer effect, but do not establish a general numerical comparison mechanism or a circuit that is robust across number formats.

Critical strengths
  1. Transparent reporting of negative results and limitations.

I appreciate that the authors report experiments that did not support their initial hypotheses, rather than presenting only the successful interventions.

  1. The relabeled-option experiment is a useful control.

The experiment in Section 6 is interesting because it distinguishes transferring an answer position from simply copying a literal answer token. By using (1)/(2) labels for the donor and (A)/(B) labels for the target, the authors show that the patched model tends to select the corresponding donor position despite the different output labels.

Critical weaknesses
  1. The main contribution is quite narrow and its novelty relative to existing work is unclear.

My biggest concern is that the paper does not establish the numerical comparison mechanism that motivates the investigation.

The aligned-donor experiments initially look promising but the swapped-donor results suggest that the patched heads mainly transfer the donor's already-selected answer. The relabeled-option experiment makes this more precise by showing that the transferred information is associated with the selected option position rather than the literal answer label.

While this is an interesting observation similar distinctions between answer selection, option position and output-symbol binding have already been studied in the mechanistic interpretability literature. The additional novelty here seems to come primarily from applying these interventions to a particular numerical comparison task.

The later experiments also weaken the numerical interpretation. In Section 7, the native knockout fails the preregistered selectivity test and the follow-up in Section 8 reports a much stronger margin effect on digits/digits comparisons than on mixed-notation comparisons. The knockout does not reduce same-format accuracy in the reported replication, despite the substantial change in margins. This makes the functional role of the heads difficult to interpret.

The broader format experiments do not establish generalization either, as most conditions fail the required competence thresholds before the causal interventions can be performed.

I appreciate that the authors acknowledge these limitations. However, after narrowing the conclusions appropriately, I am not convinced that the remaining result provides a sufficiently new insight for ICLR.

  1. Important methodological details are missing from the manuscript, making the main results difficult to assess independently.

I found the experimental presentation difficult to follow. Several sections refer to stored JSON files, experiment attempts and available records rather than explaining the experimental design in a self-contained way. Appendix A mostly lists artifact paths, which makes it harder to understand the methodology without reconstructing it from the accompanying files.

Most importantly, I could not find a sufficiently detailed explanation of how the six Llama heads and fourteen Qwen heads were originally selected. The paper specifies the final head sets but the discovery procedure, selection criteria, and separation between discovery and confirmation data are not clearly established. Since these head sets are central to the main claims, this information is important for evaluating the sufficiency results.

I also would have liked clearer descriptions of the prompt templates, numerical-pair generation procedure, and the different presentation conditions. The differences in tokenization and execution settings across experiments (for example the standalone versus chat-format BOS handling) further complicate interpretation.

The small evaluation populations are another limitation. Several central results use only 16 numerical pairs and the relabeled-option experiment acknowledges a mismatch between the preregistered inferential unit and its presentation-level decision threshold. Some of the earlier donor results also rely on summary files without trial-level records.

Finally, the reproducibility and limitations sections contain unfinished sentences, which makes the manuscript feel insufficiently polished.

Taken together, these issues make it difficult to evaluate the robustness and generality of the findings.

Questions for authors
  1. Head selection and reproducibility.

Could the authors clarify how the six Llama heads and fourteen Qwen heads were selected?

  1. Specificity to numerical comparison.

Given that swapped-donor patching appears to transfer the donor's selected answer, what evidence suggests that these heads are specifically involved in numerical comparison or cross-notation processing? Would the same transfer effect occur for non-numerical binary-choice tasks, such as word-length or alphabetical-order comparisons?

  1. Generalization of the relabeled-option result.

Can the authors replicate the cross-label experiment in Qwen or with additional answer-label pairs? I would also be interested in whether the effect survives under the original chat-format rendering, since the reported chat-format attempt stopped before patching.

Reviewer_V3gYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates why language models perform well when comparing numbers written in the numerical format but struggle when the numbers are expressed in scientific notation. First, they find a set of attention heads that seems to be causally important to improve the performance of the language models on both formats by performing activation patching. However, the follow-up experiment suggest that the intervention mainly transfers the selected answer position rather than a general numerical comparison mechanism

Critical strengths

The paper addresses an interesting problem of understanding how language models represent and use numerical information across different formats, i.e., digits and scientific notation. The authors are also transparent about their negative results and the limitations of their findings.

Critical weaknesses
  1. The paper has limited mechanistic findings on how LMs perform numerical magnitude comparison. The experiments in Section 6 show that the heads can causally influence answer-position selection, but they do not establish that the heads perform numerical magnitude comparison.
  2. The paper is difficult to follow and needs significant improvement in clarity and organization. For instance, the paper uses several jargon without properly defining them like "band effect" and "native effect". The main findings are also buried by several experimental stages, making it difficult to understand the overall contributions.
Questions for authors

Can the authors provide stronger evidence that the identified attention heads are involved in numerical magnitude comparison itself, rather than primarily transferring answer-position information?

Reviewer_kRxxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at how the attention heads in large language models learn different numerical formats, e.g. 123 versus 1.23e2. They experiment with transferring attention-head activations from a donor to a target and find that swap can the numerical comparison task on Qwen and Llama.

Critical strengths
  1. The paper looks at an interesting question about the attention head influence when producing incorrect results (particularly a numerical notation inconsistency). Transferring an answer position via an attention mask is an interesting approach and the task is a good one to showcase this approach.
  2. The paper is very clear about the negative results and limitations.
Critical weaknesses
  1. Although the authors are very clear about the limitations of their work, the novelty and contributions of the work are unclear. While the donor-target transfer is interesting, the method along with the results alone do not demonstrate any new insights about how these LLMs process information.
  2. The experiments are interesting but limited in their findings. The experimental setup is highly constrained, looking at two models with very specific prompt.s But my bigger concern is that in 6 out of 8 conditions, the models do not achieve the competence gate. Rightfully so, this is highlighted as a negative result not showing causality. But I feel that this is overclaimed; it is unclear that these results generalize beyond the highly constrained experimental setting.
Questions for authors
  1. Could the authors clarify what the main novel mechanistic insight is compared to existing work on answer-symbol binding and activation patching?
  2. How well does donor-position transfer generalize with different prompts?
  3. What are some other proposed experiments or tasks to better distinguish heads involved in numerical comparison from those involved in general multiple-choice answer selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.