acceptodds
Under review as a conference paper at ICLR 2027

When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task

Abstract

One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can tell us how models perform computations, and how to effectively intervene on them. While low dimensional manifolds have been observed for multiple concepts in the literature (e.g. numbers encoded on helices, days of the week on a circle, ...), with structure believed to reflect properties of data and tasks, the extent to which models rely on them for computation, and how they manipulate them, remains unclear. We characterize precisely the geometry of computation in a number-comparison task, as an abstraction of comparison for decision making, and how models utilize geometry in an elegant fashion to implement it. Specifically, we study the causal geometry of number comparison in Qwen2.5-7B-Instruct, a capable and widely studied open-weight model, and find Qwen largely uses linear representations of numbers despite the presence of curved geometry. To compare two numbers, the model first encodes each number along a vector and adds the two representations using attention and the residual connection, bringing them into a shared space in the residual stream. Then, the model uses MLP neurons to compare the pair of numbers on local regions in this shared space, which correspond to smaller intervals of input numbers, and combines these to obtain the position of the maximum. In fact, this reliance on linear representations for comparison also persists when the model compares three numbers. Our findings demonstrate that the manifold hypothesis can co-exist with linear representations: while concepts that are ordered may have manifold structure in representations, the model may use an underlying linear structure of the concept in certain computations.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, representation geometry, causal interventions, activation patching

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ig3jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper hypothesizes that LMs perform number comparison via computation among linear representations and propose a representation-based interpretation of the computation algorithm. The paper demonstrates that their interpretation is highly consistent with model's actual behaviors through causal intervention experiments.

Critical strengths

The paper identified vector directions for each number in an LM's number comparison task and discovered that such a subspace formed by these vectors is linearly separable according to the LM's comparison results. They use multiple causal intervention experiments to back up their hypothesis. The result is non-trivial and interesting.

Critical weaknesses
  1. The paper only use a predefined language template to conduct all experiments. It is unclear if the interpretation still works under other natural language templates.
  2. The procedure of how to discover v1 and v2 is unclear, which is important to the core soundness of the paper.
  3. Page 6., Line 316-318: "This observation is nontrivial because v2 is obtained in layer 13’s residual stream and v1 is from head H14’s output (which itself uses the layer 13 residual stream in computation), making nonlinear interactions possible." is confusing.
  4. Does the low IIA of v1 for y1 in Figure 2(f) suggest the weak dependency between comparison computation and v1?

I will increase the score if the authors address my questions.

Questions for authors

See the weakness above.

2 comments

We thank the reviewer for their feedback! We are glad that the reviewer found our paper “clear and easy to follow”, and thought our results were “non-trivial and interesting”. We appreciate the reviewer for highlighting our main contributions: 1) number comparison is performed using linear representations, 2) we identify the computation algorithm using the representations, 3) we demonstrate that the algorithm captures the model’s behaviors through causal intervention experiments.

We address specific questions in our responses below.

The paper only use a predefined language template to conduct all experiments. It is unclear if the interpretation still works under other natural language templates.

Thank you for raising this important point. We have now performed additional experiments on multiple language templates (prompt formats). Across different templates, we find that the same algorithm is used by the model for comparison, with many of the same model components–attention head H14L14 copies the first number (App. C.8 Figure 44), the same two-dimensional (v1,v2) plane enables comparison (App. C.8 Figure 45, 42), MLP neurons in L14, L15 perform comparison (App. C.8 Fig 46). While the directions u, v1, v2 also work as-is for some prompts, we note that they require refitting in other cases–the refit directions are rotated versions of the original directions. Even when we change the prompt from natural language to code (P4), the same algorithm (with refitted components) works! All the supporting figures and tables for this analysis are included in Appendix C.8.
The prompt templates used in our new results are stated below:

Table. The five prompt templates, shown with y₁ = 37 and y₂ = 58. P0 is the prompt used in the paper.

TemplatePrompt
P0 (original)Answer in the following format with a single answer. The maximum of 12 and 4 is 12. The maximum of 37 and 58 is
P1 (instruction)Reply with one number only. The maximum of 12 and 4 is 12. The maximum of 37 and 58 is
P2 ("larger")Give only the answer. The larger of 12 and 4 is 12. The larger of 37 and 58 is
P3 (question)Which number is bigger, 12 or 4? Answer: 12. Which number is bigger, 37 or 58? Answer:
P4 (code)max(12, 4) = 12<br>max(37, 58) =

The interchange intervention accuracy (IIA) for different model components across the five prompt templates is stated below. We first describe the transfer accuracy (Table 1): when the same components found from our original prompt are used for other prompts, and then refit the components (Table 2) for prompts where transfer is weak (mainly P3).

Table 1. Components fitted on the original prompt P0, applied without refitting to four reworded prompts with the same operands. IIA is measured on 400 counterfactual pairs (SE ≤ 0.025). † Below 0.8 × the P0 value.

StageMeasureP0 (original)P1 (instruction)P2 ("larger")P3 (question)P4 (code)
BehaviourAccuracy (whole number)1.0001.0001.0001.0001.000
Operands𝐮: IIA (y₁ perturbed)0.9400.8980.8430.298†0.675†
𝐯₂: IIA (y₂ perturbed)0.7900.7900.7680.573†0.683
TransportL14.H14 attention, y₂ → y₁0.7110.7310.6690.5750.724
𝐯₁: IIA (y₁ perturbed)0.4600.4850.365†0.195†0.298†
Shared plane(𝐯₁, 𝐯₂) plane: IIA (y₁ perturbed)0.8100.8150.7400.7380.798
(𝐯₁, 𝐯₂) plane: IIA (y₂ perturbed)0.9480.9480.9300.7980.923
L15 DAS: IIA (y₁ perturbed)1.0001.0001.0000.8950.930
L15 DAS: IIA (y₂ perturbed)0.6550.513†0.458†0.048†0.085†

Table 2. Components refitted on P3 and P4, the prompts where frozen transfer is weakest (Table 1). † IIA below 0.8 × the P0 value. * Below the P0 fitting noise.

ComponentMeasureP0 (original)P3 (question)P4 (code)
𝐮IIA (y₁ perturbed)0.940 (0.908–0.945)0.9580.958
|cos| to P0 fit0.8930.543*0.756*
𝐯₂IIA (y₂ perturbed)0.7900.7180.768
|cos| to P0 fit0.9950.922*0.944*
𝐯₁IIA (y₁ perturbed)0.460 (0.425–0.478)0.188†0.405
|cos| to P0 fit0.9280.459*0.941
(𝐯₁, 𝐯₂) planeIIA (y₁ perturbed)0.8100.233†0.820
IIA (y₂ perturbed)0.9480.685†0.943
Smaller principal cosine to P0 plane–0.4580.939
L15 DASIIA (y₁ perturbed)1.000 (0.998–1.000)1.0001.000
IIA (y₂ perturbed)0.655 (0.655–1.000)0.398†0.995
|cos| to P0 fit0.4660.5090.463*

In summary, across different prompt templates, the model uses the same algorithm by reusing many components found in our analysis. Major changes in prompt format require refitting the causal directions u, v1, v2, and the answer direction from L15 DAS.

(continued ...)

The procedure of how to discover v1 and v2 is unclear, which is important to the core soundness of the paper.

Thank you for pointing this out. We clarify the procedure used to obtain v1 and v2 below. Further details are included in Appendix B.3. We will also include these descriptions in the main paper for greater clarity. We obtain v1 using one-dimensional (1-d) distributed alignment search from Geiger et al 2024 (DAS) at the outputs of the copy head H14 in layer L14. 1-d DAS optimizes for a direction v1 in H14’s output space, such that the projection onto this direction has maximum causal effect on model logits. This means that performing activation patching along this direction, while keeping the remaining components of the activation fixed, changes the model’s logits toward the patched answer position.
We obtain v2 as the first principal component (PC1) of the residual stream activations at layer L13 in Qwen. We observe that this direction encodes the second number y2 monotonically, and patching along this direction has a significant causal effect on the model’s logits. For models where the first principal component (PC1) does not show causal effect, as seen in Llama-3.1-8B and Gemma-2-9B-it (Appendix C.9), we obtain v2 using a modified version of DAS which incorporates number encoding (proportional to log of the number) along with causal effect on model outputs (details in Appendix C.9).

Page 6., Line 316-318: "This observation is nontrivial because v2 is obtained in layer 13’s residual stream and v1 is from head H14’s output (which itself uses the layer 13 residual stream in computation), making nonlinear interactions possible." is confusing.

Thank you for raising this issue. We have added a Figure 2 to the main paper to illustrate the key components used in the comparison algorithm. We rephrase the lines 316-318 below: We observe that patching the rank-two shared representation space (span(v1,v2)) has a similar causal effect as patching the directions v1 and v2 individually using rank-one patches. Since v2 is obtained in layer 13’s residual stream, which feeds into head H14 of layer L14, v2 is an input to H14. Therefore, v2 can affect the outputs of H14, including v1, but patching the shared space shows that v1 and v2 behave independently in the two-dimensional space span(v1, v2). We have included this rephrasing in our updated manuscript.

Does the low IIA of v1 for y1 in Figure 2(f) suggest the weak dependency between comparison computation and v1?

Thank you for the question. While the IIA for v1 (0.46) is individually lower than IIA for u (0.94), v1 is significantly involved in subsequent steps of the comparison algorithm. The shared (v1, v2) plane has a high IIA (0.81) when y1 is perturbed. Thus, v1 is used by the model in computation, but seems to have a low IIA individually. Potential reasons for low IIA of v1 itself include the small non-monotonicity of y1 along v1 (for y1 less than 20) and higher dimensional structure at this location. It must also be noted that the baseline IIA for a random direction in the representation space is ~0, since nonzero IIA requires the model’s logits to shift such that the largest logit moves to the answer expected after patching. Therefore v1’s IIA of 0.46 is significantly higher than a random baseline. We observe high IIA scores for v1 and v2 in additional results on other models (see Appendix C.9): for Gemma-2-9B-it, v1 has IIA 0.92 and v2 has IIA of 1, while Llama-3.1-8B has v1 IIA 0.62 and v2 IIA 0.96.

Summary: We thank the reviewer for their constructive comments and feedback on writing clarity. We hope our responses and additional experiments have addressed their questions satisfactorily, and we would greatly appreciate it if the reviewer considered raising their score. We would be happy to answer any further questions during the rebuttal period!

Reviewer_XcBJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how Qwen2.5-7B-Instruct compares numbers, asking whether it uses their curved representation geometry or a simpler linear structure.

The proposed mechanism has three stages: numbers are represented along separate directions; attention brings these representations into a shared space; and MLP neurons combine local comparisons into an answer-position representation. The authors support this account with activation patching, distributed alignment search, attention-head analysis, and neuron interventions. They also extend the analysis from two to three numbers.

The main finding is that curved number representations can coexist with causally effective low-dimensional linear directions.

Critical strengths

The paper asks a focused and worthwhile question: observing a curved representation does not explain how the model computes with it. Connecting individual number directions to attention transport and comparator neurons makes this more informative than a purely geometric analysis.

The three-number extension also adds value by testing whether parts of the pairwise mechanism are reused. I appreciated the explicit acknowledgment that intervention sufficiency does not establish necessity, and that DAS may identify shortcuts.

Critical weaknesses

The output metrics provide incomplete behavioral evidence. IIA scores only the first generated digit and can count mixed-number answers as successful. Appendix C.1 explicitly shows such outputs. For three numbers, the authors switch to position recovery because IIA is weak. A logit shift then need not change the answer at all. Full-number counterfactual success should accompany the principal claims.

The three-number algorithm is only partly established. The readout mechanism is left unresolved, and the proposed sequential interpretation is a hypothesis. Moreover, the fitted “flag” directions do not transfer uniformly between perturbation cases in Figure 32. This makes a single, reusable flag representation less clear than Algorithm 2 suggests. Longer-list behavioral accuracy alone does not establish that the same algorithm scales.

The linear-versus-manifold conclusion needs narrowing. A useful linear projection can coexist with a curved representation without showing that curvature is computationally irrelevant. The manuscript demonstrates this coexistence well, but does not directly isolate the contribution of the remaining curved components. Claims that the model does not manipulate manifolds, or that the findings challenge a central premise of interpretability, go beyond the evidence.

Robustness is limited. The analysis concerns one model and one prompt format. Most learned directions use a single initialization, and only the individual-number representation analysis is replicated with three-digit inputs. Prompt paraphrases, shared-prefix numbers, and another model would help distinguish a stable comparison mechanism from a task-specific route.

Questions for authors

Do the same directions and neurons support interventions when the operands share their first digit, such as 61 versus 67?

What proportion of the main interventions produce the intended complete number, rather than only its leading digit?

Can you explain the difference between disrupting patch-mediated effects and the small ordinary-accuracy changes in Figure 35?

Can one comparison-flag direction generalize across perturbation cases and value orders without refitting?

4 comments
AuthorsOpenReview ↗

Additional Results Included in Updated Manuscript

We thank the reviewer for their thoughtful and detailed feedback! We appreciate that the reviewer “gained a substantially deeper understanding of the problem” through our paper, and found our work “asks a focused and worthwhile question”, and “appreciated the explicit acknowledgment” of limitations with regard to sufficiency versus necessity of our methods. We thank the reviewer for highlighting our paper’s main contributions: 1) causally relevant linear representations can coexist with manifold structure, 2) the algorithm using linear representations of numbers for number comparison in Qwen, and 3) extension of the algorithm to three number comparison. We provide our responses to specific questions below:

Full-number counterfactual success should accompany the principal claims. What proportion of the main interventions produce the intended complete number, rather than only its leading digit?

Thank you for the suggestion. We have now included full-number interchange intervention accuracy scores (IIA) across all intervention results where we previously reported first-digit IIA in Appendix C.7. We find that full-number IIA closely matches first-digit IIA for causal linear representations u, v1, v2, as well as for the shared plane (v1,v2) and the answer flag direction from L15 DAS. This is an expected consequence of our comparison algorithm: the model first computes the argmax position, and later copies the input at that position. We include the results in the table below, and add all relevant figures to Appendix C.7: Table. IIA scored on the first generated digit and on the whole generated number.

Patchy₁ perturbed: first digit IIAy₁ perturbed: whole number IIAy₂ perturbed: first digit IIAy₂ perturbed: whole number IIA
𝐮 (L13 residual at y₁)0.9400.940––
𝐯₁ (L14.H14 output at y₂)0.4600.4600.0000.000
𝐯₂ (L13 residual at y₂)0.0000.0000.7900.785
(𝐯₁, 𝐯₂) plane (L14 pre-MLP residual, rank 2)0.8100.8100.9480.940
L14 residual at y₂, full rank0.8250.8250.9850.115
Decision flag (L15 DAS, rank 1)1.0001.0000.6550.650
(𝐯₁, 𝐯₂) plane + 12 comparator neurons frozen0.1330.1330.2400.213
(𝐯₁, 𝐯₂) plane + 12 random neurons frozen0.8100.8100.9480.940

The three-number algorithm is only partly established. Moreover, the fitted “flag” directions do not transfer uniformly between perturbation cases in Figure 32. Can one comparison-flag direction generalize across perturbation cases and value orders without refitting?

Thank you for pointing this out. The main goal of the three number comparison is to establish our broader point: that causally relevant linear representations of numbers are used for comparison of two and three numbers. Our goal was not to claim that the two-number comparison algorithm extends as-is to the three number case. We agree that we have not completely established how the maximum position is read out by the model and further used to answer the question. However, we have demonstrated the presence of causal linear directions which are compared in a manner similar to the pairwise number comparison algorithm. We also agree that a single flag direction does not transfer across perturbation cases in Figure 32 (now Fig 33 in updated manuscript), which again relates to the readout mechanism. Please note that we have used different flag directions for each pair of numbers in Algorithm 2. We will further clarify this point in the final version of the paper, and emphasize the findings that we believe generalize: the use of causally relevant linear features for numbers in comparison. The difference between the two number comparison algorithm and the comparison algorithm of longer sequences is non-trivial and interesting, and we will highlight this in our discussion section.

Please note that in the two number comparison case, the flag direction transfers between perturbation cases: it is fit to the y1-perturbed case (IIA of 1) and tested on the y2-perturbed case (IIA 0.655).

(continued ...)

The linear-versus-manifold conclusion needs narrowing. The manuscript demonstrates this coexistence well, but does not directly isolate the contribution of the remaining curved components. Claims that the model does not manipulate manifolds, or that the findings challenge a central premise of interpretability, go beyond the evidence.

Thank you for your suggestion. We emphasize that our primary objective is to show causally relevant linear representations despite the presence of manifolds. We establish sufficiency and not necessity of linear representations for causal effects. Recent observations of concept manifolds (such as number helices, character count manifolds, temperature and age manifolds, see lines 41-43 on page 1 of the paper) led to a common belief that one requires manifold steering (Wurgaft et al 2026) and manifold manipulation methods (Gurnee et al 2026) to control model outputs. We posit that in these cases, it may be possible to find causally relevant linear subspaces. In this context, our findings challenge the premise that complicated manifold structure would require convoluted manipulation techniques: despite the presence of manifolds, steering along simple linear representations is enough for certain kinds of tasks. Therefore, our findings challenge the perspective that manifold geometry is the simplest unit of computation in large models. We will highlight this clarification, incorporate suggestions to narrow the claims by making our contribution more targeted in the final version of the paper.

Robustness is limited. The analysis concerns one model and one prompt format. Most learned directions use a single initialization, and only the individual-number representation analysis is replicated with three-digit inputs. Prompt paraphrases, shared-prefix numbers, and another model would help distinguish a stable comparison mechanism from a task-specific route.

Thank you for raising these important points. We have now performed extensive additional experiments across multiple models (Llama3.1-8B, Gemma-2-9B-it, Qwen2.5-7B-base), multiple prompt formats, using multiple random initializations for each number representation, and investigated shared first digit number comparison. We find the same high level algorithm—causally relevant linear representations for numbers, used to create a two-dimensional space on which MLP neurons perform comparison—across multiple models and prompt formats. The linear representations we learnt are stable across multiple seeds. We include further details below.

Multiple models: We have now added additional analyses on Llama-3.1-8B and Gemma-2-9B-it models, as well as Qwen2.5-7B-base (our main paper uses Qwen2.5-7B-Instruct) in Section 2.5 and Appendix C.9 of our updated manuscript. We found causally relevant linear features for numbers u, v1, v2, the shared (v1,v2) plane, and MLP neurons performing comparison on this shared plane, in all models. Therefore, our core claims of causally relevant linear number representations used in comparison hold across multiple models! The high-level causal algorithm is the same across these different models. We summarize the main takeaways from our results below:

Table R1. Interchange intervention accuracy (IIA) of the mechanism components in four models.

ComponentMeasureQwen2.5-7B-Instruct (paper)Qwen2.5-7B (base)Llama-3.1-8B (base)Gemma-2-9B-it
𝐮IIA (y₁ perturbed)0.9400.9500.7300.800
𝐯₁IIA (y₁ perturbed)0.4600.3880.6250.918*
𝐯₂IIA (y₂ perturbed)0.7900.7000.958*1.000*
(𝐯₁, 𝐯₂) planeIIA (y₁ perturbed)0.8100.7600.7630.913
IIA (y₂ perturbed)0.9480.8780.7900.928
Decision flag (y₁ fit)IIA (y₁ perturbed, fitting case)1.0001.0001.0000.970
IIA (y₂ perturbed, transfer)0.6550.1850.0000.000
Decision flag (y₂ fit)IIA (y₂ perturbed, fitting case)1.0000.9501.0000.985
IIA (y₁ perturbed, transfer)0.3880.0000.0950.000
Difference across models: In Llama and Gemma, although the decision flag can be fit separately for each perturbation case, it does not seem to transfer across perturbation cases (unlike in Qwen2.5-7B-Instruct where the flag does transfer). This means that the answer to argmax computation (decision flag) may be higher dimensional in Llama and Gemma than the simple one-dimensional flag found in Qwen, and this difference across models is interesting on its own. Importantly, this does not affect our core claims since we claim numbers have causally relevant linear structure, which does hold across models. We use a modified version of DAS to find causal direction v2 in Llama and Gemma (see Appendix C.9). The main paper figures are replicated for Llama, Gemma and Qwen-base in Appendix C.9.

(continued ...)

Multiple Prompt Formats: We have now performed additional experiments on multiple language templates (prompt formats) on the model used in our paper Qwen2.5-7B-Instruct. Across different templates, we find that the same algorithm is used by the model for comparison, with many of the same model components–attention head H14L14 copies the first number (App. C.8 Figure 44), the same two-dimensional (v1,v2) plane enables comparison (App. C.8 Figure 45, 42), MLP neurons in L14, L15 perform comparison (App. C.8 Fig 46). . While the directions u, v1, v2 also work as-is for some prompts, we note that they require refitting in other cases–the refit directions are rotated versions of the original directions. All the supporting figures and tables for this analysis are included in Appendix C.8. The prompt templates used in our new results are stated below: Table. The five prompt templates, shown with y₁ = 37 and y₂ = 58. P0 is the prompt used in the paper.

TemplatePrompt
P0 (original)Answer in the following format with a single answer. The maximum of 12 and 4 is 12. The maximum of 37 and 58 is
P1 (instruction)Reply with one number only. The maximum of 12 and 4 is 12. The maximum of 37 and 58 is
P2 ("larger")Give only the answer. The larger of 12 and 4 is 12. The larger of 37 and 58 is
P3 (question)Which number is bigger, 12 or 4? Answer: 12. Which number is bigger, 37 or 58? Answer:
P4 (code)max(12, 4) = 12<br>max(37, 58) =

The interchange intervention accuracy (IIA) for different model components across the five prompt templates is stated below. We first describe the transfer accuracy (Table 1): when the same components found from our original prompt are used for other prompts, and then refit the components (Table 2) for prompts where transfer is weak (mainly P3). Table 1. Components fitted on the original prompt P0, applied without refitting to four reworded prompts with the same operands. IIA is measured on 400 counterfactual pairs (SE ≤ 0.025). † Below 0.8 × the P0 value.

StageMeasureP0 (original)P1 (instruction)P2 ("larger")P3 (question)P4 (code)
BehaviourAccuracy (whole number)1.0001.0001.0001.0001.000
Operands𝐮: IIA (y₁ perturbed)0.9400.8980.8430.298†0.675†
𝐯₂: IIA (y₂ perturbed)0.7900.7900.7680.573†0.683
TransportL14.H14 attention, y₂ → y₁0.7110.7310.6690.5750.724
𝐯₁: IIA (y₁ perturbed)0.4600.4850.365†0.195†0.298†
Shared plane(𝐯₁, 𝐯₂) plane: IIA (y₁ perturbed)0.8100.8150.7400.7380.798
(𝐯₁, 𝐯₂) plane: IIA (y₂ perturbed)0.9480.9480.9300.7980.923
L15 DAS: IIA (y₁ perturbed)1.0001.0001.0000.8950.930
L15 DAS: IIA (y₂ perturbed)0.6550.513†0.458†0.048†0.085†

Table 2. Components refitted on P3 and P4, the prompts where frozen transfer is weakest (Table 1). † IIA below 0.8 × the P0 value. * Below the P0 fitting noise.

ComponentMeasureP0 (original)P3 (question)P4 (code)
𝐮IIA (y₁ perturbed)0.940 (0.908–0.945)0.9580.958
|cos| to P0 fit0.8930.543*0.756*
𝐯₂IIA (y₂ perturbed)0.7900.7180.768
|cos| to P0 fit0.9950.922*0.944*
𝐯₁IIA (y₁ perturbed)0.460 (0.425–0.478)0.188†0.405
|cos| to P0 fit0.9280.459*0.941
(𝐯₁, 𝐯₂) planeIIA (y₁ perturbed)0.8100.233†0.820
IIA (y₂ perturbed)0.9480.685†0.943
Smaller principal cosine to P0 plane–0.4580.939
L15 DASIIA (y₁ perturbed)1.000 (0.998–1.000)1.0001.000
IIA (y₂ perturbed)0.655 (0.655–1.000)0.398†0.995
|cos| to P0 fit0.4660.5090.463*

In summary, across different prompt templates, the model uses the same algorithm by reusing many components found in our analysis. Major changes in prompt format require refitting the causal directions u, v1, v2, and the answer direction from L15 DAS.

(continued...)

Multiple Random Initializations: We trained DAS directions (u, v1) using five random initialization seeds, as well as different training sets of the same size (see Appendix C.10). We observe that the learnt directions have high cosine similarity and similar causal effects (IIA) across random seeds. For v2, which is obtained as the first principal component at layer L13 at position of y2, we obtain PCs using multiple bootstraps, which are again very aligned (high cosine similarity) and have similar causal effects. The decision flags at layer L15 seem to vary more than the number features: but they maintain significant causal effect.

Task (score)DirectionPerturbedPaperInit seedsTraining sets|cos| init seeds|cos| training sets
Two numbers (IIA)𝐮y₁0.9400.942 ± 0.0020.923 ± 0.0150.990 ± 0.0020.917 ± 0.011
𝐯₁y₁0.4600.460 ± 0.0030.456 ± 0.0190.994 ± 0.0050.969 ± 0.017
𝐯₂ (PC1)–––––0.997 ± 0.001†
Decision flag (y₁ fit)y₁ (fit)1.0001.000 ± 0.0011.000 ± 0.0000.676 ± 0.1590.919 ± 0.016
y₂ (transfer)0.6550.877 ± 0.1460.714 ± 0.034
Three numbers (PR)𝐯₁y₁0.5890.588 ± 0.0000.585 ± 0.0031.000 ± 0.0000.996 ± 0.001
𝐯₂y₂0.2120.211 ± 0.0000.211 ± 0.0011.000 ± 0.0000.984 ± 0.005
𝐯₃ (PC1)–––––0.995 ± 0.002†
Decision flag at y₂y₂3.2053.187 ± 0.0133.172 ± 0.0330.929 ± 0.0210.929 ± 0.008
Decision flag at y₃y₃2.0181.981 ± 0.0402.001 ± 0.0330.792 ± 0.1380.943 ± 0.013

Do the same directions and neurons support interventions when the operands share their first digit, such as 61 versus 67?

Numbers with the same first digit: We chose numbers with different first digits in our original experimental setup because Qwen tokenizes each digit separately, so we analysed causal effects on first output token. We now performed additional analysis using shared first digits and included our results in Appendix C.11. Since our setup includes two pairs of numbers—clean pair (a,b) and corrupted pair (r,b)---where the clean pair activations are patched into the corrupted pair, we consider multiple cases as shown in the table below. We observe that causal effects of patching (IIA) in the shared (v1, v2) plane are high when only one of the pairs shares the first digit, i.e., when patching moves nearby numbers away, or brings farther numbers closer. Therefore, to answer the reviewer’s question on comparing 61 vs 67, if the patched number is less than 60 or greater than 70, patching works well. In all above cases, the comparator neurons have significant causal effect (Fig. 59 in Appendix C.11). However, when both pairs share the same first digit, i.e., the source and target of patching are both close together, patching doesn’t work well.

Table: IIA of the Figure 5(e) interventions when numbers share their first digit, scored on the whole generated number.

Shares first digitnClean gap a − bCorrupted gap b − r(𝐯₁, 𝐯₂) plane, y₁ perturbed(𝐯₁, 𝐯₂) plane, y₂ perturbedPlane, 12 neurons frozen, y₁ perturbedPlane, 12 neurons frozen, y₂ perturbedL15 DAS, y₁ perturbedL15 DAS, y₂ perturbed
a, b, r1,0801–81–80.0010.0440.0000.0060.9790.212
a, b only1,0801–91–850.4010.7060.0000.1130.9890.426
b, r only1,0801–861–90.5260.5910.2050.0180.9810.628
none (main text)40011–6911–690.8100.9400.1330.2131.0000.650

Can you explain the difference between disrupting patch-mediated effects and the small ordinary-accuracy changes in Figure 35?

The main goal of Figure 35 (now Figure 36) is to show ordinary accuracy changes as a function of the number of values compared: while the accuracy of the max comparison decreases with sequence length, it remains significant (80%) even for long sequences (length 20). The zero-ablation of comparator neurons only seeks to demonstrate that while these neurons are causally important for comparison, as shown elsewhere in the paper (including Fig. 5), they are not necessary and zero-ablating them does not hurt the model’s comparison performance significantly.

Summary: We again thank the reviewer for their extensive and constructive review of our paper. We have conducted a wide-ranging suite of analyses to address the questions raised: from analyzing multiple models, to multiple prompt formats, to multiple random initialization seeds and investigating our mechanism on numbers which share first digits. We hope our response has addressed the reviewer’s questions, and we would greatly appreciate it if they considered raising their score. We are happy to answer any further questions during the rebuttal period!

Reviewer_ungPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how LLMs (Qwen2.5-7B-Instruct) performs numerical comparison via its internal representations. The main finding is that, despite the exsitance of non-linear manifold structure of the numerical representations, the model can still leverage the linear directions within them. Specifically, through attention mechanism and residual connection, the information of two numbders is combined into a shared two-dimensional plane. The MLP then perform local comparisons and global aggregation, identifying the position of the maximum value. The authors provide evidence through activation patching, neuron freezing, and connectivity analysis, and also extend the comparison to three numbers.

Critical strengths
  1. This paper emphasizes that observing the geometric representation does not equate to understanding the exact computation of how a model utilizes these representations.
  2. This paper provides in-depth analysis of the numerical comparison mechanism. The authors links the numerical encoding, information flow in the attention, shared representation space, and MLP neurons together, which offers a coherent algorithm level explanation from local comparison to global answer.
  3. This paper conduct a diverse range of experimental evidence to validate the proposed explanation, which makes the findings convincing.
Critical weaknesses
  1. The experiments focused primarily on a single model, fixed prompts, and a task involving finding the maximum of 2-3 numbers. It is not clear whether the discovered mechanism can be generalized to other models, prompts, or other kind of ordered concepts.
  2. As acknowledgeed by the authors, the linear directions can influence the output but does not prove that these directions are necessary or unique computational paths.
Questions for authors
  1. When the numerical representation or the objects being compared change, such as decimals, non-decimal number systems, or alphabetical order, does the model still exhibit similar computational mechanisms?
  2. Pairwise comparison is a fundamental operation for sorting and other selection tasks. Do the numerical direction, attention heads, and comparison neurons identified in this paper also exist in such sorting tasks?
3 comments
AuthorsOpenReview ↗

Additional Results in Updated Manuscript

We are grateful to the reviewer for their thoughtful feedback! We appreciate that the reviewer found our paper “provides in-depth analysis of numerical comparison”, “offers a coherent algorithm”, with a “diverse range of experimental evidence …, which makes the findings convincing”. We thank the reviewer for highlighting the key contributions of our work: 1) Qwen uses linear representations for performing comparison of numbers despite non-linear manifold structure, 2) we identify the algorithm for comparison using linear representations, and 3) we extend the algorithm to three number comparison.

We provide answers to specific questions raised by the reviewer below.

It is not clear whether the discovered mechanism can be generalized to other models, prompts, or other kind of ordered concepts.

Thank you for highlighting this point. We have now performed experiments on multiple models (Llama-3.1-8B, Gemma-2-9B-it, Qwen2.5-7B-base), as well as using multiple prompt templates. We find the same high-level algorithm—causally relevant linear number representations, whose vector sum is used to create a two-dimensional space, followed by MLP neurons performing comparison on this space—across models and prompt templates. The specific model components involved vary across models, and the causal directions u, v1, v2 also change when the prompt template is significantly modified. Therefore, our findings on Qwen generalize at the algorithmic level, and at demonstrating causally relevant linear features despite nonlinear manifold structure. We leave generalization to other ordered concepts to future work. Multiple models: We have now included additional analyses on multiple models (Llama-3.1-8B, Gemma-2-9B-it, Qwen2.5-7B), where we find causally relevant linear features for numbers u, v1, v2, and similar algorithms for number comparison constructing the shared (v1,v2) plane, in all models. These results are included in Section 2.5 and Appendix C.9 of our updated manuscript. Therefore, our core claims of causally relevant linear number representations used in comparison hold across multiple models! The high-level causal algorithm is the same across these different models. We summarize the main takeaways from our results below: Table R1. Interchange intervention accuracy (IIA) of the mechanism components in four models.

ComponentMeasureQwen2.5-7B-Instruct (paper)Qwen2.5-7B (base)Llama-3.1-8B (base)Gemma-2-9B-it
𝐮IIA (y₁ perturbed)0.9400.9500.7300.800
𝐯₁IIA (y₁ perturbed)0.4600.3880.6250.918*
𝐯₂IIA (y₂ perturbed)0.7900.7000.958*1.000*
(𝐯₁, 𝐯₂) planeIIA (y₁ perturbed)0.8100.7600.7630.913
IIA (y₂ perturbed)0.9480.8780.7900.928
Decision flag (y₁ fit)IIA (y₁ perturbed, fitting case)1.0001.0001.0000.970
IIA (y₂ perturbed, transfer)0.6550.1850.0000.000
Decision flag (y₂ fit)IIA (y₂ perturbed, fitting case)1.0000.9501.0000.985
IIA (y₁ perturbed, transfer)0.3880.0000.0950.000
Difference across models: In Llama and Gemma, although the decision flag can be fit separately for each perturbation case, it does not seem to transfer across perturbation cases (unlike in Qwen2.5-7B-Instruct where the flag does transfer). This means that the answer to argmax computation (decision flag) may be higher dimensional in Llama and Gemma than the simple one-dimensional flag found in Qwen, and this difference across models is interesting on its own. Importantly, this does not affect our core claims since we claim numbers have causally relevant linear structure, which does hold across models. We use a modified version of DAS to find causal direction v2 in Llama and Gemma (see Appendix C.9). The main paper figures are replicated for Llama, Gemma and Qwen-base in Appendix C.9.

(continued...)

Multiple prompts: We have also performed additional experiments on multiple language templates (prompt formats). Across different templates, we find that the same algorithm is used by the model for comparison, with many of the same model components–attention head H14L14 copies the first number (App. C.8 Figure 44), the same two-dimensional (v1,v2) plane enables comparison (App. C.8 Figure 45, 42), MLP neurons in L14, L15 perform comparison (App. C.8 Fig 46). While the directions u, v1, v2 also work as-is for some prompts, we note that they require refitting in other cases–the refit directions are rotated versions of the original directions. Even when we change the prompt from natural language to code (P4), the same algorithm (with refitted components) works! All the supporting figures and tables for this analysis are included in Appendix C.8. The prompt templates used in our new results are stated below:

Table. The five prompt templates, shown with y₁ = 37 and y₂ = 58. P0 is the prompt used in the paper.

TemplatePrompt
P0 (original)Answer in the following format with a single answer. The maximum of 12 and 4 is 12. The maximum of 37 and 58 is
P1 (instruction)Reply with one number only. The maximum of 12 and 4 is 12. The maximum of 37 and 58 is
P2 ("larger")Give only the answer. The larger of 12 and 4 is 12. The larger of 37 and 58 is
P3 (question)Which number is bigger, 12 or 4? Answer: 12. Which number is bigger, 37 or 58? Answer:
P4 (code)max(12, 4) = 12<br>max(37, 58) =

The interchange intervention accuracy (IIA) for different model components across the five prompt templates is stated below. We first describe the transfer accuracy (Table 1): when the same components found from our original prompt are used for other prompts, and then refit the components (Table 2) for prompts where transfer is weak (mainly P3). Table 1. Components fitted on the original prompt P0, applied without refitting to four reworded prompts with the same operands. IIA is measured on 400 counterfactual pairs (SE ≤ 0.025). † Below 0.8 × the P0 value.

StageMeasureP0 (original)P1 (instruction)P2 ("larger")P3 (question)P4 (code)
BehaviourAccuracy (whole number)1.0001.0001.0001.0001.000
Operands𝐮: IIA (y₁ perturbed)0.9400.8980.8430.298†0.675†
𝐯₂: IIA (y₂ perturbed)0.7900.7900.7680.573†0.683
TransportL14.H14 attention, y₂ → y₁0.7110.7310.6690.5750.724
𝐯₁: IIA (y₁ perturbed)0.4600.4850.365†0.195†0.298†
Shared plane(𝐯₁, 𝐯₂) plane: IIA (y₁ perturbed)0.8100.8150.7400.7380.798
(𝐯₁, 𝐯₂) plane: IIA (y₂ perturbed)0.9480.9480.9300.7980.923
L15 DAS: IIA (y₁ perturbed)1.0001.0001.0000.8950.930
L15 DAS: IIA (y₂ perturbed)0.6550.513†0.458†0.048†0.085†

Table 2. Components refitted on P3 and P4, the prompts where frozen transfer is weakest (Table 1). † IIA below 0.8 × the P0 value. * Below the P0 fitting noise.

ComponentMeasureP0 (original)P3 (question)P4 (code)
𝐮IIA (y₁ perturbed)0.940 (0.908–0.945)0.9580.958
|cos| to P0 fit0.8930.543*0.756*
𝐯₂IIA (y₂ perturbed)0.7900.7180.768
|cos| to P0 fit0.9950.922*0.944*
𝐯₁IIA (y₁ perturbed)0.460 (0.425–0.478)0.188†0.405
|cos| to P0 fit0.9280.459*0.941
(𝐯₁, 𝐯₂) planeIIA (y₁ perturbed)0.8100.233†0.820
IIA (y₂ perturbed)0.9480.685†0.943
Smaller principal cosine to P0 plane–0.4580.939
L15 DASIIA (y₁ perturbed)1.000 (0.998–1.000)1.0001.000
IIA (y₂ perturbed)0.655 (0.655–1.000)0.398†0.995
|cos| to P0 fit0.4660.5090.463*

In summary, across different prompt templates, the model uses the same algorithm by reusing many components found in our analysis. Major changes in prompt format require refitting the causal directions u, v1, v2, and the answer direction from L15 DAS.

(continued...)

As acknowledged by the authors, the linear directions can influence the output but does not prove that these directions are necessary or unique computational paths.

Yes! We identify a sufficient mechanism for number comparison that allows us to modify model outputs, but do not claim that it is necessary. As noted in prior work, large models have many redundant paths for the same computation (Hydra effect, McGrath et al 2023). Our paper is about sufficiency of linear representations for certain computations (comparison), not necessity. We defer investigations into necessary components for performing numerical comparison to future work. We will add this discussion into a revision of our manuscript.

When the numerical representation or the objects being compared change, such as decimals, non-decimal number systems, or alphabetical order, does the model still exhibit similar computational mechanisms?

Thank you for the question. Our paper’s experimental design includes comparison of pairs and triples of two digit numbers. While we do believe that other number formats and other ordered objects may be compared using similar computational mechanisms, provided the models can perform the comparison with good accuracy, we leave a detailed investigation into these mechanisms to future work. As described in responses to previous questions, we have demonstrated generalization of our main claims across multiple prompt formats and models.

Do the numerical direction, attention heads, and comparison neurons identified in this paper also exist in such sorting tasks?

This is an interesting and important question, but it is outside the scope of our paper. We demonstrate the mechanism for number comparison using causal interventions on representation geometry (linear representations and vector addition) and model components (copy attention heads, MLP neurons performing comparison). We leave an investigation into the reuse of discovered components and mechanisms in more complex tasks using comparison, like sorting and selection, to future work.

Summary: We thank the reviewer for their feedback which has helped us generalize our claims to multiple models and prompt templates. We hope our responses and our additional analyses within the scope of our paper answered the reviewer’s questions. We would be happy to address any further questions during the discussion period.

Reviewer_cLa3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Past work has shown that numbers are not represented linearly inside of language models, on manifolds like a spiral. In this paper, the authors ask, if a concept is represented on a certain manifold, does that influence how the language model processes the numbers? The answer is no -- complex representations don’t necessarily imply complex computations. The authors use DAS and PCA to find the directions inside of the model that are most associated with the computation and then show that the model uses simple magnitude signals along these directions to compare numbers: it brings the signals together, performs comparisons over smaller ranges of values, and combines those results to decide which number is larger.

Critical strengths
  • This paper complements some existing prior work exploring number representation

  • The paper is well-written and was enjoyable to consume .

Critical weaknesses
  • I would have liked to see more models compared.

  • The paper clearly demonstrates that simple linear signals can support number comparison despite curved representations. However, the broader implications remain underspecified: it is unclear when we should expect this mechanism to generalize, or what the findings rule out about the computational role of manifold geometry.

You are missing a couple of important citations:

  • Ku, Alexander Y., Declan Campbell, Xuechunzi Bai, Jiayi Geng, Ryan Liu, Raja Marjieh, R. Thomas McCoy et al. "Levels of analysis for large language models." arXiv preprint arXiv:2503.13401 (2025).

  • Marjieh, Raja, Veniamin Veselovsky, Thomas L. Griffiths, and Ilia Sucholutsky. "What is a number, that a large language model may know it?." arXiv preprint arXiv:2502.01540 (2025).

Questions for authors

Minor points:

  • Line 90 need to properly cite El-Shangiti w/o parentheses.

  • Line 194 and around there the opening quote ‘ is wrong.

1 comment
AuthorsOpenReview ↗

Included additional results in updated manuscript

We thank the reviewer for their review. We appreciate the reviewer’s comments about our paper being “clear and easy to follow”, “enjoyable to consume”, and our work “complements existing prior work exploring number representations”. We are grateful that the reviewer highlighted the key contributions of our work: 1) we find causally relevant linear representations of numbers performing comparison despite the presence of manifold structure, 2) we identify the algorithm for number comparison using linear representations.

We address specific questions below.

I would have liked to see more models compared.

Thank you for raising this issue. We have now added additional analyses on Llama-3.1-8B and Gemma-2-9B-it models, as well as Qwen2.5-7B (our main paper uses Qwen2.5-7B-Instruct) in Section 2.5 and Appendix C.9 of our updated manuscript. We found the same high-level algorithm—causally relevant linear features for numbers u, v1, v2, the shared (v1,v2) plane, and MLP neurons performing comparison on this shared plane—in all models. Therefore, our core claims of causally relevant linear number representations used in comparison hold across multiple models! We summarize the main takeaways from our results below:

Table R1. Interchange intervention accuracy (IIA) of the mechanism components in four models.

ComponentMeasureQwen2.5-7B-Instruct (paper)Qwen2.5-7B (base)Llama-3.1-8B (base)Gemma-2-9B-it
𝐮IIA (y₁ perturbed)0.9400.9500.7300.800
𝐯₁IIA (y₁ perturbed)0.4600.3880.6250.918*
𝐯₂IIA (y₂ perturbed)0.7900.7000.958*1.000*
(𝐯₁, 𝐯₂) planeIIA (y₁ perturbed)0.8100.7600.7630.913
IIA (y₂ perturbed)0.9480.8780.7900.928
Decision flag (y₁ fit)IIA (y₁ perturbed, fitting case)1.0001.0001.0000.970
IIA (y₂ perturbed, transfer)0.6550.1850.0000.000
Decision flag (y₂ fit)IIA (y₂ perturbed, fitting case)1.0000.9501.0000.985
IIA (y₁ perturbed, transfer)0.3880.0000.0950.000
Difference across models: In Llama and Gemma, although the decision flag can be fit separately for each perturbation case, it does not seem to transfer across perturbation cases (unlike in Qwen2.5-7B-Instruct where the flag does transfer). This means that the answer to argmax computation (decision flag) may be higher dimensional in Llama and Gemma than the simple one-dimensional flag found in Qwen, and this difference across models is interesting on its own. Importantly, this does not affect our core claims since we claim numbers have causally relevant linear structure, which does hold across models. We use a modified version of DAS to find causal direction v2 in Llama and Gemma (see Appendix C.9). Main results are replicated for all models in Appendix C.9.

It is unclear when we should expect this mechanism to generalize, or what the findings rule out about the computational role of manifold geometry.

Thank you for raising this important point. Our findings indicate that causally relevant geometry can be task-specific, and one must NOT rule out linear representations simply by observing curved manifold structure. While we don’t claim that manifold geometry is not useful in its entirety, we show that certain tasks do not need the entire manifold structure, and simpler sub-structures, such as linear representations, suffice for the model’s computation. Therefore, our findings challenge the perspective that manifold geometry is the simplest unit of computation in large models. We posit that in multiple cases where manifold steering (Wurgaft et al 2026) and manifold manipulation techniques (Gurnee et al 2026) have been used and proposed, it may be possible to find causally relevant linear subspaces.

Generalization: We hypothesize that other concepts which have an intrinsic order, i.e., can be mapped to the number line such as object weights/ sizes, color intensity, may have causally relevant linear representations for simple decision making tasks that involve comparison, even if they have manifold structure in representations. Our broader implication is that the interpretability community may have to treat representation geometry and causally relevant geometry as separate objects of study, without assuming that the former implies the latter. We will include the above implications in the discussion section of our updated manuscript.

Minor points: Line 90 need to properly cite El-Shangiti w/o parentheses. Line 194 and around there the opening quote ‘ is wrong.

Thank you for pointing this out. We had incorporated the formatting and citations in our manuscript.

Summary: We thank the reviewer for their constructive and useful suggestions, which have strengthened the paper. We hope our additional experiments have addressed the reviewer’s questions. We are happy to answer any further questions the reviewer may have!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Appreciate reviewer engagement!

Dear reviewers,

Thank you for taking the time to review our article. We would appreciate your comments on our rebuttal and new results.

Warm Regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.