acceptodds
Under review as a conference paper at ICLR 2027

Linear Probes to Bayesian Manifolds: Geometry of Arithmetic in Large Language Models

Abstract

Linear probes are a standard method to study geometric structures, such as helices and circles, from the activations of large language models (LLMs), and such structure is often taken as evidence that the model computes with it. Probe accuracy alone cannot establish this, because it conflates four separate claims. These are (1) that the concept is decodable from the activations, (2) that they form a specific shape, (3) that the shape belongs to the probed concept rather than to a related concept the model also represents, and (4) that the output of the model depends on the shape. We call these claims decodability, shape, ownership, and causal use, and we test them separately with a four-stage Bayesian pipeline. Stage 1 verifies that a concept is decodable and that the decoding is not an artefact. Stage 2 compares candidate shapes and selects the best one only when its evidence passes a predefined threshold. Stage 3 removes a set of related concepts, specified in advance, and tests whether the shape is owned or inherited. Stage 4 intervenes on the shape, at the final token of the prompt, to test whether the output of the model depends on it. Because every stage issues its verdict on the same data, the four answers can be compared directly. We apply the pipeline to two-digit addition and multiplication in GPT-J 6B, Llama 3.1 8B, and Pythia 6.9B. The results were, nearly every concept is decodable, but the decoded directions overlap substantially across concepts. Every strongly supported comparison selects a ribbon, which is a helix whose radius changes with the value, or a torus, which refines rather than contradicts the helix account of prior work. Of 1,330 cells with an ownership verdict, 19% of shapes are owned and 81% are inherited. Owned and inherited shapes move the output about equally often. Intervening on the shape retains approximately 61% of the effect of intervening on the full probe while using at most five dimensions, against a median of nine for the probe. Our results indicate that a probe verdict alone overstates what a model computes with, and we propose that probe results be reported as verdicts on all four claims rather than as a single accuracy.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, linear probes, representation geometry, manifolds, Bayesian model comparison, arithmetic, activation patching, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fGCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a four-stage Bayesian pipeline to study arithmetic representations in LLMs, separating decodability, geometric shape, ownership, and causal use. Experiments on two-digit addition and multiplication across three models show that concepts are often decodable, but their geometric structures are frequently shared with related concepts. The results suggest that probe accuracy alone does not establish how a model performs computation.

Critical strengths

This work provides clear conceptual framework. Separating four distinct claims helps avoid equating successful decoding with causal use, offering a useful perspective for interpreting probe results. The paper combines shape comparison, related-concept removal, and causal interventions across multiple models and tasks, evaluating all four claims on the same analysis units.

Critical weaknesses
  1. The scope is limited. The experiments cover only two-digit addition and multiplication at the final prompt token. It does not fully explain the computation across token positions or establish generalization to more complex tasks.
  2. The ownership criterion needs stronger justification. If shape evidence does not increase after removing related concepts, this does not necessarily mean the shape is inherited. The removal may also damage information about the target concept.
  3. Formatting could improve readability. Example tokens, such as 2 and 1 in lines 73–75, should be formatted distinctly to make them easier to distinguish from the surrounding text.
Questions for authors

Refer to Weakness.

Reviewer_o39wOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a four-stage framework for understanding arithmetic representations in large language models. The framework tests whether arithmetic concepts can be decoded from internal activations, identifies their geometric structure, assesses whether this structure belongs to the target concept or is inherited from related concepts, and evaluates its causal influence on the model’s predictions.

Critical strengths
  1. The paper has a clear motivation and addresses the important question of how large language models internally represent and perform arithmetic.

  2. The paper proposes a systematic four-stage framework that distinguishes decodability, geometric structure, ownership, and causal use, enabling a more rigorous analysis of arithmetic representations.

Critical weaknesses
  1. The broader significance of the findings remains unclear. While the framework systematically characterizes arithmetic representations, the connection between these geometric findings and the model’s underlying computational mechanisms is not fully established. Demonstrating concrete mechanistic insights and clarifying their relevance beyond the studied arithmetic tasks would strengthen the paper’s contribution.

  2. The ownership criterion requires further validation. When arithmetic concepts are strongly correlated, removing related-concept directions may also erase information belonging to the target concept itself. A decrease in geometric evidence therefore does not necessarily establish that the structure is inherited. Although the authors acknowledge this limitation, the frequency and impact of such misclassifications remain unquantified.

Questions for authors

See Weaknesses.

Reviewer_CLsFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks a good question: if a clean geometric structure is found in an LLM's hidden states, does that show the model actually computes with it? Starting from a linear probe that predicts an arithmetic concept (e.g. the carry in addition) from hidden states, the authors split the question into four stages: decodability, shape, ownership, and causal use. They propose a Bayesian pipeline that compares candidate geometries (circle, helix, ribbon, torus) by marginal likelihood, then tests ownership by removing the subspaces of related concepts, and causal use by subspace ablation and patching. Experiments cover GPT-J 6B, Llama 3.1 8B and Pythia 6.9B on two-digit addition and multiplication (4,860 cells). Headline numbers: ribbon wins most shape comparisons, 81% of concepts are "inherited" rather than "owned", and ablating the geometric subspace retains 61% of the full-subspace effect.

Critical strengths
  1. The paper's main value is that it corrects a common logical error in interpretability work: decodable does not mean causally used. Separating decodability, shape, ownership and causal use into explicit stages, with pre-registered criteria, is the right discipline, and the pipeline is applied consistently across three models and two tasks.
Critical weaknesses
  1. The "81% inherited" result is close to definitional. Answer digits and carries are deterministic functions of their listed related concepts (the operand digits), so their subspaces necessarily lie in the span of the related concepts' subspaces and are guaranteed to be classified as inherited. The number therefore reflects the pre-registered related-concept lists and the strict ownership rule more than a property of the models, and the ownership verdict does not yet have a stable causal interpretation.

  2. The shape comparison is driven by model misspecification and by the candidate set. The ribbon family nests the circle and the open helix, so under the paper's isotropic noise model and large N the most flexible family wins almost by construction; evidence gaps in the thousands of nats make the stated 5-nat selection bar irrelevant. The helix candidate is single-period, so the claim that the ribbon "refines" the helix of prior work is never tested against the actual multi-period model. More generally, it is unclear whether the Bayesian comparison distinguishes geometries or merely ranks a pre-chosen list.

Finally, an intervention effect on hidden states does not by itself show that the model was computing along that geometry; the causal stage needs a stronger control than random rank-matched subspaces in the full 4,096-d space, which carry almost no variance.

Questions for authors
  1. How many of the "inherited" verdicts are forced by the related-concept lists (concept is a deterministic function of its listed relatives)? Please report ownership restricted to concepts that are not functions of their relatives.
  2. Does the ribbon still win when the helix candidate is the multi-period generalized helix of Kantamneni & Tegmark, and when the noise model is non-isotropic?
  3. For the causal stage, what is the retention when the control is the top-k PCA directions of the concept subspace, or random k-dimensional subspaces inside it, rather than random subspaces of the full space?
  4. What is the intended reading of "owned" when subspace removal takes 85-89% of the variance in both owned and inherited cells?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Message to Area Chair and Senior Area Chair (Submission 37194)

Dear Area Chair and Senior Area Chair,

Thank you for handling our submission, and thank you to the three reviewers for their time. All three reviewers agree on the value of our main idea, that probe accuracy alone does not show what a model computes with, and that decodability, shape, ownership, and causal use must be tested separately.

We have posted a full response to each reviewer. While we prepared it, we found that most of the weaknesses and questions are already addressed in the submitted paper. Our responses point to the exact sections, tables, figures, and lines. In short:

  1. Reviewer CLsF asks whether the inherited verdicts are forced by the related-concept lists, whether the shape comparison favours the most flexible shape, and whether the causal control is strong enough. Section 4, Sections 5.2 to 5.4, and Appendices G and H, with Figure 4 and Table 18, answer each of these points.
  2. Reviewer o39w asks about the broader significance of the findings and about the validation of the ownership criterion. Sections 5.4, 6, and 7 state the mechanistic findings and their relevance beyond arithmetic. Section 4 and Appendix G define the ownership rule and state how its one possible error works.
  3. Reviewer fGCM asks about the scope of the study, the justification of the ownership criterion, and the formatting of example tokens. Section 1 and Appendix B state the scope and the reason for it. Sections 4 and 5.3 and Appendix G justify the ownership criterion. We will apply the formatting change in the revised version.

We fully respect the review process and the judgement of the reviewers. We would be grateful if you could encourage the reviewers to read our responses before the discussion closes on November 18. If they find that their concerns are resolved, we hope that their final reviews and scores can reflect this. If any point remains open, we will gladly answer it.

Kind regards, The authors of Submission 37194

AuthorsOpenReview ↗

Concern Regarding Lack of Reviewer Engagement During the Discussion Period – Submission 37194

Dear Area Chairs and Senior Area Chairs,

We would like to express our sincere concern and disappointment regarding the lack of reviewer engagement during the discussion period for our submission (37194).

We submitted detailed, point-by-point responses on November 8, addressing all the concerns raised, with supporting evidence and references to the paper. We also posted gentle reminders on November 9 and November 13. Unfortunately, none of the three reviewers has acknowledged or engaged with our responses, and only two days remain before the discussion period closes.

As authors, we have invested considerable time and effort in this research and have made every sincere attempt to address the reviewers' concerns. It is disheartening to receive no engagement despite these efforts. We do not expect reviewers to necessarily change their opinions or scores, but we do hope our responses will be read and fairly considered before a final decision.

We kindly request you to take note of this situation and, if possible, encourage the reviewers to engage before the discussion period closes.

Thank you for your consideration.

Kind regards, The authors of Submission 37194

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.