acceptodds
Under review as a conference paper at ICLR 2027

How to track your tangent space: the geometry of the linear representation hypothesis

Abstract

We examine how local geometry shapes interpretability by studying the tangent structure of the activation manifold of language models: we show that hidden states can be decomposed into linear and multidimensional components by projection onto the tangent space. Using this decomposition, we then show that linear components dominate the interpretable signature of the network. We then perform a similar decomposition of the recently proposed -lens, to concentrate its on-manifold versus off-manifold data. We give a full analytic description of a tangent-native -lens, and show that the normal contribution to the -lens depends on the off-manifold extension of the network: in particular, the workspace readout can be changed arbitrarily subject to mild technical constraints, which could prevent a misaligned network from being detected. Empirically, we show that there is not a simple fix for this: the on-manifold Jacobian shows substantial cancellation in mid-to-late layers, and the -lens outputs materially rely on their off-manifold projection.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, lenses, activation geometry, activation manifolds, manifold structures

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A7SNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the geometry of the Linear Representation Hypothesis, specifically its implications of how activations should decompose into the tangent and normal spaces of the representation manifold. They show three key findings: the tangent component of activations contains most of the interpretable information measured by the J-lens, a theoretical analysis that shows J-lens can be modified without changing on-manifold model behavior and lastly empirical evidence that J-lens’s performance relies more on its normal component.

Critical strengths
  • The paper is well motivated, providing a new analysis of the linear representation hypothesis. The use of the projection to tangent and normal spaces is interesting and may provide motivation beyond this work.

  • The discrepancy between the interpretable information being tangent to the activation manifold and J-lens’s greater use of the normal space is interesting, and provides insight into a new method that is less understood and may have implications for future research.

Critical weaknesses
  • Given that the results show that the push-forward and PCA approximations of the tangent space capture different subspaces and in 4.3 had differences in results (Tables 1 and 2), it would be more convincing to see for section 5.1 the results with the PCA approximation as well. Seeing consistent trends across both methods would strengthen the claim of 5.1 and show it’s not an artifact of the tangent space approximation.

  • In Section 4.2, it is shown that the pushforward and PCA approximations capture substantially different subspaces. Therefore, it is hard to understand from the paper how well the approximations represent the tangent space, and by extension the reliability of the empirical findings.

Questions for authors

Both approximation methods contain parameters such as neighborhood size and energy (for PCA). How sensitive are the empirical results to the choice of hyperparameters? Also, the dataset used for generating activations might have an effect on the quality of the approximation, have the authors tried increasing/decreasing the number of sequences to validate their choice?

Reviewer_sQyrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the behavior of internal representations of large language models, motivated by the linear representation hypothesis and on-manifold vs. off-manifold behavior. The authors find that much of the interpretable information captured by the J-lens is preserved by tangent space projections of the current hidden state, while the J-lens itself relies on normal/off-manifold components. The authors also theoretically show that such off-manifold components can be modified without altering a network’s on-manifold behavior and output distribution.

Critical strengths

The central tension between the importance of on-manifold and off-manifold information to the behavior and interpretability performance of J-lens is very interesting.

The effect is clearly shown across multiple real language models in the synthetic lens evaluation suite of Gurnee et al. 2026.

The formal mathematical analysis of J-lens on-manifold and off-manifold behavior is also helpful.

Critical weaknesses

Overall, the connection between the results of the work and the linear representation hypothesis is established somewhat less well. While the LRH is an interesting concept, most results in the paper seem only to rely on the fact that the representation manifold is generally smooth, not necessarily that the representations themselves are linear.

Furthermore, while the mathematical theorems are appreciated, some of the central theorems seem closely related to prior work, such as Anders et al. 2020, as the authors point out. Thus, it is somewhat unclear as to what the novel components of the new theorems are, as well as the broader implications of those components.

Finally, the implication that altering off-manifold information can change J-lens behavior without changing on-manifold behavior, while interesting, has not been shown to be actually realizable in real language models, so the risks in lines 397 seem a bit speculative, though the authors do acknowledge that that remains future work.

Questions for authors

No additional questions beyond weaknesses.

2 comments

Hello! We just wanted to follow-up briefly, to see if you've had a chance to review the updated material. The deadline is in ~2 days, and we'd love to know if there are any more improvements we can make, or if we've satisfactorily resolved any reservations you had!

I would like to thank the authors for providing the clarifications to both the LRH as well as the details of the proof. The new results on the LLMs in figure 4 are informative, and I have increased my score as a result.

Reviewer_jhJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper's primary investigations are along the recently introduced J-lens, where the main result is a distinction between intrinsic interpretability and derivative-based interpretability, i.e., a model’s behavior on its activation manifold does not uniquely determine its Jacobian-based explanations, and the off-manifold extension can carry substantial explanatory signal. Overall, the paper has interesting geometric insights about the representations, a potentially important identifiability result in Theorem 3, and an empirical validation that is less conclusive than the primary claims made at the start of the paper.

Critical strengths
  • The strongest contribution/interesting finding is an identifiability warning for Jacobian-based interpretability, on-manifold behavior alone does not determine ambient gradients, so a derivative-based readout can depend on an unconstrained extension of the model. This result may be useful for the community if validated thoroughly,

  • The geometric decomposition is useful (considering that the tangent is only a proxy for linearity). The experiments establish that normal-projected Jacobians can be highly informative on the tested benchmarks, not that normal directions universally encode more meaning or that the J-lens is practically easy to manipulate in trained transformers. Some results along those lines will make a paper more impactful.

Critical weaknesses
  • The tangent space is estimated/approximated from a finite empirical corpus, but the theorems are stated for smooth manifolds. The bridge between the empirical estimator and the mathematical object requires validation, especially in high dimensions, which is not completely clear.

  • The stated Theorem 1 is straightforward, which is a geometric integration-by-parts identity. It says that the average tangent derivative is constrained by the on-manifold function values, the sampling density, and the manifold geometry, however this itself is not enough to claim that the tangent derivative vanishes/that it is always semantically informative. Similarly, for Theorem 2, the results strongly depend on a strong set of assumptions (like orthogonality, cone geometry, and a specific coefficient distribution) that may be violated in the LLM representation setting. Furthermore, the implication of Theorem 3 is left for future work, and claims about a misaligned agent manipulating its J-lens are a demonstrated vulnerability of the mathematical abstraction framed by the paper, not yet an end-to-end transformer attack.

Questions for authors
  • The tangent and normal subspaces are estimated/approximated, the global Jacobian is estimated from a finite sample, and the benchmark covers six synthetic tasks. I was wondering about the reason not to consider a stronger causal test that would vary the number of Jacobian samples, compare local and global lenses at matched computational budgets, and assess whether the observed cancellation persists under alternative tangent estimators. This would be straightforward to perform and would reveal more reliability in the paper's claims.
3 comments

Hello! We just wanted to follow-up briefly, to see if you've had a chance to review the updated material. The deadline is in ~2 days, and we'd love to know if there are any more improvements we can make, or if we've satisfactorily resolved any reservations you had!

Thank you for acknowledging the primary concerns regarding the theorems, the revised theoretical framing is more precise, the empirical methodology is better validated, and the normal bundle attack further strengthens the practical relevance of the paper. I think these validate my initial assessment of the paper, and with the modified definitions, I am inclined to view the revision favorably and upgrade my soundness assessment.

Thank you for your updated review and recommendation! We appreciate your time and suggestions, which were very helpful!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.