acceptodds
Under review as a conference paper at ICLR 2027

Short Horizons and Sparse Concepts: a Mathematical View of the Readout in the J-lens

Abstract

The Jacobian lens (J-lens) has been proposed as a way to read verbalizable representations from language models. However, its principle and meaning lack a detailed and theoretical discussion. We provide a mathematical view of this interpretation and of its assumed causal structure. Besides treating the J-lens as a heuristic probe, we further regard it as a first-order causal transfer operator from intermediate activations to expected future readouts. We study the Jacobian matrix as the optimal local linear approximation of the downstream mapping, analyze its global approximation behavior and bias, and identify its mathematical meaning as an expectation over anticipated future readouts. Further analysis of the Jacobian energy distribution reveals that its causal geometry is highly sparse. The energy decays with depth, concentrates in an extremely small proportion, and decomposes into diagonal pathways and specific critical positions. This decomposition further resolves the expectation of the J-lens over future outputs into short-horizon and sparse concept predictions, providing a more intuitive attribution and explanation for the ability of the J-lens to visualize concepts during the thinking process. Based on the theory, we propose a simple but effective improvement strategy, which significantly enhances the ability of the J-lens to read out correct intermediate concepts.

Primary area:
interpretability and explainable AI
Keywords:
J-lens, Mathematical Interpretation, Short Horizon, Sparse Concept

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4T6nOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to build a mathematical explanation for the J-lens, and proposes a filtering step to increase its effectiveness.

Critical strengths
  • Exploring the J-lens is a timely topic, and getting a foundational understanding of how the method works would be useful for the field.
Critical weaknesses
  • The notion that the J-lens is an expectation over future readouts is not particularly novel, since it is what the original J-lens paper intends to isolate [1]. Further, the central theorem assumes Gaussian inputs, but transformer activations are not Gaussian [2]! This in my view very much weakens the strength of the analysis in this work.

  • The paper does not explain how much additional time/FLOPs are required to implement their proposed method atop the J-lens, or justify how it would be actually desired in practice. How much compute is needed for the energy calculation and ranking them?

  • The paper does not check whether improvements from the proposed method consistent with different noise seeds, nor does it evaluate whether the improvements are statistically significant and meaningful.

Minor Weaknesses:

  • Claims will generalize poorly because all experiments have been conducted on small checkpoints on just 2 model families.

  • There are some potential mathematical issues with Eq. 18 and 20, please verify.

[1] https://arxiv.org/abs/2607.15495 [2] https://arxiv.org/abs/2402.17762

Questions for authors
  • There are no controls to show that ranking high energy (source, target) and pruning is what improved the ICR score. What would have happened if they were ranked randomly?

  • Further, how is the proposed method actually supposed to help down the line? Does pruning while preserving high energy source, target pairs result in J-lens readouts that reveal more ideal steering targets?

Reviewer_YaaiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to provide a mathematical interpretation of the Jacobian Lens (J-Lens). The authors first connect the averaged Jacobian to a linear prediction operator through Taylor expansion, least-squares regression, and Stein's identity. They then analyze Jacobian energy across layers and token positions, interpreting high-energy regions as short-horizon predictions and sparse concepts. Finally, an energy-based position filtering strategy is proposed and evaluated on Qwen3 and Gemma3 models.

Critical strengths

The problem addressed in this paper is meaningful. Understanding the mathematical properties of averaged Jacobians and the contributions of different token positions could help explain the effectiveness and limitations of J-Lens. The authors also evaluate the proposed filtering strategy across several models and report improvements in intermediate-concept recall under certain settings.

Critical weaknesses

I find the theoretical novelty rather limited. The Jacobian as the optimal local linear approximation, as well as the relationship between the expected Jacobian and least-squares regression under Gaussian assumptions, are established mathematical results. The paper mainly applies these results to J-Lens rather than developing a fundamentally new theory.

More importantly, Equation (18) in Section 3.4.3 is mathematically questionable. The authors directly add the squared nonlinear approximation error and the squared non-Gaussian bias without sufficient justification.

For a general input covariance matrix, the contribution of slope mismatch to prediction mean-squared error should involve covariance weighting rather than simply the squared Frobenius norm. Consequently, Equation (18) is not a valid general error decomposition as currently stated.

The proposed ICR counts the number of position-layer cells whose top-K predictions contain a target concept, rather than evaluating whether the concept is recovered at the correct location and layer. Consequently, an increased ICR does not necessarily imply more accurate interpretability. It may simply indicate that the target token appears at more positions. The metric does not adequately distinguish correct concept localization from broad but nonspecific activation.

In addition, the main paper contains only approximately eight pages of technical content, with the conclusion beginning on page 8. The experimental details and analysis are not sufficiently developed. More importantly, the authors explicitly acknowledge in the conclusion that the work is still in progress and that the reported findings are not final.

Questions for authors

see above

Reviewer_o2JNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the mathematical interpretation of the Jacobian lens. It connects averaged Jacobians to optimal affine prediction through a Gaussian Stein identity and discusses deviations arising from nonlinearity and non-Gaussian activations. It analyzes positional Jacobian energy, identifying diagonal and concentrated cross-position patterns interpreted as short-horizon predictions and sparse concepts. Motivated by this concentration, the authors propose constructing the lens using only high-energy positions. Validation includes layer-wise bias and energy analyses, visualizations, and evaluations on six tasks across five Qwen3 and Gemma3 models. Filtering improves reported average intermediate-concept recall substantially for Qwen3, with smaller improvements for Gemma3, alongside modest gains in short-horizon prediction.

Critical strengths
  1. This provides a productive framework for understanding and investigating J-lens approximation error.
  2. Their modification yields encouraging results across multiple models. Energy filtering improves average ICR from 19.41 to 27.93 for Qwen3-4B and from 18.71 to 22.14 for Qwen3-8B, while also modestly improving SHL.
Critical weaknesses
  1. Stein identity is valid for the stated function y=f(h)y=f(h), but transformer outputs also depend on contextual activations. The implemented Jacobian varies one activation while holding others fixed, whereas the global regression averages over correlated contexts. The paper does not establish conditions connecting these two objects; marginal Gaussianity of the selected activation alone is insufficient. Additionally, as a prediction-MSE decomposition, Eq. (18) requires covariance weighting: with Δ=W∗−E[J]\Delta=W^*-\mathbb{E}[J], the excess error is tr⁡(ΔΣΔ⊤)\operatorname{tr}(\Delta\Sigma\Delta^\top), rather than ∣Δ∣F2|\Delta|_F^2, unless an additional condition such as whitening applies. These issues affect the central optimal-readout interpretation.

  2. Concentrated Jacobian norms establish sensitivity patterns, but do not by themselves identify concept-specific causal roles or establish which contributions dominate the averaged operator, where directional cancellation matters. Improved concept-token recall supports filtering’s usefulness but does not independently validate the proposed explanation. Assessment is also limited by missing evaluation sample counts and uncertainty estimates, and unclear protocol details: Eq. 22 defines ICR as a count, Section 5 describes position-pair filtering while Appendix D describes source-position filtering, and the full-grid metric description is not reconciled with Appendix C’s fixed readout positions.

Questions for authors
  1. How is context handled in eq 2, and what assumptions connect the implemented intervention Jacobian to the derivative required by the Stein argument?
  2. Were activations whitened in the bias experiments? How were Figure 1’s quantities estimated, and does the claimed agreement hold using the covariance-weighted error decomposition?
  3. Please clarify the filtering unit, ICR normalization, evaluated positions, evaluation sample counts, uncertainty estimates, and whether filtering ratios were selected on separate validation data. What evidence distinguishes the proposed concept-specific explanation from a generic benefit of retaining high-sensitivity contributions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.