acceptodds
Under review as a conference paper at ICLR 2027

Model Internals Speak Volumes: J-Lens Loudness Predicts Decodability

Abstract

When probing the activations of a Large Language Model (LLM), several design decisions arise. Besides the probe design itself, one must choose which layer(s) and token position(s) to probe. These choices are often made heuristically or through expensive sweeps over candidate layers and positions, and to our knowledge, no method yet predicts the decodability of a signal directly from the activations. We take a step in this direction by introducing J-Lens loudness, a new application of the Jacobian lens (J-Lens) that guides these choices without labels or probe training. We represent a signal of interest by a small subset of the model's vocabulary, and define J-Lens loudness as the log probability mass that the J-Lens assigns to this subset at a given activation. In the reasoning chains of two LLM agents navigating grid worlds, the balanced accuracy of next-action probes increases steadily with direction signal loudness. Taken together, we show that J-Lens loudness can be used as a criterion to find layer–token positions whose activations contain underlying computation relevant to a signal of interest.

Primary area:
interpretability and explainable AI
Keywords:
J-Lens, Jacobian Lens, probing, probes, decodability, representation, mechanistic, interpretability, llm, layer selection, token selection, localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n2ntOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work adopts J-Lens loudness to identify useful layers and token positions for probing LLM activations without training probes. Experiments on two LLMs in grid-world tasks show that higher loudness is generally associated with better decoding accuracy for movement directions.

Critical strengths
  1. The idea is simple and interesting.
  2. The experimental results seems encouraging.
Critical weaknesses
  1. My main concen is that this work does not directly verify whether the selected layers achieve the best probing accuracy, which weakens one of its main claims.
  2. The experiments are limited to one environment and mainly one successful signal, so the generality of the method remains unclear.
Questions for authors
  1. Please add at least one experiment to demonstrate the optimality of the selected layers.
  2. Please consider more environments and signal types.
2 comments

We thank the reviewer for the time spent on reviewing our paper submission, and for the concise summary of flaws. We added a general comment aimed to reply to all concerns raised by all reviewers, and we will reference it here while addressing your concerns.

  1. Optimality of the selected layer. As stated in the general reply, we agree this was a potential over-claim. We tested it by running a sweep and find that, in short, the loudest layer is also the most decodable for Qwen3.6-35B-A3B and close to the most decodable for GPT-OSS-20B. It is near-optimal, not optimal, and we will revise the claim to match these results.

  2. More environments and signals. Further signals and environments are unfortunately not feasible in this reviews cycle. To make up for this, we will add a second probing task with labels from the environment rather than labels verbalized by the model (experiment A). We will state this limitation in the paper, and treat our work as a first step towards understanding how to better select token positions for probing.

Would reducing the layer-selection claim resolve your main concern? And is there something else you would consider necessary for acceptance? Looking forward to your reply!

Thank you again for your review. We've uploaded the revised PDF, and the general comment above lists everything that changed.

Optimality of the selected layers: We ran the experiment you asked for: probes at every layer of both models (Section 4.1, Figure 3, with full tables in Appendix C.8). For Qwen the loudest layer is also the best one. For GPT-OSS it sits on the high-accuracy plateau, two layers from the best. Therefore, we have softened the claim throughout the paper to "a layer on the plateau" rather than "the optimal layer", and we now present position selection as the main contribution.

More environments and signals: We added a second probing task whose labels come from the environment rather than from the model (Section 4.5), and loudness predicts decodability there as well. We couldn't add new environments in this cycle, and we state that as a main limitation.

Aside from the request for additional environments/signals, do these changes settle your concerns?

Reviewer_j7uvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes J-Lens loudness, a label-free measure based on the probability mass assigned by the Jacobian Lens to a signal-related vocabulary. The method is used to select layers and token positions for probing without training probes during selection. Experiments on two LMs performing grid-world navigation show that positions with higher direction loudness generally yield better probe accuracy than those selected by Logit Lens or random sampling.

Critical strengths
  • J-Lens loudness provides a signal-specific criterion for identifying informative activations without requiring labeled probe training during selection. The improvements over random and Logit-Lens-based selection are consistent across the two evaluated models.

  • The cross-selection experiments help separate position quality from probe training effects. The loudness-decile analysis and removal of explicit direction words provide additional support for the observed relationship between loudness and decodability.

Critical weaknesses
  • Generalizability is limited. The main findings are demonstrated for only one successfully decoded signal in a single grid-world environment, while the grid-state experiment remains inconclusive. Furthermore, the proposed layer-selection criterion is not directly validated against a layer-wise probe sweep, leaving an important part of the central claim unverified.

  • The method primarily applies an existing Jacobian Lens to a heuristically constructed signal vocabulary, and the sensitivity to vocabulary construction is not evaluated. Fitting the J-Lens also requires substantial computation, while no direct cost comparison establishes its advantage over conventional probe selection. The lack of pracitical efficiency is downweighting the paper quality.

Questions for authors
  1. Have you checked whether the layers selected by J-Lens loudness actually achieve the best probe accuracy across layers?

  2. Does the relationship between loudness and decodability hold for another decodable signal, given that the grid-state experiment was inconclusive?

  3. Given the cost of fitting the J-Lens, is the overall procedure cheaper than training probes at a few candidate layers and positions?

3 comments

Thank you for the careful review and for the questions. Since criticism from other reviewers was similar, we issued a general comment which aims to answer all questions. Here we provide answers to your questions specifically, and hope to satisfy your concerns.

  1. Layer validation. We performed a layer sweep with varying results. The loudest layer is also the best for Qwen and close to the best for GPT-OSS-20B. We will narrow the claim from "finds the optimal layer" to "finds a layer on the plateau". We provided more detail in the general comment, and will update the paper with the added probe sweep.

  2. Additional signal. Unfortunately this is not possible in this reviews cycle, but we will state this as an explicit limitation in the paper. To try to make up for this, we will instead add a second probing task with labels coming from the environment rather than the model's verbalization (experiment A in general comment) .

  3. Cost comparison. We will add an estimated comparison (see revision proposal E in general comment). For one signal on one model, a probe sweep over layers is cheaper than fitting a J-Lens. That said, the fit is a one-off cost per model and is reused across signals. Given a fitted lens, a loudness profile is one pass over the activations, which is cheaper than training and testing a probe at each layer. The main saving is over positions: in chains of up to 18k tokens, a position sweep is the expensive part. For GPT-OSS-20B, positions ranked at a single layer were more decodable at every layer, which suggests that a lens at one plateau layer may be enough.

  4. The second critical weakness puts in question the metric's sensitivity to the vocabulary built. To address this, we will add the ablation C (general comment above), where we will measure the change in loudness across variations of the vocabulary.

Would the narrowed layer claim, the cost estimate and the vocabulary ablation (C) address your concerns about soundness and efficiency? If one additional result would matter most to you, which is it? Looking forward to your reply!

Thank you for the layer-sweep results. My other concerns are still unresolved:

  1. Can you provide experimental results on a genuinely different decodable signal?
  2. Can you provide the vocabulary sensitivity results and a quantitative cost comparison against conventional probe selection?

The current PDF does not yet include these results. Could you please update it or share the results here?

Thank you for following up, and sorry the results weren't in the PDF yet when you asked. They are now, and the general comment above summarises all the changes.

A genuinely different signal, ideally in another environment, unfortunately was not feasible in this cycle, and we now name it as the main limitation of the paper. Instead, to add more evidence, we added a second task related to the direction signal: predicting whether the goal is to the left or the right of the agent. The labels of this task come from the grid itself rather than from the model's answer, and it has different classes. Loudness predicts decodability for this task as well (Section 4.5, Figure 6, and Appendix Figures 13 and 16).

Vocabulary sensitivity: Appendix C.10 and Figure 15 show the ablation. We removed a random 20% of the direction words under 5 different random seets and recomputed the layer profile. The profile remained robust to these changes.

Cost: We wrote this up in the Discussion under "Practical use and cost". We compare the processes rather than wall-clock times, since those depend heavily on the hardware. Fitting the J-Lens is the expensive part, but it is a one-off per model, needs no labels, and isn't needed at all if a fitted lens already exists. After that, ranking positions takes a single forward pass with no labels or probe training, whereas a sweep trains a probe for every candidate layer or position and needs labels for each one. There is also an additional advantage at inference time: since loudness correlates with accuracy, the loudness measure can be used as a metric for how much a probe's prediction can be trusted.

Apart from the main limitation of the lack of an additional signal, do these changes address your remaining concerns?

Reviewer_drxpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes J-Lens loudness, a label-free method for selecting layers and token positions to probe in LLMs. The main idea is to measure how strongly a signal of interest is reflected in the model's predictions using the Jacobian lens, and select positions where this signal is most prominent. Experiments on grid-world reasoning tasks with two LLMs show that J-Lens-selected positions generally lead to better probing accuracy than Logit Lens, random, or end-of-sentence positions. The authors suggest that loudness can serve as an efficient alternative to searching over many layers and positions.

Critical strengths
  • The method addresses a practical problem. Choosing where to probe often requires labels or an expensive search, and I like that the paper offers a relatively simple, label-free alternative.

  • The selectivity experiment is a useful addition. Showing that loudness for an unrelated signal does not predict direction accuracy helps support the signal-specific nature of the approach.

  • The results are encouraging. J-Lens performs consistently better than Logit Lens as a selection criterion in the reported experiments, suggesting that it could be useful for identifying informative probing locations.

Critical weaknesses
  • It is unclear whether loudness captures decodability beyond the model's output tendency. Since both the probed direction and its label are derived from the model's own output, high accuracy at loud positions may partly reflect what the model is already likely to say. I found it difficult to tell whether loudness identifies richer internal representations or simply positions where the output is more predictable. An evaluation that separates these two effects would help clarify this distinction.

  • The practical advantage over simpler selection strategies is not fully established. The comparisons focus on Logit Lens and a few position baselines, but it would be useful to know how J-Lens compares with a conventional probing sweep or a simple middle-layer heuristic. In particular, the finding that informative positions tend to occur in intermediate layers seems consistent with existing intuition. Some comparison with these alternatives, ideally on a broader range of probing tasks, would help establish the added value of loudness.

  • The stability of the selected layers and reported improvements is unclear. The loudness profiles appear fairly flat across several layers, making it difficult to tell how meaningful the choice of the loudest layer is. It would be helpful to see how probe accuracy changes when nearby layers are selected instead. The probing results also lack variance estimates, which makes some of the smaller improvements difficult to assess.

Questions for authors

N/A

3 comments

Thank you for the careful review. We added new results in the general comment, together with a plan for additional experiments. Here we respond to your three concerns.

  1. Decodability versus output tendency. We agree the current direction prediction probe experiment leaves this open, since the labels come from the model's own forced answer. Two results may answer this question. First, in our preliminary experiments, we tried using the J-Lens output as a direct classifier (point 3 in general reply above). This leads to random chance prediction (RIGHT almost always has higher log probability), which points to loudness not being limited to simply predicting the model's verbalization. Second, the loudness-decodability trend persists after removing verbalized direction words and their neighbours from the evaluated tokens (Appendix Figure 8). In addition to this, we will also add direction probes with labels computed from the grid, which do not depend on the model's output at that position (see experiment proposal A in general reply).

  2. Advantage over simpler strategies. We performed a sweep for each model, please find it in the general comment. For the optimal-layers claim, loudness does not clearly beat the middle-layer heuristic: it succeeds in selecting the optimal layer for Qwen3.6-35B-A3B, but fails on GPT OSS 20B, by selecting a layer close-by. We will update the paper's claims accordingly. The advantage we claim is for positions, where no comparable heuristic exists: loud positions beat random and end-of-sentence positions (Tables 1 and 2).

  3. Stability. The sweep shows how accuracy changes at nearby layers: it barely moves across the plateau, from layer 11 to layer 22. We agree with the criticism that the log-probability profile seems too flat, but the sweep shows that this matters little. The sweep intervals are over seeds; the decile analyses (Figures 3 and 8) already carry bootstrap intervals over chains.

Would these results, revisions, and proposed experiments settle your concerns? If perhaps you have an additional proposal, we would be happy to look into it. Looking forward to your reply!

Thank you again for your review. We've now uploaded the revised PDF with the experiments we promised, and the general comment above gives an overview of everything that changed. Here is where to find the parts that relate to your concerns.

Decodability versus output tendency: This was the concern we found most important, so we added a probe whose labels come from the environment instead of the model's own answer: whether the goal is to the left or the right of the agent. The relationship between loudness and decodability holds for this label too (Section 4.5, Figure 6). We also added the J-Lens-as-classifier baseline (Section 4.1, Appendix C.9). It sits near chance where loudness is highest, which suggests the lens tells us where the information is but doesn't predict what the model will say. A small correction to our first reply: the classifier's predictions are mostly DOWN, not RIGHT.

Simpler strategies: We added layer sweep results to the paper in Figure 3, and the full results as tables in the tables in Appendix C.8. As we said before, loudness doesn't clearly beat the middle-layer heuristic for picking a layer, and the paper now includes this as a limitation. Where it helps is with positions: loud positions are more decodable than random ones at every layer, not only the one where we ranked them. We now have two probing tasks, but we haven't been able to add more environments in this cycle, and we list that as an additional limitation.

Stability: Tables 7–8 give the per-layer results with seed variation. The vocabulary ablation in Appendix C.10 also shows that the loudest layer doesn't move when the word list is perturbed.

Thanks again for pushing on these points; we think the paper is clearer for it. Do the changes address your concerns?

Thank you for the detailed rebuttal and additional experiments.

I appreciate the new results addressing W1 and W3, particularly the environment-derived labels and layer-wise analysis, which clarify several of my concerns.

However, my main concern regarding W2 remains. Loudness-based layer selection does not clearly outperform simple heuristics, and the advantage of position selection is mainly demonstrated against relatively weak baselines within a single environment. I remain unconvinced of the method's practical significance and generalizability.

Therefore, I will maintain my original rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision proposal

We thank the reviewers for their careful reading. All three asked for a direct validation of layer selection, so we report it here together with the changes we plan.

  1. Layer sweep, GPT-OSS-20B We trained linear and MLP direction probes at every layer from 3 to 22, on two sets of positions: the 20 loudest per chain (ranked once, at layer 15) and random ones. Values are final-epoch balanced accuracy on 720 held-out chains, averaged over 5 seeds (seed std at most 0.008), given as linear / MLP.

20 loudest per chain positions (lr / mlp):

  • Layer 12 (middle): 0.798 / 0.846
  • Layer 15 (loudest): 0.800 / 0.857
  • Layer 17 (best): 0.818 / 0.873
  • Layer 22: 0.795 / 0.848

20 random positions (lr / mlp):

  • Layer 12: 0.658 / 0.716
  • Layer 15: 0.656 / 0.716
  • Layer 17: 0.668 / 0.725
  • Layer 22: 0.640 / 0.685

Three observations:

  • The loudest layer (15) is not the best layer (17), but both sit on the same plateau: accuracy barely changes between layers 11 and 22 (will show the full table in the revised pdf due to character limits on openreview). Layer choice matters little for this signal.
  • Random positions give a curve of the same shape, peaking at layers 16 to 17.
  • Loud positions are more decodable than random ones at every layer, even if they were ranked at layer 15 only.
  1. Layer sweep, Qwen3.6-35B-A3B The same setup, at every layer from 3 to 38. Values are final-epoch balanced accuracy on 52 held-out chains, averaged over 5 seeds (seed std at most 0.007), given as linear / MLP.

20 loudest per chain positions (lr / mlp):

  • Layer 19: 0.506 / 0.610
  • Layer 20 (middle): 0.500 / 0.595
  • Layer 27 (loudest, best): 0.507 / 0.626
  • Layer 35: 0.475 / 0.554

20 random positions (lr / mlp):

  • Layer 19: 0.351 / 0.390
  • Layer 20: 0.345 / 0.385
  • Layer 27: 0.345 / 0.395
  • Layer 35: 0.344 / 0.352

Two observations:

  • The loudest layer (27) is also the most decodable on loud positions. For linear probes, layer 19 is level with it.
  • Loud positions are more decodable than random ones at every layer; random positions stay near chance.

Given the layer sweep results, we revise our claims: Loudness finds a layer on the high-decodability plateau, not necessarily the best one, and for this signal it does not clearly beat the middle-layer heuristic. Position selection is the main contribution, and it does not depend on finding the right layer. We will rewrite the abstract, Section 4.1 and the discussion accordingly.

  1. J-Lens decodability prediction is not limited to tokens about to be verbalized: In previous experiments, we tried using the lens directly as a classifier, predicting whichever of UP, DOWN, LEFT and RIGHT it rates highest. In several attempts the lens ranked RIGHT first, so this J-Lens-based classifier's balanced accuracy was close to chance (around 0.25), whereas trained probes reach 0.80 to 0.86. We read this as a division of labour: the lens shows where the information lies, and a probe is needed to extract the details. If decodability was limited to the model's output tendencies, we would expect to see a stronger correlation between the J-Lens' output space and the model's own verbalized choice of direction.

Planned revisions:

  • A. Additional experiment: Direction probes with labels taken from the grid, not from the model's output: whether the goal lies to the left or to the right of the agent.
  • B. Additional preliminary results: We will incorporate the lens-based classification argument from point 3 into the paper. We hope this, together with (A), will address the concern about the decodability prediction being limited to the model's output tendencies.
  • C. Vocabulary sensitivity: we will recompute the loudness metric with subsets of the vocabulary and report the deviations from the full-vocabulary metric.
  • D. Layer-optimality claim: We soften this claim, and will add the full per-layer tables for both models and both position sets in the revised paper.
  • E. Cost comparison: we will provide an estimate for the cost of the J-Lens loudness method against a probe sweep.

We hope these results and the planned revisions address the main concerns, and we welcome the reviewers’ views on whether further changes are needed.

AuthorsOpenReview ↗

Additional experiments and PDF update

We thank the reviewers again. We have uploaded a revised PDF that implements every revision proposed (A to E) in our earlier official comment. All new results are in the paper; below we summarize what changed and where to find it.

Corrections to our earlier comment. (1) The Qwen3.6-35B-A3B layer sweep used 3 seeds, not 5, and its largest seed s.d. is 0.0135, not 0.007. The GPT-OSS-20B figures (5 seeds, s.d. ≤ 0.008) were correct. (2) We had described the J-Lens classifier as mostly predicting RIGHT. That was indeed the case during our preliminary experiments, but now having re-ran those experiments with the most up-to-date setup, we find that the predictions are dominated by DOWN at the middle layers (73–88% of predictions in GPT-OSS-20B layers 11–16) and by UP at early layers. Neither conclusion is changed, but we apologise for both errors.

A. Labels from the environment (new Section 4.5, Figure 6; Appendix Figures 13 and 16). A second probing task for the direction signal: is the goal to the left or the right of the agent? The labels come from the grid coordinates, not from the model's output. The results mirror the direction probes. Loud positions beat random ones at every layer in both models. The trend survives removing direction words, and accuracy is flat across the grid-signal loudness. Caveat, stated in the paper: the goal side correlates with LEFT/RIGHT moves, so this task removes the dependence on the model's output but does not fully separate the goal from the plan. See figure 16 in the appendix (very bottom of the document)

B. The J-Lens as a classifier (Section 4.1; Appendix C.9, Figure 14). We use it as a label-free baseline: the argmax of the J-Lens over the four action tokens, with no probe. At the loudest layers it is close to chance, whereas trained probes on the same positions reach much higher accuracies. The lens indicates where direction information is present, but a probe is needed to extract it.

C. Vocabulary sensitivity (Section 4.1; Appendix C.10, Figure 15). We removed a random 20% of the GPT-OSS-20B direction vocabulary (from 446 tokens to 35, five separate draws with different random seeds) and recomputed the layer profile. The argmax stays at layer 15 and the runner-up at layer 13 in every draw. The loudness metric remains consistent for the direction signal.

D. Layer sweep and narrowed claims (Section 4.1, Figure 3; full tables in Appendix C.8, Tables 7–8). Direction probes at every layer, on loud and random positions. The loudest layer lies on the high-decodability plateau: it is the best layer for Qwen3.6-35B-A3B (.507 linear / .626 MLP), and for GPT-OSS-20B it is 0.016–0.017 below the best layer (layer 17). For this signal, loudness does not clearly beat the middle-layer heuristic. This was now added in the abstract, the introduction, Section 4.1 and conclusion. Positions matter more than layers: loud positions are more decodable than random ones at every layer, even though they were ranked based on loudness measured at one layer.

E. Cost (Discussion, "Practical use and cost"). We compare the processes rather than wall-clock times, which depend heavily on infrastructure:

  • Fitting a J-Lens is the most expensive step. It is done once per model, needs no labels, is reused across signals, and is unnecessary when a fitted lens is already available.
  • Given a lens, ranking positions takes one forward pass plus one lens readout per token. That is comparable to applying a probe at every token, with no labels and no probe training.
  • A sweep trains a probe per candidate layer (and per position) and needs a label at every candidate. For local belief, each label is a separate forced generation.
  • With a fixed middle layer, both approaches train one probe. The difference is at inference: loudness comes from the same forward pass as the probe and, since accuracy rises with loudness, it indicates how far each per-token probe output can be trusted.

Other changes. The limitations paragraph is rewritten to cover the fact that the goal-side probes use labels which are equivallent at times with the forced verbalizations of the model. It is also extended with mentions of the single environment, the vocabulary ablation being limited to one model, and the single held-out dataset per sweep.

We would be grateful to hear whether these revisions address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.