- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
State-of-the-art vocabulary lenses can predict and verbalize what an intermediate state represents. However, they do not answer which of the many states available for a given concept will be actually routed towards a desired recipient position. For example, in the sentence “The capital of France is BLANK_A. The capital of China is BLANK_B”, lenses can read many states associated with the “source” concept France, but what flows from France to the “recipient” BLANK_A should differ from what flows to BLANK_B, even though the France state is the same. To tackle this problem the authors propose a Contextual Cross-Jacobian (CCJ) lens. CCJ computes a map for a given prompt and layer that depends on both the source position and the target recipient.
Specifically, the map measures, for a prompt X, how much the residual in the hidden space of the last layer L in the recipient position (q) changes, when small changes to the residual at layer are applied to the position of the source (p). This map is in practice a Jacobian and can be used to obtain a transport vector from the source state to the target recipient: . Finally, this transport is passed to the final normalization layer and then to the unembedding of the model , to obtain . Sorting gives a vocabulary ranking for the tuple .
On two diffusion models, Dream-7B and LLaDA-8B, with concept-disjoint layer selection over all layers, CCJ beats globally averaged J- and R-lens baselines (Global-J and Global-R) on target-blind multihop recovery. Further experiments, also show that changing only the recipient changes the readout (which supports the claim that the readout depends on the recipient), and that CCJ can rank a concept highly before the output does during denoising. Two causal tests are also reported. A test on an auto-regressive model (LLaMA-3.1-8B) also shows positive gains.
Critical strengths
- The idea to build lenses that are specific to a given source-recipient rout is interesting. By construction, no averaged lens can show this (and the results confirm it).
- Some of the evaluations are careful. For example, concepts are split into disjoint dev and test sets, all layers are searched for every method in the Dream/LLaDA ranking comparison, and results are repeated over 100 splits, and paired differences have clustered bootstrap CIs.
Critical weaknesses
The baselines used are built for a different question therefore might not be the best comparison for this specific task. While this comparison is useful to prove the authors’ point that conditioning helps, it does not show that CCJ is a good tool for reading what flows to a specific recipient. For that claim, the paper needs baselines built for the same question.
- Averaged lenses are not built to perform the task at hand. Evidence of this is also shown in some results (e.g., in the four-layer check Global-J is worse than the plain logits lens on Dream).
- Global-J builds one map per layer, it decodes the same contextual source state, but with a transport that ignores the prompt, the source and the recipient (while CCJ uses this info). See questions for some ideas about how to make this fairer.
The contributions link the readout to causal effects. However, as designed, neither test shows that CCJ's readout or its layer choice is causally meaningful: the finite-dose test checks the same derivative that defines CCJ-J, and the swap test shows that the source states matter, but not that CCJ picked the right layer or ranked the right words. In the terms of Herrmann & Levinstein ("Standards for belief representations in LLMs", Minds and Machines, 2025), CCJ shows that information is decodable along a route and that the recipient is locally sensitive to the source, but it does not show "use": i.e., that the model actually relies on the information CCJ decodes along this route to produce its answer.
- CCJ-J is the exact derivative, as the authors also note, so the finite-dose test is close to circular for CCJ-J. So for 1–10% perturbations it should predict the observed logit change almost by definition. Additionally, this test evaluates the logit change not the vocabulary ranking. So it does not show that the highly ranked words reflect what the model uses. A stronger test would check whether the concepts CCJ ranks highest are the ones that change most when the source is removed or replaced.
- The binding swap experiment shows a known effect (at least for auto-regressive models). But it does not show that CCJ picked the correct layer. A better test would be to check the swap at a random layer, the Global-J-selected layer, or a sweep over layers. At early layers, swapping states is close to swapping the input words, so success is expected. The test would support CCJ only if its chosen layer succeeds where others fail.
- The layer used in the causal tests is not reported. The linear approximation should get worse the earlier the layer, because more nonlinear layers remain between source and output. Neither the finite-dose test nor the swap test says which layer it uses. If the layer is close to the output that could explain the high correlation.
The way some results are presented could be misleading (and some key comparisons lack error bars).
- Under the norm-based rule the global lenses score close to zero, but with the fixed-last-token source they recover substantially (at least on Dream). When each method is compared at its own best source rule, CCJ still wins, but the gap is much smaller than Table 1 suggests, especially on Dream. The size of the advantage reported in Table 1 therefore depends heavily on the source rule, and each method should be compared at its best-performing setting.
- The absolute recovery is modest: test MRR is about 0.07–0.09 and P@10 9.7–14.5%, although this is above the global lens, the gap to oracle MRR is large, which points to layer choice as the bottleneck (see below).
- During denoising, the model's own output logits beat CCJ on MRR-AUC. The threshold for a concept counting as “weak” in the current output (a hidden-reveal event) is not defined, and the number of such events is not reported.
The algorithm might not be practical: picking the layer requires labelled development concepts, and it is unclear how to deal with targets (and even sources) that span multiple tokens (either because they are longer words or because the concept is spread across the sentence).
- The unstable behavior of the profiles (Fig. 5) shows that the layer choice is the main bottleneck. The paper selects the layer by development-set MRR, which needs labelled concepts for the task, and the authors note that some selected layers remain split-sensitive. In practice, a user analyzing a new prompt has no label-free way to pick the layer.
- CCJ is clearly defined for one source position. When no source is given (or I guess when the source is spread across multiple tokens of a single word or across the whole sentence), the token with the largest transported norm is picked. The authors call this a heuristic that “does not guarantee semantic relevance”. The only check is the fixed-last control which shows that the heuristic helps CCJ in some cases and hurts the global lens. The paper never reports which tokens are picked, or whether they are the meaningful ones. In Fig. 3a the source that reads out “China” is the function word “in”, and the paper does not say how that source was chosen.
- Some ideas for stronger tests are suggested in the questions.
Presentation. Some information is missing (in addition to what is already mentioned above).
- “Mean-state CCJ-R” is never defined.
- The recipient-control sample size is not given.
- The authors claim the method is generic (i.e., works for diffusion as well as autoregressive models) but the autoregressive test is weak (it uses the four-layer grid instead of all layers, and has no recipient-control, finite-dose or swap results).
- The authors claim that CCJ can “reveal” hidden concepts but “hidden” is currently measured only against the final output.
Suggestions (not needed to increase my score but would strengthen the paper)
- I find that the theory section adds little. As the authors say, it gives no rank guarantees. The preregistered heterogeneity measure even correlates negatively with the ranking gain so the theory does not seem to explain the gains…. That space could be used for more thorough evaluation?
- Run some tests with sources that have multiple-tokens (you could test first, sum, last-token).
- What happens with concepts that are spread out? You could test prompts where the concept is only implied by several words (e.g. “the city with the Eiffel Tower and the Louvre is [MASK]” —> Paris). You could compare the best single source with the sum over all positions.
- Causal effect: ablate or patch the selected position and measure the drop in the recipient's target logit. Is the norm-picked position also the one with the largest causal effect?
Questions for authors
First the questions that could move my score.
-
Could you report Table 1:
- with each method using its own best source rule, chosen on development data only?
- using the same bootstrap technique used for Table 6 also for the fixed-last control (Table 5) and for this best-rule comparison?
- the standard deviation of the paired difference across the 100 concept-disjoint splits, not only the mean?
- Global-J/R fitted on at least three independent corpus samples and mask seeds, to show that their low scores are stable. Is CCJ still clearly better, especially Dream CCJ-R vs. Global-R?
-
Can you add stronger baselines to table 1? Here some ideas
- As explained above CCJ has access to specific info about the prompt while global-J does not. You could build something intermediate that has access at least to where the source and recipient are: e.g., one map per layer and singed offset . This can tell you if this specific knowledge makes a difference.
- A learned cross-position transport without derivative: a linear map from , fitted by regression on generic text (one per layer, optionally per offset p − q), then decoded with . This is also an ablation study because it tests whether the per-example derivative is really needed.
- An appended-question readout, in the style of Chen et al. (2024, “Designing a Dashboard for Transparency and Control of Conversational AI”), who append “I think the {attribute} of this user is” and read that position with trained probes. The paper already appends “The implied concept is [MASK]” for association, but the recipient’s own prediction is reported only in the denoising analysis, not in Tables 1/3. For multihop asking explicitly is not tested.
- Patchscopes (Ghandeharioun et al., 2024). This is probably the closest algorithm to CCJ. It copies into a separate “inspection” prompt and lets the model decode it. It is training-free and full-vocabulary like CCJ.
-
On the source token selection
- On items where there is a known “source” (e.g., the country/function swap grid, or multihop items with a known bridge entity), how often does the norm rule pick a token inside that span? Compare with random, last-token, attention-max and gradient x input selection.
- Report all methods with the gold source span given, so that the effect of the selection rule can be separated from the effect of the lens.
- Which token types does the norm rule pick? Please add a norm-matched control, to show that the rule finds meaning and not just large-norm positions.
-
Can you add a casual test that shows that the concepts CCJ ranks highest are the ones that change most when the source is removed or replaced?
-
Can you add the swap results for a random layer, the Global-J-selected layer, or a sweep over layers, and the CCJ-selected layer?
The following are also interesting questions to better understand or strengthen the work but not strictly needed to increase the evaluation.
- How exactly is C averaged in the bidirectional models (all p, q pairs? masked recipients only?)?
- Why is Global-J worse than the logit lens on Dream? Please add the logit lens to Table 1.
- Which layer do the finite-dose and swap tests use?
- Which direction is written in the finite-dose test?
- Can you report r and the slope per dose and per layer (early, middle, late), instead of pooled over doses and signs?
- If the authors want to keep the claim about revealing “hidden” concepts, then they should show that a logit lens and a tuned lens at the recipient position's own intermediate layers cannot reveal those concepts.