acceptodds
Under review as a conference paper at ICLR 2027

Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form

Abstract

Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly. What causes a representation to enter that form is open, and the word workspace invites an admission story: a gate that decides what gets in. Testing it on open-weight models with Jacobian lenses, over a benchmark whose five arms share an identical context, we find no gate where it predicts one. Demand raises a concept's lens visibility beyond what applying an operator to a supplied value produces: +0.050 [+0.045, +0.057] in percentile rank on our primary checkpoint, positive on all four we measure, though that arm answers at ceiling and the accuracy-matched contrast is stronger under that readout. At the same time one shared linear map decodes the variable from every arm, the control included, at 6.4-9.0x its selection-corrected floor. What produces the later readable form at the queried position is attention-mediated gathering inside a mid-depth window: separating patch depth from readout depth puts transport there at least 17x above anywhere shallower under non-saturating readouts, with no tested MLP output contributing positively inside it. Under the saturating percentile rank the same grid does not localise the window, which is a fact about that measure. An arm that needs the variable for nothing concentrates sevenfold less, so the window is demand-specific. That window has two measured edges, a survival failure below and destruction above, and it falls at the same fractional depth in a 64-layer hybrid and a 62-layer dense model from another family. We localise where the variable is installed and read, not the route from the passage, which transports nothing. But the readout is not a calibrated measure of use: three components move it to within 12% of one another and differ 7.4x in what they do to the answer.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, language models, verbalizable representations, global workspace, Jacobian lens, activation patching, attention, latent variables, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yZ1XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how task demand makes a latent variable readable at the query position. Five task conditions share a context, and the analysis combines Jacobian lenses, linear probes, activation patching, ablations, and direction removal. Demand increases lens visibility, while donor-substitutable content concentrates at intermediate depth. Similar readout changes also produce substantially different behavioral effects. The authors interpret these findings as attention-mediated gathering and contrast them with their own formalization of admission at the query position.

Critical strengths
  1. The regularizer is straightforward to integrate. Its weights remain fixed during residual optimization. Top-k selection limits historical gradient evaluations, and Appendix D.6 correctly acknowledges that key scanning and storage still grow with the history.
  2. The experiments include useful comparisons. Tables 1–2 cover three models and two editing datasets, including AlphaEdit and HSE. Tables 4 and 6 examine selection strategies, response signals, and weighting functions. The reported LLaMA CounterFact generalization improvement from AlphaEdit's 78.15 to 93.95 would be meaningful if verified.
Critical weaknesses

For me, the primary issue with this paper is its lack of readability. Starting from the abstract, the text introduces technical terms without any explanation. Furthermore, while a significant portion of the abstract is dedicated to presenting empirical results, it offers very little insight into the implications of those findings. I do not believe this is an effective way to present research; moreover, the paper reads more like an empirical report than a formal academic paper.

To be honest, after reading the introduction, it seems to me that the phenomenon discussed is similar to the "attention sink" effect. However, the research relies heavily on "probing" methods—a technique used in interpretability studies—to examine the inconsistencies and commonalities among various existing probes. In my view, this does not constitute a substantive contribution.

Questions for authors

See Weakness

Reviewer_T5LbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why a task-relevant latent variable becomes more visible through a Jacobian lens. JGateBench varies instructions over shared contexts, principally language identification followed by a randomized lookup. Across several checkpoints, demand increases visibility even when a shared probe decodes the variable from control activations. Varying patch and readout depths localizes especially strong donor-substitutable content around mid-depth on Qwen3.6-27B; a behavioral window also appears on Gemma-4. In that window, attention patches shift the lens more than MLP patches do. Concept-direction removal changes counterfactual answers, while similar lens shifts across components produce very different behavioral effects.

Critical strengths

Separating patch depth from readout depth is a useful advance over a single late readout: it exposes survival as a confound and makes the concentration of substitutable content much more convincing. The same-context benchmark, label-symmetry check, held-out confirmation and reseeded donor pairings provide informative controls. Relative to Gurnee et al. (2026), the added value is this demand-conditioned intervention analysis rather than the existence of workspace-like representations or relay heads.

The matched-trial comparison in Figure 5 is also consequential: comparable lens shifts produce substantially different answer-margin changes. Together with the concept-direction controls, this shows why a readable concept can be behaviorally relevant without making readout magnitude a calibrated causal measure. These methodological findings could improve how subsequent workspace and patching studies interpret their measurements.

Critical weaknesses

The evidence localizes a substitutable representation more convincingly than its proposed origin. Appendix H patches attention after its query-dependent output gate, mixing incoming values with local gating; this does not distinguish gathering from making already local information readable. Short-distance readouts reduce survival confounding but cannot exclude an earlier, differently encoded representation. Neither the unsuccessful passage-tail patch nor the observational attention-mass analysis resolves this. The rejection of query-position admission therefore exceeds the demonstrated causal contrast.

The necessity claim also needs narrowing: Table 6 establishes selective dependence on resid.L39 under mean ablation, while attention is significant only under zero ablation. This supports some attention dependence, but does not identify the proposed gather as the necessary mechanism.

Questions for authors

Can you separate the weighted-value contribution from the attention output gate and the query position's own value at L39, or provide an equivalent intervention showing that the demand effect requires information from another position? The distinction would determine whether the result identifies transport or local re-expression. Given the four-position result in Appendix C, can the relevant behavioral and necessity comparisons be made under that less destructive span?

Reviewer_HZgLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is doing something that is not clearly defined in the abstract or the introduction.

Critical strengths

N/A

Critical weaknesses

The paper is AI slop. I have tried to read the abstract and introduction 4 times, but I still don't know what the paper is talking about. I will fight to reject the paper.

Questions for authors

Please rewrite the paper properly for human readers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.