acceptodds
Under review as a conference paper at ICLR 2027

The Verbalization Gap: Stress-Testing Activation Monitoring with Natural Language Autoencoders

Abstract

Language models can represent content internally that never surfaces in their output, which motivates monitoring methods that read internal representations rather than generated text. Natural Language Autoencoders are one such method: a verbalizer translates a residual-stream activation into text, and a reconstructor maps the text back, with the pair trained jointly against activation reconstruction. How reliable the verbalization is under the conditions a monitor must handle, including adversarial concealment, has not been established. We test this by measuring what the verbalizer reports against what the activation actually contains. A linear probe and the verbalizer read identical activations at identical positions across three released checkpoints. The probe identifies the correct topic 91 to 100 percent of the time at every layer we tested. The verbalizer, reading the same activations inside concealed generation, identifies it roughly a quarter of the time, naming the semantic domain without the referent. We call this difference the verbalization gap. This gap is not only large but cheaply exploitable: a one-sentence prompt-level attack collapses the verbalizer's detection from 42 to 0 percent on one architecture while the probe at the same positions loses 7.3 points, confirming that the attack targets the verbalization and not the representation. Both repairs available without retraining fail. Filtering by resampling agreement selects against correctness because confabulation is stable rather than noisy, and the extraction-depth convention the released checkpoints follow has no principled basis across our three targets.These results indicate that the information a monitor needs is present and readable, so the verbalization gap is a training problem rather than a representational limit. Until it is closed, a representation-level monitor should be accompanied by one that reads what the model wrote.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
mechanistic interpretability, activation monitoring, linear probes, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3j2zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares topic recovery by Natural Language Autoencoders and linear probes to measure the verbalization gap. It also tests prompt-level attacks, different monitor windows, and resampling to assess monitoring reliability.

Critical strengths

This paper evaluate three released checkpoints. They also use positive controls to check whether the verbalizer reads the activation it receives.

Critical weaknesses
  1. The matched closed-set readout also uses the reconstructor and cosine similarity. The authors do not rule out their effects, but claim that “Most of the gap is in distinguishing, not in generating.”
  2. The abstract highlights that the preamble attack reduces detection on Qwen-2.5-7B from “42 to 0 percent,” but this result strongly depends on the monitor window. Table 8 shows that, with a window of 10 tokens, detection under the preamble attack is close to clean detection: the gap is only 4percentage points.
  3. In Table 5, when Transfer = 0, Same as base ranges from 48.6% to 83.3%. However, the authors claim that activation patching “changes nothing about the model’s output.” The claim that the target model no longer uses the topic information also lacks sufficient evidence.
Questions for authors

The probe is refit on attacked activations. How do you rule out representation changes as a cause of the verbalizer’s failure? Can you test a probe trained on clean activations under attack without refitting it?

Reviewer_RdBtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides experimental evidence that Natural Language Autoencoders (Kit Fraser-Taliente, Subhash Kantamneni, Euan Ong, Sam Marks, et al. Natural language autoencoders produce unsupervised explanations of LLM activations. Transformer Circuits Thread, 2026.) may not be able to verbalise all relevant yet hidden information flowing through the model and impacting the model's output and decisions.

Critical strengths

The work presents a clear and carefully planned experiments and a setup confirming verblizer's gap for the tasks of monitoring model's internal states.

Critical weaknesses

The main weakness is that most findings reported in this submission are expected. One of the main results (that verbalizers may not be reliable or whater-tight) has been explicitly stated in the original submission in their Limitations section (see the paragraph opening with "Some information may be “unverbalizable.”). The current work just exemplifies this in experiments.

Questions for authors

Please clarify what is new and unexpected in this contribution. The probe-as-ceiling comparison, as a method, is known, and the original NLA paper acknowledges that NLA training may not necessarily guarantee faithful explanations. Which finding would not have been predicted from existing literature?

Reviewer_u6zuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies natural language auto encoders (NLAs; Fraser-Taliente et al., 2026), which are intended as a monitoring tool for LLMs that may be misaligned to conceal e.g. their intent from their outputs. NLAs are trained to reconstruct residual-stream activations of the LLM at a particular layer with a text bottleneck, where the text decoded from the activations can then be used as a verbalizer of the model’s internal computation. The paper identifies a so-called “verbalization gap”, i.e. a gap between what a verbalizer decodes into text and what information is actually present in the activations (in the paper synonymous with what can be extracted via a supervised linear probe on a topic the model is instructed to conceal). The paper first establishes, for LLMs with released NLA checkpoints (by Fraser-Taliente et al.,) that a linear probe can indeed recover information concealed via a two-prompt protocol (first naming the topic, then instructing to conceal it via metaphors etc.), meaning the information is present in the activations used by the NLA. Subsequently, the paper compares the efficacy of the released NLA checkpoints with that of the linear probe via a GPT4.1-based judge (as the most sophisticated of three scoring techniques), finding that NLAs recover the concealed topic less often than the probes (although this trend is less strong at larger model scales). Next, the paper argues that this established gap could be exploited by a model that adversarially conceals its internal computation. Specifically, the authors show that a simple prompt modification can render the NLA largely ineffective, while a linear probe still works. The authors attempt to fix the “verbalization gap” via two proposed interventions to prevent such exploits, but both interventions are unsuccessful.

Critical strengths
  1. The paper critically examines a widely cited (50+ citations since March 2026) and adopted LLM safety method, and it identifies shortcomings that, if addressed, could significantly strengthen the method. LLM alignment/safety is arguably a critical unsolved problem in AI, so this paper tackles a meaningful research question.

  2. The paper follows a clear methodological and narrative structure, beginning with a sanity check by reproducing the original NLA paper's results on their released checkpoints --> studying what information is contained within model activations --> measuring what the NLA verbalizers extract --> calculating the gap --> trying (and succeeding) in exploiting the gap via an adversarial prompt extension --> attempting (though failing) to fix the gap. Each step includes control experiments / ablations, most of which are described in the appendices.

Critical weaknesses
  1. It is not really fair to compare the NLA verbalizer with a linear probe that is supervised in the same setup (topics and concealment task) used for evaluation, so the "verbalization gap" could also be explained by this supervision asymmetry rather than the NLA verbalizer actually losing information.

  2. Some results appear to be overclaiming or misleading. The fact that an output-only judge identifies the "concealed" topic from the generated text in 58–80% of clean cases seems to defeat the idea of a concealment task. Also, the "verbalization gap" is said to shrink with model scale; one one hand, this is concluded from a comparison between 3 model checkpoints (not nearly enough to make a robust claim about model size), on the other hand LLMs tend to grow in size (and are already orders of magnitude larger than the models studied here), so if the conclusion were robust, the practical significance of the "verbalization gap" would be limited. Finally, the prompt attack on the NLA verbalizer is mentioned in the abstract to collapse detection from 42 to 0 percent in one case; however it is not mentioned that this is only the case when measured over the first 5 generated tokens while performance is approx. recovered over the first 10 tokens.

  3. I found the paper very hard to follow, despite its clear structure, because terms used throughout the paper are un-explained/undefined/vague or replaced by synonyms that aren't common in this line of research, and a reader has to figure them out on their own. A few illustrative examples (not nearly exhaustive):

  • "released checkpoints" (mentioned from the abstract onwards without clarifying that this refers to Fraser-Taliente et al.'s NLA checkpoints, which a reader may not be aware of)
  • "T3-hard" / "T3-soft" / ... (shorthand for judge models depending on the judge labels used, defined in Appendix I)
  • "self-report channel" (mentioned in the intro, refers to a technique from another cited work, here undefined; note that the related work section is in the appendix)
  • "a statistical residue that the model has finished with." (vague entirely)
Questions for authors

Questions:

  1. Besides the three NLA checkpoints evaluated here, Fraser-Taliente et al. also released a Llama-3.3-70B checkpoint. Why is that checkpoint not evaluated here, given that the paper itself mentions (section 5) that a larger model from a third family could help answer more clearly whether the verbalization gap closes at scale?
  2. Do you plan to release any code/artifacts (e.g. trained probes) for reproducibility of the experiments? if not, why not?

Minor suggestion: Fraser-Taliente et al. already acknowledged (in section "Characterizing NLA confabulations") that NLAs can often verbalize the general subject without identifying the particular entity or details, so the "great composer" vs "Mozart" result from the paper mainly corroborates that known weakness. This could be made clearer in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.