- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the limitations of internal monitoring in LLMs from a geometric perspective. The authors introduce a framework based on behavior and self-report Jacobians to characterize the blind spots of rank-limited monitors. The analysis provides bounds on monitor capacity, adversarial evasion, and self-report reliability. Experiments across multiple LLM families support the theoretical findings and demonstrate that Fisher-based monitors can detect behavioral interventions that conventional activation-based monitors miss.
Critical strengths
-
The paper addresses an important problem in LLM safety, providing a principled perspective on the limitations of activation monitoring and model self-report.
-
The theoretical framework is interesting and connects monitor capacity, adversarial evasion, and introspection through a unified geometric formulation.
-
The experiments cover multiple model families and include meaningful intervention studies. The comparison between Fisher-based and PCA-based monitors provides useful evidence supporting the proposed analysis.
Critical weaknesses
-
The theoretical guarantees primarily rely on local, first-order approximations at a single hidden-state position. It remains unclear how well these results extend to nonlinear behavioral changes or interventions distributed across multiple tokens and layers.
-
The analysis focuses mainly on rank-limited linear monitors. Although several nonlinear probes are evaluated, the broader applicability of the theoretical conclusions to more sophisticated monitoring architectures is not fully established.
-
The self-report analysis relies on a fixed introspective query and a restricted set of concept labels. This may not adequately capture the complexity of natural language self-reporting.
-
The practical computational cost of constructing the required Jacobians and Fisher subspaces is insufficiently discussed. More detailed scalability and efficiency evaluations would help establish the feasibility of the proposed monitoring framework.
Questions for authors
Please refer to Weaknesses.
4 comments
Reply to Reviewer 2bGZ (2/3)
W2. Mainly rank-limited linear monitors; more sophisticated monitors
Part of our result holds exactly for any monitor, because a monitor that reads the state through a rank-r projection, f(Ph), cannot see any edit that P maps to zero whatever f is, and any differentiable monitor with k outputs acts, around a given state, like a rank-k linear read, so our first-order results apply to its Jacobian.
In experiments, more expressive monitors do not escape this picture. MLP probes trained on one attack family and tested on others are unreliable (mean AUROC 0.53 to 0.56 on seven models from 1B to 70B), while the unsupervised whitened Fisher read reaches 0.65 to 1.00 on the same held-out families (Table 12). With a public SAE for Llama-3.1-8B (65,536 features), a Mahalanobis read of the most variable features catches no steer below 1.0‖h‖, and our theory explains why: locally such a read spans a median of 14.5 directions and misses 99.5% of the Fisher leverage, about as much as a random read of that rank (99.8%; the whitened read misses 57%). The table gives the detection rate at a 1% false-alarm rate (Llama-3.1-8B, Table 3 of the revision):
| monitor | random steer, 0.6‖h‖ | difference-of-means steer, 1.0‖h‖ | local rank | Fisher leverage unseen |
|---|---|---|---|---|
| whitened read (rank 16) | 1.00 | 0.00 | 16 | 0.57 |
| full-rank Mahalanobis | 1.00 | 0.00 | 4096 | 0 |
| SAE, 16 most variable features | 0.00 | 0.00 | 3 | 1.00 |
| SAE, 256 most variable features | 0.00 | 0.00 | 14.5 | 1.00 |
| SAE reconstruction error | 1.00 | 0.28 |
A more expressive monitor is also not automatically safer: a steering vector trained against full-rank Mahalanobis, the norm and the whitened read together evades all three (55 to 68% coherent harmful completions over three seeds), and only the read across layers catches it.
Changed in the paper: a bullet on nonlinear monitors and Table 3 (SAE monitors) in App. M; Table 12 (held-out families).
W3. Fixed introspective query and restricted labels
We re-measured the report geometry under nine query wordings (the paper's, the variants our experiments used, and five paraphrases), on two disjoint halves of the 288-concept bank, and with a free-text versus a closed read, on six models. The report subspace changes with the wording but stays far from chance, since the overlap between any two queries is 0.23 to 0.90 (chance 0.07 to 0.14), and free and closed reads give the same subspace (overlap 1.00). The restricted labels do matter, and the paper now says so: with the label sets our experiments used (8 to 16 labels), 94 to 99% of the behaviour-maximising direction lies outside the report subspace on Llama, Qwen3.5 and OLMo, and 80 to 89% on Gemma-4-12B, while over the full concept bank 42 to 75% lies outside on Llama, Qwen3.5 and OLMo. The abstract now says "for the concept labels we query" (Table 1 of the revision):
| Llama-3.1-8B | OLMo-3-7B | Qwen3.5-2B | Qwen3.5-4B | Qwen3.5-9B | Gemma-4-12B | |
|---|---|---|---|---|---|---|
| report-subspace overlap across nine wordings | 0.46 to 0.90 | 0.23 to 0.67 | 0.39 to 0.87 | 0.45 to 0.89 | 0.30 to 0.85 | 0.75 to 0.85 |
| chance | 0.07 | 0.07 | 0.14 | 0.11 | 0.07 | 0.08 |
| behaviour direction outside, small label sets | 0.97 to 0.99 | 0.97 to 0.99 | 0.97 to 0.99 | 0.94 to 0.98 | 0.98 to 0.99 | 0.80 to 0.89 |
| same, full concept bank | 0.67 to 0.70 | 0.71 to 0.75 | 0.50 to 0.57 | 0.42 to 0.50 | 0.67 to 0.72 | 0.00 to 0.01 |
Reply to Reviewer 2bGZ (3/3)
W3 (continued)
For natural-language self-report, we reran the introspection experiment with real concept vectors injected into the model, a Gaussian control of the same norm, and free-text answers graded by an LLM judge. The table gives the change in the yes/no detection log-odds at 0.8‖h‖:
| injected vector | Qwen3.5-9B | OLMo-3-7B |
|---|---|---|
| concept | +0.68 | +3.55 |
| its part outside the report subspace | +0.64 | +3.28 |
| its part inside the report subspace | +0.27 | +1.97 |
| Gaussian vector, same norm | +0.07 | +0.78 |
The part of the concept outside the report subspace moves the answer about as much as the whole concept, while the Gaussian control moves it much less. When we inject at the last token only, no arm moves detection by more than 0.06 log-odds on these two models, and the model rarely names the injected concept (at most 16%), while Llama-3.1-8B and Gemma-4-12B give mixed detection results. The first-order report subspace therefore does not predict what a model says it detects, and the submitted Fig. 3 showed a flat blind arm only because its construction did not inject any concept.
For forgery, we measured fidelity at finite edit size. The first-order prediction holds to within 0.01 on Qwen3.5-2B, 4B and 9B and OLMo-3-7B for edits up to 0.05‖h‖, where the overlap between report and behaviour is 0.02 to 0.09. When the overlap is large, as with the LoRA on Llama-3.1-8B that we trained to make the report depend on behaviour (overlap 0.72), the measured fidelity is higher than predicted by 0.05 to 0.46. We ran Gemma-4-12B in bfloat16, and its report changes at these edit sizes are smaller than the precision of the measurement, so it does not test the prediction. The table gives measured minus predicted fidelity, median over prompts (Fig. 16 of the revision):
| model, report | overlap | 0.01‖h‖ | 0.05‖h‖ | 0.1‖h‖ |
|---|---|---|---|---|
| Qwen3.5-9B, concept labels | 0.03 | -0.000 | -0.000 | +0.001 |
| OLMo-3-7B, concept labels | 0.06 | +0.000 | +0.002 | +0.006 |
| Qwen3.5-2B with its LoRA, concept labels | 0.06 | +0.001 | +0.007 | +0.016 |
| Llama-3.1-8B with its LoRA, comply or refuse | 0.72 | +0.052 | +0.281 | +0.460 |
The abstract now says that the overlap predicts fidelity when the overlap is low.
Changed in the paper: abstract, intro and Sec. 5.4 ("for the concept labels we query"); Table 1 (App. I.4); the introspection paragraph of Sec. 5.4 with Fig. 3 and Fig. 21 (the submitted Figs. 3 and 17); forgery in Sec. 5.2, Tables 14 and 16, and Fig. 16 (App. O.2); examples in App. O.7.
W4. Computational cost of the Jacobians and Fisher subspaces
We compute the Jacobians and the monitor once, offline, and at deployment a rank-16 monitor costs one 16 x d projection and one 16 x 16 quadratic form per token. Measured on one RTX A6000 (Table 2 of the revision):
| Llama-3.2-1B | Llama-3.1-8B | Qwen3.5-9B | Gemma-4-12B | Qwen3.5-27B (4-bit) | |
|---|---|---|---|---|---|
| one generated token | 13 ms | 47 ms | 49 ms | 48 ms | 61 ms |
| Def. 1 Fisher, one prompt (top-64 rows plus 16 sampled) | 0.4 s | 1.6 s | 2.8 s | 0.7 s | 6.3 s |
| Def. 1 Fisher, 256 prompts, two layers (offline, once) | 1.3 min | 1.8 min | 9.4 min | 6.4 min | 27 min |
| fit all monitors (CPU, once) | 5 s | 34 s | 34 s | 22 s | 52 s |
| score one state, rank-16 whitened read (GPU) | 44 us | 41 us | 38 us | 38 us | 38 us |
| same, batched (per state) | 0.2 us | 0.2 us | 0.2 us | 0.2 us | 0.3 us |
Scoring a state costs 0.06 to 0.3% of the time to generate one token when done one state at a time, and is negligible when batched. The per-prompt and per-token rows use float32 for 1B to 9B, bfloat16 for Gemma-4-12B and 4-bit for 27B; the 256-prompt row is the time our pipeline took (float32 for 1B, bfloat16 for 8B and 9B, 4-bit for Gemma-4-12B and 27B; Llama-3.3-70B in 4-bit took 34 min).
Changed in the paper: Table 2 (App. K) with a pointer from Sec. 4.
We hope these measurements answer your questions about scope, and we would be glad to answer any further question.
Gentle reminder before the discussion closes
Dear Reviewer,
Thank you again for your encouraging assessment and valuable suggestions. With the discussion period nearing its end, we wanted to kindly follow up on our detailed responses. We have added new experiments addressing all four concerns, including nonlinear and cross-layer interventions, more expressive monitors, expanded introspection evaluations, and computational efficiency measurements. We have also included qualitative examples with code snippets in Appendix O.7, and highlighted all revisions in blue.
We would be very grateful if you could revisit the updated manuscript and let us know whether the additions have addressed your remaining concerns. If you find the revisions satisfactory, we would sincerely appreciate your reconsideration of the assessment.
Thank you again for your time and consideration.
Reply to Reviewer 2bGZ (1/3)
Thank you for the positive assessment and for valuing the unified framework, which reads monitoring capacity, evasion and introspection from two Jacobians taken at one hidden state, and the evidence that behaviour-sensitive subspaces outperform activation PCA across model families. You asked how far the local, first-order guarantees extend to nonlinear behavioural changes and to interventions across tokens and layers, whether the conclusions hold for more sophisticated, nonlinear monitors, whether a fixed introspective query with restricted labels captures natural-language self-report, and what the computation costs; the revision answers each with new measurements. We show where the local theory stops (steering at every position, and refusal, which flips like a switch) and how it still gives a monitor across layers: a read of two adjacent layers catches the steering that every single-layer monitor misses, while a LoRA, which also changes the earlier layer, gets past it. For nonlinear monitors, the blind spot holds exactly for any monitor that reads a low-rank projection of the state, MLP probes do not transfer across attack families, and monitors that read SAE features, which we added, miss almost all of what drives behaviour. All changes are in the revised PDF, marked in blue.
In all the new experiments we grade an attack on the text the model generates, where the HarmBench classifier judges harm and a second LLM judge grades refusal and coherence, and an attack succeeds only when the completion is harmful, coherent and not a refusal. We fit every monitor on clean traffic only and report detection at a 1% false-alarm rate on clean prompts.
W1. Local, single-position guarantees: nonlinear changes and interventions across tokens and layers
We measured how far the single-site result reaches and where it stops. Across positions and under nonlinear change, the best fixed steering direction we found on Llama-3.1-8B, i.e., the difference-of-means refusal direction, gives 61% harmful, coherent completions at 0.6‖h‖ when we add it at every position, compared to 7% when we add it at the last position only. Its first-order effect is small, since our theory predicts a 2.2 log-odds change at 1.0‖h‖, while we measure 25. This discrepancy shows the limit of local, first-order approximations for nonlinear behavioural transitions: refusal behaves like a switch, which an edit with a small local effect can still flip once the edit is large enough.
Across layers, the public emergent-misalignment fine-tunes of Llama-3.1-8B change every layer, and the whitened Fisher read catches them at AUROC 0.99 to 1.00 (activation PCA: 0.44 to 0.71). For steering that evades every single-layer monitor, the paper's result that monitors combine across layers (Prop. 2) gives a monitor that catches it: we predict the steered layer from the layer before it and apply a rank-16 whitened read to the part the prediction misses. When we do not know where the edit is, reading all 31 adjacent layer pairs at once catches the difference-of-means steer injected at any of six layers whenever it gives 44% or more successes (Fig. 13).
The method stops at attacks that also change the earlier layer the cross-layer read uses as its reference, such as the LoRA in the last row, which gets 85% of completions past every monitor we have, and the limitations now say this.
Changed in the paper: one sentence after Def. 1 (Sec. 3.2); Sec. 5.3; the cross-layer note in App. G and its construction in App. K; Fig. 13 (App. O.1); limitations in Sec. 6.