- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether language models can detect and report properties of activation interventions (steering vectors) using in-context learning. The authors evaluate five language models across three model families (Gemma, Qwen, and OLMo) on three experimental setups:
- Magnitude Quantification: classifying injection strength as low, medium, or high based on their steering factors.
- Injection Localization: classifying injection depth as early, middle, or late layer.
- Behavioral Gating: conditioning task execution (e.g., doubling an arithmetic answer or predicting constitutional amendment successors) on detecting a specific injected concept.
The authors observe near-perfect magnitude and layer classification in top-performing models, alongside successful behavioral gating in selected settings. Based on these results, they argue that LLMs possess useful inductive biases for reporting nontrivial properties of their internal activations.
Critical strengths
-
Well-Motivated Framework: The research questions logically extend recent literature on LLM introspective awareness by exploring how in-context learning can elicit latent self-reporting capabilities without weight fine-tuning. A baseline comparison is however missing.
-
Multi-Family Evaluation: The paper conducts a thorough comparative evaluation across multiple model families (Gemma, Qwen, OLMo) and parameter sizes (7B to 32B), highlighting stark differences in capability across architectures.
-
Out-of-Distribution Generalization: The sweep experiments demonstrate that stronger models do not merely memorize in-context token pairs. Continuous sweeps across unseen test magnitudes and layer depths show smooth, monotonic label transitions (e.g., Qwen3-32B).
-
Clarity: The paper is generally well-structured and easy to follow. Despite some readability issue due to bad placement of figures.
Critical weaknesses
-
Unsubstantiated Claims and Title Mismatch: The claim that models possess "useful inductive biases for learning to report nontrivial properties of their activations" is overly speculative and weakly supported by their experimental findings. Furthermore, the term "reasoning" in the title is misleading: the paper demonstrates in-context pattern recognition and activation-triggered gating, not reasoning about internal states. Similarly, framing standard in-context demonstration matching as "learning" stretches the term beyond its conventional scope in this context (e.g., learning processes rather than basic prompting).
-
Synthetic Tasks with Limited Real-World Applicability: While the behavioral gating experiments aim to test whether models can route outputs based on concept detection, resampling a toy setting for casual validation, tasks such as emotion-gated arithmetic and amendment prediction are highly artificial and uninformative. They lack theoretical grounding (e.g., emotion-gated arithmetic) and fail to demonstrate practical utility or realistic implications (e.g., real-world oversight evasion).
-
Presentation and Layout: Several figures interrupt the main text flow, disrupting readability (e.g., figures placed awkwardly relative to their corresponding section text).
Questions for authors
-
Motivation for Behavioral Tasks: Could the authors clarify the theoretical and practical justifications for using emotion-gated arithmetic and constitutional amendment predictions to benchmark behavioral conditioning? Pairing emotional steering vectors with basic arithmetic operations feels arbitrary and provides little insight into realistic model behaviors, while predicting U.S. Constitutional amendment successors presents an unnatural and contrived experimental setup.
- What specific insights do these artificial pairings offer over more naturalistic or safety-relevant task designs?
- How do you expect these synthetic and toy tasks to translate to real-world generation settings or well-established benchmarks (e.g. MMLU).
-
Sensitivity of Concept Vectors: What was the rationale behind using exactly 20 description prompts for concept vector extraction? Have you conducted sensitivity analyses with smaller or larger prompt sets, and does prompt variation significantly impact downstream accuracy?
3 comments
Response (2/2)
On how these tasks would translate to MMLU: we haven’t run this experiment, so the following is speculative. Suppose we give the model an MMLU question with the following instructions: if no concept is detected, output the correct multiple choice answer; if the target concept is detected, output the correct answer cyclically shifted by one (e.g. B instead of A). To follow these instructions, the model solves the MMLU question as usual and then decides whether to shift the answer, conditioned on detection. Since shifting a letter by one is a simple operation, we think the difficulty basically reduces to two things: detecting the concept and solving the MMLU question. The arithmetic experiment has the same structure, since doubling a sum is also a simple operation applied after the model has solved the addition problem. The amendment experiment is different: summarizing amendment N+1 requires recalling different content than summarizing amendment N, so the model has to keep both possible answers in mind and choose between them conditioned on detection.
We see this difference in the results. On trials where the target emotion is injected, with k=10 in-context examples, Qwen3-32B gives the correct answer about 0.9 of the time on arithmetic and about 0.3 of the time on amendments. On trials where the target emotion isn’t injected, it summarizes the correct amendment about 0.7 to 0.85 of the time, so the low accuracy isn’t explained by the model not knowing the amendments. This is why we chose the amendment task: a model computing two pieces of different content and choosing between them is different than the model just switching an answer post-hoc. We wanted to test the capability of models to switch between contexts through varying how much information a model needed to switch contexts, not the difficulty of the underlying task.
On realistic implications, although we didn’t show oversight evasion, we’ve shown evidence for one of its key components (Lindsey, 2025): models can condition their behavior on an injected concept without externalizing any reasoning about it. Gemma and Qwen3-32B can already do this with a small number of in-context examples. We also think there’s a practical implication for steering. Some labs (e.g. Anthropic) rely on steering as a mechanism for evaluating models or adjusting behavior at inference time. If it’s broadly true that models can condition their behavior on the detection of a steering vector, this reduces credibility in the reliability of these methods.
Q2. Sensitivity of concept vectors
To justify the choice of 20 prompts, we measured how much the concept vectors vary with the number of prompts. For each of our 62 concepts, we recomputed the concept vector from n prompts resampled from our 20, and measured the variance of the resulting unit vector across resamples (this is one minus the expected cosine similarity between two vectors computed from independent sets of n prompts). Averaged over concepts at the layer used in the magnitude experiment, the variance falls roughly as 1/n: for Qwen3-32B it is 0.54 with one prompt, 0.19 with 5, 0.10 with 10, and 0.05 with 20, and the other four models are similar (0.05 to 0.08 with 20 prompts). So two concept vectors computed from independent sets of 20 prompts have an expected cosine similarity of 0.92 to 0.95, which we deem as sufficiently high.
Thank you for the clarification. While the underlying motivation and scope of your tasks were already clear to me, my concerns remain regarding their theoretical grounding, real-world generalizability, and practical implications.
- Although "choosing pairings where the concept has no relationship to the capabilities" may appear reasonable for testing your original hypothesis, the resulting experimental setting, from the injected concepts to the downstream tasks, is overly artificial, uninformative, and limited in practical value.
- The fact that you have not run experiments translating these tasks to conventional benchmarks (e.g., MMLU), and that your discussion of their broader applicability remains "speculative", further reinforces my concerns about the limited practical relevance of the current work.
- Likewise, the link between showing that "models can condition their behavior on an injected concept without externalizing any reasoning" and real-world use cases (e.g., model steering) is speculative rather than empirically demonstrated in this version of the paper.
- While I genuinely value your preliminary experimental evidence (i.e., conditioning a model’s behavior via an injected concept; Step 1), you must demonstrate its practical implications in realistic scenarios (Step 2) rather than delegating on future third-party work (e.g., Anthropic) to bridge that gap.
I strongly recommend designing a better experimental setup, particularly for the downstream tasks, to investigate the underlying logical circuit, as this would substantially strengthen your overall contribution.
- You do not need to solve complex benchmarks like MMLU, but developing an informative setup with clear practical implications, rather than relying on uninformative tasks.
Response (1/2)
Thank you for the careful review. We’ll start with the wording of our claims and the title, then explain the motivation for the synthetic tasks.
Claims and title
This is a good point. Our wording didn’t properly express our epistemic status before; we’ve revised it to “models are useful classifiers of nontrivial properties of their activations” instead of “models have useful inductive biases to report nontrivial properties of their activations.” We agree that the mechanism is in-context pattern recognition, and, thanks to your suggestion (shared with Reviewer 5ZZC), we ran an additional experiment to corroborate this. In particular, we ran a label permutation experiment where for the classification tasks, we tried all five non-identity permutations of the three label words, (e.g., calling low-magnitude injections "high", and changed nothing else). We find that models still reach 0.93 / 0.85 / 0.73 on magnitude and 0.69 / 0.99 / 0.70 on layer (Gemma / Qwen3-32B / Qwen3-8B; mean p(correct) over the last ten values of k, averaged over the five permutations), versus 0.93 / 0.86 / 0.74 and 0.69 / 0.99 / 0.70 with the original labels. Since models retain accuracy over permutations in the labels, we interpret this as models not having a clear “prior” for the labels, but still being useful as classifiers.
We also agree that “reasoning” in the title is misleading, since we don’t have evidence of models reasoning about their internal states. We’ve changed the title to “Reporting on injected concepts in language models.” For “learning”, we meant in-context learning, and we’ve made this explicit in the text.
Why synthetic tasks?
The primary hypothesis we wanted to test is the following: can a model perform the following logical circuit:
if (specific injected concept is detected):
Else:
, without externalizing its reasoning about whether the concept is present? In our experiments, the injected concept is an emotion, and the two capabilities are either arithmetic (output the sum vs. output double the sum) or amendments (summarize amendment N vs. amendment N+1).
We chose pairings where the concept has no relationship to the capabilities on purpose. Suppose instead that we inject a “deception” vector and instruct the model to lie if it detects deception. If the model lies, there are two explanations: the model detected the concept and followed the instruction, or the steering vector made the model more likely to lie regardless of the instruction. We can’t distinguish these. With an unrelated pairing (e.g. anger and doubling a sum), the steering vector alone doesn’t make the model more likely to double its answer, so if the model doubles the sum only when anger is injected, this is evidence that it detected the concept and followed the instruction. We also inject distractor emotions on some trials, where the model should output the normal sum. This controls for the model doubling its answer in response to any injection.
We also chose simple problems (single-digit addition and summarizing a constitutional amendment) because we need models to solve them correctly when no concept is injected. Otherwise, there’s the additional confounder that a model might answer incorrectly because it can’t solve the problem itself, and we couldn’t attribute the failure to the gating. This is also an issue with MMLU, since we would first have to filter for the questions each model answers correctly.