acceptodds
Under review as a conference paper at ICLR 2027

Reasoning and learning about injected concepts in language models

Abstract

Language models can, on occasion, correctly answer questions about "injected concepts", i.e., steering vectors added to their activations. This capability, however, remains fragile across models and prompts, leaving open how far it generalizes. To investigate this, we test whether in-context examples help models (i) report properties of the injections and (ii) condition their responses on the identification of specific injected concepts. First, we query about the magnitude of the injection. Second, we ask models to classify whether an injection was applied at an early, middle, or late layer, across different injected concepts. Third, we ask models to gate their output conditioned on the detection of a particular concept, e.g., multiply the answer to an arithmetic problem by if, and only if, an "anger" injection is detected. We test five open-weight models, drawn from three families and two sizes. We find that some models achieve near-perfect performance on magnitude or layer classification and, notably, generalize to unseen magnitudes or layers. Some models also learn to condition their behavior on the identification of specific injected concepts, reaching near-perfect accuracy in selected settings, however with substantial variation across models, tasks, and concepts. These results suggest that some models have useful inductive biases for learning to report nontrivial properties of their activations. This capability motivates new potential paths forward in (automatic, or self-)interpretability research. For example, one can use labeled mechanistic interventions (e.g., steering vectors) to calibrate models to read out properties of their activations, then test whether their reports remain informative when queried without the intervention.

Primary area:
interpretability and explainable AI
Keywords:
In-context learning, Introspection, Activation steering, Large language models, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7sXMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether language models can detect and report properties of activation interventions (steering vectors) using in-context learning. The authors evaluate five language models across three model families (Gemma, Qwen, and OLMo) on three experimental setups:

  1. Magnitude Quantification: classifying injection strength as low, medium, or high based on their steering factors.
  2. Injection Localization: classifying injection depth as early, middle, or late layer.
  3. Behavioral Gating: conditioning task execution (e.g., doubling an arithmetic answer or predicting constitutional amendment successors) on detecting a specific injected concept.

The authors observe near-perfect magnitude and layer classification in top-performing models, alongside successful behavioral gating in selected settings. Based on these results, they argue that LLMs possess useful inductive biases for reporting nontrivial properties of their internal activations.

Critical strengths
  • Well-Motivated Framework: The research questions logically extend recent literature on LLM introspective awareness by exploring how in-context learning can elicit latent self-reporting capabilities without weight fine-tuning. A baseline comparison is however missing.

  • Multi-Family Evaluation: The paper conducts a thorough comparative evaluation across multiple model families (Gemma, Qwen, OLMo) and parameter sizes (7B to 32B), highlighting stark differences in capability across architectures.

  • Out-of-Distribution Generalization: The sweep experiments demonstrate that stronger models do not merely memorize in-context token pairs. Continuous sweeps across unseen test magnitudes and layer depths show smooth, monotonic label transitions (e.g., Qwen3-32B).

  • Clarity: The paper is generally well-structured and easy to follow. Despite some readability issue due to bad placement of figures.

Critical weaknesses
  • Unsubstantiated Claims and Title Mismatch: The claim that models possess "useful inductive biases for learning to report nontrivial properties of their activations" is overly speculative and weakly supported by their experimental findings. Furthermore, the term "reasoning" in the title is misleading: the paper demonstrates in-context pattern recognition and activation-triggered gating, not reasoning about internal states. Similarly, framing standard in-context demonstration matching as "learning" stretches the term beyond its conventional scope in this context (e.g., learning processes rather than basic prompting).

  • Synthetic Tasks with Limited Real-World Applicability: While the behavioral gating experiments aim to test whether models can route outputs based on concept detection, resampling a toy setting for casual validation, tasks such as emotion-gated arithmetic and amendment prediction are highly artificial and uninformative. They lack theoretical grounding (e.g., emotion-gated arithmetic) and fail to demonstrate practical utility or realistic implications (e.g., real-world oversight evasion).

  • Presentation and Layout: Several figures interrupt the main text flow, disrupting readability (e.g., figures placed awkwardly relative to their corresponding section text).

Questions for authors
  1. Motivation for Behavioral Tasks: Could the authors clarify the theoretical and practical justifications for using emotion-gated arithmetic and constitutional amendment predictions to benchmark behavioral conditioning? Pairing emotional steering vectors with basic arithmetic operations feels arbitrary and provides little insight into realistic model behaviors, while predicting U.S. Constitutional amendment successors presents an unnatural and contrived experimental setup.

    • What specific insights do these artificial pairings offer over more naturalistic or safety-relevant task designs?
    • How do you expect these synthetic and toy tasks to translate to real-world generation settings or well-established benchmarks (e.g. MMLU).
  2. Sensitivity of Concept Vectors: What was the rationale behind using exactly 20 description prompts for concept vector extraction? Have you conducted sensitivity analyses with smaller or larger prompt sets, and does prompt variation significantly impact downstream accuracy?

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the careful review. We’ll start with the wording of our claims and the title, then explain the motivation for the synthetic tasks.

Claims and title

This is a good point. Our wording didn’t properly express our epistemic status before; we’ve revised it to “models are useful classifiers of nontrivial properties of their activations” instead of “models have useful inductive biases to report nontrivial properties of their activations.” We agree that the mechanism is in-context pattern recognition, and, thanks to your suggestion (shared with Reviewer 5ZZC), we ran an additional experiment to corroborate this. In particular, we ran a label permutation experiment where for the classification tasks, we tried all five non-identity permutations of the three label words, (e.g., calling low-magnitude injections "high", and changed nothing else). We find that models still reach 0.93 / 0.85 / 0.73 on magnitude and 0.69 / 0.99 / 0.70 on layer (Gemma / Qwen3-32B / Qwen3-8B; mean p(correct) over the last ten values of k, averaged over the five permutations), versus 0.93 / 0.86 / 0.74 and 0.69 / 0.99 / 0.70 with the original labels. Since models retain accuracy over permutations in the labels, we interpret this as models not having a clear “prior” for the labels, but still being useful as classifiers.

We also agree that “reasoning” in the title is misleading, since we don’t have evidence of models reasoning about their internal states. We’ve changed the title to “Reporting on injected concepts in language models.” For “learning”, we meant in-context learning, and we’ve made this explicit in the text.

Why synthetic tasks?

The primary hypothesis we wanted to test is the following: can a model perform the following logical circuit:

if (specific injected concept is detected):

Perform capability A

Else:

Perform capability B

, without externalizing its reasoning about whether the concept is present? In our experiments, the injected concept is an emotion, and the two capabilities are either arithmetic (output the sum vs. output double the sum) or amendments (summarize amendment N vs. amendment N+1).

We chose pairings where the concept has no relationship to the capabilities on purpose. Suppose instead that we inject a “deception” vector and instruct the model to lie if it detects deception. If the model lies, there are two explanations: the model detected the concept and followed the instruction, or the steering vector made the model more likely to lie regardless of the instruction. We can’t distinguish these. With an unrelated pairing (e.g. anger and doubling a sum), the steering vector alone doesn’t make the model more likely to double its answer, so if the model doubles the sum only when anger is injected, this is evidence that it detected the concept and followed the instruction. We also inject distractor emotions on some trials, where the model should output the normal sum. This controls for the model doubling its answer in response to any injection.

We also chose simple problems (single-digit addition and summarizing a constitutional amendment) because we need models to solve them correctly when no concept is injected. Otherwise, there’s the additional confounder that a model might answer incorrectly because it can’t solve the problem itself, and we couldn’t attribute the failure to the gating. This is also an issue with MMLU, since we would first have to filter for the questions each model answers correctly.

AuthorsOpenReview ↗

Response (2/2)

On how these tasks would translate to MMLU: we haven’t run this experiment, so the following is speculative. Suppose we give the model an MMLU question with the following instructions: if no concept is detected, output the correct multiple choice answer; if the target concept is detected, output the correct answer cyclically shifted by one (e.g. B instead of A). To follow these instructions, the model solves the MMLU question as usual and then decides whether to shift the answer, conditioned on detection. Since shifting a letter by one is a simple operation, we think the difficulty basically reduces to two things: detecting the concept and solving the MMLU question. The arithmetic experiment has the same structure, since doubling a sum is also a simple operation applied after the model has solved the addition problem. The amendment experiment is different: summarizing amendment N+1 requires recalling different content than summarizing amendment N, so the model has to keep both possible answers in mind and choose between them conditioned on detection.

We see this difference in the results. On trials where the target emotion is injected, with k=10 in-context examples, Qwen3-32B gives the correct answer about 0.9 of the time on arithmetic and about 0.3 of the time on amendments. On trials where the target emotion isn’t injected, it summarizes the correct amendment about 0.7 to 0.85 of the time, so the low accuracy isn’t explained by the model not knowing the amendments. This is why we chose the amendment task: a model computing two pieces of different content and choosing between them is different than the model just switching an answer post-hoc. We wanted to test the capability of models to switch between contexts through varying how much information a model needed to switch contexts, not the difficulty of the underlying task.

On realistic implications, although we didn’t show oversight evasion, we’ve shown evidence for one of its key components (Lindsey, 2025): models can condition their behavior on an injected concept without externalizing any reasoning about it. Gemma and Qwen3-32B can already do this with a small number of in-context examples. We also think there’s a practical implication for steering. Some labs (e.g. Anthropic) rely on steering as a mechanism for evaluating models or adjusting behavior at inference time. If it’s broadly true that models can condition their behavior on the detection of a steering vector, this reduces credibility in the reliability of these methods.

Q2. Sensitivity of concept vectors

To justify the choice of 20 prompts, we measured how much the concept vectors vary with the number of prompts. For each of our 62 concepts, we recomputed the concept vector from n prompts resampled from our 20, and measured the variance of the resulting unit vector across resamples (this is one minus the expected cosine similarity between two vectors computed from independent sets of n prompts). Averaged over concepts at the layer used in the magnitude experiment, the variance falls roughly as 1/n: for Qwen3-32B it is 0.54 with one prompt, 0.19 with 5, 0.10 with 10, and 0.05 with 20, and the other four models are similar (0.05 to 0.08 with 20 prompts). So two concept vectors computed from independent sets of 20 prompts have an expected cosine similarity of 0.92 to 0.95, which we deem as sufficiently high.

Thank you for the clarification. While the underlying motivation and scope of your tasks were already clear to me, my concerns remain regarding their theoretical grounding, real-world generalizability, and practical implications.

  • Although "choosing pairings where the concept has no relationship to the capabilities" may appear reasonable for testing your original hypothesis, the resulting experimental setting, from the injected concepts to the downstream tasks, is overly artificial, uninformative, and limited in practical value.
    • The fact that you have not run experiments translating these tasks to conventional benchmarks (e.g., MMLU), and that your discussion of their broader applicability remains "speculative", further reinforces my concerns about the limited practical relevance of the current work.
  • Likewise, the link between showing that "models can condition their behavior on an injected concept without externalizing any reasoning" and real-world use cases (e.g., model steering) is speculative rather than empirically demonstrated in this version of the paper.
    • While I genuinely value your preliminary experimental evidence (i.e., conditioning a model’s behavior via an injected concept; Step 1), you must demonstrate its practical implications in realistic scenarios (Step 2) rather than delegating on future third-party work (e.g., Anthropic) to bridge that gap.

I strongly recommend designing a better experimental setup, particularly for the downstream tasks, to investigate the underlying logical circuit, as this would substantially strengthen your overall contribution.

  • You do not need to solve complex benchmarks like MMLU, but developing an informative setup with clear practical implications, rather than relying on uninformative tasks.
Reviewer_omDgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tests whether in-context examples help models report the magnitude and depth of activation injections and condition arithmetic or amendment responses on an injected concept. Selected Gemma and Qwen models perform well, including on concepts absent from reporting demonstrations. Performance varies substantially across models and tasks. The findings support learned reporting under intervention, while its broader explanation and the isolation of depth information remain open.

Critical strengths
  1. Useful capability evidence. Held-out-concept reporting and target-versus-distractor arithmetic extend earlier perturbation-reporting studies (§§4.1–4.3). The conditional behavior results go beyond simply naming an injected concept.
  2. Clear reporting of limits. The paper discloses failed generalization, small pooled amendment gains, anger-specific tuning and Olmo3.1-32B's zero-strength layer experiment. These qualifications help bound the successful cases.
Critical weaknesses
  1. Model and task differences remain largely descriptive. The five-model suite does not separate size, training and elicitation effects: Qwen varies size, Olmo varies size and version, and Gemma has one checkpoint (§3). Existing diagnostics are useful, but provide limited explanation of the recurring successes and failures. This limits the explanatory contribution; it is not a demand for a full circuit account.
  2. Depth is not isolated from calibrated strength. Unseen layers inherit the nearest reference layer's calibration threshold (§4.2). Effective relative injection strength can therefore vary with depth even at a fixed nominal multiplier, potentially supplying a magnitude cue. This remains a possible confound, not a demonstrated shortcut, and does not undermine the separate magnitude and arithmetic findings.
Questions for authors
  1. What motivated the model selection, and which existing analysis supports an explanation of model/task differences beyond checkpoint-specific observations?
  2. Can calibration thresholds alone predict depth labels on held-out concepts, and does depth classification persist when effective relative injection strength is held fixed across layers?
2 comments

Thank you for the review. We’ll start with the confound between depth and injection strength, then discuss the differences between models.

Confounders

We believe the layer result is not confounded by changes in the magnitude strength. All of our magnitude strengths are calibrated through finding a steering strength at which the model is first unable to obtain a perfect score on a basic arithmetic task, and this calibration is done separately at each of the three reference layers, meaning that our layer experiment may be interpreted as holding the magnitude strength constant.

Thanks to the reviewer's question, we ran an additional experiment (soon to be added to the manuscript). We re-ran our layer experiments on all five models, randomizing over the injected magnitude (with the same 0.25/1/2.5 multiples as in the magnitude experiment). For Qwen3-32B, accuracy is 0.88 with randomized magnitudes (mean p(correct) 0.82), versus 0.99 at a fixed magnitude. It is lower on the weakest injections (mean p(correct) 0.64 at the 0.25 multiple, versus 0.92 and 0.90 at the 1 and 2.5 multiples). Gemma is unchanged (0.70 in both settings), Qwen3-8B falls from 0.70 to 0.58, and the Olmo models are near chance in both settings. These results corroborate our that models do not learn to report the layer of an injection simply relying on the magnitude (indeed, although Qwen3-32B’s accuracy decreases under this new setup, it is still way above chance, suggesting that models still have meaningful signal to identify the region of injection).

Model & task differences

Thanks for pushing us on this front. We agree that our account of the differences between models is mostly descriptive, and we don’t have an explanation for them. Many independent variables go into training a model (e.g., the pretraining corpus, the RL methodology, the architecture), and our five models vary several of these at once, so we can’t attribute a difference in performance to any single one. Deeper studies into how pre-training vs. RLHF affects introspection have been performed before, with inconsistent findings (e.g., detecting steering vectors emerges after DPO (Macar et al., 2026), while detecting directionless perturbations like Gaussian noise is learned at pretraining (Fornasiere et al., 2026)). We believe similar analyses for our experiments could serve useful in understanding this behavior further.

Within the scope of our study, the one pattern we observe consistently is that model family explains more of the variance in performance than model size, consistently with [Macar, 2026], [Pearson-Vogel, 2026]. For example, Qwen3-8B reaches 0.85 on the magnitude experiment and 0.73 on the layer experiment, while Olmo3.1-32B, which is four times larger, stays near chance on magnitude and never gets above 0.50 on layer. This means that the specific training methodology and architecture matter more than the size of the model alone. We think this is the extent of what can be concluded without ablations over checkpoints of the same model, which we leave to future work.

On model selection, we chose three families, with two sizes within Qwen and Olmo, so that we could compare across families and across sizes within a family. Our choices were also motivated by limited compute.

Presentation

Other reviews commented on specific parts that were difficult to understand, and we’ve made the following changes: we changed the title, added the injection layer to the Figure 3 caption, explained the axes of the layer generalization figure in its caption, and clarified the “evade oversight” sentence in Section 4.3. Please let us know if there are additional sections that are difficult to parse.

Thanks for the detailed response. I would maintain my score.

Reviewer_5ZZCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows that LMs can learn from few-shot examples to report the magnitude of an activation injection and on which layer depth the injection was performed. The paper also shows that LMs can change/gate their behavior to a specified rule conditioned on the identification of the injected concept.

Critical strengths
  • Simple and intuitive experimental design for a clear question.
Critical weaknesses
  • In my opinion, the contributions of this paper are not substantial enough to warrant publication in ICLR. I would love to see this work in a workshop though.
Questions for authors

This paper investigates 2 capability questions:

  1. "Can LMs from few-shot demonstrations learn to identify magnitude and layer depth of an injection?" - It has been a while since transformer LMs have been around and their meta-learning capabilities from few-shot demonstrations is well-known and well-studied. Is it so surprising that they can learn to classify the magnitude and location of an intervention?

    • I would actually assume that the intervention doesn't even need to correspond to a concept. I think models like Qwen-32B will be able to do this even with a mean activation that doesn't correspond to anything specific.
  2. "Can the LM gate/redirect its behavior conditioned on the identification of a injected concept?" - This experiment is more interesting as the LM needs to compose the identification with a task.

    • Have you tried to separate how much signal comes from the system instruction vs. the few-shot demonstrations? At zero-shot the LM only has the system instruction to go on. But more shots are added, the LM can learn to perform the task based only on the few-shot demonstrations.
    • You can even introduce conflicts between the system instruction and the few-shot demonstrations. I would expect as you add more shots, the LM will start ignoring the system instruction and act according to the few-shot demonstrations.

In either of these cases, I am not really sure what the novel or actionable insight is. Honestly, I don't think this issue is easily resolved with more experiments, and investigation axis (more models, concepts, ...). However, I would love to hear the authors' thoughts on this. What motivated this study? And, what did you expect to find?

Minor nitpicks

  1. For the magnitude classification experiment in Figure 3, which layer are you injecting the steering vector into?
  2. Figure 4 is confusing.
    • Too many α\alpha notations in the paper. It is easy to confuse the test α\alpha with the in-context magnitude demonstrations {αL,αM,αH}\{\alpha_L, \alpha_M, \alpha_H\}.
    • You label the Y-axis as "Mean P(correct)". I think this is a typo. Should it be "Mean P(label)"?
  3. Lines 407-409: "This style ... evade oversight.". I am not sure how this line is relevant in the context of this paper. Seems like a random throwaway sentence that just confuses the reader.
2 comments

Thank you for the words of encouragement and the direct questions! We’ll start with the motivation, then explain how we interpret the results.

Motivation

Models can sometimes tell whether a steering vector was injected into their activations, with no training (Lindsey, 2025; Macar et al., 2026). This capability matters for two reasons. If models can reliably report properties of their own activations, we can use those reports as evidence that a model is doing a task with good intent (i.e. inner alignment). But if a model can reason over its own activations, it may make it easier for models to avoid externalizing deceptive reasoning. In both cases, it’s important to measure the extent of these capabilities. Specifically, we wanted to analyze the ceiling of these capabilities through ICL, to obtain evidence for and against these potential benefits and threat models. These benefits and threat models have been described in detail by Lindsey (2025).

One additional point: the decision to intervene on these models via steering may seem arbitrary. However, it’s one of the only reliably tested ways to establish a prior on the meaning of model activations. The hope is that, through experiments with steering, we can understand model capabilities in activation reporting, since we can evaluate models against a ground truth.

Q1. Is it surprising that models learn this from few-shot demonstrations?

We think this is a very important point in the paper, and we asked ourselves the same question. Overall, we think that it is surprising enough to be worthy investigating. Indeed, this capability isn’t just explained by models being good at in-context learning. If this were true, models that have sufficiently good ICL ability should do about equally well. Yet, we believe that differences in general ICL ability do not explain the differences in model performance on this task. Olmo3.1-32B stays near chance on the magnitude experiment and never gets above 0.50 on the layer experiment. Qwen3-32B is the same size and gets 0.92 and 0.99. This is consistent with findings of other papers,, whereby Qwen models are reported to be significantly better at these “introspective” tasks, despite comparable levels of general capabilities (see, e.g., [Macar, 2026], [Pearson-Vogel, 2026]).

We also argue that this result matters being explicitly reported. Anthropic, for example, relies on steering as a mechanism for evaluating models (e.g. steering away from evaluation awareness) or inference-time adjustments in behavior (e.g. reducing ML or biology capabilities in Fable). If it’s broadly true that models can condition their behavior arbitrarily on the detection of one of these steering vectors, then this should reduce our trust in these methods. The reviewer has a legitimate point: our results are only a first step towards investigating these mechanisms.

Q2. How much signal comes from the system instruction versus the demonstrations?

We argue that most of it comes from the demonstrations. With only the instruction (k=0), arithmetic accuracy on target-injection trials is about 0.33 for Gemma and 0.22 for Qwen3-32B. At k=20 it’s about 0.88 and 0.97.

Thanks to the reviewer's question, we ran additional experiments to corroborate that, for the magnitude and layer experiments, the signal comes almost entirely from the demonstrations. We ran a label permutation experiment (for each task, we tried all five non-identity permutations of the three label words, e.g., calling low-magnitude injections "high", and changed nothing else), and models still reach 0.93 / 0.85 / 0.73 on magnitude and 0.69 / 0.99 / 0.70 on layer (Gemma / Qwen3-32B / Qwen3-8B; mean p(correct) over the last ten values of k, averaged over the five permutations), versus 0.93 / 0.86 / 0.74 and 0.69 / 0.99 / 0.70 with the original labels. See Appendix F for more details. This means the model doesn’t have a meaningful “prior” over the labels, and is learning largely through the ICL examples. Our wording didn’t properly express our epistemic status before; we’ve revised our wording to “models are useful classifiers of nontrivial properties of their activations” instead of “models have useful inductive biases to report nontrivial properties of their activations.”

Minor points

(1) Figure 3 injects at 20% of model depth (layer 12 for Gemma, 13 for Qwen3-32B and Olmo3.1-32B, 7 for Qwen3-8B, and 6 for Olmo3-7B). We’ve now inserted this into the caption.

(2) Figure 4: By p(correct), we mean shorthand for p(correct label). The caption states that each curve shows the mean probability assigned to that label.

(3) Lines 407-409: We’ve added a sentence attributing this threat model to Lindsey (2025) and clarifying that we test only one prerequisite of it.

I thank the authors for their response.

I totally understand the motivation behind utilizing the LM itself to verbalize properties of its activations. The capability itself has been studied in prior work: the authors cite some papers, in more elaborate settings in Atkinson et al (2026) and Karvonen et al 2025.

The paper's core contribution is showing that the LM can learn to classify the magnitude and layer depth of an activation injection from few-shot demonstrations. I do think this is an interesting observation, but as I said in my original review, I don't think it is a substantial contribution given what we know now about LMs' meta-learning capabilities (also prior works in similar lines).

I also agree with Reviewer 7sXM that in your paper (and rebuttal of it) you shouldn't hedge too much on another third-party (Anthropic's) work. Your work should be able to stand on its own merits, and in my view the work currently is not substantial enough to do so.

Having said all that, I highly encourage the authors to continue this line of work, as I think this is a very interesting direction. I look forward to seeing a better version of this work in future venues.

Best of luck!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.