- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLMs can control their own activations in context via a neurofeedback-style protocol. It separates three hypotheses — (i) in-context pattern matching over sentences, (ii) sentence selection based on privileged access to activations, and (iii) changing how a fixed sentence is processed — using a writer/subject decoupling (Exp. 1), a fixed-sentence setting (Exp. 2), and fixed-sentence sessions against six safety monitors with and without CoT (Exp. 3–4). Across 10 open-weight models and 42 readouts, free-generation effects are explained by writer competence rather than self-access; with the sentence fixed, three models (Qwen3.5-27B, Qwen3.6-27B, Gemma-4-31B) show small but significant effects (≤2 points). Against monitors, only Qwen3.5-27B shows an effect, and only in the direction of raising the score.
Critical strengths
- The writer/subject decoupling and the additive fit (competence + susceptibility) cleanly separate (i) from (ii), closing the gap left by Aoki et al., who refuted the earlier positive finding but did not directly establish its cause.
- Thorough evaluation: 42 readouts × 2 depths × disclosure conditions × 10 models, 168,000 sessions.
- Effect sizes are reported without exaggeration, and the safety implications are stated conservatively.
Critical weaknesses
- Effect size and control validity (Exp. 2). The 1.8-point effect is 2% of the scale. That activations on a fixed sentence differ when the preceding numbers differ is trivial; the question is only whether the shift depends on the instruction. A significant objective difference could still arise from an interaction between the “max”/”min” wording and the different statistics of real vs. uniform-random scores. The Limitations section acknowledges this, but it is a fundamental weakness of the control design.
- Additivity assumption. The diagonal excess depends on s_o τ = m + a_W + b_S. Any W × S interaction (a writer that is specifically effective on a particular subject) would invalidate it. Fig. 5 is consistent with additivity, but no test is reported.
- No account of model dependence. Why only Qwen3.5/3.6 and Gemma-4-31B? Training data, post-training, and scale are all plausible, but the paper offers no mechanistic analysis or discussion.
Questions for authors
- How would the results generalize to other model families and larger sizes?
- Do training phases (base / SFT / RL) differ in their ability to control activations?
- What are the practical or safety implications that developers should monitor today, and what would constitute a real threat?
3 comments
[2/2] Additivity, model dependence, and safety implications
Additivity assumption. ... Fig. 5 is consistent with additivity, but no test is reported.
We thank the reviewer for pointing this out. We agree that this needed a test, and we added one (new paragraph "Additivity" and Table 6 in Appendix E.1). We predicted every pair with W ≠ S by fitting the additive model on the other 19 pairs with W ≠ S, in all four combinations of objective and disclosure (we exclude the pairs with W = S, since they are the ones whose excess we measure). The prediction errors are within sampling noise, so we find no evidence of a W × S interaction, that is, of a writer that is specifically effective on a particular subject. We built the additive model to separate two reasons why a model can move its own readouts without privileged access. It can be a good writer, which moves the readouts of any subject (a_W), or a susceptible subject, whose readouts move whoever writes the sentences (b_S). However, the fit itself shows a posteriori that subjects differ little in susceptibility, so our conclusion does not actually need the model, and when we compare each model's effect on itself with its average effect on the other four subjects, no model shows a significant advantage.
No account of model dependence. Why only Qwen3.5/3.6 and Gemma-4-31B?
How would the results generalize to other model families and larger sizes?
Do training phases (base / SFT / RL) differ in their ability to control activations?
We thank the reviewer for these questions, which also led us to test a larger model. We honestly cannot say yet what explains the differences across models. Training data and post-training are plausible causes, but the training data of these models is not public, and no base checkpoint is released for Qwen3.5-27B or Qwen3.6-27B, so we cannot test them. OLMo-3.1-32B releases a checkpoint after each training phase (SFT, DPO, and reinforcement learning), but its final model doesn't show control, so comparing them would not tell us when control emerges. We also find that scale does not explain it, because within the Qwen3.5 family the effect appears in the 27B model but neither in the smaller Qwen3.5-9B nor in the larger Qwen3.5-122B-A10B, which we added for this revision (−0.1 points, Section 3.2, Figure 3). Kowalski et al. [2] also find that the ability of models to control their activations does not track their number of parameters. Across the five families we tested (Qwen, Gemma, GPT-OSS, Mistral, and OLMo), robust control appears only in the Qwen family, so we cannot claim that it generalizes to other families.
However, when we observe a control effect, we characterize it. Our paper already shows that it appears within the first few turns and does not require telling the model what is measured (Section 3.4, Figures 3 and 7), and the analyses we added for the revision show that it is targeted (new Appendix F.2) and that it only appears in the layers between 55 and 85% of the depth (new Appendix F.3).
What are the practical or safety implications that developers should monitor today, and what would constitute a real threat?
We appreciate this question, since it is the motivation of our work. The threat we study is a model that evades an activation monitor by controlling what its activations encode. McGuinness et al. [4] show that finetuning can teach this, and that the evasion transfers to unseen monitors. We ask whether it can also arise in context, where neurofeedback gives an upper bound, since the model sees the monitor's score at every turn. A real threat would be a model that lowers the score of a monitor enough to change its decisions, and no model does so in our experiments (Section 4). Developers should therefore track this capability as new models are released: we find control only in recent models, and our fixed-sentence protocol is cheap to rerun with the released code.
References
[2] Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar, and David Demitri Africa. Measuring Activation Control in Large Language Models. arXiv:2608.21664, 2026.
[4] Max McGuinness, Alex Serrano, Luke Bailey, and Scott Emmons. Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors. arXiv:2512.11949, 2025.
Thank you for the detailed response and the additional experiments. My main concerns have been addressed.
- Control validity (Exp. 2): The shuffled control directly tests the confound I raised. I appreciate that you report the outcome transparently: the effect for Qwen3.5/3.6 is roughly halved but remains significant, and the Gemma-4-31B effect disappears. The breakdown by depth also makes clearer where the effect is concentrated.
- Additivity: The leave-one-pair-out test resolves my concern, and the comparison of each model's effect on itself against its average effect on other subjects shows that the conclusion does not depend on the additive model.
- Model dependence: The cause remains open, but the addition of Qwen3.5-122B-A10B rules out scale as a simple explanation, and the limits of what can be tested are stated clearly.
One suggestion for the final version: since the shuffled control changes the picture for Gemma-4-31B, please state this in the main text (not only in Appendix F.1) and adjust the claim that three models show the effect accordingly.
I am satisfied with the rebuttal and maintain my positive score.
[1/2] Effect size and control validity
We thank the reviewers for their careful reviews. We are glad that they found that our cross-model writer comparison resolves an ambiguity left by prior work (Am4h, gj4v), that the fixed-sentence experiments provide new insights (Am4h, XxMZ), and that our evaluation is thorough (gj4v).
We thank the reviewer for raising this point, which led us to run a new control. We agree that the effect is small on average, and we claim that it is statistically significant, not that it is large.
To test the interaction you describe, we reran part of our experiments with a shuffled control, in which each session is shown the scores of a real session in a random order, so that they have the same statistics as real feedback (new Appendix F.1 and Figure 12). We find that, with this control in place, the effect of Qwen3.5-27B and Qwen3.6-27B is weaker, about half of what we reported, but remains significant, while the effect of Gemma-4-31B disappears. The half that remains requires the feedback to come from the model's own activations, turn by turn. The half that disappears does not depend on the feedback, but we cannot tell whether it comes from the interaction you describe or from the model trying to follow the instruction without using the scores, which would mean zero-shot control. Since our question is whether models use the feedback, our metric should not count it.
The average over our 42 readouts, however, understates this control, since some of the readouts we designed are especially hard to move. As our paper already shows (Figure 8, and now also in the main text, Figure 3, left), the effect is concentrated on the readouts read at 66% of the depth, in agreement with previous work [1, 2]. The shuffled control confirms this: the readouts defined as a direction in the residual stream space move by two to three times the average at 66% of the depth, and by less than 1 point at 33% in every model (the metric is the objective difference at turn 30, measured against the shuffled control, as in the new Figure 12):
As is well known, no control condition is perfect [3], and we keep the random scores as the main control of the paper, because they allow a comparison at every turn and, since they do not depend on the real sessions, can be shared by all the readouts and run for every experiment.
Our main finding still holds: with the sentence fixed, a few models show a small but significant control effect.
References
[1] Ji-An Li, Hua-Dong Xiong, Robert C. Wilson, Marcelo G. Mattar, and Marcus K. Benna. Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations. NeurIPS 2025.
[2] Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar, and David Demitri Africa. Measuring Activation Control in Large Language Models. arXiv:2608.21664, 2026.
[3] Bettina Sorger, Frank Scharnowski, David E. J. Linden, Michelle Hampson, and Kymberly D. Young. Control freaks: Towards optimal selection of control conditions for fMRI neurofeedback studies. NeuroImage 2019.