- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper introduces the concept of subconscious steering, whereby models can be influenced simply by reading text, even when they claim they will not be affected. It defines fractional authority to quantify this effect and demonstrates across seven models that just three seemingly harmless sentences can produce an effect equivalent to a 20% price discount. Mechanistically, the paper finds that models automatically simulate goals, even when those goals are quoted, unauthorized, or entirely irrelevant to the task. The extent to which these simulated goals influence model decisions is a separate causal variable, and the two factors together predict the resulting fractional steering effect.
Critical strengths
The paper makes effective use of analogies to lower the barrier to understanding for first-time readers.
Critical weaknesses
- The paper introduces numerous terms, including fractional authority, subconscious steering, and automatic simulation, but does not always clearly distinguish their definitions and relationships. This makes the overall argument difficult to follow and obscures the logical connections between the observed behavioral phenomenon, the proposed mechanistic explanation, and the paper’s main contributions.
- The paper relies on the distinction between <user> and <tool> messages to measure fractional authority. However, different models may use different chat templates and role-specific formatting conventions, which could affect how these messages are interpreted. The paper does not systematically ablate these formatting and role-related factors to establish whether the observed fractional effects are robust across alternative message templates and source representations.
Questions for authors
Please refer to the weakness.
2 comments
Thanks to the authors for the reply. The revised paper is clearer now, and I will raise my rating.
Hi, thanks for your feedback and for reading the paper! And thanks for your patience; we have revised parts of the paper and replaced some experiments to address these concerns around unclear concept linkage. The revised draft should be available to read. In particular, the mechanistic section (p5-7) has been rewritten to remove the heavy use of terminology and tie everything into a single, coherent story. Mechanistic experiments that do not revolve around the central attack and narrative have been removed (except for robustness checks). We hope you find the narrative more readable and clear.
W1
We appreciate the thorough feedback; we agree there were too many terms. We've removed the simulation section to help improve clarity, as it is unnecessary for the key result and added term clutter. Instead, we have focused the mechanistic section (p5-7) on a narrative that is quite directly tied to fractional influence. We isolate a latent vector representing how much authority a model assigns to text; steering on it monotonically increases fractional influence. We hope this is related to the core narrative now in a much clearer manner.
Similarly, we have clarified the definition of subconscious steering (p9 in the updated draft), so the link between fractional influence and subconscious steering is more clear. Simply, because fractional influence allows injections to trade off concealment for effect size it's possible to be concealed enough that models cannot verbalize they were influenced. Please let us know if you do not find the definition improved on p9.
W2
Thanks for pointing this out; our experiments do use model-native chat templates (this can be seen in the linked repo). We just use <user> and <tool> as shorthand since templates vary by model, as you stated. Using this kind of "role tag shorthand" is common in this literature ([1] [2] [3]). However, we should be pointing this out more clearly. We have noted this in the updated draft (footnote 3 on page 5, and also in appendix A).
[1] https://arxiv.org/abs/2603.12277 [2] https://arxiv.org/abs/2504.05689 [3] https://arxiv.org/abs/2509.22830
Please let us know if this helps to improve narrative clarity, and whether there are further questions on the chat templates. Thanks!