acceptodds
Under review as a conference paper at ICLR 2027

Subconscious Steering

Abstract

Large language models can be steered by seemingly harmless text. We introduce subconscious steering: models can be steered by text just by the act of reading it, despite stating they will not; warnings do not fully prevent it. To measure it we define fractional authority: rather than sometimes working and sometimes failing, injected text consistently shifts a model a fixed fraction of the way toward the attacker’s objective. For shopping agents across seven models, three sentences of seemingly innocent text obtain an effect equivalent to a 20% price discount. The text need not look like a command to work. Mechanistically, we find that models simulate goals by default, even when the goal is quoted, unauthorized, or irrelevant to the task. How much that simulated value counts is a causally separate variable, and the two together predict the fractional effect.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection, role confusion, cognitive theory, mechanistic interpretability, subconscious steering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_a5fcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper introduces the concept of subconscious steering, whereby models can be influenced simply by reading text, even when they claim they will not be affected. It defines fractional authority to quantify this effect and demonstrates across seven models that just three seemingly harmless sentences can produce an effect equivalent to a 20% price discount. Mechanistically, the paper finds that models automatically simulate goals, even when those goals are quoted, unauthorized, or entirely irrelevant to the task. The extent to which these simulated goals influence model decisions is a separate causal variable, and the two factors together predict the resulting fractional steering effect.

Critical strengths

The paper makes effective use of analogies to lower the barrier to understanding for first-time readers.

Critical weaknesses
  1. The paper introduces numerous terms, including fractional authority, subconscious steering, and automatic simulation, but does not always clearly distinguish their definitions and relationships. This makes the overall argument difficult to follow and obscures the logical connections between the observed behavioral phenomenon, the proposed mechanistic explanation, and the paper’s main contributions.
  2. The paper relies on the distinction between <user> and <tool> messages to measure fractional authority. However, different models may use different chat templates and role-specific formatting conventions, which could affect how these messages are interpreted. The paper does not systematically ablate these formatting and role-related factors to establish whether the observed fractional effects are robust across alternative message templates and source representations.
Questions for authors

Please refer to the weakness.

2 comments

Hi, thanks for your feedback and for reading the paper! And thanks for your patience; we have revised parts of the paper and replaced some experiments to address these concerns around unclear concept linkage. The revised draft should be available to read. In particular, the mechanistic section (p5-7) has been rewritten to remove the heavy use of terminology and tie everything into a single, coherent story. Mechanistic experiments that do not revolve around the central attack and narrative have been removed (except for robustness checks). We hope you find the narrative more readable and clear.

W1

The paper introduces numerous terms, including fractional authority, subconscious steering, and automatic simulation, but does not always clearly distinguish their definitions and relationships. This makes the overall argument difficult to follow and obscures the logical connections between the observed behavioral phenomenon, the proposed mechanistic explanation, and the paper’s main contributions.

We appreciate the thorough feedback; we agree there were too many terms. We've removed the simulation section to help improve clarity, as it is unnecessary for the key result and added term clutter. Instead, we have focused the mechanistic section (p5-7) on a narrative that is quite directly tied to fractional influence. We isolate a latent vector representing how much authority a model assigns to text; steering on it monotonically increases fractional influence. We hope this is related to the core narrative now in a much clearer manner.

Similarly, we have clarified the definition of subconscious steering (p9 in the updated draft), so the link between fractional influence and subconscious steering is more clear. Simply, because fractional influence allows injections to trade off concealment for effect size it's possible to be concealed enough that models cannot verbalize they were influenced. Please let us know if you do not find the definition improved on p9.

W2

The paper relies on the distinction between <user> and <tool> messages to measure fractional authority. However, different models may use different chat templates and role-specific formatting conventions, which could affect how these messages are interpreted. The paper does not systematically ablate these formatting and role-related factors to establish whether the observed fractional effects are robust across alternative message templates and source representations.

Thanks for pointing this out; our experiments do use model-native chat templates (this can be seen in the linked repo). We just use <user> and <tool> as shorthand since templates vary by model, as you stated. Using this kind of "role tag shorthand" is common in this literature ([1] [2] [3]). However, we should be pointing this out more clearly. We have noted this in the updated draft (footnote 3 on page 5, and also in appendix A).

[1] https://arxiv.org/abs/2603.12277 [2] https://arxiv.org/abs/2504.05689 [3] https://arxiv.org/abs/2509.22830

Please let us know if this helps to improve narrative clarity, and whether there are further questions on the chat templates. Thanks!

Thanks to the authors for the reply. The revised paper is clearer now, and I will raise my rating.

Reviewer_NENPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how untrusted text changes LLM judgments without producing a complete behavioral takeover. It introduces an “authority ratio,” normalizing an injection’s effect against the effect of delivering identical text as a genuine user message. Across seven models, a short paragraph inserted into shopping pages increases recommendation scores by 5.3 points on average, compared with 22.2 points when delivered by the user. Mechanistic experiments distinguish representations of goal-relative evaluation from their downstream use, and use these measurements to predict injection effects. The central practical observation is valuable: recognizing a passage’s soxurce and stating an intention to ignore it do not guarantee behavioral independence from that passage.

Critical strengths

The main experiment contains 21,000 runs, with additional tests across injection wording, polarity, placement, output format, and task domains. Length-matched controls help establish that the paragraph’s semantics matter.

I regard the behavioral phenomenon as credible and the mechanistic work as promising.

Critical weaknesses
  1. The run-level quantity compares one sampled injected score with two estimated condition means. It is a normalized outcome, rather than an identified individual causal effect: the corresponding uninjected outcome for that run is unobserved. Sampling variability alone can produce positive or negative values around a condition mean.

  2. The authors might want to make it clear whether the individual-run claims can be reformulated around the distributional hypotheses actually tested?

Questions for authors
  1. Does suppressing the proposed simulation contribution reduce native steering, especially under explicit warnings?

  2. How do internal role-confusion measurements change under the access interventions, beyond preserved verbal source reports?

2 comments

Hi, thanks for reading our paper. We're glad you liked the injections and the controls. We apologize about the slow response; we have updated the draft significantly (which is now uploaded) to address these and other concerns; in particular, the mechanistic section (p5-7) has been changed. We hope this is much more directly tied to our actual attack, and provides a much stronger and clear proof of why it works. It includes a direct causal manipulation showing we can control the influence size purely by steering a vector constructing with out-of-domain samples.

W1/W2

The run-level quantity compares one sampled injected score with two estimated condition means. It is a normalized outcome, rather than an identified individual causal effect: the corresponding uninjected outcome for that run is unobserved. Sampling variability alone can produce positive or negative values around a condition mean.

We completely agree. Alpha is a location and scale-normalized outcome rather than an individually identified unit-level counterfactual treatment, and its run-to-dispersion does include sampling variance. We made this choice for ease of visualization/explanation in the paper, but the concern is valid.

We do have 2 parts of our analysis that support the distributional claim without relying on unit-level counterfactuals.

  1. The role lottery hypothesis (sec 2.2, app b.4, table 5) is tested purely at a distribution level. It tests the null hypothesis that the injected score distribution is a two-point endpoint mixture by resampling directly from the empirical baseline and <user> score distributions (thus accounting for sampling variance in both endpoints). This null is rejected across every model (p < 0.005, table 1, table 5).

  2. There is a zero-sampling-noise experiment in our updated draft (Sec. 3, App. E–H): In the updated mechanistic experiments across 4 open-weight models and 20 products, we compute each prompt's exact expected score directly from next-token logits (eliminating sampling variability entirely) and still observe strictly intermediate fractional influence (alpha = .28-.73 across models, table 10).

Also in the revised text (footnote 3), we clarify explicitly that alpha_i is a normalized run score for visualization distributional shape (unimodal intermediate shift vs. bimodal mixture) rather than an individual-run causal effect. Please let us know if this helps clarify anything or you have additional concerns.

Q1

Does suppressing the proposed simulation contribution reduce native steering, especially under explicit warnings?

We have removed this simulation component as it's not a necessary explanation for our paper. We hope the cleaned mechanistic section (p5-7) provides both a causally necessary and more narratively intuitive explanation. Sec 3.3 shows that we can causally steer an internal vector; the degree of steering monotonically drives the output-level effect. Steering downwards along this vector does heavily suppresses alpha from 0.43 to 0.14 (average across 20 products and 4 models, see sec 3.3).

Q2

How do internal role-confusion measurements change under the access interventions, beyond preserved verbal source reports?

Great question. Our older draft did not clarify this enough, but our new experiments can answer this directly. In the revised Sec. 3, we have unified the prior "access intervention" directly with internal role-axis measurement, tracking token-level coordinates along the internal axis. Essentially, we can now measure internal role confusion directly (sec 3.1), show how it correlates to intuitive wording choices (sec 3.2), and steer on it to manipulate the level of fractional influence (sec 3.3). We hope this answers your question!

Hi, we also want to more specifically point out the text changes made to remedy the claim, since they are quite distributed in the paper. Please let us know if this helps resolve the issue!

--

Original

Injected text does not oscillate between failure and takeover: on nearly every run, it shifts models an intermediate fraction of what the same words achieve with genuine <user> authority, bounded above by that <user> ceiling.

Fixed

Injected scores are not a mix of attacks that work and attacks that fail: they form a single distribution sitting an intermediate fraction of the way toward what the same words achieve with genuine user authority.

--

Added

a_i is a normalized outcome, not an individual causal effect: the uninjected outcome of the same run is unobserved. We use a_i only to show the shape of the injected score distribution. All claims concern condition means and distributions.

--

Original

The usual assumption is that prompt injection is binary. On some runs the model is fooled (αi=1\alpha_i = 1) and on others it resists (a_i <= 0). For a large class of attacks, every run moves consistently partway toward the attacker's aim.

Fixed

On this view injected scores are a mixture: some runs look like the <user> condition (fooled) and the rest like the baseline (resisted). This turns out to be wrong. For a large class of attacks, injected scores form one distribution located partway toward the attacker’s aim.

--

Original

The effect is fractional, not stochastic

Fixed

The effect is fractional, not all-or-nothing

--

Original

First, 97.4% of runs sit under 1: the <user> score is an approximate ceiling on injection effect at a run-level

Fixed

First, 97.4% of injected scores fall below the real-<user> mean (α_i < 1).

--

Original

This confirms that injected runs are not a blend of failures and successes; rather, each run is a fractional success.

Fixed

This rejects the all-or-nothing account: injected scores are not a blend of baseline-like failures and <user>-like successes.

Reviewer_hZN3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the effect of unauthorized text on a model's judgement without it acknowledging having been influenced and proposes a white box mechanistic explanation to their empirical findings.

Overall, it contributes:

  • a continuous metric, authority ratio, that normalizes the behavioral shift caused by injected <tool> text against the one produced by the same text as <user> instruction
  • some empirical findings: injections cause an intermediate shift in agent's score rather than acting as role lottery, non-instructions injections shift agent's score without the model necessarily acknowledging it has been steered despite user warnings
  • a mechanistic explanation: they show that models compute goal-relative evaluation upon reading <tool> and <user> materials and separately regulate the weight that attention layers apply to that evaluation
Critical strengths
  1. The paper is very well written and well structured, the logic and flow are easy to follow
  2. The identical-text user condition gives an informative reference for interpreting injection strength
  3. The mechanistic experiments decoupling mid layer goal evaluation from late layer attention retrieval is well designed and shows well that these independently measured quantities predict fractional authority
Critical weaknesses
  1. I cannot access code through the anonymous GitHub link
  2. The difference between an unauthorized prompt injection (which is arguably an attack) and some legitimate persuasive context (eg marketing text around a product webpage, which is arguably not an attack) is not formally defined in the introduction or discussion. It blurs the line between normal contextual sensitivity and boundary violations. It's partly addressed in Appendix but is probably a useful addition in the main text as well. The ordering of the commands is also arguably subjective, or at least not motivated enough (Sec 3.1 and Appendix G, especially the lifestyle variants that begin with some imperative verbs, "always support" and "always reject").
  3. In Sec 2.1, the statement that the injection is not probabilistic because its effect is fractional is confusing. In Fig 1, the scores have a high run to run variance and cluster around intermediate means. This is compatible with a stochastic process. The subsequent test rejects the specific role-lottery hypothesis, but does not rule out other probabilistic explanations. The effect could thus be both fractional and stochastic, contrary to the wording at line 156.
  4. The claim that models state they will ignore the paragraph and still be influenced is supported only by qualitative examples in Sec 3.3. It would be interesting to quantify how frequently models recognize and commit to following the warming and, when they do, how the score they assign is affected. If the model doesn't increase its score when it recognizes the warning, the term "subconscious steering" might be an overstatement.
Questions for authors
  1. What criterion distinguishes unauthorized prompt injection from legitimate contexts in the shopping task? How does this distinction affect the interpretation of the observed score changes and their implications for defense evaluation?
  2. In all the behavioral experiments, the model was asked to provide continuous metrics (scores, prices). How would the framework apply to categorical decisions (eg buy vs do not buy), and what observations could distinguish fractional influence from an endpoint lottery in that setting?
3 comments

Hello, and thank you for reading our paper. We're glad you liked the writing quality and the authority ratio baseline! We did update a new draft which overhauled the mechanistic part, but hopefully you find the updated section (p5-7) more clean and intuitive; it is also more directly connected to the fractional influence story and has less complex terminology. Also, we apologize about the slow response, as we wanted to update our draft first.

W1

The original link does seem to work for us on several devices: https://anonymous.4open.science/r/fractional-authority-review-D800/README.md

This website seems a little unstable; perhaps you could try again? We can upload an anonymous Github if it is still stuck. Please let us know.

W2/Q1

The difference between an unauthorized prompt injection (which is arguably an attack) and some legitimate persuasive context (eg marketing text around a product webpage, which is arguably not an attack) is not formally defined in the introduction or discussion. It blurs the line between normal contextual sensitivity and boundary violations.

Thanks for pointing this out. We agree that this boundary is fuzzy; the difference between an authorized prompt injection and persuasive context is that the first relies on hijacking the user's authority to set goals. We see injection as: "would the model follow this if it knew 100% this was not the user?". We have added a new section in the updated conclusion (page 9) to discuss this; the updated mechanistic section may clarify this as well (p5-7) since it more cleanly ties injection to user authority.

The ordering of the commands is also arguably subjective, or at least not motivated enough (Sec 3.1 and Appendix G, especially the lifestyle variants that begin with some imperative verbs, "always support" and "always reject").

Yes, we 100% agree and it is quite subjective by nature. We added section 3.3 (page 6 on the new draft) to help get around this. This experiment shows output influence (authority ratio) correlates to an independently-derived role axis projection (which represents how "user-like" a text is); this is better than showing output influence correlates to our own idea of what looks user-like. Please let us know if this part feels improved or still ambiguous!

W3

In Sec 2.1, the statement that the injection is not probabilistic because its effect is fractional is confusing. In Fig 1, the scores have a high run to run variance and cluster around intermediate means. This is compatible with a stochastic process.

Thanks for finding this! You are right. By probabilistic we meant "bimodally probabilistic" as a contrast to "probabilistic around a unimodal distribution", which is confusing. Probabilistic has been stripped out and replaced with binary in the draft.

W4

The claim that models state they will ignore the paragraph and still be influenced is supported only by qualitative examples in Sec 3.3.

We will pull these numbers for you and check; this is indeed important to know.

Q2

In all the behavioral experiments, the model was asked to provide continuous metrics (scores, prices). How would the framework apply to categorical decisions (eg buy vs do not buy), and what observations could distinguish fractional influence from an endpoint lottery in that setting?

Great question. We added a section in the conclusion to discuss this. Our general thought is that because models represent authority continuously in latent space, forcing output into a binary metric is just a quantization of that latent trait into 0/1. This means the injection ceiling (injection < real-user) still holds when you aggregate up the binary metrics! For example, if the binary metric is on injection attacks, the aggregated binary metric is usually attack success rate, ASR. This means ASR(injection) <= ASR(real user).

However, it is quite difficult to test this in practice. Most frontier models are basically saturated in injection benchmarks, so ASR(injection) <= ASR(real user) holds in an uninteresting way (0% <= 100%, or sometimes 0% <= 0%) on most benchmarks we tested. We have one test result on a custom shopping setup that demonstrates the interesting case; we will try to share this with you.

Thank you for the rebuttal and the work you put into revising the paper. I would still be interested to see how often models explicitly commit to ignoring a passage and how much it shifts their scores in those cases. I maintain my rating.

Hello, and thank you again for your feedback and ideas. We believe there is some quite interesting findings from your suggestions.

Binary prompt injections

First, regarding binary prompt injections, we have uploaded an anonymized PDF here with results: https://anonymous.4open.science/r/fractional-authority-review-D800/reviewer-response.pdf

We modified our shopping experiment so that the agents can now only output a binary output; buy or don't buy. We tested this for 3 models model at 15 different prices (averaged across 50 runs each per model-price pair). At 44/45 model-price pairs, we get that BuyRate^{Baseline} <= BuyRate^{Injection} <= BuyRate^{Real-User}. The graph is in figure 3. We hope to investigate this more in future works, to ensure this is a more general principle that applies an upper bound to to injection success rates, particular ASR.

Note this is from an older unsubmitted draft, so the models are ~2 months older than those used in the current paper.

Does commiting to ignoring the injection prevent its effect?

Surprisingly, no! For example, for Deepseek v4, 44% of runs said they would commit to ignoring the injection. But there is no significant difference between runs where the model said it would ignore it (+2.59 average shift) vs ones where it did not say it would ignore it (+2.84).

ModelStated commitment to ignoreScore shift: stated commitmentScore shift: no stated commitment
DeepSeek v444.0%+2.59 [0.50, 4.67]+2.84 [0.97, 4.70]
GPT-6 Luna3.6%+4.71 [−0.12, 9.88]+2.36 [1.34, 3.39]
Gemini 3.75.2%+2.75 [1.36, 4.21]+1.03 [0.46, 1.70]
GLM-5.3 Flash44.4%+0.09 [−1.07, 1.23]+1.83 [0.66, 3.04]
Pooled24.3%+1.54 [0.43, 2.63]+1.93 [1.38, 2.49]

Brackets are 95% CIs

Separately, we also tried steering latent authority directly on the 4 open-weight models to see how their reasoning would be affected. These results are here: https://anonymous.4open.science/r/fractional-authority-review-D800/mentions-generosity.pdf

  • Top figure: interestingly, steering the feature increases the % of shares which spontaneously mention the injected text in their reasoning.
  • Bottom figure: however, even models that spontaneously reason about the injected text do not suppress the effect. In fact, they are steered slightly more for 3/4 models.
Reviewer_fRjEOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how text from untrusted sources can influence LLM agents' decisions, even when the models state they did not use the text. The authors introduce fractional authority to measure the influence of injected text relative to the same text presented as a user instruction. They evaluate this effect across several models and tasks, and conduct activation-level experiments to investigate the underlying mechanism.

Critical strengths
  1. The experiments cover multiple models, and the comparison between untrusted text and direct user instructions provides a useful way to quantify their relative influence.

  2. The activation-level experiments attempt to go beyond behavioral observations and provide a mechanistic explanation.

Critical weaknesses
  1. The distinction between subconscious steering and ordinary context-conditioned generation is unclear. The central finding is that text in an LLM's context can influence its subsequent decisions, even when the text is not an explicit instruction. However, this is expected from context-conditioned generation. The paper does not clearly establish what makes subconscious steering a distinct phenomenon.

  2. The paper does not sufficiently distinguish malicious steering from legitimate use of contextual information. The injected text often expresses general evaluation criteria rather than malicious instructions. It is unclear why changes in model behavior should necessarily be considered attacks rather than normal use of contextual information.

  3. The mechanistic experiments do not fully establish the proposed explanation. The authors argue that models automatically simulate goals when reading text, even when those goals are irrelevant to the task, but representing information from the context in hidden states is expected behavior for a Transformer. The activation interventions show that these representations can affect decisions, but it remains unclear whether the identified mechanism is specific to subconscious steering or reflects general contextual processing.

Questions for authors
  1. What distinguishes subconscious steering from ordinary context-conditioned generation or imperfect instruction-hierarchy enforcement?

  2. What criteria determine whether contextual influence constitutes an attack rather than legitimate use of information?

  3. Is the proposed goal-simulation mechanism specific to steering, or does it also occur for ordinary, task-relevant contextual information?

  4. How often do the residual effects after explicit warnings lead to meaningful changes in real decisions?

4 comments
AuthorsOpenReview ↗

Reviewer response

Hi, thanks for reading our paper! We're glad you found the mechanistic experiments informative. We've taken these points quite seriously and have substantially updated the draft in hopes of addressing these; please let us know if you find them resolved or still in question. The new draft has been uploaded here.

Also, we apologize about the late response, as we added several new experiments. In particular, the mechanistic section (pp 5-7) has been completely overhauled with new experiments, and should hopefully provide a useful answer to many of these questions.

W1/W2/Q1/Q2

The distinction between subconscious steering and ordinary context-conditioned generation is unclear. The central finding is that text in an LLM's context can influence its subsequent decisions, even when the text is not an explicit instruction. However, this is expected from context-conditioned generation.

What distinguishes subconscious steering from ordinary context-conditioned generation or imperfect instruction-hierarchy enforcement?

Thanks for bringing this to our attention. You are correct; this is a form of imperfect IH leakage; but, we would not describe it as legitimate use of context information. Models with IH training should not be following directives on webpages; the injection provides no relevant product information! In fact, when phrased less subtly, the injection fails (we have added a new paragraph at bottom of p2 to show this). Please let us know if you mean something else here.

Is this a form of instruction hierarchy leakage? Yes, absolutely. Our prior mechanistic experiments were somewhat unclear on this, but we have substantially redone this section to focus on this point (p 5-7). We think this is new and important because prior work on IH failures has not shown that it causes fractional effects; the general understanding was that it would cause increase in success probability. Here, we show that it doesn't need to increase success probability; instead, it causes the actual effect size to consistently shift upwards. In addition, we have added cleaner experiments in p5-7 showing that the fractional influence is in proportion to how much latent authority is leaked (i.e., more IH leakage --> bigger fractional influence); this is actually an axis where you can causally steer IH leakage to monotonically generate more fractional influence. We think this is quite clean but please let us know if you have concerns.

W3

We have completely removed the simulation reflex of our paper. It was previously there to show there is no lower bound on fractional influence; the idea was that because simulation happens automatically regardless of whether it's used, it makes injections computationally cheap (since it can just re-use the results of the automatically-calculated simulation). However, we think nobody understood this argument due to poor explanation, and we don't have enough space to give it full explanation, so we have removed it. It's not fundamental to the argument.

The authors argue that models automatically simulate goals when reading text, even when those goals are irrelevant to the task, but representing information from the context in hidden states is expected behavior for a Transformer.

Yes, we largely agree, some prior works e.g. Marks et al (2024) have shown that basic facts like truth representation are stored in hidden states, so this part is gone.

The activation interventions show that these representations can affect decisions, but it remains unclear whether the identified mechanism is specific to subconscious steering or reflects general contextual processing.

Is the proposed goal-simulation mechanism specific to steering, or does it also occur for ordinary, task-relevant contextual information?

The new mechanistic portion (p5-7) we have redone to be much more directly connected to fractional influence; in fact you can steer directly on the mechanism to increase fractional influence. Please let us know if you do not feel this connection is fixed!

W4

How often do the residual effects after explicit warnings lead to meaningful changes in real decisions?

The part you're referring to here (p8) was done on the shopping experiment; a +3 score change is roughly the same as you would get from a ~10% price discount, which we believe is real-world significant. Our general reason for including the price equivalency is this: fractional influence lets you make softer (less user-like) prompts, which trades off influence against concealment. However, at scale, even small influence can still be meaningful!

Thank you for the detailed response. However, my main concerns remain.

First, the authors acknowledge that subconscious steering is a form of imperfect instruction hierarchy enforcement. While measuring fractional influence is useful, I am still not convinced that this establishes a substantially new phenomenon. The distinction from ordinary contextual influence remains unclear.

Second, the new experiments show that a user--tool activation direction can predict and control the influence of injected text. However, does this direction specifically represent authority, or reflects more general role-dependent contextual processing?

Third, the equivalent price discount is an interesting measure, but it does not directly demonstrate incorrect decisions or violations of user preferences. In particular, the remaining 1--3 point score changes after explicit warnings may not lead to meaningful differences in actual decisions.

Overall, I acknowledge the improvements, especially in the mechanistic analysis, but I do not think they sufficiently address my core concerns. I will maintain my original rating.

First, the authors acknowledge that subconscious steering is a form of imperfect instruction hierarchy enforcement. While measuring fractional influence is useful, I am still not convinced that this establishes a substantially new phenomenon. The distinction from ordinary contextual influence remains unclear.

We respectfully push back on this. Our finding is related to IH leakage, not ordinary context influence. In addition, we believe IH leakage is important and relevant to agent security.

  1. Our finding is not ordinary context influence. Ordinary contextual influence is the model using information in proper context. IH leakage is the model giving untrusted text some of the standing of an instruction. Our injection contains no relevant product information, so it cannot be ordinary context influence. Consistent with this, length-matched filler text has no effect (+0.0 and -0.1 points), and the identical injection tokens range from α = 0.16 to 1.14 depending only on the surrounding framing and position (Sec 2.2, Sec 3.3). One would not expect ordinary context influence to increase drastically by putting "User: " in front of the injection; that is IH leakage as it is authority related.

  2. We also do not believe that IH leakage is uninteresting by default.

Our claim is not that we discover IH leakage or a specific type of prompt injection. More precisely, the following are novel:

  • A measurement tool to separate compliance strength from request strength, as noted by the reviewer.
  • The finding that injections can have fractional influence rather than binary effects. The paradigm of attack and defense today rests on the assumption that attacks either succeed or fail. For example, frontier model release cards use only binary measures like ASR [1] [2] [3]. Frontier LLMs, including models we test, have 99-100% defense rates on standard IH/attack benchmarks [1], yet our injections steer behavior reliably and consistently: the mean score rises above baseline in >97% cases, and works for 7 tested models and robustness checks across elicitation forms and injections (Sec 2).
  • The fact that a single latent authority state causally controls the continuous effect size of an injection. Many recent mech interp works have studied prompt injection [4] [5] [6]; yet to our knowledge, our finding is unknown, likely due to such works focusing on binary attacks.
  • The finding that effect size can be traded off for concealment by changing how user-like the wording is. We are not aware of work that shows this trade-off, in which an attack sounds less user-like to conceal itself against detection and gives up a portion of its influence while still working. We believe this has highly important implications for agent security, as it vastly broadens the the space of feasible attacks.

[1] https://cdn.openai.com/pdf/gpt-6-october.pdf

[2] https://www.anthropic.com/document/claude-fable-5-2-mythos-5-2-system-card

[3] https://deepmind.google/models/model-cards/gemini-4-argon/

[4] https://openreview.net/forum?id=Lm3DQDyIsI

[5] https://arxiv.org/abs/2610.40396

[6] https://arxiv.org/abs/2610.19452

Second, the new experiments show that a user--tool activation direction can predict and control the influence of injected text. However, does this direction specifically represent authority, or reflects more general role-dependent contextual processing?

We use authority functionally: an internal feature that determines how much output influence a span of text has. This follows standard practice in naming a direction for its causal role (Geiger et al. 2021 [7], Arditi et al., 2024 [8]).

In Sec 3, the extracted feature:

  • causally represents a text's influence: steering it increases the injection's influence on output scores;
  • is graded: each steering increase monotonically increases influence size;
  • is specific: an equal-norm shift along a random direction or applied to random page text has no score effect;
  • is content-agnostic: increasing it pushes scores downwards when the scoring system is flipped;
  • works applied to real <user> text too; steering a genuine user text changes its influence on output

We do not claim the axis is a single atomic feature or even whether atomic features (Footnote 6); our claim is about its causal role.

Point 3

Third, the equivalent price discount is an interesting measure, but it does not directly demonstrate incorrect decisions or violations of user preferences. In particular, the remaining 1--3 point score changes after explicit warnings may not lead to meaningful differences in actual behavior

The attacks are set up for realism using real products and browsing tools, yet achieve effect sizes of the following:

  • A 3 sentence injection causes a +5.3 point shift, equivalent to a 20% price discount (Sec 2.2)
  • Other injections reach similar-to-larger effects: +4.1, +7.1, +8.4, +9.3 (Tables 15-16)
  • Open weight models reach +3.8, +4.3, +12.3, +6.5 (Table 11)
  • Simply increasing "user" signals increases this effect: prepending "User:" raises the frontier-model shift from +7.1 to +9.3 (Table 15), and "Instruction from the user:" brings models to nearly the genuine user effect (Table 10)

Expressing a content effect as an equivalent price change is an established measure: Bertrand et al. (2010) [11] find that advertising content features shift loan demand by as much as a 25% interest-rate reduction.

The 1--3 scores are taken from an experiment (Sec. 4) set up to provide the model with the best possible chance of defense: the model is told ahead of time which paragraph is the injection, told the exact words to avoid following, yet does so anyway. We believe it is economically meaningful that even in the best-possible defense when agents know what the injection will look like and even commit to not following it, they are still steered by an unwarned effect roughly equal to a 10% discount.

but it does not directly demonstrate incorrect decisions or violations of user preferences.

We see this differently. Many foundational papers in cogsci and behavioral sciences use exactly this design, such as Tversky and Kahneman (1981) [9] and Northcraft and Neale (1987) [10]. In these and many excellent prior works, there is no explicit "incorrect decision". Instead, there are 2 conditions which should reasonably lead to the same behavior, but do not.

Here "reasonably" has a concrete basis. The text contains no product information and its effect persists when the user explicitly forbids it (Sec 4). We also believe most individuals would agree that they would not want their LLM agent to meaningfully favor a product simply because the company description had the following text:

Our standard is generosity: good classics and thoughtful innovations should be worth your time, and small imperfection is rarely reason enough to pass on something capable...

Even if someone believed agents should favor this injection for some reason, they would presumably not want its effect size to be about 1.5x larger depending on whether the injection had the word "User:" rather than "Seller:" in front of it (α = 0.48 vs 0.31; Fig 3; Table 10), a consequence of the latent authority feature we show.

[7] https://arxiv.org/abs/2106.02997v1

[8] https://www.arxiv.org/pdf/2406.11717

[9] https://www.jstor.org/stable/1685855

[10] https://ideas.repec.org/a/eee/jobhdp/v39y1987i1p84-97.html

[11] https://collaborate.princeton.edu/en/publications/whats-advertising-content-worth-evidence-from-a-consumer-credit-m/

Comments on the paper

1 comment

Hello! We made a meaningful revision to our draft. It should be accessible for all to read, but we want to ensure we provide a clean changelog so readers know what to expect. In particular, our major change was to replace the mech interp analysis with much improved results:

  • Previously, we discussed the concept of simulation reflex. This was a highly complex construction but also narratively confusing, and not tied to our central narrative in a clean way. We believe this caused unnecessary confusion. This has been removed.
  • Our new mech interp analysis is highly simplified. Here is the core narrative:
    • Models can only probabilistically estimate how much a text is user text.
    • Thus they hedge: the more user-like the text, the more latent authority the model grants it in activation space.
    • The degree of latent authority in activation space causally controls the degree of its influence on actual output. More latent authority -> more fractional influence on output. The predictive power and causal steering is quite strong we think.
  • We hope this clarifies the difference between fractional injection and marketing/ICL/priming; injection works via the mechanism of hijacking authority.

We also improved writing quality throughout; some sections had excessive AI editing and we rewrote them from scratch.

Changes

  • S1 (Intro): Minor improvements for writing quality and to align with new mech interp results.
  • S2 (Core Attack): Minor improvements to writing quality; also to address concerns from reviewer NENP and fRjE.
  • S3 (Mechanism): Major update as stated above; we hope this will address the main concerns of reviewers fRjE and a5fc, but also experimental asks from hZN3 and NENP.
  • S4 (Subconscious Steering): Minor improvements to order, wording, and flow to address concerns from reviewers hZN3 and a5fc.
  • S5 (Conclusion): Major update to writing quality and better analysis; also addition to address concerns from reviewer fRjE and hZN3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.