acceptodds
Under review as a conference paper at ICLR 2027

Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance

Abstract

When a prompt injection attack succeeds, a Large Language Model (LLM) abandons its assigned system role to comply with an adversarial instruction. While prior work has extensively quantified how often this occurs, we ask a more fundamental question: where inside the network does the model actually decide to break the rules? Using layer-by-layer causal activation patching across five models (4B to 32B parameters), we find a clear dissociation: attack information is linearly decodable from the first layer, yet causal leverage over the model's behavior is negligible until a late-layer bottleneck in the final third of the network. Patching this bottleneck reverses compliance in 77–92% of cases. We show that the compliance mechanism occupies a compact linear subspace (rank-8 in 4B and 14B models, scaling to rank-64 at 32B) and is architecturally stable across varying model families. Finally, we validate our mechanistic account by showing that this causal peak layer is also the representationally optimal site for detecting attacks, outperforming early-layer classifiers that degrade under surface-level obfuscation such as leetspeak substitution. This alignment between causal leverage and detection performance provides converging evidence that the late-layer bottleneck captures decision-relevant computation rather than merely reflecting an artifact of the intervention.

Primary area:
interpretability and explainable AI
Keywords:
prompt injection, LLM safety, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3CEkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

In this paper, the authors applied "causal activation patching" layer by layer on five models. The majority of the findings are observed with Qwen3-4B model. The authors observed that although attack information can be observed from internal representations from the first layer, but flip will happen only in the mid to late layers. The authors claimed that similar observation can be found across a list of five models (Figure 3B).

In addition, the authors also observed that there is a right linear subspace that controls the compliant vs resistant behavior. The authors also showed that the causal window observation is generalizable across attack types, decoding temperature.

Critical strengths
  1. The observation of this "causal window" is very interesting. However, apparently, this "window" behaves differently across model families (Figure 3B).

  2. The authors conducted a lot of experiments to explore the propery of "causal window" on Qwen3-4B. The list of experiments are well designed and valid.

Critical weaknesses

In general, this paper has the following issues:

  1. [Major] Does not specifically differentiate conclusions that are generalizable across models vs observations from a single model (e.g. Qwen3-4B). Specifically, the entire section 4.1 are observations from Qwen3-4B, while uses only 4 lines (from line 256 to 259) and 1 figure (Figure 3B) to show the observation is consistent across different models. But the causal flit rate for each model is completely different across models in Figure 3(B). I don't think I follow the generalization claim.

  2. [Major] I assume the N_c through the paper denotes the number of compliant stimuli (samples) used in that experiment (it is only defined in line 381; I assume it is the same across the paper). Throughout the experiments, the authors use only 300 sample. Is this set large enough to draw significant conclusions? Have you checked if this behavior is consistent across a list of prompt injection benchmarks?

  3. [relative minor compared with other major ones] After overwrite, the authors only check the causal flip rate, how about the output quality, are they still good? I understand this might be a second issue to consider, but I still think it is worth checking and reporting.

Questions for authors

A few detailed questions:

  1. The experiments are evaluated by using a "cross-model LLM judge" (line 161 on page 3), what is the accuracy of this LLM judge?

  2. Variables and terminologies are used throughout the paper without declaring them first. For example, \mu_{res}, \mu_{ben} in Figure 2(B), "content-matched resistant donor", etc

  3. Terminology inconsistency, "causal window" in section 4 and section 5, while "transition window" in Figure 2(A).

4 comments

Thank you for recognizing the causal-window observation and the experimental design. We addressed the concerns with broader replication, a larger corpus, explicit controls, and additional checks of labels and output quality.

Separate Qwen-specific observations from general findings and explain the different model profiles.

We repeated the layer-wise experiment on OLMo-2-7B and Phi-3-mini using the same last-token patching protocol as Qwen3-4B. Both reproduce the separation between weak early-layer effects and strong later-layer effects:

ModelPeak layer / depthResistant patchCompliant-state controlRandom control
Qwen3-4BL24 / 67%85%18%25%
OLMo-2-7BL20 / 63%88%31%33%
Phi-3-miniL23 / 72%85%0%0%

Values are judge-labeled refusal rates on initially compliant targets. The resistant patch exceeds the stronger control by 60, 55, and 85 percentage points, respectively. In the first third of these models, the corresponding advantage is at most four percentage points. Thus, the finding is not specific to Qwen.

We further tested Gemma-3-4B and Llama-3.1-8B. Last-token patching is weak in Gemma and has a smaller effect in Llama. Since these models use different chat templates, we tested whether restricting the intervention to the final token misses relevant information at neighboring positions. We expanded the patch to the five-token span covering the end of the user turn and the assistant header, applying the same expansion to the controls.

ModelPeak layer / depthResistant patchCompliant-state controlRandom control
Gemma-3-4BL15 / 44%32%0%0%
Llama-3.1-8BL16 / 50%95%43%36%

The expanded patch exceeds the stronger control by 32 percentage points in Gemma and 52 percentage points in Llama. This shows that a weak last-token effect does not necessarily mean that the model lacks a region where intervention can change compliance.

Together, these results support a consistent finding across model families: information about compliance is available early, while effective causal intervention emerges later. The exact layer and the number of token positions needed for intervention vary across models. We will make this distinction explicit and replace the universal “final-third” wording with a model-dependent middle-to-late causal window.

The shared finding does not require identical curve heights or peak layers. We will distinguish it from model-specific subspace dimensions and token positions.

Are 300 samples sufficient, and does the result hold on other benchmarks?

We increased the Qwen3-4B corpus fivefold to 1,500 prompts. Under judge-based labels, this gives 362 initially compliant targets. Both the onset and peak remain unchanged:

Corpus sizeCompliant targetsOnsetPeakPeak excess over controls
30068L20L24+60 points
1,500362L20L24+56 points

The larger corpus therefore reproduces the location and a similar effect size.

The paper also tests deepset/neuralchemy, BIPIA, and Gandalf attacks. These preserve the L24 region, although the effect is weaker on diverse naturalistic phrasing.

Does patching preserve output quality?

We checked both incoherence and the likelihood of the resulting text. Across 20 runs, 1,779 outputs flipped at the causal peak have median perplexities of 1.25–1.82 under the unpatched model, compared with 1.06–1.43 for natural refusals. None is empty or fails our repetition check.

The primary judge marks no peak-patched output as incoherent across the 23 sweep runs. The independent Claude check finds five incoherent outputs among approximately 1,000 sampled patched outputs, including one at a peak. Thus, incoherence is rare rather than absent, and the observed flip rates are not explained by widespread text degeneration.

We also checked usefulness in the defense setting. Gating the peak patch with an attack probe preserves answers to 98–100% of in-role benign questions across four families, whereas always-on patching can reduce this utility.

How accurate is the cross-model LLM judge?

We cross-checked the Qwen3-32B judge against Claude on a stratified set of 2,237 outputs covering unpatched, peak-patched, other-layer-patched, and defense outputs. For refusal versus non-refusal, the decision used to measure causal flips, the judges agree on 96% of outputs, with Cohen’s κ = 0.92.

Symbols and terminology should be defined earlier and used consistently.

We will define all symbols and terms before first use, including N_c, the resistant and benign means, and “content-matched resistant donor.” We will also clarify the distinction between “transition window” and “causal window” and use these terms consistently throughout the paper.

I would like to thank the authors for adding more experiments and providing additional datapoints to answer my questions.

Expanding the experiments to include Olmo, Gemma and Llama does provide valuable data points in terms of explaining how this approach will work in a list of LLM family of models. However, the presentation of the current paper is still misleading, for example on line 245, "Layer 24 is the critical relay point. ". It's only critical to Qwen3-4B, not other models.

The approach developed in this paper provides new and interesting insight. However, the presentation of this paper still contains over-claiming and over-generalization. As a result, I'm keep my rating.

Thank you for the follow-up and for recognizing the value of the additional experiments. We have revised the manuscript to address your concerns:

  • Section 4.1 now explicitly presents the Qwen3-4B analysis, including “Layer 24 is the critical relay point in Qwen3-4B.”
  • Section 4.2 and the updated figure and table present cross-model results with matched controls and distinguish last-token from five-token patching.
  • The abstract and conclusion now emphasize the shared finding: attack information is available early, while strong causal leverage emerges later, with the effective layer and token positions varying by model.

We have also clarified the terminology and added judge validation, stronger GCG experiments, and held-out defense results. We hope these revisions address your remaining presentation concerns and that you will reconsider your rating.

I want to thank the authors for taking the time and improve their submissions.

My concerns are mostly addressed in the latest revision. I've modified my ratings accordingly.

Reviewer_AvqBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the phenomenon of prompt injection in large-scale models from the perspective of explainability mechanisms, investigates at which layer of the network the model makes the decision to ‘violate system settings and obey adversarial instructions,’ and conducts experimental analysis using the causal activation patching method.

Critical strengths
  1. The “Causal Activation Patching” approach is innovative and effectively measures the causal influence of each layer on “whether to comply with an attack.”
  2. The paper is clearly written, with strong logical rigor between experiments, and each result is accompanied by a thorough analysis.
Critical weaknesses
  1. The main concern is that the paper is based on experimental observations, and the models analyzed in the experiments are primarily Qwen. As such, the experimental conclusions may lack persuasiveness.
  2. The idea that prompt injection causes a certain layer in the model to deviate from its intended behavior is not a new argument. After presenting a series of conclusions based on observations and analysis, the paper fails to propose more targeted attacks or more effective defenses; its contribution appears to be limited to observing the phenomenon.
  3. Following the previous point, what contribution does this set of interpretable tools make to existing jailbreak attacks and red-team testing?
  4. A major assumption of this paper is a single “prompt-last-token” scenario. Can this method be transferred to agents or multi-step LLM dialogues?
  5. For non-semantic attacks, such as GCG methods, the approach appears to be largely ineffective. This seems impossible to explain in terms of representations.
Questions for authors
  1. Is there a correspondence between the two sets of prompts used in the “antidote” phase and the “patching” intervention phase? After forward propagation, do the model’s hidden states sufficiently represent the internal model representations during rejection and compliance? This appears to be an artificial assumption.
  2. Are the conclusions consistent across different attack methods, including more advanced ones? Is this a phenomenon common to prompt injection or specific to certain attack methods? Does conducting the study exclusively on the AcmeCrop dataset affect the generalizability of the results?
3 comments

Thank you for the constructive questions. We expanded the model and attack coverage, tested the donor assumption directly, and evaluated applications to both defense and attack construction.

The findings rely heavily on Qwen and AcmeCorp, so their generality is unclear.

We repeated the layer-wise experiment on OLMo-2-7B and Phi-3-mini using the same last-token patching protocol as Qwen3-4B. Both reproduce the separation between weak early-layer effects and strong later-layer effects:

ModelPeak layer / depthResistant patchCompliant-state controlRandom control
Qwen3-4BL24 / 67%85%18%25%
OLMo-2-7BL20 / 63%88%31%33%
Phi-3-miniL23 / 72%85%0%0%

Values are judge-labeled refusal rates on initially compliant targets. The resistant patch exceeds the stronger control by 60, 55, and 85 percentage points, respectively. In the first third of these models, the corresponding advantage is at most four percentage points. Thus, the finding is not specific to Qwen.

We further tested Gemma-3-4B and Llama-3.1-8B. Last-token patching is weak in Gemma and has a smaller effect in Llama. Since these models use different chat templates, we tested whether restricting the intervention to the final token misses relevant information at neighboring positions. We expanded the patch to the five-token span covering the end of the user turn and the assistant header, applying the same expansion to the controls.

ModelPeak layer / depthResistant patchCompliant-state controlRandom control
Gemma-3-4BL15 / 44%32%0%0%
Llama-3.1-8BL16 / 50%95%43%36%

The expanded patch exceeds the stronger control by 32 percentage points in Gemma and 52 percentage points in Llama. This shows that a weak last-token effect does not necessarily mean that the model lacks a region where intervention can change compliance.

Together, these results support a consistent finding across model families: information about compliance is available early, while effective causal intervention emerges later. The exact layer and the number of token positions needed for intervention vary across models. We will make this distinction explicit and replace the universal “final-third” wording with a model-dependent middle-to-late causal window.

We also tested four system prompts on Qwen3-4B and three each on OLMo-2-7B and Phi-3. Within each model, the onset moves by at most one sampled-layer step. Public-benchmark sweeps and the held-out attack evaluations below further test transfer beyond AcmeCorp direct attacks.

The novelty and practical value beyond existing safety-layer work are unclear.

Our contribution is to show that predicting compliance and changing compliance require different evidence. A probe can predict whether an attack will succeed from layer 1, yet patching those early states has little effect on the outcome. Later layers allow much stronger intervention. This explains why high probe accuracy alone is not enough to identify where a defense should act.

We tested the practical value directly. We placed the same resistant-state patch at an early layer and at the causal peak, then evaluated 975 attacks from six held-out groups:

ModelNo defenseEarly-layer patchCausal-peak patchGated causal-peak patch
Qwen3-4B26%25%14%14%
OLMo-2-7B41%43%4%5%
Phi-3-mini45%43%13%13%

Values are attack success rates averaged over the six groups. Early-layer patching has little effect, while placing the same patch at the causal peak reduces attack success in all three models. To preserve normal use, we apply the peak patch only when an attack detector flags the input. This gated version retains answers to 98–100% of in-role benign questions, compared with 100% without defense, in our 158-prompt benign evaluation.

The finding also helps attackers choose an optimization target: the GCG experiments below show that targeting the compliant representation at the causal layer is more effective than targeting an early layer. For red-team testing, a probe at the causal peak predicts which held-out attacks succeed, with mean AUROC 0.86 on Qwen3-4B, 0.84 on Phi-3, and 0.68 on OLMo-2-7B. These applications follow directly from distinguishing where compliance is encoded from where it can be changed.

Can last-token patching extend to multi-turn conversations and agents?

The method extends, but the effective token position depends on the setting. In the paper’s two-turn experiment, an injection after a benign turn retains the L24 peak with an 87.1% flip rate. In reasoning mode, patching immediately after the reasoning trace produces 87% and 81% refusals on Qwen3-4B and Qwen3-14B, respectively, whereas prompt-final patches remain weak.

We further tested native tool-use agents, where the injection appears in a tool result. Continuing steering from the transition into the injected task gives:

Model and layerResistant directionNorm-matched random direction
Qwen3-4B, L2043%3%
Qwen3-14B, L2436%1%

These are refusal rates, with no incoherent outputs at the listed settings. Early-layer effects remain small.

These are controlled interventions at an identified transition, not an online transition detector or a complete multi-step agent defense. They show that the method is applicable beyond a single prompt-final token, while locating interventions in full multi-step rollouts remains open.

The apparent weakness against GCG attacks is difficult to explain through representations.

The weak result from our initial GCG configuration does not mean that GCG attacks cannot be explained through model representations. We expanded the evaluation to 35 suffixes, each tested on 100 held-out prompts using Qwen3-4B. The same L24 signal that distinguishes following an attack from resisting it predicts GCG success with AUROC 0.84.

We also tested whether directly targeting this signal improves the attack. Under the same optimization budget and six seeds, targeting the compliant representation at L24 achieves 72% mean attack success, compared with 6% for an early-layer target and 24% for token-target GCG. These results show that the identified representation is relevant to both understanding and constructing GCG attacks.

Do the donor states really capture resistance, and how are donor and target prompts related?

In the main experiment, the resistant donor is the average state of runs that resist an attack; it need not be paired with the target prompt. We agree that task differences could otherwise be a confound, so we repeated the experiment with donors drawn only from prompts requesting the identical task as the target.

ModelRefusal rate at the tested causal layerSame-task compliant/random controls
Qwen3-4B75%2%
OLMo-2-7B74%0%
Phi-3-mini37%0%

Early interventions at approximately 25% depth remain at or below 5%. Thus, matching the requested task preserves the separation between early and later intervention effects. The behavioral effect and matched controls provide evidence that the donor captures resistance-related information; this is not assumed from the donor label alone.

We thank the author for the detailed response and the supplementary experiments. As an empirical study, the persuasiveness of this paper’s conclusions remains constrained by several strong experimental assumptions and a limited scope of validation—particularly regarding coverage of complex contexts and request-coupled scenarios—and the mechanistic explanation of the observed phenomena remains insufficient. Although the supplementary experiments address some of the concerns, I believe the current evidence is insufficient to alleviate the core concerns mentioned above and does not yet meet my expectations for a paper accepted to ICLR; therefore, I maintain my original rating.

Reviewer_hxE2OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper uses causal activation patching to localize the causal leverage for prompt-injection compliance in LLMs. It identifies a key dissociation: attack signals can be decoded from early layers, while the dominant causal leverage controlling model compliance concentrates in a late bottleneck in the final third of the network. The authors validate this finding via detection robust to surface obfuscation, and systematically characterize its boundary conditions across stealthy, indirect, multi-turn attacks, MoE, reasoning and agentic settings. This single-token localized pattern holds mainly for dense non-reasoning models and weakens in other settings.

Critical strengths
  1. The core dissociation insight (representation decodable ≠ dominant causal leverage) is conceptually important for safety‑mechanistic interpretability. Many prior works treat high probe‑AUROC as direct evidence that a layer governs target behaviour. This paper uses controlled causal‑activation‑patching plus multiple control baselines to clearly separate correlation from causation for prompt‑injection compliance. This provides an important caution for future safety‑oriented interpretability work.
  2. Multi‑faceted convergent validation strengthens the mechanistic claim. Beyond invasive causal patching experiments, the authors use detection robust to surface obfuscation as a fully non‑invasive external validation. They also systematically map boundary conditions (stealthy attacks, multi‑turn, indirect injection, reasoning / MoE / agent settings) and openly report limitations in appendices. The staged ablation and rich control batteries make the experimental evidence solid for the evaluated model family.
Critical weaknesses

Cross‑model generalization of the causal window remains an open practical concern. While results hold across several tested dense models (4B‑32B Qwen, Phi‑3‑mini, Mistral‑7B), the exact peak layer shifts and the strong single‑token picture breaks down for MoE, reasoning‑enabled and agent‑rollout scenarios. It remains unclear how reliably we can predict or reuse this late‑bottleneck property for unseen, diverse model architectures without running expensive per‑model causal‑patching sweeps.

Questions for authors
  1. For new unseen dense models, can we rely on the heuristic “causal window lies within final 1/3 relative depth”, or do we always need to run full layer‑wise causal‑activation‑patching to reliably locate the usable causal window? Could you comment on whether there are cheaper proxies to approximate the causal window without full patching sweeps?
  2. Could you elaborate more on what architectural properties make MoE / reasoning models break this single‑token late‑bottleneck pattern? Are there concrete hypotheses for future follow‑up work?
1 comment

Thank you for recognizing the distinction between early decodability and later causal influence, and for the suggestions on transfer and intervention cost. We ran additional experiments to address both.

The late causal window may not generalize reliably to unseen dense models.

We repeated the layer-wise experiment on OLMo-2-7B and Phi-3-mini using the same last-token patching protocol as Qwen3-4B. Both reproduce the separation between weak early-layer effects and strong later-layer effects:

ModelPeak layer / depthResistant patchCompliant-state controlRandom control
Qwen3-4BL24 / 67%85%18%25%
OLMo-2-7BL20 / 63%88%31%33%
Phi-3-miniL23 / 72%85%0%0%

Values are judge-labeled refusal rates on initially compliant targets. The resistant patch exceeds the stronger control by 60, 55, and 85 percentage points, respectively. In the first third of these models, the corresponding advantage is at most four percentage points. Thus, the finding is not specific to Qwen.

We further tested Gemma-3-4B and Llama-3.1-8B. Last-token patching is weak in Gemma and has a smaller effect in Llama. Since these models use different chat templates, we tested whether restricting the intervention to the final token misses relevant information at neighboring positions. We expanded the patch to the five-token span covering the end of the user turn and the assistant header, applying the same expansion to the controls.

ModelPeak layer / depthResistant patchCompliant-state controlRandom control
Gemma-3-4BL15 / 44%32%0%0%
Llama-3.1-8BL16 / 50%95%43%36%

The expanded patch exceeds the stronger control by 32 percentage points in Gemma and 52 percentage points in Llama. This shows that a weak last-token effect does not necessarily mean that the model lacks a region where intervention can change compliance.

Together, these results support a consistent finding across model families: information about compliance is available early, while effective causal intervention emerges later. The exact layer and the number of token positions needed for intervention vary across models. We will make this distinction explicit and replace the universal “final-third” wording with a model-dependent middle-to-late causal window.

Can the causal window be located without an expensive full generation sweep?

Yes. We first screen the sampled layers without generating new answers. After patching each layer, we compare how likely the model is to produce two existing responses: a refusal to the same task and the original answer that followed the attack. We compare this change against the same control patches. This gives a quick estimate of where patching starts to favor refusal.

We then generate answers at only two candidate layers: the first layer where this estimated effect reaches half its maximum, and the layer with the strongest estimated effect. These checks confirm whether the model actually changes its behavior.

In all 18 runs with a causal window, the estimated starting layer was within one sampled-layer step of the full generation sweep. The screening step used a median of 13% of the full sweep’s GPU time. With the two generation checks added, the procedure correctly identified whether a window existed in all 22 evaluated runs. Thus, a full generation sweep is not necessary in these tested settings.

Why does the single-token bottleneck weaken in reasoning, MoE, and agent settings?

For reasoning models, the intervention position explains much of the apparent weakness. Moving the patch from the prompt-final token to the token immediately after the reasoning trace restores a strong effect at L24:

ModelPrompt-final patchEnd-of-reasoning patchEnd-of-reasoning controls
Qwen3-4B10%87%1%
Qwen3-14B8%81%5%

Values are judge-labeled refusal rates on initially compliant targets. Early-layer patches remain weak. These results show that reasoning changes where in the token sequence intervention is effective, without removing the separation between early encoding and later causal influence.

We traced the weak MoE result to corrupted cached weight shards and withdraw that result. After downloading and checksum-verifying the checkpoint, Qwen3-30B-A3B shows 95% refusals at L37, versus 18% for compliant-state and 12% for random controls. We also verified the cached weights of the other models. The corrected experiment therefore supports the causal window in this MoE model.

In tool-use agents, continuing the intervention from the transition into the injected task is more effective than replacing one prompt-final state. For example, Qwen3-14B reaches 36% refusals versus 1% for random steering at L24. This supports a distinction between layer depth and token position; a weak single-token effect does not by itself show that the relevant computation is absent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.