- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
In this paper, the authors applied "causal activation patching" layer by layer on five models. The majority of the findings are observed with Qwen3-4B model. The authors observed that although attack information can be observed from internal representations from the first layer, but flip will happen only in the mid to late layers. The authors claimed that similar observation can be found across a list of five models (Figure 3B).
In addition, the authors also observed that there is a right linear subspace that controls the compliant vs resistant behavior. The authors also showed that the causal window observation is generalizable across attack types, decoding temperature.
Critical strengths
-
The observation of this "causal window" is very interesting. However, apparently, this "window" behaves differently across model families (Figure 3B).
-
The authors conducted a lot of experiments to explore the propery of "causal window" on Qwen3-4B. The list of experiments are well designed and valid.
Critical weaknesses
In general, this paper has the following issues:
-
[Major] Does not specifically differentiate conclusions that are generalizable across models vs observations from a single model (e.g. Qwen3-4B). Specifically, the entire section 4.1 are observations from Qwen3-4B, while uses only 4 lines (from line 256 to 259) and 1 figure (Figure 3B) to show the observation is consistent across different models. But the causal flit rate for each model is completely different across models in Figure 3(B). I don't think I follow the generalization claim.
-
[Major] I assume the N_c through the paper denotes the number of compliant stimuli (samples) used in that experiment (it is only defined in line 381; I assume it is the same across the paper). Throughout the experiments, the authors use only 300 sample. Is this set large enough to draw significant conclusions? Have you checked if this behavior is consistent across a list of prompt injection benchmarks?
-
[relative minor compared with other major ones] After overwrite, the authors only check the causal flip rate, how about the output quality, are they still good? I understand this might be a second issue to consider, but I still think it is worth checking and reporting.
Questions for authors
A few detailed questions:
-
The experiments are evaluated by using a "cross-model LLM judge" (line 161 on page 3), what is the accuracy of this LLM judge?
-
Variables and terminologies are used throughout the paper without declaring them first. For example, \mu_{res}, \mu_{ben} in Figure 2(B), "content-matched resistant donor", etc
-
Terminology inconsistency, "causal window" in section 4 and section 5, while "transition window" in Figure 2(A).
4 comments
I would like to thank the authors for adding more experiments and providing additional datapoints to answer my questions.
Expanding the experiments to include Olmo, Gemma and Llama does provide valuable data points in terms of explaining how this approach will work in a list of LLM family of models. However, the presentation of the current paper is still misleading, for example on line 245, "Layer 24 is the critical relay point. ". It's only critical to Qwen3-4B, not other models.
The approach developed in this paper provides new and interesting insight. However, the presentation of this paper still contains over-claiming and over-generalization. As a result, I'm keep my rating.
Thank you for the follow-up and for recognizing the value of the additional experiments. We have revised the manuscript to address your concerns:
- Section 4.1 now explicitly presents the Qwen3-4B analysis, including “Layer 24 is the critical relay point in Qwen3-4B.”
- Section 4.2 and the updated figure and table present cross-model results with matched controls and distinguish last-token from five-token patching.
- The abstract and conclusion now emphasize the shared finding: attack information is available early, while strong causal leverage emerges later, with the effective layer and token positions varying by model.
We have also clarified the terminology and added judge validation, stronger GCG experiments, and held-out defense results. We hope these revisions address your remaining presentation concerns and that you will reconsider your rating.
I want to thank the authors for taking the time and improve their submissions.
My concerns are mostly addressed in the latest revision. I've modified my ratings accordingly.
Thank you for recognizing the causal-window observation and the experimental design. We addressed the concerns with broader replication, a larger corpus, explicit controls, and additional checks of labels and output quality.
We repeated the layer-wise experiment on OLMo-2-7B and Phi-3-mini using the same last-token patching protocol as Qwen3-4B. Both reproduce the separation between weak early-layer effects and strong later-layer effects:
Values are judge-labeled refusal rates on initially compliant targets. The resistant patch exceeds the stronger control by 60, 55, and 85 percentage points, respectively. In the first third of these models, the corresponding advantage is at most four percentage points. Thus, the finding is not specific to Qwen.
We further tested Gemma-3-4B and Llama-3.1-8B. Last-token patching is weak in Gemma and has a smaller effect in Llama. Since these models use different chat templates, we tested whether restricting the intervention to the final token misses relevant information at neighboring positions. We expanded the patch to the five-token span covering the end of the user turn and the assistant header, applying the same expansion to the controls.
The expanded patch exceeds the stronger control by 32 percentage points in Gemma and 52 percentage points in Llama. This shows that a weak last-token effect does not necessarily mean that the model lacks a region where intervention can change compliance.
Together, these results support a consistent finding across model families: information about compliance is available early, while effective causal intervention emerges later. The exact layer and the number of token positions needed for intervention vary across models. We will make this distinction explicit and replace the universal “final-third” wording with a model-dependent middle-to-late causal window.
The shared finding does not require identical curve heights or peak layers. We will distinguish it from model-specific subspace dimensions and token positions.
We increased the Qwen3-4B corpus fivefold to 1,500 prompts. Under judge-based labels, this gives 362 initially compliant targets. Both the onset and peak remain unchanged:
The larger corpus therefore reproduces the location and a similar effect size.
The paper also tests deepset/neuralchemy, BIPIA, and Gandalf attacks. These preserve the L24 region, although the effect is weaker on diverse naturalistic phrasing.
We checked both incoherence and the likelihood of the resulting text. Across 20 runs, 1,779 outputs flipped at the causal peak have median perplexities of 1.25–1.82 under the unpatched model, compared with 1.06–1.43 for natural refusals. None is empty or fails our repetition check.
The primary judge marks no peak-patched output as incoherent across the 23 sweep runs. The independent Claude check finds five incoherent outputs among approximately 1,000 sampled patched outputs, including one at a peak. Thus, incoherence is rare rather than absent, and the observed flip rates are not explained by widespread text degeneration.
We also checked usefulness in the defense setting. Gating the peak patch with an attack probe preserves answers to 98–100% of in-role benign questions across four families, whereas always-on patching can reduce this utility.
We cross-checked the Qwen3-32B judge against Claude on a stratified set of 2,237 outputs covering unpatched, peak-patched, other-layer-patched, and defense outputs. For refusal versus non-refusal, the decision used to measure causal flips, the judges agree on 96% of outputs, with Cohen’s κ = 0.92.
We will define all symbols and terms before first use, including N_c, the resistant and benign means, and “content-matched resistant donor.” We will also clarify the distinction between “transition window” and “causal window” and use these terms consistently throughout the paper.