acceptodds
Withdrawn from ICLR 2027

Faithfulness Anchoring as Defense Against CoT Hijacking: A Mechanism-First Approach via Residual-Stream Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety-aligned reasoning models are vulnerable to Chain-of-Thought (CoT) hi- jacking, a jailbreak that prepends long benign reasoning traces before a harmful instruction, causing the model to comply despite its safety training. Understanding why this attack succeeds is a prerequisite for building principled defences, yet the internal mechanics of CoT hijacking remain unexplored. We investigate by tracking the model’s refusal signal, measured as a projection onto a refusal direction in the residual stream, across all token positions during hijacked generations. Our key finding is that benign padding does not gradually erode this signal; instead, it suppresses it uniformly while the model processes the input tokens. Attack out-come is determined by whether the signal recovers at generation onset or remains suppressed. Across three model families, we confirm that early-generation signal strength correlates with attack outcome (Pearson r ≈ −0.60, p < 0.0001, n=150 per model) and causally validate it via activation patching. Building on this finding, we propose faithfulness anchoring, a prompt-level intervention that restores the refusal signal at generation onset without any access to model internals, reducing mean ASR by 22.7pp across three models while keeping benign task accuracy close to baseline. We further show that activation-level steering cuts ASR further (mean −32.0pp) but trades capability for safety in a model-dependent way, motivating the prompt-level defence. Finally, we find the defence is not adaptively robust: attacks targeting the anchor’s own instruction recover much of the original attack surface.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm jailbreaking, AI Safety, Chain-of-though reasoning, Mechanistic interpretability, Refusal direction, Activation steering, Adaptive attacks, Prompt injection, Adversarial robustness, Residual stream analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FvpQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies “chain-of-thought hijacking” jailbreaks, in which several benign puzzles and their solutions are prepended to a harmful request to induce safety-aligned models to abandon refusal. Building on the refusal direction proposed by Arditi et al., the authors define a token-level signal, F(t), and track it in Llama-3.1-8B, Llama-Nemotron-8B, and Qwen2.5-7B. The results show that the padding uniformly suppresses F(t) during prompt processing, regardless of the eventual outcome. Attack success instead depends on whether F(t) recovers at the start of generation (r ≈ −0.52). The authors also use activation patching for causal validation. Based on these findings, they propose two defenses: “faithfulness anchoring,” which injects a safety reminder at the first response token and reduces attack success rates by an average of 22.7 percentage points; and activation steering, which adds the refusal direction back into the residual stream and achieves an average reduction of 11.3 percentage points, but incurs a larger capability cost. Adaptive evaluation shows that prompt injection directly targeting the anchor text substantially weakens Defense A.

Critical strengths
  1. The empirical evaluation is comprehensive and transparent. The paper includes cross-validation with two judges, per-example outcome-flip analysis, adaptive attacks, and multiple controls designed to isolate the mechanism.
Critical weaknesses
  1. The mechanistic claims lack sufficient causal support. The correlation between generation-stage F(t) and the outcome is subject to reverse causality: the refusal text generated by the model can itself increase F(t). Activation patching replaces the entire layer state, rather than just the refusal direction, and patching control layers also restores refusal.

  2. The validity of the experimental setup is questionable. The instruction-tuned models are evaluated without chat templates, and their benign-task baseline performance is conspicuously abnormal: Llama achieves only 23% on GSM8K, and Nemotron only 7%. The main models are not dedicated reasoning models, and attack success rates without the attack prefix are also missing.

Questions for authors
  1. Chat templates and the evaluation pipeline. Footnote 5 states that prompts are “raw, template-free strings.” Were the official chat templates omitted for all models? What explains the GSM8K accuracies of 23% for Llama and 7% for Nemotron?
Reviewer_HSzZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies chain-of-thought (CoT) hijacking jailbreaks and finds that attack success depends on whether the refusal signal recovers at the onset of response generation. Based on this finding, it proposes two defenses: inserting a safety reminder at the beginning of the response (Defense A) and adding the refusal direction back into the residual stream (Defense B). Defense A reduces the attack success rate by an average of 22.7 percentage points across three models, with minimal degradation in general capabilities.

Critical strengths
  1. The paper analyzes CoT hijacking through the model’s internal refusal direction, offering an intuitive mechanistic explanation of how attack success relates to the recovery of the refusal signal at the onset of response generation.

  2. Defense A is straightforward to implement and does not require access to model internals.

  3. The experimental evaluation is reasonably extensive.

Critical weaknesses
  1. The novelty appears limited, and several closely related works are missing. The attack and mechanistic analysis substantially overlap with [1], while the two defenses resemble existing approaches such as [2] and [3]. None of these works are cited or included in the comparisons.

  2. The motivation for using KV-cache insertion in Defense A is unclear. Table 12 shows no meaningful difference between KV-cache insertion and plain concatenation on Llama and Nemotron, raising questions about the necessity of this design choice.

  3. The experimental setup does not fully support the paper’s framing around reasoning models: two of the three main models are not reasoning models. Moreover, the baseline GSM8K accuracies of 23% for Llama and 7% for Nemotron raise concerns about the capability evaluation setup.

[1] Chain-of-Thought Hijacking. Jianli Zhao et al.

[2] Effectively Controlling Reasoning Models through Thinking Intervention. Tong Wu et al.

[3] Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models. Guobin Shen et al.

Questions for authors
  1. Could the authors discuss and compare their approach with [1]–[3], clarifying the paper’s contributions beyond these works?

  2. Why does plain concatenation perform as well as KV-cache insertion on Llama and Nemotron? Why does placing the reminder in the user prompt yield better results on Qwen?

  3. Could the authors evaluate their approach on reasoning models such as Qwen3 and DeepSeek-R1, and verify the GSM8K evaluation setup?

Reviewer_t3tzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the internal dynamics of chain-of-thought (CoT) hijacking, an attack in which benign puzzle–solution traces are prepended to a harmful request. Building on the refusal-direction analysis of Arditi et al. (2024), the authors extract a model-specific refusal direction from residual-stream activations and track its projection, denoted F(t)F(t), across prompt processing and response generation. They report that, under their principal five-puzzle setting, the refusal signal does not progressively deteriorate during the prompt but instead differs between successful and unsuccessful attacks around generation onset. Across Llama-3.1-8B, Llama-Nemotron-8B, and Qwen2.5-7B, the generation-stage signal correlates with whether the model refuses, and activation patching can restore refusals for some attacked examples.

Motivated by this analysis, the paper examines two defenses. Defense A, faithfulness anchoring, introduces an instruction emphasizing the original request and safety obligations around response onset; Defense B adds a scaled refusal direction to the residual stream during generation. On 150 HarmBench instances per primary model, Defense A lowers the reported non-refusal-based attack success rate (ASR) by an average of 22.7 percentage points. The paper further evaluates prompt phrasing, injection position, steering strength, benign-task performance, alternative judges, and three adaptive attack variants. It finds substantial model-dependent costs, including increased over-refusal, and shows that an adaptive prompt-injection override can substantially weaken the anchoring defense.

Strengths. The focus on token-level refusal dynamics offers a potentially useful diagnostic perspective beyond measuring jailbreak rates alone. The paper reports meaningful empirical defense gains and includes extensive ablations, negative controls, benign-capability checks, and adaptive evaluations. Its willingness to report inconsistent and unfavorable findings is valuable.

Weaknesses. The evidence does not isolate the claimed single-direction causal mechanism: full-activation patching changes more than the measured refusal component, and off-target layers also produce substantial recovery. Crucially, the mechanism-isolation controls show that ordinary text insertion often performs as well as the proposed KV-cache delivery and can outperform it on Qwen, weakening the claim that the specialized intervention follows necessarily from the mechanistic analysis. Finally, the broad claims about uniform suppression and CoT hijacking are not fully matched by the padding-length controls, attack baselines, or the non-refusal-based ASR definition.

Critical strengths

1. Tracking refusal-related activations throughout generation provides a useful mechanistic observation.

Rather than relying exclusively on behavioral jailbreak outcomes, the paper follows F(t)=hℓ∗(t)⊤uRF(t)=h_{\ell^*}(t)^\top u_R across token positions and contrasts successful with unsuccessful attacks. In Section 5.1 and Table 2, the reported outcome correlations are r=−0.570r=-0.570, −0.320-0.320, and −0.675-0.675 for Llama-3.1-8B, Llama-Nemotron-8B, and Qwen2.5-7B, respectively. Appendix A.5, Table 6 shows similar correlations under cosine normalization, and Appendix A.9, Table 10 reports significance after multiple-testing correction. These findings support a reproducible association between refusal-related representations and attack outcomes. While the refusal direction itself comes from earlier work, characterizing its dynamics in this particular attack setting is a meaningful empirical extension.

2. The defense evaluation demonstrates practical gains and examines relevant safety–capability trade-offs.

For Defense A, Table 2 reports ASR changes of 58.0% to 25.3% for Llama-3.1-8B, 53.3% to 26.7% for Nemotron, and 41.3% to 32.7% for Qwen2.5-7B, corresponding to a mean reduction of 22.7 percentage points. Appendix A.12, Table 13 shows that a second judge, GPT-4o-mini, reproduces the direction of these reductions. The phrasing and timing sweeps in Table 1 and Appendix A.10 help characterize when anchoring is effective. The Defense B coefficient sweep in Appendix A.13–A.14 further illustrates that stronger activation steering can lower ASR substantially while degrading benign accuracy and increasing over-refusal. These experiments support the practical relevance of the trade-offs studied.

3. The boundary tests and negative results make the empirical account more credible.

The authors do not conceal outcomes that undermine a simple universal story. Appendix A.8 reports that the prompt-stage outcome-independence pattern breaks at K=8K=8 for two models, despite holding in the primary K=5K=5 configuration. Appendix A.11, Table 12 shows that simpler delivery controls can equal or outperform the main anchoring implementation. Appendix A.15, Table 15 reveals substantial per-example accuracy churn even when aggregate benign accuracy changes little. Finally, Table 3 shows that adaptive override attacks increase defended ASR by 9.3–24.0 percentage points across the three models. These results identify concrete limitations rather than presenting the defense as universally robust.

Critical weaknesses

1. The proposed causal explanation is stronger than the interventions actually establish.

The paper characterizes suppression of a single refusal direction at generation onset as a causal driver of CoT hijacking. The correlations in Table 2 and Appendix A.5 establish predictive association, but not causation. The activation-patching experiment in Section 5.1 replaces the attacked activation with a clean activation vector at ℓ∗\ell^*, altering many features besides its projection onto uRu_R. Consequently, restored refusal after patching establishes that the intervened representation contains behaviorally relevant information, but does not isolate the refusal direction as the causal mediator. Layer specificity is also incomplete: for Llama, an off-target layer restores refusal in 46.8% of instances against 72.5% at the selected layer; for Qwen, one control layer reaches 44.3% against 52.9% at the selected layer. Nemotron additionally exhibits a weaker correlation (r=−0.320r=-0.320) and a different best patching layer in the attack-specific sweep (Appendix A.3). A direction-only patch or ablation, with norm-matched orthogonal and random-direction controls, is needed to distinguish the proposed mechanism from broader restoration of a clean internal state.

2. The mechanism-isolation experiments do not establish a unique advantage for the proposed anchoring implementation.

The main narrative emphasizes prompt-level anchoring at response onset, delivered through KV-cache injection, as a targeted remedy for the inferred failure point. However, Appendix A.11, Table 12 shows that Visible Onset, a simple concatenation control using the same text and position without the specialized KV-cache mechanism, matches Defense A exactly on Llama (25.3% ASR) and Nemotron (26.7%). Moreover, placing the text in the user prompt yields 22.0% ASR on Qwen, better than Defense A's 32.7%, while an earliest-position control yields 24.7%. Thus, the effectiveness of a safety-prioritization reminder is reasonably supported, but the necessity of the proposed delivery mechanism or specific timing is not. Since related safety reminders and goal-prioritization prompts already exist, this weakens both the claimed mechanistic attribution and the distinct methodological contribution. The authors acknowledge mixed control results in Section 5.2 but do not adequately reconcile them with the stronger claims in the conclusion.

3. The attack mechanism and outcome metrics are insufficiently isolated, and the uniform-suppression claim has a documented boundary.

The reported baseline ASR in Table 2 concerns padded harmful requests without defense, but an otherwise matched unpadded harmful-request baseline is not clearly reported. As a result, the paper does not quantify how much additional failure is caused by the puzzle–solution padding itself, separately from the models' ordinary propensity to answer the selected harmful requests. Furthermore, Section 4 defines success using the judge output is_refusal = False; a non-refusal is not necessarily a harmful, instruction-completing response. Cross-validating two refusal judges is useful but does not fully resolve this construct-validity issue. In Appendix A.5, Table 7, the lack of a significant difference in average prompt-stage F(t)F(t) between outcome groups at K=5K=5 does not itself demonstrate that the signal remains constant over token positions. Appendix A.8 additionally finds significant prompt-stage outcome differences at K=8K=8 for Llama and Qwen, a non-monotonic exception to the proposed uniform-suppression pattern. Table 9 notes that Llama's K=8K=8 ASR reuses the K=5K=5 run rather than an independent execution, making it particularly important to clarify the provenance of each K=8K=8 statistic. The conclusions should be restricted to what these controls directly establish.

Questions for authors

1. The proposed causal explanation is stronger than the interventions actually establish.

The paper characterizes suppression of a single refusal direction at generation onset as a causal driver of CoT hijacking. The correlations in Table 2 and Appendix A.5 establish predictive association, but not causation. The activation-patching experiment in Section 5.1 replaces the attacked activation with a clean activation vector at ℓ∗\ell^*, altering many features besides its projection onto uRu_R. Consequently, restored refusal after patching establishes that the intervened representation contains behaviorally relevant information, but does not isolate the refusal direction as the causal mediator. Layer specificity is also incomplete: for Llama, an off-target layer restores refusal in 46.8% of instances against 72.5% at the selected layer; for Qwen, one control layer reaches 44.3% against 52.9% at the selected layer. Nemotron additionally exhibits a weaker correlation (r=−0.320r=-0.320) and a different best patching layer in the attack-specific sweep (Appendix A.3). A direction-only patch or ablation, with norm-matched orthogonal and random-direction controls, is needed to distinguish the proposed mechanism from broader restoration of a clean internal state.

2. The mechanism-isolation experiments do not establish a unique advantage for the proposed anchoring implementation.

The main narrative emphasizes prompt-level anchoring at response onset, delivered through KV-cache injection, as a targeted remedy for the inferred failure point. However, Appendix A.11, Table 12 shows that Visible Onset, a simple concatenation control using the same text and position without the specialized KV-cache mechanism, matches Defense A exactly on Llama (25.3% ASR) and Nemotron (26.7%). Moreover, placing the text in the user prompt yields 22.0% ASR on Qwen, better than Defense A's 32.7%, while an earliest-position control yields 24.7%. Thus, the effectiveness of a safety-prioritization reminder is reasonably supported, but the necessity of the proposed delivery mechanism or specific timing is not. Since related safety reminders and goal-prioritization prompts already exist, this weakens both the claimed mechanistic attribution and the distinct methodological contribution. The authors acknowledge mixed control results in Section 5.2 but do not adequately reconcile them with the stronger claims in the conclusion.

3. The attack mechanism and outcome metrics are insufficiently isolated, and the uniform-suppression claim has a documented boundary.

The reported baseline ASR in Table 2 concerns padded harmful requests without defense, but an otherwise matched unpadded harmful-request baseline is not clearly reported. As a result, the paper does not quantify how much additional failure is caused by the puzzle–solution padding itself, separately from the models' ordinary propensity to answer the selected harmful requests. Furthermore, Section 4 defines success using the judge output is_refusal = False; a non-refusal is not necessarily a harmful, instruction-completing response. Cross-validating two refusal judges is useful but does not fully resolve this construct-validity issue. In Appendix A.5, Table 7, the lack of a significant difference in average prompt-stage F(t)F(t) between outcome groups at K=5K=5 does not itself demonstrate that the signal remains constant over token positions. Appendix A.8 additionally finds significant prompt-stage outcome differences at K=8K=8 for Llama and Qwen, a non-monotonic exception to the proposed uniform-suppression pattern. Table 9 notes that Llama's K=8K=8 ASR reuses the K=5K=5 run rather than an independent execution, making it particularly important to clarify the provenance of each K=8K=8 statistic. The conclusions should be restricted to what these controls directly establish.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.