acceptodds
Under review as a conference paper at ICLR 2027

Chain-of-Thought Hijacking

Abstract

Large Reasoning Models (LRMs) improve task performance through extended inference-time reasoning. Although previous studies suggest that longer reasoning should lead to more robust safety behavior, we find evidence to the contrary: over-extended reasoning can instead be exploited to systematically weaken refusal behavior. We propose *Chain-of-Thought Hijacking*, a simple yet effective black-box jailbreak attack that induces LRMs to engage in prolonged benign puzzle-solving reasoning, often lasting more than five minutes, before eliciting harmful compliance. Across HarmBench, CoT Hijacking achieves attack success rates of **99%, 94%, 100%, and 94%** on Gemini 2.5 Pro, ChatGPT o4 Mini, Grok 3 Mini, and Claude 4 Sonnet, respectively. To understand why this attack succeeds, we conduct activation probing, attention-pattern analysis, and causal interventions on open-source reasoning models. Our results indicate that refusal behavior depends on a low-dimensional safety signal whose expression weakens as reasoning traces grow longer. In particular, extended benign reasoning shifts attention away from harmful intentions and attenuates refusal-related activations, producing what we call *refusal dilution*. These findings demonstrate that excessively prolonged reasoning can introduce a systematic jailbreak attack surface. We release our evaluation materials to support reproducibility and further research.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, jailbreak, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MrQPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a black-box jailbreak method called cot hijacking, which induces LLMs to perform long benign puzzle-solving reasoning before responding to harmful requests. The authors report high attack success rates across multiple reasoning models. They further investigate the underlying mechanism, called refusal dilution, through activation probing, attention analysis, and causal interventions on attention heads, showing that longer cot may weaken refusal-related signals.

Critical strengths

(1) The authors present an effective jailbreak method with high attack success rates across multiple models, outperforming previous methods.

(2) The paper provides comprehensive experiments, including ablation studies and comparisons with existing methods.

(3) The mechanistic analysis is interesting, especially the investigation of refusal directions and attention heads, providing some insights into why the attack works.

Critical weaknesses

(1) I don't think the paper sufficiently isolates CoT length as the main reason for jailbreak success. The shorter CoT ablation also changes puzzle difficulty and type, so it is unclear whether the improvement is really caused by longer reasoning.

(2) The proposed refusal dilution mechanism is not fully justified. The attention head ablation shows these heads are important for refusal, but does not directly prove that CoT Hijacking works by weakening these specific heads.

(3) It is unclear whether this vulnerability is specific to reasoning models, since the paper lacks comparisons with non-reasoning models under similar attack settings.

(4) The attack uses iterative refinement, but it is unclear whether the query budgets are fairly matched with previous jailbreak methods.

Questions for authors

See weakness, please resolve them.

2 comments

Response to Reviewer MrQP

We sincerely thank you for recognizing the effectiveness of our attack, the comprehensive experiments, and the insights of our mechanistic analysis.

1. Isolating CoT Length

Weakness 1: The shorter-CoT ablation also changes puzzle difficulty and type, so the effect of CoT length is unclear.

Section 5.2 is designed to make CoT length the only variable: the prompt is identical for each instruction type, and the puzzle CoT is manually truncated to different lengths (lines 355–358). To address your concern, we further test the compliance rate of Qwen3-14B on stealthy harmful instructions under different CoT lengths (as complementary to Figure 4):

CoT Length1k3k11k21k31k47k
Compliance Rate12%15%23%38%39%65%
  • Validity of Truncation: Naturally shorter traces would change the reasoning content; truncation only removes redundant, independent puzzle steps and keeps the trace coherent.
  • Factorial Ablation: Among all components, shortening the CoT causes the largest ASR drop (99% → 71%, Table 5).

These results demonstrate that longer reasoning itself drives compliance.

2. Attention-Head Ablation

Weakness 2: The head ablation does not prove that CoT Hijacking works by weakening these specific heads.

The 60 heads are selected by the "longer CoT, lower ratio" pattern (Figure 7), i.e., heads whose attention to harmful tokens is diluted by long CoT. Ablating them simulates extreme dilution, and reproduces the attack effect (6% → 73%).

  • Random-Head Control: Selected heads weaken refusal far more than random heads (Figure 8).
  • Attention under Attack: On real jailbreak samples, harmful tokens receive little attention (Appendix J).

3. Non-Reasoning Models

Weakness 3: Is this vulnerability specific to reasoning models?

Following prior baselines (Mousetrap, H-CoT, AutoRAN), our attack design and analysis center on the reasoning process of LRMs. As you requested, we test Gemini 2.5 Flash in non-reasoning mode under the same setting (Section 3.3, 100 HarmBench samples):

ModelGemini 2.5 Pro (reasoning)Gemini 2.5 Flash (non-reasoning)
ASR99%72%

From the table we can observe that the attack still works on non-reasoning models, but is much stronger on models with CoT, even though Gemini 2.5 Pro is claimed to be safer.

4. Query Budget

Weakness 4: Are query budgets fairly matched with prior methods?

To address your concern, we have compiled the normalized budget metrics for different baseline methods. The table below details the average resources required per harmful instruction when attacking Gemini 2.5 Pro on the first 100 samples of the HarmBench subset:

MethodASRCandidate Prompts (Rounds)Total Input TokensTotal Output TokensAttacker-Model CallsLatencyCost ($) / Sample
CoT Hijacking (Ours)99%12 (6 if 1st iter succeeds)~0.19M~0.10M12 (max 36)5–12 min3.50
AutoRAN69%~3 (range 1–6)~0.07M~0.02M~42–3 min1.50
H-CoT60%~2 (range 1–5)~0.03M~0.02M–0.03M32–4 min0.80
Mousetrap44%~6 (range 3–9)~0.07M~0.05M01–2 min1.80

Our attack uses slightly more resources but yields substantially higher ASR, with a strict stop after 3 refinement turns. Moreover, a NINJA baseline given the same 12 candidate prompts reaches only 23% (Table 5).

Reviewer_MrQPOpenReview ↗

Response to the Authors' New Evidence

I thank the authors for the response. Only few of my concerns have been resolved, but I don't think it is convincing as the response mostly restates results already in the submission.

1. Isolating CoT Length

Weakness 1: The shorter-CoT ablation also changes puzzle difficulty and type.

  • Same Number Reused: The "Factorial Ablation" point cites the 99% → 71% drop in Table 5, which is the result being questioned (Table 5 is also a one-at-a-time ablation, not factorial).
  • Different Setting: The new compliance table uses Qwen3-14B with DAN-style prompts, not CoT Hijacking prompts on the attacked models.
  • Truncation & Context Length: Algorithm 1 uses trim_mid, not "only redundant steps." The largest jump (39% → 65%) occurs from 31k to 47k, beyond Qwen3-14B's native 32k context. Was YaRN used? Appendix B also states an 8k max generation length.

2. Attention-Head Ablation

Weakness 2: The head ablation does not prove that CoT Hijacking works by weakening these specific heads.

No new evidence is provided. Figures 7–8 and Appendices H and J are all in the submission.

  • Selection Confound: Heads with the strongest length trend likely already attend heavily to harmful tokens. A random-head control cannot rule this out; a control matched on baseline harmful attention is needed.
  • Ablation ≠ Dilution: Ablating 6 heads reduces the refusal component by ~135 (Figure 8a), while 47k CoT reduces it by only ~30 (Figure 4). The 6% → 73% result is also measured without the attack (Appendix H).

3. Other Points

  • Non-Reasoning Models: Flash vs. Pro is a cross-model comparison; a same-model thinking on/off comparison would isolate reasoning. Also, 72% (Flash), 72% (GPT-5-mini, min effort), and 71% (shorter CoT) suggest most of the success comes from the puzzle framing rather than long reasoning.
  • Query Budget: 12 candidates vs. 2–6 is more than "slightly more".
Reviewer_17CQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Chain-of-Thought Hijacking (CoT Hijacking), a black-box jailbreak method that induces large reasoning models to generate extended benign puzzle-solving reasoning before responding to a harmful request, thereby weakening the models' refusal behavior. Across a 100-sample subset of HarmBench, the method achieves ASRs exceeding 90% on four reasoning models, demonstrating that safety mechanisms degrade under prolonged CoT generation. Furthermore, the authors conduct mechanistic analyses on open-source reasoning models (e.g., Qwen3-14B) using refusal-direction probing, attention-ratio analysis, and causal interventions on attention heads. The experimental results indicate that as benign CoT length increases, projections onto the refusal direction weaken and relative attention to harmful instruction tokens decreases—a phenomenon termed "refusal dilution."

Critical strengths
  1. Empirical observation of safety degradation under extended CoT: The paper provides empirical evidence showing that scaling inference-time reasoning length can reduce refusal rates rather than strengthen safety behavior, contrasting with prior assumptions regarding reasoning length and robustness.
  2. Mechanistic interpretability analysis: The authors connect behavioral attack success to internal model states by identifying a low-dimensional refusal direction in LRMs, tracking the decay of the refusal component across CoT lengths (1k to 47k tokens), and verifying the causal role of specific attention heads through targeted ablation experiments.
  3. Quantitative attack performance across evaluated models: CoT Hijacking achieves 94%–100% ASR across the evaluated proprietary and open-source reasoning models on HarmBench, outperforming the compared LRM jailbreak baselines (Mousetrap, H-CoT, and AutoRAN) under the reported evaluation protocol.
Critical weaknesses
  1. Evaluation limited to earlier-generation reasoning models: The empirical evaluation is conducted on relatively earlier reasoning models (e.g., Gemini 2.5 Pro, ChatGPT o4 Mini, Grok 3 Mini, and Claude 4 Sonnet). It remains unclear whether CoT Hijacking remains effective against more recent frontier models equipped with stronger reasoning capabilities and updated safety defenses, such as GPT-5.6-sol, GPT-6-sol, Claude Opus 5, Claude Opus 5.5, or Gemini 4. Evaluating the method on these newer models is necessary to verify whether refusal dilution persists in current frontier systems.
  2. Entanglement of attack components and unclear distinction from prior work: The proposed CoT Hijacking pipeline does not solely increase benign CoT length. As described in Section 3.2 (Figure 2), the attack combines three distinct components: (i) wrapping the prompt in a complex puzzle preface, (ii) mapping or encoding the harmful goal into abstract puzzle variables, categories, or protocols (which resembles classical obfuscation/encoding jailbreaks), and (iii) employing a multi-turn black-box iterative refinement loop driven by Gemini 2.5 Pro based on refusal scores and step counts.
    • Consequently, the individual contribution of each component—specifically, variable obfuscation versus iterative search versus CoT length itself—is not fully isolated. For instance, in Table 2c (GPT-5-mini), the ASR under Minimal (72%) and Low (76%) reasoning effort is higher than under High effort (68%), which raises questions about the direct relationship between reasoning length and attack success.
    • Furthermore, prior work such as Mousetrap [1] also utilizes iterative reasoning chains and obfuscated mappings to jailbreak LRMs. The authors should clarify the methodological differences between their prompt construction/refinement pipeline and [1], and more precisely delineate the specific technical contribution of CoT Hijacking beyond existing iterative obfuscation frameworks.
Questions for authors

refer to weakness

2 comments

Response to Reviewer 17CQ

We sincerely thank you for recognizing our empirical finding on safety degradation, the mechanistic analysis, and the strong attack performance.

1. Newer Frontier Models

Weakness 1: Does CoT Hijacking remain effective on newer frontier models?

Our experiments were conducted on the frontier models available at the time, and the evaluation already extends to newer generations:

  • GPT-5 Family: CoT Hijacking reaches 68–76% ASR on GPT-5-mini (Table 2c), and successfully jailbreaks GPT-5 (Appendix D).

More importantly, following responsible disclosure, we reported this vulnerability to OpenAI, Anthropic, Google DeepMind, and xAI before release, and several of them are evaluating mitigations (Ethics Statement). Therefore, newer models may have been specifically patched against our attack, which reflects the practical impact of our findings rather than the absence of the vulnerability.

2. Attack Components & CoT Length

Weakness 2.1: The contributions of obfuscation, iterative search, and CoT length are not isolated; GPT-5-mini is not more vulnerable under high effort.

To isolate each component, we ablate one at a time when attacking Gemini 2.5 Pro (Appendix C, Table 5):

SettingASRDrop in ASR
Full CoT Hijacking99%–
w/o Role Framing92%−7%
Puzzle → GPQA Reasoning89%−10%
w/o Variable Mapping (Obfuscation)77%−22%
Shorter CoT71%−28%

These results demonstrate that CoT length is the largest contributor, while obfuscation alone does not explain the attack.

  • CoT Length without Search: With a fixed prompt and no refinement, compliance of Qwen3-14B rises from 12% to 65% as CoT grows from 1k to 47k tokens (same setting as Figure 4, Section 5.2).

3. Difference from Mousetrap

Weakness 2.2: How does CoT Hijacking differ from iterative obfuscation methods such as Mousetrap?

As illustrated in Figure 1, the two attacks differ fundamentally in what the model reasons about:

  • Reasoning Target: Mousetrap asks the model to decode a harmful request encrypted by 1–3 cipher mappings (e.g., Caesar cipher, reversal), so its reasoning is spent on reconstructing the harmful request itself. In CoT Hijacking, around 90% of the prompt is a benign puzzle (grey in Figure 1), and the model reasons for minutes on puzzle deduction without touching the harmful request.
  • Role of the Harmful Request: Mousetrap hides the harmful request, so the model cannot recognize it before decoding. CoT Hijacking keeps it readable as the final "practical narrative" requirement (red in Figure 1): it is not hidden, but drowned out by long benign reasoning.
  • Source of Attack Strength: Mousetrap is bounded by decoding ability, as its success drops beyond 3 cipher steps when the model fails to reconstruct the request. CoT Hijacking grows with reasoning length, as compliance rises from 12% to 65% when CoT grows from 1k to 47k tokens.

Thanks for the author's detailed response, and my concerns have largely been solved. I will maintain my score

Reviewer_cDaGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study whether longer reasoning makes reasoning models safer, as recent alignment work suggests. They propose CoT Hijacking, a black-box jailbreak that hides a harmful request inside a hard benign puzzle, such as a logic grid, and asks for a “practical example” of the solution at the end. An attacker model writes and refines these prompts, using the target’s refusals and reasoning length as feedback. On 100 HarmBench behaviors, the attack reaches 99%, 94%, 100%, and 94% success on Gemini 2.5 Pro, o4-mini, Grok 3 Mini, and Claude 4 Sonnet, compared with at most 69% for prior reasoning-model jailbreaks. A length-matched long-input baseline without a puzzle reaches only 23%. To explain the attack, the authors study Qwen3-14B. They measure how strongly its hidden state points along a refusal direction. This signal weakens as the puzzle reasoning grows longer, and attention to the harmful request falls relative to the puzzle. Ablating 60 attention heads selected by this attention drop raises harmful compliance from 6% to 73%. The authors call this effect refusal dilution and conclude that long reasoning opens a new attack surface.

Critical strengths
  1. Prior works on deliberative alignment [1] and the o1 system card [2] have shown that reasoning can makes models safer. The authors rather find that reasoning length can work against refusal. This is an interesting finding, as safety training that relies on reasoning must then account for long traces.

  2. I found the ablations insightful. Specifically putting the harmful request at the end of about 100k tokens of ordinary benign text, as in NINJA, the attack succeeds only 23% of the time, compared with 99% for the full method. Also, with swapping the puzzles for GPQA science questions, it still succeeds 89% of the time. So it shows that the reasoning itself, not how long the input is or which kind of puzzle is used, is important.

[1] Guan, M.Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J. and Chung, H.W., 2024. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339.

[2] Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A. and Iftimie, A., 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.

Critical weaknesses

The authors hypothesize that long reasoning weakens an internal refuse signal, and that this is the reason for model compliance. However, the current experiments do not clearly support this hypothesis. The authors do not check whether the model complies more often as the signal weakens. The signal drops only slightly, from about 160 to about 130, and stays far above the level for harmless requests, which is around 14. Also, the test in Appendix H where the authors switch off 60 attention heads is run on plain harmful prompts without the attack, so it shows only that these heads help with refusal in general.

Questions for authors
  1. With the prompt held fixed, does harmful compliance on Qwen3-14B increase as the puzzle reasoning grows longer?
2 comments

Response to Reviewer cDaG

We sincerely thank you for recognizing our counterintuitive finding and the insightfulness of our ablations.

1. Refusal Signal & Compliance Rate

Weakness 1 & Question 1: Does compliance increase as the refusal signal weakens with longer CoT?

Section 5.2 is designed to isolate the effect of CoT length, not to reproduce the full attack. To address your concern, we measure the compliance rate of Qwen3-14B on stealthy harmful instructions, with the prompt held fixed:

CoT Length1k3k11k21k31k47k
Compliance Rate12%15%23%38%39%65%

From the table we can observe that compliance rises from 12% to 65% simply as the CoT grows longer.

Note that this setting varies CoT length only, while the full attack also uses role framing, puzzle structure, and variable mapping (Table 5). Hence the signal stays above the harmless level, and compliance does not reach 99%.

2. Attention-Head Ablation

Weakness 2: The head ablation uses plain harmful prompts, so it only shows these heads help refusal in general.

The 60 heads are selected by the "longer CoT, lower ratio" pattern (Figure 7), i.e., heads whose attention to harmful tokens is diluted by long CoT. Ablating them simulates extreme dilution without any puzzle, and reproduces the attack effect (6% → 73%).

This link to the attack is further supported by:

  • Head Selection Protocol: Heads are selected by their CoT-length trend, not by their effect on refusal.
  • Random-Head Control: Selected heads weaken refusal far more than random heads (Figure 8).
  • Attention under Attack: On real jailbreak samples, harmful tokens receive little attention (Appendix J).

These results demonstrate that CoT Hijacking and head ablation act on the same safety heads. We will clarify this in Section 5.4.

Dear Reviewer cDaG,

Thank you again for your thoughtful feedback. We posted our response a few days ago. As the discussion period is approaching its end, we would greatly appreciate any further comments or suggestions.

In our response, we:

  • added compliance rates across CoT lengths (12% → 65%);
  • clarified how the ablated heads link to CoT length;
  • provided further evidence linking these heads to the attack.

We hope these additions address your main concerns and look forward to your further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.