- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies “chain-of-thought hijacking” jailbreaks, in which several benign puzzles and their solutions are prepended to a harmful request to induce safety-aligned models to abandon refusal. Building on the refusal direction proposed by Arditi et al., the authors define a token-level signal, F(t), and track it in Llama-3.1-8B, Llama-Nemotron-8B, and Qwen2.5-7B. The results show that the padding uniformly suppresses F(t) during prompt processing, regardless of the eventual outcome. Attack success instead depends on whether F(t) recovers at the start of generation (r ≈ −0.52). The authors also use activation patching for causal validation. Based on these findings, they propose two defenses: “faithfulness anchoring,” which injects a safety reminder at the first response token and reduces attack success rates by an average of 22.7 percentage points; and activation steering, which adds the refusal direction back into the residual stream and achieves an average reduction of 11.3 percentage points, but incurs a larger capability cost. Adaptive evaluation shows that prompt injection directly targeting the anchor text substantially weakens Defense A.
Critical strengths
- The empirical evaluation is comprehensive and transparent. The paper includes cross-validation with two judges, per-example outcome-flip analysis, adaptive attacks, and multiple controls designed to isolate the mechanism.
Critical weaknesses
-
The mechanistic claims lack sufficient causal support. The correlation between generation-stage F(t) and the outcome is subject to reverse causality: the refusal text generated by the model can itself increase F(t). Activation patching replaces the entire layer state, rather than just the refusal direction, and patching control layers also restores refusal.
-
The validity of the experimental setup is questionable. The instruction-tuned models are evaluated without chat templates, and their benign-task baseline performance is conspicuously abnormal: Llama achieves only 23% on GSM8K, and Nemotron only 7%. The main models are not dedicated reasoning models, and attack success rates without the attack prefix are also missing.
Questions for authors
- Chat templates and the evaluation pipeline. Footnote 5 states that prompts are “raw, template-free strings.” Were the official chat templates omitted for all models? What explains the GSM8K accuracies of 23% for Llama and 7% for Nemotron?