- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates shortcut learning in the safety alignment of large reasoning models. The authors argue that although standard SFT/RL alignment can achieve near-perfect safety scores, models learn two types of spurious shortcuts: format shortcuts, where refusal behavior is tied to chat-template tokens, and lexical shortcuts, where sensitive words reflexively trigger refusals. They support this argument through template-stripping experiments and a three-phase analysis of RL training dynamics. To address these issues, they propose DeShortcut-Align, a three-stage framework: (1) refusal-sensitivity attribution, which identifies highly sensitive words by masking input tokens and measuring the absolute change in the mean teacher-forced log-likelihood of the final refusal response; (2) attribution-guided contrastive augmentation, which uses GPT-4o to rewrite harmful queries into benign queries that preserve sensitive words while reversing the intent, then incorporates these queries into the training data; and (3) counterfactual consistency regularization, which constructs template-stripped inputs through attention masking and enforces consistent decisions between the original and template-stripped inputs during both SFT and RL. Evaluated on DeepSeek-R1-Distill-Qwen 7B/14B and Qwen3-4B against SFT and GRPO, and, for the 7B model only, PPO, STAR-1, RealSafe-R1, and TARS, the method reduces the template-dependence gap by approximately 18–72% and the average false refusal rate by 30–58% relative to the corresponding SFT/GRPO baselines in the main table.
Critical strengths
-
The diagnosis is compelling and broadly relevant. Rather than merely proposing a new method, the paper first uses template-stripping experiments (Figure 1) and an analysis of RL training dynamics (Figure 2) to reveal a systematic failure across methods and benchmarks: the safety of existing alignment methods relies heavily on superficial cues such as formatting and vocabulary.
-
The method directly targets the diagnosed failure modes. Each component corresponds to a specific failure mode, and the ablation studies support these connections.
Critical weaknesses
-
Baseline coverage is insufficient, and the method’s contributions are not cleanly isolated. The main table compares only against standard SFT/GRPO, while specialized methods are evaluated only on the 7B model. SafeChain, SafeKey, and STAIR, all cited in the paper, are absent from the main quantitative performance comparisons, although SafeChain appears in the auxiliary KL-divergence analysis. There are also no inference-time defenses, such as activation steering or prompt-based methods, for comparison.
-
The gains in capability retention are uneven, and the safety–utility trade-off deserves clearer discussion. On the 7B model, Ours (SFT) improves Avg2 only modestly over standard SFT (62.83 vs. 62.13) and remains below the base model (67.70), while MMLU falls from 46.49 under standard SFT to 40.61. Its templated average DSR also decreases from 90.20 to 86.11. These results support partial mitigation of the alignment tax, but not the stronger wording in Section 5.2 that the method “circumvents” it.
Questions for authors
-
Could inference-time defenses offer a cheaper alternative? Activation-steering approaches, such as AlphaSteer, likewise aim to maintain safety without impairing benign behavior and require no additional fine-tuning of the underlying LLM. How do they perform under the paper’s template-stripping setting? If they achieve comparable DSR/FRR, the training cost of the proposed method would require further justification.
-
Can AGCA’s contribution be distinguished from that of adding arbitrary benign data? What happens to over-refusal and reasoning performance if the benign queries generated by AGCA are replaced with an equal number of ordinary benign queries—for example, Alpaca or XSTest-safe-style samples—without sensitive-word preservation or attribution guidance?
4 comments
W2. Capability Retention and Safety–Utility Trade-off
We acknowledge that the original SFT results showed limited capability retention and reduced standard-template safety. One contributing factor may be the original batch size of 8, which results in approximately 16 times more optimizer updates than the effective batch size of 128 used by STAR-1 [4], given identical data and epochs. As further analyzed in our response to Reviewer 9S8Q, training configurations substantially affect over-refusal. We therefore reevaluate both standard SFT and DeShortcut-Align (SFT) using batch size 128 under identical training configurations.
Table R3. Updated SFT comparison with batch size 128 (%).
| Method | Safety ↑ | Δavg ↓ | XSTest ↓ | OKTest ↓ | FR-test ↓ | FRR ↓ | MMLU ↑ | General ↑ | H-mean ↑ |
|---|---|---|---|---|---|---|---|---|---|
| SFT (bs128) | 89.90 | 14.08 | 35.00 | 21.00 | 51.00 | 35.67 | 46.86 | 65.60 | 74.70 |
| Ours (SFT, bs128) | 86.51 | 2.75 | 28.00 | 14.00 | 38.00 | 27.00 | 48.39 | 67.35 | 79.38 |
Under the matched configuration, DeShortcut-Align improves MMLU and overall general capabilities while substantially reducing template dependence and over-refusal. Although standard-template safety remains slightly lower, the results indicate a more favorable overall safety–utility trade-off, as reflected in the improved H-mean. We also agree that "mitigates the alignment tax" is more accurate than "circumvents the alignment tax" and will revise Section 5.2 accordingly.
Q1. Can Inference-Time Defenses Offer a Cheaper Alternative?
(1) Complementary roles of safety alignment and inference-time defenses. Training-time safety alignment and inference-time defenses such as AlphaSteer [6] serve complementary roles. Safety alignment, typically through supervised fine-tuning (SFT) or reinforcement learning (RL), aims to establish intrinsic safety behavior by training models to recognize harmful intent and generate appropriate responses. In contrast, inference-time methods such as AlphaSteer enhance safety by manipulating existing internal representations without updating the underlying model parameters. While these approaches can provide additional protection at relatively low training cost, their effectiveness depends on the safety-related representations and capabilities already present in the model. Inference-time defenses therefore do not necessarily replace alignment training; both can be combined.
DeShortcut-Align specifically addresses a limitation of the safety alignment process itself: models may achieve high safety rewards by learning superficial formatting and lexical shortcuts rather than sufficiently relying on the underlying harmful intent. Our objective is to mitigate these shortcuts during SFT/RL, thereby improving the robustness of the model's intrinsic safety behavior. Although DeShortcut-Align introduces additional training costs, the resulting aligned model requires no additional steering operations or safety probing during inference. The approaches thus involve different training–inference cost trade-offs.
(2) Empirical comparison and limitations. Table R2 reports standard-template results; template-removal results are summarized below. AlphaSteer improves the average DSR over Base (54.40% vs. 46.42%) with templates, but drops to 46.01% when templates are removed. AlphaSteer was originally developed for instruction-tuned models, whereas our evaluation focuses on reasoning models with more complex chain-of-thought (CoT) generation, where activation-level interventions may transfer less reliably and potentially interfere with reasoning. Moreover, such interventions operate on the existing policy and do not directly eliminate its learned template dependence. In contrast, ADA-LP improves from 69.73% to 73.37% without templates. This may be related to its mechanism of re-injecting assistant-header safety tokens to probe the model's internal safety signals: we observe more frequent injections without templates (47.44 vs. 37.95 on average), potentially strengthening its inference-time protection rather than eliminating the underlying shortcuts. DeShortcut-Align (RL) achieves substantially higher DSR in both settings (97.07% / 93.44%) without additional inference-time interventions. We acknowledge that these approaches offer complementary benefits and that our comparison is not compute-matched. Nevertheless, inference-time defenses do not necessarily address the underlying shortcut learning: excessive reliance on superficial cues, rather than harmful intent, can leave safety behavior brittle. Mitigating such shortcuts during safety alignment is therefore important for improving the policy's intrinsic robustness.
Q2. AGCA vs. Ordinary Benign Data Augmentation
We agree that the contribution of Attribution-Guided Contrastive Augmentation (AGCA) should be distinguished from simply adding benign data. We therefore conduct equal-budget benign-data comparisons, replacing AGCA-generated samples with the same number of benign samples from Alpaca [8] or STAR-1 [4], while keeping harmful training data and optimization settings fixed.
Table R4. Comparison of benign data augmentation strategies (%).
| Benign Data Source | Safety ↑ | Δavg ↓ | XSTest ↓ | OKTest ↓ | FR-test ↓ | FRR ↓ | H-mean ↑ |
|---|---|---|---|---|---|---|---|
| Alpaca [8] | 91.33 | 6.23 | 32.00 | 21.00 | 59.00 | 37.33 | 79.85 |
| STAR-1 Benign [4] | 86.29 | 9.24 | 34.00 | 17.00 | 46.00 | 32.33 | 80.25 |
| AGCA (Ours) | 86.51 | 2.75 | 28.00 | 14.00 | 38.00 | 27.00 | 84.41 |
Bold values indicate the best result in each metric column; ties are bolded.
H-mean is the harmonic mean of Safety, 100 − Δavg, and 100 − FRR.
AGCA achieves the lowest template gap (2.75) and over-refusal rate (27.00%), along with the highest H-mean (84.41). Although Alpaca achieves higher standard-template safety, these results indicate that AGCA's benefits cannot be explained solely by adding benign data.
Furthermore, the original Table 3 compares Random, Gradient, Attention, and Token Sensitivity Score (TSS) selection within the same augmentation framework. TSS achieves the lowest over-refusal while maintaining competitive safety performance. Together, these experiments support targeting refusal-sensitive vocabulary and constructing benign counterfactuals that preserve such vocabulary while changing the overall intent, rather than relying on ordinary benign-data augmentation or arbitrary token selection.
References
[1] Kaiwen Zhou et al. SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning. EMNLP, 2025. https://aclanthology.org/2025.emnlp-main.1291/
[2] Fengqing Jiang et al. SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities. Findings of ACL, 2025. https://aclanthology.org/2025.findings-acl.1197/
[3] Yi Zhang et al. Internalizing Safety Understanding in Large Reasoning Models via Verification. ICML, 2026. https://proceedings.mlr.press/v306/zhang26n.html
[4] Zijun Wang et al. STAR-1: Safer Alignment of Reasoning LLMs with 1K Data. AAAI, 2026. https://arxiv.org/abs/2504.01903
[5] Yichen Sun et al. SAVeR²: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards. arXiv:2611.07482, 2026. https://arxiv.org/abs/2611.07482
[6] Leheng Sheng et al. AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint. arXiv:2506.07022, 2025. https://arxiv.org/abs/2506.07022
[7] Jiawei Zhang et al. Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth. arXiv:2510.18081, 2025. https://arxiv.org/abs/2510.18081
[8] Rohan Taori et al. Stanford Alpaca: An Instruction-following LLaMA Model. Stanford CRFM, 2023. https://github.com/tatsu-lab/stanford_alpaca
Dear Reviewer,
Thank you again for your thoughtful and constructive feedback, which we have carefully addressed in our author response. As the discussion deadline is approaching, we would greatly appreciate any further feedback you may have. If our responses have resolved your main concerns, we would also be grateful if you could update your overall assessment accordingly.
Sincerely,
The Authors
We thank the reviewer for recognizing our analysis of shortcut learning and the motivation behind DeShortcut-Align. We address the concerns regarding baseline coverage, capability retention, and the two additional questions below.
W1. Insufficient Experimental Baselines
We agree that broader baseline comparisons would strengthen the evaluation. The original paper already includes STAR-1, RealSafe-R1, TARS, and PPO in Appendix Table 5, alongside jailbreak comparisons in Table 4. We further evaluate SafeKey [1], SafeChain [2], and SInternal-7B [3], and update both standard supervised fine-tuning (SFT) and DeShortcut-Align (SFT) using batch size 128, matching the official STAR-1 configuration [4]. Following SAVeR² [5], we adopt a four-dimensional harmonic mean (H-mean) to jointly evaluate standard-template safety, template robustness, over-refusal (measured by the false refusal rate, FRR), and general capabilities.
Table R1. Expanded safety alignment baseline comparison (%).
Bold values indicate the best result in each metric column within each training paradigm; ties are bolded.
H-mean is calculated over Safety, 100−Δavg, 100−FRR, and General.
To address the missing inference-time baselines and examine their safety effectiveness as alternatives to training-time alignment, we additionally evaluate AlphaSteer [6] and Any-Depth Alignment with a linear probe (ADA-LP) [7] on DeepSeek-R1-Distill-Qwen-7B under standard chat templates.
Table R2. Comparison with inference-time safety interventions (defense success rate, DSR, %).
Bold values indicate the best result for each benchmark or average across methods; ties are bolded.
The expanded comparisons show that DeShortcut-Align achieves the highest four-dimensional H-mean under both SFT and RL, while also outperforming the evaluated inference-time defenses in standard-template DSR. Although specialized methods differ in training budgets and intervention mechanisms, these results support the competitiveness of shortcut mitigation across multiple safety objectives. We will incorporate the expanded comparisons into the revised paper.