acceptodds
Under review as a conference paper at ICLR 2027

DeShortcut-Align: Decoupling Spurious Shortcuts for Robust Safety Alignment in Large Reasoning Models

Abstract

Safety alignment of large reasoning models (LRMs) via supervised fine-tuning (SFT) and reinforcement learning (RL) often yields near-perfect safety scores, yet this apparent success comes at the cost of severe over-refusal and degraded general capabilities. Through systematic empirical analysis, we find that these failures are closely associated with the learning of spurious shortcuts rather than robust intent-sensitive safety evaluation. Specifically, we identify two dominant shortcuts: formatting shortcuts, where refusal behaviors are overly bound to structural prompt templates that frequently appear in safety alignment corpora; and lexical shortcuts, where sensitive keywords reflexively trigger refusals on benign queries. To mitigate reliance on these shortcuts, we propose DeShortcut-Align, a shortcut-decoupling alignment framework that reduces dependence on superficial cues. DeShortcut-Align operates across three coordinated stages: (1) Refusal Sensitivity Attribution, which masks input tokens to quantify their impact on the final refusal response distribution; (2) Attribution-Guided Contrastive Augmentation, which constructs benign contrastive samples using high-sensitivity tokens to mitigate lexical shortcuts; and (3) Counterfactual Consistency Regularization, which constructs template-ablated states via attention blinding to enforce decision consistency across SFT and RL, mitigating formatting shortcut dependence. Experiments on 7B and 14B models demonstrate that DeShortcut-Align significantly improves robustness against template-stripping bypass attacks (reducing performance drops by up to ), substantially reduces over-refusal by over , and better preserves general-purpose reasoning capabilities, thereby mitigating the alignment tax commonly observed in safety training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Fh3iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates shortcut learning in the safety alignment of large reasoning models. The authors argue that although standard SFT/RL alignment can achieve near-perfect safety scores, models learn two types of spurious shortcuts: format shortcuts, where refusal behavior is tied to chat-template tokens, and lexical shortcuts, where sensitive words reflexively trigger refusals. They support this argument through template-stripping experiments and a three-phase analysis of RL training dynamics. To address these issues, they propose DeShortcut-Align, a three-stage framework: (1) refusal-sensitivity attribution, which identifies highly sensitive words by masking input tokens and measuring the absolute change in the mean teacher-forced log-likelihood of the final refusal response; (2) attribution-guided contrastive augmentation, which uses GPT-4o to rewrite harmful queries into benign queries that preserve sensitive words while reversing the intent, then incorporates these queries into the training data; and (3) counterfactual consistency regularization, which constructs template-stripped inputs through attention masking and enforces consistent decisions between the original and template-stripped inputs during both SFT and RL. Evaluated on DeepSeek-R1-Distill-Qwen 7B/14B and Qwen3-4B against SFT and GRPO, and, for the 7B model only, PPO, STAR-1, RealSafe-R1, and TARS, the method reduces the template-dependence gap by approximately 18–72% and the average false refusal rate by 30–58% relative to the corresponding SFT/GRPO baselines in the main table.

Critical strengths
  1. The diagnosis is compelling and broadly relevant. Rather than merely proposing a new method, the paper first uses template-stripping experiments (Figure 1) and an analysis of RL training dynamics (Figure 2) to reveal a systematic failure across methods and benchmarks: the safety of existing alignment methods relies heavily on superficial cues such as formatting and vocabulary.

  2. The method directly targets the diagnosed failure modes. Each component corresponds to a specific failure mode, and the ablation studies support these connections.

Critical weaknesses
  1. Baseline coverage is insufficient, and the method’s contributions are not cleanly isolated. The main table compares only against standard SFT/GRPO, while specialized methods are evaluated only on the 7B model. SafeChain, SafeKey, and STAIR, all cited in the paper, are absent from the main quantitative performance comparisons, although SafeChain appears in the auxiliary KL-divergence analysis. There are also no inference-time defenses, such as activation steering or prompt-based methods, for comparison.

  2. The gains in capability retention are uneven, and the safety–utility trade-off deserves clearer discussion. On the 7B model, Ours (SFT) improves Avg2 only modestly over standard SFT (62.83 vs. 62.13) and remains below the base model (67.70), while MMLU falls from 46.49 under standard SFT to 40.61. Its templated average DSR also decreases from 90.20 to 86.11. These results support partial mitigation of the alignment tax, but not the stronger wording in Section 5.2 that the method “circumvents” it.

Questions for authors
  1. Could inference-time defenses offer a cheaper alternative? Activation-steering approaches, such as AlphaSteer, likewise aim to maintain safety without impairing benign behavior and require no additional fine-tuning of the underlying LLM. How do they perform under the paper’s template-stripping setting? If they achieve comparable DSR/FRR, the training cost of the proposed method would require further justification.

  2. Can AGCA’s contribution be distinguished from that of adding arbitrary benign data? What happens to over-refusal and reasoning performance if the benign queries generated by AGCA are replaced with an equal number of ordinary benign queries—for example, Alpaca or XSTest-safe-style samples—without sensitive-word preservation or attribution guidance?

4 comments

We thank the reviewer for recognizing our analysis of shortcut learning and the motivation behind DeShortcut-Align. We address the concerns regarding baseline coverage, capability retention, and the two additional questions below.

W1. Insufficient Experimental Baselines

We agree that broader baseline comparisons would strengthen the evaluation. The original paper already includes STAR-1, RealSafe-R1, TARS, and PPO in Appendix Table 5, alongside jailbreak comparisons in Table 4. We further evaluate SafeKey [1], SafeChain [2], and SInternal-7B [3], and update both standard supervised fine-tuning (SFT) and DeShortcut-Align (SFT) using batch size 128, matching the official STAR-1 configuration [4]. Following SAVeR² [5], we adopt a four-dimensional harmonic mean (H-mean) to jointly evaluate standard-template safety, template robustness, over-refusal (measured by the false refusal rate, FRR), and general capabilities.

Table R1. Expanded safety alignment baseline comparison (%).

MethodSafety ↑Δavg ↓FRR ↓General ↑H-mean ↑
SFT-based Methods
SFT (bs128)89.9014.0835.6765.6074.70
STAR-1 [4]90.6715.4035.0063.5074.10
RealSafe-R196.688.9969.0063.4557.68
SafeKey [1]88.9711.8230.0060.0574.74
SafeChain [2]66.408.733.3364.1176.98
Ours (SFT, bs128)86.512.7527.0067.3579.38
Reinforcement learning (RL)-based Methods
PPO98.2417.4179.3363.9146.34
GRPO99.916.5474.3363.8553.10
TARS79.0015.1512.3365.2078.14
SInternal-7B [3]95.6113.0325.0062.2577.89
Ours (RL)97.073.6331.0065.8079.41

Bold values indicate the best result in each metric column within each training paradigm; ties are bolded.

H-mean is calculated over Safety, 100−Δavg, 100−FRR, and General.

To address the missing inference-time baselines and examine their safety effectiveness as alternatives to training-time alignment, we additionally evaluate AlphaSteer [6] and Any-Depth Alignment with a linear probe (ADA-LP) [7] on DeepSeek-R1-Distill-Qwen-7B under standard chat templates.

Table R2. Comparison with inference-time safety interventions (defense success rate, DSR, %).

BenchmarkBase ↑AlphaSteer ↑ADA-LP ↑Ours (SFT) ↑Ours (RL) ↑
WildJailbreak45.2054.8077.2077.2097.20
StrongREJECT35.1438.6653.3599.3698.61
WildChat58.9269.7378.6582.9795.41
Average46.4254.4069.7386.5197.07

Bold values indicate the best result for each benchmark or average across methods; ties are bolded.

The expanded comparisons show that DeShortcut-Align achieves the highest four-dimensional H-mean under both SFT and RL, while also outperforming the evaluated inference-time defenses in standard-template DSR. Although specialized methods differ in training budgets and intervention mechanisms, these results support the competitiveness of shortcut mitigation across multiple safety objectives. We will incorporate the expanded comparisons into the revised paper.

W2. Capability Retention and Safety–Utility Trade-off

We acknowledge that the original SFT results showed limited capability retention and reduced standard-template safety. One contributing factor may be the original batch size of 8, which results in approximately 16 times more optimizer updates than the effective batch size of 128 used by STAR-1 [4], given identical data and epochs. As further analyzed in our response to Reviewer 9S8Q, training configurations substantially affect over-refusal. We therefore reevaluate both standard SFT and DeShortcut-Align (SFT) using batch size 128 under identical training configurations.

Table R3. Updated SFT comparison with batch size 128 (%).

MethodSafety ↑Δavg ↓XSTest ↓OKTest ↓FR-test ↓FRR ↓MMLU ↑General ↑H-mean ↑
SFT (bs128)89.9014.0835.0021.0051.0035.6746.8665.6074.70
Ours (SFT, bs128)86.512.7528.0014.0038.0027.0048.3967.3579.38

Under the matched configuration, DeShortcut-Align improves MMLU and overall general capabilities while substantially reducing template dependence and over-refusal. Although standard-template safety remains slightly lower, the results indicate a more favorable overall safety–utility trade-off, as reflected in the improved H-mean. We also agree that "mitigates the alignment tax" is more accurate than "circumvents the alignment tax" and will revise Section 5.2 accordingly.

Q1. Can Inference-Time Defenses Offer a Cheaper Alternative?

(1) Complementary roles of safety alignment and inference-time defenses. Training-time safety alignment and inference-time defenses such as AlphaSteer [6] serve complementary roles. Safety alignment, typically through supervised fine-tuning (SFT) or reinforcement learning (RL), aims to establish intrinsic safety behavior by training models to recognize harmful intent and generate appropriate responses. In contrast, inference-time methods such as AlphaSteer enhance safety by manipulating existing internal representations without updating the underlying model parameters. While these approaches can provide additional protection at relatively low training cost, their effectiveness depends on the safety-related representations and capabilities already present in the model. Inference-time defenses therefore do not necessarily replace alignment training; both can be combined.

DeShortcut-Align specifically addresses a limitation of the safety alignment process itself: models may achieve high safety rewards by learning superficial formatting and lexical shortcuts rather than sufficiently relying on the underlying harmful intent. Our objective is to mitigate these shortcuts during SFT/RL, thereby improving the robustness of the model's intrinsic safety behavior. Although DeShortcut-Align introduces additional training costs, the resulting aligned model requires no additional steering operations or safety probing during inference. The approaches thus involve different training–inference cost trade-offs.

(2) Empirical comparison and limitations. Table R2 reports standard-template results; template-removal results are summarized below. AlphaSteer improves the average DSR over Base (54.40% vs. 46.42%) with templates, but drops to 46.01% when templates are removed. AlphaSteer was originally developed for instruction-tuned models, whereas our evaluation focuses on reasoning models with more complex chain-of-thought (CoT) generation, where activation-level interventions may transfer less reliably and potentially interfere with reasoning. Moreover, such interventions operate on the existing policy and do not directly eliminate its learned template dependence. In contrast, ADA-LP improves from 69.73% to 73.37% without templates. This may be related to its mechanism of re-injecting assistant-header safety tokens to probe the model's internal safety signals: we observe more frequent injections without templates (47.44 vs. 37.95 on average), potentially strengthening its inference-time protection rather than eliminating the underlying shortcuts. DeShortcut-Align (RL) achieves substantially higher DSR in both settings (97.07% / 93.44%) without additional inference-time interventions. We acknowledge that these approaches offer complementary benefits and that our comparison is not compute-matched. Nevertheless, inference-time defenses do not necessarily address the underlying shortcut learning: excessive reliance on superficial cues, rather than harmful intent, can leave safety behavior brittle. Mitigating such shortcuts during safety alignment is therefore important for improving the policy's intrinsic robustness.

Q2. AGCA vs. Ordinary Benign Data Augmentation

We agree that the contribution of Attribution-Guided Contrastive Augmentation (AGCA) should be distinguished from simply adding benign data. We therefore conduct equal-budget benign-data comparisons, replacing AGCA-generated samples with the same number of benign samples from Alpaca [8] or STAR-1 [4], while keeping harmful training data and optimization settings fixed.

Table R4. Comparison of benign data augmentation strategies (%).

Benign Data SourceSafety ↑Δavg ↓XSTest ↓OKTest ↓FR-test ↓FRR ↓H-mean ↑
Alpaca [8]91.336.2332.0021.0059.0037.3379.85
STAR-1 Benign [4]86.299.2434.0017.0046.0032.3380.25
AGCA (Ours)86.512.7528.0014.0038.0027.0084.41

Bold values indicate the best result in each metric column; ties are bolded.

H-mean is the harmonic mean of Safety, 100 − Δavg, and 100 − FRR.

AGCA achieves the lowest template gap (2.75) and over-refusal rate (27.00%), along with the highest H-mean (84.41). Although Alpaca achieves higher standard-template safety, these results indicate that AGCA's benefits cannot be explained solely by adding benign data.

Furthermore, the original Table 3 compares Random, Gradient, Attention, and Token Sensitivity Score (TSS) selection within the same augmentation framework. TSS achieves the lowest over-refusal while maintaining competitive safety performance. Together, these experiments support targeting refusal-sensitive vocabulary and constructing benign counterfactuals that preserve such vocabulary while changing the overall intent, rather than relying on ordinary benign-data augmentation or arbitrary token selection.

References

[1] Kaiwen Zhou et al. SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning. EMNLP, 2025. https://aclanthology.org/2025.emnlp-main.1291/

[2] Fengqing Jiang et al. SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities. Findings of ACL, 2025. https://aclanthology.org/2025.findings-acl.1197/

[3] Yi Zhang et al. Internalizing Safety Understanding in Large Reasoning Models via Verification. ICML, 2026. https://proceedings.mlr.press/v306/zhang26n.html

[4] Zijun Wang et al. STAR-1: Safer Alignment of Reasoning LLMs with 1K Data. AAAI, 2026. https://arxiv.org/abs/2504.01903

[5] Yichen Sun et al. SAVeR²: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards. arXiv:2611.07482, 2026. https://arxiv.org/abs/2611.07482

[6] Leheng Sheng et al. AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint. arXiv:2506.07022, 2025. https://arxiv.org/abs/2506.07022

[7] Jiawei Zhang et al. Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth. arXiv:2510.18081, 2025. https://arxiv.org/abs/2510.18081

[8] Rohan Taori et al. Stanford Alpaca: An Instruction-following LLaMA Model. Stanford CRFM, 2023. https://github.com/tatsu-lab/stanford_alpaca

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback, which we have carefully addressed in our author response. As the discussion deadline is approaching, we would greatly appreciate any further feedback you may have. If our responses have resolved your main concerns, we would also be grateful if you could update your overall assessment accordingly.

Sincerely,

The Authors

Reviewer_9S8QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that although standard SFT and RLVR pipelines achieve near-perfect safety scores on in-domain benchmarks, they frequently induce spurious shortcut learning—specifically formatting shortcuts (binding refusal behavior to structural prompt templates) and lexical shortcuts (reflexively refusing benign requests containing sensitive keywords). To mitigate these issues, the authors propose DeShortcut-Align, a three-stage alignment framework that: (1) performs Refusal Sensitivity Attribution by masking input tokens to quantify their influence on the refusal distribution, (2) applies Attribution-Guided Contrastive Augmentation by constructing benign contrastive prompts containing high-sensitivity keywords, and (3) introduces Counterfactual Consistency Regularization by blinding attention on template tokens to enforce consistent predictions between original and template-ablated inputs. Empirical results on 7B and 14B models suggest that DeShortcut-Align improves robustness against template-stripping attacks and reduces over-refusal while maintaining general reasoning performance.

Critical strengths
  • Problem Formulation and Training Dynamics Analysis: The paper investigates the trade-off between safety alignment and over-refusal in Large Reasoning Models (LRMs) from the perspective of shortcut learning. It categorizes the observed failure modes into structural (formatting) and token-level (lexical) shortcuts, and tracks how these behaviors evolve across three stages of RL training.
  • Integration with Standard Alignment Pipelines: The proposed framework combines data-level contrastive augmentation with attention-level consistency regularization, which can be applied to standard alignment algorithms (SFT, PPO, and GRPO) and evaluated across 7B and 14B model scales on safety, over-refusal, and general reasoning benchmarks.
Critical weaknesses

W1: Questionable Motivation for Formatting Shortcuts and Anomalous Baseline Over-Refusal

  1. Practicality of the Formatting Shortcut Threat Model: From a motivation standpoint, it is questionable whether relying on a default chat template during alignment is genuinely undesirable in practice. Using a fixed structural template during SFT and RLVR is standard practice that promotes training stability and clear role separation. From a practical safety perspective, closed-source frontier models enforce opaque chat templates and fix special template tokens at the system/API level, meaning external attackers cannot arbitrarily strip or manipulate the chat template during inference (and even for open-weight models, standard deployment frameworks enforce fixed chat templates). The authors should better justify this motivation: if attackers cannot manipulate the template in realistic deployment scenarios, trading off standard in-distribution safety performance for robustness under template-stripping conditions seems poorly motivated.

  2. Unreliably High Over-Refusal in Baselines: In the experimental evaluations (Tables 1, 2, and 3), the over-refusal rates reported for the SFT and GRPO baselines are anomalously high. For instance, in Table 1, Distill-Qwen-7B trained with SFT on the STAR-1 harmful split (even when paired with contrastive benign data) exhibits an XSTest over-refusal rate of ~56%, which deviates substantially from the results reported in the original STAR-1 paper (even those trained exclusively on harmful data) [1] and other prior reproductions [2] . Such abnormally high over-refusal strongly suggests that the baselines may be improperly configured—for example, due to over-training (e.g., an excessively high learning rate or too many training epochs) or an imbalanced data mixture (e.g., a disproportionately high ratio of harmful-to-benign data). This over-optimization is also evident in the training dynamics plotted in Figure 2. Furthermore, even after applying DeShortcut-Align, the resulting over-refusal rates remain unacceptably high for practical deployment.

W2: Ambiguities in Refusal Sensitivity Attribution and Token Masking

In the problem formulation, harmful prompts are conceptually divided into tokens representing Harmful Intention and those representing Spurious Shortcut Features. However, Stage 1 (Refusal Sensitivity Attribution) relies on masking input tokens to measure refusal sensitivity, which introduces two critical issues that are not adequately addressed:

  1. Disentangling Harmful Intention from Spurious Shortcuts (Label Flipping): If token masking perturbs the core harmful intention tokens, the semantic label of the prompt should effectively flip from harmful to benign (making a non-refusal appropriate). Conversely, if masking only removes spurious shortcut features, the harmful label should remain unchanged. It is unclear how the attribution mechanism distinguishes between true harmful-intent tokens and spurious lexical triggers, or how the framework accounts for label flipping during attribution and subsequent training.
  2. Semantic Corruption under Token Masking: Masking arbitrary tokens can easily break grammatical structure and corrupt the semantics of the original prompt (e.g., turning a harmful query into a nonsensical or meaningless sequence). Measuring refusal sensitivity or regularizing the model on semantically broken inputs appears ill-posed. The authors should clarify how semantic validity is maintained during token masking and how spurious tokens are reliably separated from core intent tokens during training.

W3: Insufficient Experimental Baselines

The experimental comparisons in the main tables are restricted to vanilla alignment methods (standard SFT, PPO, and GRPO) and lack comparisons with prior data-augmentation or shortcut-mitigation baselines for safety alignment. More competitive baselines should be included—for example.

[1] Wang, Zijun, et al. "Star-1: Safer alignment of reasoning llms with 1k data." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 44. 2026.

[2] Zhang, Yi, et al. "Internalizing Safety Understanding in Large Reasoning Models via Verification." arXiv preprint arXiv:2605.08930 (2026).

Questions for authors

refer to weakness

8 comments

We thank the reviewer for the detailed feedback. We address the concerns regarding research motivation, baseline configurations, token attribution, and experimental comparisons below.

W1.1. Motivation and Practical Significance of Formatting Shortcuts

(1) Research motivation: From shortcut learning to safety robustness. Our motivation stems from an observation in Figure 1: removing chat templates substantially degrades safety performance even when the underlying user instructions remain unchanged. This prompted us to investigate the underlying training dynamics. Figure 2 reveals that models tend to learn formatting shortcuts first, followed by lexical shortcuts, while reward saturation and diminishing policy gradients lead to gradient starvation. These findings suggest that models may optimize safety rewards through superficial refusal patterns without sufficiently learning the relationship between harmful intent and safe refusal, potentially limiting their robustness against challenging jailbreak attacks. To examine this broader concern, we summarize the jailbreak results from the original Table 4 and additionally evaluate SInternal-7B.

Table R1. Jailbreak defense success rates (%) under supervised fine-tuning (SFT) and reinforcement learning (RL).

MethodPAIR ↑GCG ↑TAP ↑Avg. ↑
STAR-1 [1]84.084.092.086.7
RealSafe-R194.089.098.093.7
SInternal-7B94.092.096.094.0
SFT70.092.086.082.7
Ours (SFT)86.095.094.091.7
GRPO96.084.098.092.7
Ours (RL)100.095.098.097.7

Bold values indicate the best result in each metric column; ties are bolded.

DeShortcut-Align improves average jailbreak defense by 9.0 points under SFT and 5.0 points under RL, supporting the broader value of shortcut mitigation beyond template stripping. Our goal is not to defend against template removal in deployment, but to mitigate shortcut learning during safety alignment. The template gap serves as a diagnostic measure of formatting dependence, while the broader benefits extend to jailbreak robustness, reduced over-refusal, and better preservation of general capabilities.

(2) Counterfactual Consistency Regularization (CCR) preserves role separation and standard-template safety. We agree that fixed chat templates are important for training stability and role separation. CCR preserves the original templates in standard SFT/RL training and introduces only an auxiliary consistency regularizer to reduce reliance on formatting cues. Moreover, mitigating formatting shortcuts does not necessarily sacrifice standard-template safety: the CCR-only ablation (without Attribution-Guided Contrastive Augmentation, AGCA) in the original Table 2 improves standard-template defense success rate (DSR) from 90.20% to 91.06% while reducing the template gap from 12.83 to 4.82. Thus, CCR encourages greater reliance on query intent without removing the structural benefits of chat templates.

W1.2. Baseline Over-Refusal and Training Configurations

(1) SFT training configurations and over-refusal. We acknowledge that the original SFT baseline exhibits higher over-refusal than reported in STAR-1 [1]. Our initial experiments used batch size 8, whereas the official STAR-1 configuration uses an effective batch size of 128. With identical data and epochs, this corresponds to approximately 16 times more optimizer updates. We therefore conducted additional batch-size comparisons. Increasing the batch size to 128 reduces the SFT baseline's XSTest over-refusal from 56% to 35% and average false refusal rate (FRR) from 48.67% to 35.67%. Under the same updated configuration, DeShortcut-Align further reduces FRR to 27.00% and the template gap to 2.75 (Global Table G2). These results confirm that optimization configurations materially affect over-refusal, while the improvements from shortcut mitigation persist under the revised setting.

(2) Training dynamics and over-optimization. According to the training configuration in the Appendix, our reported RL baseline is trained for 100 steps, not 150. Figure 2 extends training to 150 steps solely to investigate optimization dynamics after reward saturation. To address the concern about over-optimization, we additionally evaluate the Step-50 checkpoint, where the training reward is already close to 1, and compare it with the final results.

Table R3. Safety performance at different training checkpoints (%).

MethodSafety ↑Δavg ↓FRR ↓PAIR ↑GCG ↑TAP ↑Attack Avg. ↑
Step 5089.2219.9518.0091.080.092.087.7
Step 100 (Final)99.916.5474.3396.084.098.092.7
Ours (RL)97.073.6331.00100.095.098.097.7

Bold values indicate the best result in each metric column; ties are bolded.

Despite near-saturated training rewards at Step 50, the model exhibits lower safety performance, substantial template dependence, and weaker jailbreak robustness. Further optimization improves safety but sharply increases over-refusal. These results suggest that reward saturation does not necessarily indicate robust safety learning, and early stopping alone cannot achieve the same safety–utility trade-off as DeShortcut-Align.

(3) Over-refusal evaluation protocol and research scope. As detailed in Appendix D.1, we evaluate 100 benign samples each from XSTest, OKTest, and FalseReject [2]. FalseReject contains contextually benign but seemingly harmful requests and is substantially more challenging than XSTest. To contextualize the concern about high absolute over-refusal rates, we report published results from FalseReject [2], where even strong models exhibit high refusal rates:

Table R4. Over-refusal rates reported in FalseReject [2] (%).

ModelXSTest ↓FalseReject-Test ↓
Claude-3.5-Sonnet13.2058.13
Claude-3.5-Haiku17.6084.33
Llama-3.1-8B-Instruct8.8048.10

Bold values indicate the best result in each metric column; ties are bolded.

Under our evaluation protocol, the officially released STAR-1 model also exhibits 35.00% average FRR, including 51.00% on FalseReject (original Appendix Table 5). These results highlight the importance of evaluation distributions when interpreting absolute over-refusal rates, although cross-study results are not directly comparable.

Our primary objective is to investigate and mitigate shortcut learning during safety alignment, rather than extensively optimize data mixtures or training hyperparameters for the lowest possible FRR. We therefore focus on relative improvements under consistent training and evaluation settings. We acknowledge that further tuning of data composition and optimization configurations would be necessary to meet practical deployment requirements.

W2. Clarification of Refusal Sensitivity Attribution and Token Masking

(1) Conceptual formulation of harmful intent and shortcut features. In Section 3.2, the decomposition into semantic intent (CC) and shortcut features (SS) is conceptual rather than a token-level classification. Harmful intent is determined by the compositional semantics and context of the entire query, rather than individual tokens. Therefore, our framework does not assume that tokens can be explicitly separated into harmful-intent and spurious-shortcut categories.

(2) How Token Sensitivity Score (TSS) and AGCA mitigate lexical shortcuts.

We first clarify that Stage 1 does not aim to distinguish harmful-intent tokens from spurious shortcut tokens. Instead, Refusal Sensitivity Attribution (RSA) is an offline procedure that identifies tokens whose removal substantially changes the model's refusal behavior. A refusal-sensitive token is not necessarily a shortcut or a harmful-intent token; even when masking changes the query's intent and consequently the model's refusal decision, this indicates refusal sensitivity rather than proving a spurious shortcut. The motivation for RSA is that such tokens represent potential lexical cues that the model may over-rely on for refusal during safety alignment, making them valuable targets for counterfactual augmentation.

Shortcut mitigation occurs during the subsequent augmentation and training stages. Stage 2 applies Attribution-Guided Contrastive Augmentation (AGCA) to construct benign queries that preserve these refusal-sensitive tokens while changing the overall intent. Stage 3 trains on harmful and benign queries containing overlapping sensitive vocabulary, encouraging the model to rely more on contextual intent rather than isolated lexical cues. To verify whether targeting refusal-sensitive tokens is more effective than alternative selection strategies, we compare TSS with random, gradient-based, and attention-based selection under the same augmentation framework.

Table R5. Comparison of token selection strategies (%).

StrategyWildJailbreak ↑StrongReject ↑WildChat ↑Safety Avg. ↑XSTest ↓OKTest ↓FR-test ↓FRR ↓
Standard SFT84.8099.0486.7690.2056.0024.0066.0048.67
+ Random62.8098.4079.4680.2240.0021.0045.0035.33
+ Gradient61.6097.4481.3580.1339.0014.0050.0034.33
+ Attention71.2099.0485.6885.3136.0019.0043.0032.67
+ TSS (Ours)76.0098.8782.7085.8634.008.0042.0028.00

Bold values indicate the best result in each metric column; ties are bolded.

Among the augmentation strategies, TSS achieves the lowest over-refusal rate (28.00%) while maintaining competitive safety performance. These results suggest that targeting refusal-sensitive tokens provides more effective counterfactual augmentation than random or alternative token selection strategies, supporting our motivation for using RSA to mitigate lexical shortcuts during training.

(3) Semantic validity and label flipping under token masking. We acknowledge that token masking can potentially disrupt syntax or alter the original intent. However, token erasure and masking are established perturbation-based attribution techniques for measuring how individual tokens influence model predictions [3–5]. To directly assess semantic corruption, we sampled 50 queries from our attribution data, applied single-token masking and rewriting as comparison settings, and regenerated model responses. We measured changes in response log-perplexity (Δlog-PPL) and conducted human evaluations of prompt understandability and semantic invalidity.

Table R6. Semantic validity of perturbed prompts.

SettingΔlog-PPLUnderstandable ↑Invalid ↓
Base0.000096.0%4.0%
Mask−0.097594.0%6.0%
Rewrite−0.0744100.0%0.0%

Single-token masking preserves prompt understandability in 94% of cases, close to the original 96%, with only a modest change in response log-PPL. The results suggest that masking typically does not turn queries into meaningless sequences. Importantly, masked inputs are used only for offline refusal-sensitivity attribution, not as labeled SFT or RL training samples. Thus, potential intent changes do not directly introduce incorrect training labels, although they may affect attribution accuracy. We use single-token masking to limit such perturbations, with Appendix Table 10 further showing better downstream performance than longer masking spans. The actual training samples are complete benign counterfactuals generated by Attribution-Guided Contrastive Augmentation (AGCA), with 99.33% independent-model and 98.67% human verification accuracy (Appendix Table 11).

Reviewer_sGaQOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a solution for short-cut learning during alignment. They propose a method to handle over-refusal of benign tokens, and degraded general reasoning during training for alignment. Specifically addressing models refusal behavior to superficial correlates rather than genuine semantic intent. Two shortcut types are focussed in this work.

  1. Formatting shortcuts
  2. Lexical shortcuts

The main method includes 3 distinct steps:

  1. Refusal sensitivity attribution (RSA): Token Sensitivity Score (TSS) computed via single-token masking + teacher-forced LL shift on the refusal answer tokens
  2. Attribution-Guided Contrastive Augmentation (AGCA): Uses top-K TSS tokens to constrain GPT-4o to generate intention-inverted benign queries that retain the trigger vocabulary but flip intent.
  3. Counterfactual Consistency Regularization (CCR): Create a counterfactual input, then penalizes divergence between factual/counterfactual predictions, applied in both SFT and RL.

The paper is well written and the experiments cover diverse datasets.

Critical strengths

The attention-blinding counterfactual mechanism is novel operationalization of "format invariance" that is differentiable and pluggable into existing SFT/PPO/GRPO pipelines without architecture change.

Critical weaknesses

Judge-based reward circularity risk: Both the RL reward and CCR depend on GPT-4o-as-judge; while there's a human-agreement check for Llama-Guard (eval metric), there isn't an equivalent check reported for GPT-4o-as-training-judge's reliability/consistency, needs a check since training signal quality directly drives all downstream results.

Small training data (1000 SFT prompts, 340-400 RL prompts), the generalizability of TSS-based attribution and AGCA beyond the narrow harmful-query distributions seems questionable.

Questions for authors

A comparison with refusal direction / "circuit breakers" work, representation-level safety interventions is absent, even though CCR's attention-blinding is conceptually adjacent to representation-level interventions. Such comparison would make this paper stronger.

2 comments

We thank the reviewer for the constructive feedback. We address the concerns regarding reward reliability, training-data generalization, and representation-level safety interventions below.

W1. Reliability of GPT-4o Training Rewards

We clarify that Counterfactual Consistency Regularization (CCR) reuses the GPT-4o safety judgments obtained during reinforcement learning (RL) reward evaluation, rather than introducing a separate GPT-4o evaluation. To assess the reliability of this shared training signal, we conduct a human agreement study on 50 harmful and 50 benign responses sampled from RL training rollouts, with human annotators and GPT-4o independently applying the same evaluation criteria. We report raw agreement and Cohen's κ.

Table R1. Human–GPT-4o agreement on training rewards.

Reward DimensionAgreement ↑Cohen's κ ↑
Harmful — Think Safety92.0%0.84
Harmful — Answer Safety96.0%0.92
Benign — Helpful Response94.0%0.88

All three dimensions achieve at least 92% agreement, with Cohen's κ ranging from 0.84 to 0.92, indicating strong consistency with human judgments. These results provide direct evidence supporting the reliability of the GPT-4o training reward, complementing our original Llama-Guard evaluation and human validation in Appendix E.8.

W2. Generalization Beyond Small Training Datasets

Our original training scale follows data-efficient safety alignment settings such as STAR-1 [1], enabling comparisons under limited data budgets. To examine whether our improvements persist beyond this setting, we conduct new data-scaling experiments using harmful queries from SafeChain [2] and corresponding benign samples constructed through Attribution-Guided Contrastive Augmentation (AGCA). We compare standard supervised fine-tuning (SFT) with DeShortcut-Align (SFT) across training-data scales from 40% to 200%, consistently using batch size 128.

Table R2. Generalization across training-data scales.

Data ScaleSFT Δavg ↓Ours Δavg ↓SFT XSTest ↓Ours XSTest ↓
40%17.2111.704341
80%14.233.913129
100%14.082.753528
120%14.943.604030
160%13.313.323332
200%14.462.893628

Across all evaluated data scales, DeShortcut-Align consistently reduces template dependence and over-refusal, with performance remaining relatively stable as the dataset expands. These results support the effectiveness of our approach across larger training scales and an additional harmful-data source, rather than only under the original small-data setting.

Q1. Comparison with Representation-Level Safety Interventions

(1) Distinction from representation-level interventions. Existing methods such as Refusal Direction [3] and AlphaSteer [4] investigate or manipulate refusal-related representations, while Circuit Breakers [5] modifies internal representations through additional training. More recently, Any-Depth Alignment (ADA) [6] leverages safety signals in assistant-header hidden states to detect and interrupt harmful generation. In contrast, DeShortcut-Align targets shortcut learning during safety alignment. Its Counterfactual Consistency Regularization (CCR) uses attention blinding to reduce formatting dependence during training, rather than directly steering refusal representations or applying inference-time safety checks. These approaches are therefore complementary rather than direct substitutes.

(2) Additional representation-level baseline experiments. To address the requested comparison with representation-level safety interventions, we evaluate two representative inference-time methods on DeepSeek-R1-Distill-Qwen-7B: AlphaSteer [4], which steers refusal-related activations, and ADA with a linear probe (ADA-LP) [6], which uses a linear probe over Safety-Token representations to detect harmful continuations. For AlphaSteer, we follow the original steering procedure with λ = +0.45. We additionally evaluate ADA-LP under standard chat templates.

Table R3. Comparison with representation-level safety interventions (defense success rate, DSR, %).

BenchmarkBase ↑AlphaSteer ↑ADA-LP ↑Ours (SFT) ↑Ours (RL) ↑
WildJailbreak45.2054.8077.2077.2097.20
StrongREJECT35.1438.6653.3599.3698.61
WildChat58.9269.7378.6582.9795.41
Average46.4254.4069.7386.5197.07

Under the evaluated configurations, AlphaSteer provides limited improvements, while ADA-LP achieves more substantial gains through inference-time intervention. These results illustrate the potential of representation-based defenses, although their effectiveness depends on the intervention design and evaluation setting. Unlike these approaches, DeShortcut-Align improves the aligned policy during training without requiring additional inference-time steering or safety probing. We will incorporate these comparisons and clarify their complementary roles in the revised paper.

References

[1] Zijun Wang et al. STAR-1: Safer Alignment of Reasoning LLMs with 1K Data. AAAI, 2026. https://arxiv.org/abs/2504.01903

[2] Fengqing Jiang et al. SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities. Findings of ACL, 2025. https://aclanthology.org/2025.findings-acl.1197/

[3] Andy Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS, 2024. https://arxiv.org/abs/2406.11717

[4] Leheng Sheng et al. AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint. arXiv:2506.07022, 2025. https://arxiv.org/abs/2506.07022

[5] Andy Zou et al. Improving Alignment and Robustness with Circuit Breakers. NeurIPS, 2024. https://arxiv.org/abs/2406.04313

[6] Jiawei Zhang, Andrew Estornell, David D. Baek, Bo Li, and Xiaojun Xu. Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth. arXiv:2510.18081, 2025. https://arxiv.org/abs/2510.18081

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Global Response

We thank the reviewers for their constructive feedback. Here, we clarify our research motivation, address shared concerns, and summarize the additional experimental evidence. Reviewer-specific concerns are addressed in the individual responses.

G0. Research Motivation: From Shortcut Learning to Safety Robustness

Our motivation stems from an observation in Figure 1: removing chat templates substantially degrades safety performance even when the underlying user instructions remain unchanged. This prompted us to investigate the underlying training dynamics. Figure 2 reveals that models tend to learn formatting shortcuts first, followed by lexical shortcuts, while reward saturation and diminishing policy gradients lead to gradient starvation. These findings suggest that models may optimize safety rewards through superficial refusal patterns without sufficiently learning the relationship between harmful intent and safe refusal, potentially limiting their robustness against challenging jailbreak attacks. To examine this broader concern, we summarize the jailbreak results from the original Table 4 and additionally evaluate SInternal-7B.

Table G1. Jailbreak defense success rates (%).

MethodPAIR ↑GCG ↑TAP ↑Avg. ↑
STAR-1 [1]84.084.092.086.7
RealSafe-R1 [2]94.089.098.093.7
SInternal-7B [6]94.092.096.094.0
SFT70.092.086.082.7
Ours (SFT)86.095.094.091.7
GRPO96.084.098.092.7
Ours (RL)100.095.098.097.7

Bold values indicate the best result in each metric column; ties are bolded.

DeShortcut-Align improves average jailbreak defense by 9.0 points under SFT and 5.0 points under RL, supporting the broader value of shortcut mitigation beyond template stripping. Our goal is not to defend against template removal in deployment, but to mitigate shortcut learning during safety alignment. The template gap serves as a diagnostic measure of formatting dependence, while the intended benefits extend to stronger jailbreak robustness, reduced over-refusal, and better preservation of general capabilities. Standard chat templates remain unchanged during deployment.

G1. Expanded Safety Alignment Baselines and Safety–Utility Trade-offs

To address limited baseline coverage and assess the overall safety–utility trade-off, we extend the comparisons in Appendix Table 5 by including SafeKey [3], SafeChain [4], and SInternal-7B [6], alongside previously evaluated methods [1,2,5]. To address concerns about the original SFT configuration, we additionally reevaluate both standard SFT and DeShortcut-Align (SFT) with batch size 128, matching the official STAR-1 configuration [1], while keeping their optimization settings consistent.

Following SAVeR² [8], which employs the harmonic mean to evaluate safety–utility trade-offs, we extend this metric to four dimensions: standard-template safety, template robustness, over-refusal (measured by the false refusal rate, FRR), and general capabilities.

H4=HM⁡(Safety,100−Δavg,100−FRR,General).H_4=\operatorname{HM}(\mathrm{Safety},100-\Delta_{\mathrm{avg}},100-\mathrm{FRR},\mathrm{General}).

Table G2. Expanded comparison on DeepSeek-R1-Distill-Qwen-7B.

MethodSafety ↑Δavg ↓FRR ↓General ↑H-mean ↑
SFT-based methods
SFT (bs128)89.9014.0835.6765.6074.70
STAR-1 [1]90.6715.4035.0063.5074.10
RealSafe-R1 [2]96.688.9969.0063.4557.68
SafeKey [3]88.9711.8230.0060.0574.74
SafeChain [4]66.408.733.3364.1176.98
Ours (SFT, bs128)86.512.7527.0067.3579.38
RL-based methods
PPO98.2417.4179.3363.9146.34
GRPO99.916.5474.3363.8553.10
TARS [5]79.0015.1512.3365.2078.14
SInternal-7B [6]95.6113.0325.0062.2577.89
Ours (RL)97.073.6331.0065.8079.41

Bold values indicate the best result in each metric column within each training paradigm; ties are bolded.

DeShortcut-Align achieves the highest four-dimensional H-mean among the evaluated methods in both SFT and RL settings. While its standard-template safety is slightly lower than that of the corresponding vanilla baselines, it substantially reduces template dependence and over-refusal while better retaining general capabilities. These results support an improved overall trade-off rather than uniform improvements on every metric.

G2. Is Attribution-Guided Contrastive Augmentation (AGCA) More Effective Than Ordinary Benign Data Augmentation?

To distinguish the effect of AGCA from simply adding benign data, we conduct a controlled comparison using equal numbers of benign samples from Alpaca [7], STAR-1 [1], and AGCA. The harmful training data and training configurations are held fixed, with only the benign data source changed.

Table G3. Controlled benign-data augmentation comparison under SFT (batch size 128). H-mean (3D) combines Safety, 100−Δavg, and 100−FRR, and is reported only for this ablation.

Benign Data SourceSafety ↑Δavg ↓FRR ↓H-mean (3D) ↑
Alpaca [7]91.336.2337.3379.85
STAR-1 Benign [1]86.299.2432.3380.25
AGCA (Ours)86.512.7527.0084.41

Bold values indicate the best result in each metric column; ties are bolded.

Under the same benign-data budget, AGCA achieves the lowest template-dependency gap and false refusal rate, although Alpaca retains higher standard-template safety. This suggests that the benefits of AGCA are not explained solely by the addition of benign samples. Together with the attribution-strategy ablation in the original Table 3, these results support the value of constructing benign counterfactuals that preserve refusal-sensitive tokens while changing the overall intent.

References

[1] Zijun Wang et al. STAR-1: Safer Alignment of Reasoning LLMs with 1K Data. AAAI, 2026. https://arxiv.org/abs/2504.01903

[2] Yichi Zhang et al. RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability. arXiv:2504.10081, 2025. https://arxiv.org/abs/2504.10081

[3] Kaiwen Zhou et al. SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning. EMNLP, 2025. https://aclanthology.org/2025.emnlp-main.1291/

[4] Fengqing Jiang et al. SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities. Findings of ACL, 2025. https://aclanthology.org/2025.findings-acl.1197/

[5] Taeyoun Kim et al. Reasoning as an Adaptive Defense for Safety. NeurIPS, 2025. https://arxiv.org/abs/2507.00971

[6] Yi Zhang et al. Internalizing Safety Understanding in Large Reasoning Models via Verification. ICML, 2026. https://proceedings.mlr.press/v306/zhang26n.html

[7] Rohan Taori et al. Stanford Alpaca: An Instruction-following LLaMA Model. Stanford CRFM / GitHub, 2023. https://github.com/tatsu-lab/stanford_alpaca

[8] Yichen Sun et al. SAVeR²: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards. arXiv:2611.07482, 2026. https://arxiv.org/abs/2611.07482

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.