acceptodds
Under review as a conference paper at ICLR 2027

ShallowDefend: Harnessing Shallow Safety Alignment for Multi-Turn Defender

Abstract

Safety alignment has enabled large language models to reliably refuse harmful requests directly, yet the same request distributed across a multi-turn conversation still succeeds far too often. This brittleness has a mechanistic root: Alignment is shallow, with safety behavior concentrated in a model's first few output tokens, and existing defenses inherit the weakness, supervising the surface form of the response without ever giving the model an explicit safety decision to learn. We take the opposite view of shallowness: if a short span determines the trajectory of an entire response, that span is the highest-leverage place to intervene, and the defender can occupy it first. We propose Assess Decide Respond, a framework in which the defender assesses the user's intent, commits to an explicit, parseable safety decision, and only then generates a reply conditioned on it, with the commitment trained by reinforcement learning at the payload turn. Because the decision is a token span, the reward can be a rule: a self-verified reward checks the committed label against the conversation's known nature, requiring no judge and no annotation, while the same format also supports an LLM-as-judge reward over a graded decision space with safe redirection. Across two model families and fourteen benchmarks spanning safety, utility, and helpfulness, our defenders achieve the strongest multi-turn safety in a controlled comparison, maintaining their lead in safety while preserving utility at a modest cost to helpfulness. A per-token KL analysis and interventional probes confirm why this works: training concentrates its effect on the brief deliberation that sets a response's trajectory, and the decision committed there is a causal control point. The very shallowness attackers exploit becomes the mechanism of defense.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety alignment, multi-turn jailbreak defense, reinforcement learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_exHZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents ShallowDefend, an RL training framework that requires a defender LLM to explicitly verbalize the user's intent and commit to a safety decision (COMPLY, REFUSE, or SAFE-REDIRECT) before generating a response in multi-turn conversations.

The policy is optimized using rewards computed at the final assistant turn. The authors explore two reward formulations: a static, self-verified reward and an LLM-as-judge reward. They also ablate different components of the framework and investigate which token positions receive policy-gradient updates, including an experiment that masks out the response tokens.

With LLM-as-judge rewards, ShallowDefend improves average safety on Llama-3.1-8B-Instruct from 59.9% to 92.9%, and on Qwen3-8B from 46.1% to 87.6%. The self-verified reward achieves 93.4% safety on Qwen3-8B. The method largely preserves general benchmark performance but noticeably increases over-refusal.

The ablations suggest that the explicit safety decision is a crucial component. Without it, the model may generate unsafe responses even after correctly identifying malicious intent. The partial-gradient experiment further suggests that directly optimizing response tokens is not necessary to achieve substantial safety improvements, as the generated response generally follows the preceding safety decision.

Critical strengths
  1. Addresses multi-turn jailbreaks, which remain among the most effective attacks against safety-aligned LLMs.

  2. Provides informative ablations and mechanistic analyses that support the proposed connection to shallow safety alignment.

  3. Demonstrates effectiveness on both reasoning and non-reasoning LLMs, while largely preserving general benchmark performance.

Critical weaknesses
  1. Utility evaluation bypasses the Assess → Decide → Respond framework. The general utility benchmarks are evaluated in completion format, where the proposed framework is not activated. However, in deployment, the defender cannot know a priori whether a conversation will be benign or adversarial, so the framework must remain active. It is therefore unclear whether the reported utility is preserved under realistic deployment conditions.

  2. The method substantially increases over-refusal. Moreover, Appendix B's helpfulness metric excludes the XSB benchmark, despite reporting it in the main results. Including XSB changes the conclusions of the safety–helpfulness trade-off analysis, particularly the claimed Pareto dominance over competing methods.

  3. The evaluation is limited to unrealistic static multi-turn jailbreaks and compares only against internal defense baselines. The authors do not evaluate against adaptive jailbreaks like X-teaming, where automated attackers adjust their strategies based on the defender's responses. Such attacks better reflect realistic adversarial interactions. The paper also lacks comparisons with external defenses, such as Llama Guard and Qwen3Guard, making it difficult to assess the advantages of integrating safety decisions into the defender rather than using a separate guard model.

  4. The self-verified reward is potentially vulnerable to reward hacking. It only checks whether the generated decision matches the conversation's ground-truth label, without verifying whether the actual response is consistent with that decision. Although the authors provide empirical evidence that this mismatch is not a significant problem for Llama-3.1-8B and Qwen3-8B, it remains unclear whether the same behavior would hold for larger, more capable models that might learn to exploit this reward structure.

Questions for authors

Please see weaknesses.

6 comments

We thank the reviewer for recognizing our model as among the most effective safety-aligned LLMs, and for acknowledging our evaluation as well as its generalization performance. We also thank the reviewer for the feedback, which helped us revise our paper to make it stronger. We hope the following additional experiments and clarifications can address the reviewer's remaining concerns:

1. Clarification on utility evaluation setting

We would like to clarify that our utility evaluation protocol is, in fact, one of the strengths of our paper, reflecting a deliberately rigorous evaluation. Capability benchmarks are commonly evaluated in completion format, which bypasses the chat template and, with it, the safety behavior that alignment modifies. Such evaluations are therefore largely insensitive to safety training and can overstate how well utility is preserved. Had we only followed this practice, our defenders would appear to match the base model (Table 3), and we could have claimed safety gains at no utility cost. Instead, we explicitly highlight this limitation (Figure 2) and complement it with a helpfulness evaluation in chat format with the framework active, covering both conversational quality (MT-Bench) and over-refusal (XSTest, XSB). In short, we follow the standard protocol identically for all methods, and point out its limitation precisely so that it is not taken as the sole evidence of preserved capability.

That said, to further confirm that capabilities are preserved with the framework active, we re-run two suites from Table 3 in the deployment setting: GSM8K for reasoning, and IFEval for instruction following, the axis most sensitive to chat behavior. Each benchmark is queried through the chat template; for the unaligned and aligned models the framework prompt is active and the Intent/Decision preamble is stripped before answer extraction, exactly as a user would receive the reply. GSM8K is evaluated zero-shot with chain-of-thought, and IFEval with its standard prompt-level strict and loose accuracies. Since the chat template unlocks instruction-tuned behavior, absolute scores are higher than in Table 3, so comparisons should be made within the table below, where all rows share the same chat setting.

BackboneMethodGSM8KIFEval (prompt-level strict)IFEval (prompt-level loose)
Llama3.1-8BBase84.571.576.0
Unaligned83.373.476.0
Self reward – aligned82.070.673.4
External reward – aligned79.858.863.0
Qwen3-8BBase94.084.187.2
Unaligned95.078.078.6
Self reward – aligned95.575.277.8
External reward – aligned94.675.276.2

The results confirm that capabilities are largely preserved in the deployment setting. Reasoning is essentially unaffected, and most of the change in instruction following already appears in the unaligned model, indicating that it stems from the deliberation format rather than from training; our self-verified defender stays within three points of its unaligned counterpart on both backbones. Regarding over-refusal, including XSB in the trade-off leaves the trend unchanged (Appendix B, Figure 8): baselines that substantially improve safety incur a comparable helpfulness cost, while ours achieve clearly higher safety at a similar cost.

Revision: We clarify the utility evaluation protocol in Section 3.2 and add the deployment-setting results to Appendix E.1 (Table 4).

2. Clarification on helpfulness cost

We would like to clarify that our helpfulness evaluation targets two aspects: multi-turn conversational quality (MT-Bench) and over-refusal on harmful-looking but benign requests (XSTest, XSB). The original Appendix B took one benchmark per aspect (MT-Bench and XSTest) so that the helpfulness axis weighs quality and over-refusal equally. Including all three would count over-refusal twice and bias the axis toward it. XSB remains fully reported in Table 1.

To address the concern directly, we add Figure 8 to Appendix B, where Helpfulness is [(100 − XSTest) + (100 − XSB) + 10·MT-Bench] / 3. The trend is unchanged. Baselines that substantially improve safety while still falling short of ours (X-Boundary and SFT on Llama3.1-8B; MTSA, MUSE-D and DPO on Qwen3-8B) incur a comparable helpfulness cost, so this cost is inherent to effective multi-turn defense rather than specific to our approach, and at a similar cost our variants achieve clearly higher safety on both backbones. On Qwen3-8B, our external-reward variant is both safer and more helpful than every high-safety baseline (DPO, MTSA, MUSE-D), and our self-verified variant outperforms MTSA, the safest baseline, on both axes.

Revision: Appendix B now explains why the helpfulness axis takes one benchmark per aspect, and adds Figure 8, which repeats the trade-off plot with XSB included. Section 3.2 points to both versions.

3. Comparison with external guard models

We would like to clarify that our method operates on the model side, shaping how the defender itself reasons and responds, whereas external guard models act as filters around a model. The two are therefore complementary rather than direct competitors. Nonetheless, to quantify how the two approaches compare and interact, we evaluate Llama-Guard-3-8B and Qwen3Guard-Gen-4B as input and output filters, both on the base models and on top of our self-verified defenders: the guard classifies every user turn and every reply, and any flagged turn or reply is replaced by a fixed refusal, while the answering model is otherwise unchanged. We use the same suites and judge panel as Table 1 (the guard used as a filter is excluded from that row's judge consensus, and MHJ is judged by gpt-5-mini as in Table 1). We report SAFE % and helpfulness:

BackboneModelAdvBench ↑HarmBench ↑MHJ ↑XSTest ↓XSB ↓MT-Bench ↑
Llama3.1-8BBase54.562.538.010.030.06.20
Base + Llama-Guard-358.766.067.411.237.66.09
Base + Qwen3Guard66.072.844.510.838.56.09
Ours (self-verified)98.899.558.520.454.45.86
Ours + Llama-Guard-399.699.183.120.854.75.77
Ours + Qwen3Guard99.699.767.020.855.05.72
Qwen3-8BBase49.252.522.37.215.66.08
Base + Llama-Guard-362.270.069.18.830.96.05
Base + Qwen3Guard71.177.939.19.232.96.17
Ours (self-verified)97.197.675.226.456.87.36
Ours + Llama-Guard-395.096.787.728.457.47.20
Ours + Qwen3Guard98.197.981.627.257.17.25

The results confirm both points. On their own, guard models are no substitute for our defender: they improve the base model only modestly on AdvBench and HarmBench and fail precisely where attacks are gradual, since each decomposed turn appears harmless to a per-turn filter, so their lower over-refusal comes with little multi-turn robustness. Combined with our defender, however, they are clearly complementary. Adding a guard on top preserves our defender's robustness on AdvBench and HarmBench while substantially improving MHJ, where output filtering is particularly effective, and it does so at almost no additional over-refusal or helpfulness cost, because our defender already refuses most of what the guard would flag.

Revision: We have added this comparison to the new Appendix E.8 (Table 11), which also recommends deploying our defender together with an output-level guard in practice, and referenced it from Section 3.3 and Appendix E.5.

4. More experiments with adaptive attackers

Please see Point 1 of the General Response for the new experiments with four live adaptive attackers, including one that knows the defense, together with the corresponding revisions.

5. Agreement between self-verified results and external evaluation

Please see Point 2 of the General Response for the new label–reply analysis over all 3,738 held-out safety dialogues, together with the corresponding revisions.

6. Generalization across model sizes

Please see Point 3 of the General Response for the new results at 3B and 14B, together with the corresponding revisions.

Dear Reviewer exHZ,

As the discussion period closes soon, we want to kindly follow up on our response above. We hope our point-by-point explanations and experiments regarding utility evaluation, helpfulness metrics, generalization to adaptive attackers and self-verified rewards address your concerns. We are happy to clarify any remaining questions. If our reply has resolved your concerns, we would be grateful if you could consider updating your score.

Thank you again for your time and feedback.

Reviewer_dftiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ShallowDefend, a multi-turn jailbreak defense based on an Assess–Decide–Respond framework. The defender explicitly assesses user intent, commits to a safety decision, and generates a response conditioned on that decision. The authors introduce a self-verified reinforcement learning reward that checks the committed decision against the conversation's known label without requiring an external judge. Experiments across two model families and fourteen benchmarks demonstrate substantial improvements in multi-turn safety. Additional ablations and interventions investigate the role of explicit decisions and their relationship to shallow safety alignment.

Critical strengths

S1. A simple and interesting approach to multi-turn safety alignment.

The paper introduces an explicit Assess–Decide–Respond framework that requires the defender to commit to a safety decision before generating its final response. Motivating this design through shallow safety alignment provides an interesting perspective on how early response tokens can influence subsequent generation.

S2. Strong empirical results and comprehensive evaluation.

The proposed method demonstrates substantial safety improvements across two model families and multiple multi-turn jailbreak benchmarks. The authors compare against several training-based and representation-level defenses using the same training data. Additional evaluations with multiple independent judges, statistical confidence intervals, and component ablations strengthen the empirical evidence.

S3. A practical reward formulation that reduces dependence on external judges.

The self-verified reward uses the model's explicit decision and the known conversation label rather than requiring an external LLM judge during training. This simplifies reward computation and reduces dependence on potentially inconsistent judge feedback. The decision-intervention experiments and gradient ablations also provide useful insights into how the proposed structure affects response behavior.

Critical weaknesses

W1. The self-verified reward does not guarantee response safety.

The proposed reward only checks whether the decision label is correct, without evaluating the actual response. Therefore, a model could receive a positive reward even if it outputs unsafe content after selecting REFUSE. Although the authors provide evidence of decision–response consistency, this may not hold under stronger attacks.

W2. The utility evaluation does not reflect the actual deployment setting.

The authors evaluate general capabilities in completion format, which bypasses the proposed Assess–Decide–Respond framework. Therefore, these results cannot fully demonstrate that the defender preserves utility in real interactions. Moreover, the substantial increase in over-refusal raises concerns about the safety–helpfulness trade-off.

W3. The evaluation lacks adaptive multi-turn attacks.

The experiments rely on pre-generated attack conversations, where attackers cannot adjust their strategies based on the defender's responses. However, real multi-turn jailbreak attacks are often adaptive. It remains unclear whether the proposed defense is effective against attackers that dynamically exploit its explicit safety decisions.

W4. The necessity of reinforcement learning is not sufficiently justified.

If the key idea is to explicitly make a safety decision before responding, could similar improvements be achieved through simple prompting or SFT on structured decision labels? Although the authors evaluate a prompt-only baseline, they do not compare against SFT using the same decision structure. This makes it difficult to determine whether the improvements come from the explicit decision mechanism or the proposed RL training.

Questions for authors

Q1. How reliable is the self-verified reward?

How often does the model produce unsafe responses despite selecting REFUSE? Does decision–response consistency remain reliable under stronger attacks?

Q2. Does the defender preserve utility in the actual deployment setting?

Could the authors evaluate general capabilities with the Assess–Decide–Respond framework enabled, rather than bypassing it through completion format?

Q3. How robust is the method against adaptive multi-turn attacks?

Would the defense remain effective if attackers could adjust their prompts based on the defender's previous responses and safety decisions?

Q4. Is reinforcement learning necessary?

Could similar improvements be achieved through prompting or SFT using the same explicit decision structure? In particular, how does the proposed RL method compare with a matched decision-supervised SFT baseline?

3 comments

We thank the reviewer for finding the approach simple and interesting, and recognizing our comprehensive evaluation and practical reward formulation. We also thank the reviewer for the feedback, which helped us revise our paper to make it stronger. We hope the following additional experiments and clarifications can address the reviewer's remaining concerns:

1. Clarification on utility evaluation setting

We would like to clarify that our utility evaluation protocol is, in fact, one of the strengths of our paper, reflecting a deliberately rigorous evaluation. Capability benchmarks are commonly evaluated in completion format, which bypasses the chat template and, with it, the safety behavior that alignment modifies. Such evaluations are therefore largely insensitive to safety training and can overstate how well utility is preserved. Had we only followed this practice, our defenders would appear to match the base model (Table 3), and we could have claimed safety gains at no utility cost. Instead, we explicitly highlight this limitation (Figure 2) and complement it with a helpfulness evaluation in chat format with the framework active, covering both conversational quality (MT-Bench) and over-refusal (XSTest, XSB). In short, we follow the standard protocol identically for all methods, and point out its limitation precisely so that it is not taken as the sole evidence of preserved capability.

That said, to further confirm that capabilities are preserved with the framework active, we re-run two suites from Table 3 in the deployment setting: GSM8K for reasoning, and IFEval for instruction following, the axis most sensitive to chat behavior. Each benchmark is queried through the chat template; for the unaligned and aligned models the framework prompt is active and the Intent/Decision preamble is stripped before answer extraction, exactly as a user would receive the reply. GSM8K is evaluated zero-shot with chain-of-thought, and IFEval with its standard prompt-level strict and loose accuracies. Since the chat template unlocks instruction-tuned behavior, absolute scores are higher than in Table 3, so comparisons should be made within the table below, where all rows share the same chat setting.

BackboneMethodGSM8KIFEval (prompt-level strict)IFEval (prompt-level loose)
Llama3.1-8BBase84.571.576.0
Unaligned83.373.476.0
Self reward – aligned82.070.673.4
External reward – aligned79.858.863.0
Qwen3-8BBase94.084.187.2
Unaligned95.078.078.6
Self reward – aligned95.575.277.8
External reward – aligned94.675.276.2

The results confirm that capabilities are largely preserved in the deployment setting. Reasoning is essentially unaffected, and most of the change in instruction following already appears in the unaligned model, indicating that it stems from the deliberation format rather than from training; our self-verified defender stays within three points of its unaligned counterpart on both backbones. Regarding over-refusal, including XSB in the trade-off leaves the trend unchanged (Appendix B, Figure 8): baselines that substantially improve safety incur a comparable helpfulness cost, while ours achieve clearly higher safety at a similar cost.

Revision: We clarify the utility evaluation protocol in Section 3.2 and add the deployment-setting results to Appendix E.1 (Table 4).

2. Agreement between self-verified results and external evaluation

Please see Point 2 of the General Response for the new label–reply analysis over all 3,738 held-out safety dialogues, together with the corresponding revisions.

3. More experiments with adaptive attackers

We agree that this is an important test. Please see Point 1 of the General Response for the new experiments with four live adaptive attackers, including one that knows the defense, together with the corresponding revisions.

4. Necessity of reinforcement learning

We would like to note that supervised data in our Assess → Decide → Respond format does not naturally exist: off-the-shelf models do not produce an explicit Intent/Decision preamble with correct safety labels, and no existing dataset provides such annotations. For this reason, our controlled comparison includes plain SFT as a baseline (Table 1, "SFT" row), trained on safe dialogues self-distilled from the base model, while prompting alone corresponds to the unaligned rows of Table 1. Nonetheless, to verify whether our gains stem from the explicit decision structure itself or from reinforcement learning, we additionally construct the matched decision-supervised SFT baseline that the reviewer suggests.

Construction. We use the same 832 training conversations, the same framework prompt, and the same LoRA budget (r = 32), replacing RL with supervised fine-tuning. For each defender turn, the Intent line is written by the base model; the Decision is set to the ground-truth label, namely REFUSE on the harmful payload turn and COMPLY on every earlier turn and on all benign turns (416 REFUSE / 3,492 COMPLY turns per backbone); and the reply is sampled from the base model conditioned on that decision. Training converges to a final loss of 0.08 on both backbones, and the resulting models emit the format on 99% of turns. All models are evaluated with the same framework prompt and judges as in Table 1:

BackboneMethodAdvBench ↑MHJ ↑XSTest ↓XSB ↓MT-Bench ↑
Llama3.1-8BBase54.538.010.030.06.20
Framework prompt only63.044.38.033.85.61
Plain SFT69.455.114.045.65.67
Decision-SFT (new)76.132.86.421.25.48
Framework + RL (ours)98.858.520.454.45.86
Qwen3-8BBase49.222.37.215.66.08
Framework prompt only69.046.914.837.97.33
Plain SFT59.935.617.640.97.53
Decision-SFT (new)79.246.09.229.16.44
Framework + RL (ours)97.175.226.456.87.36

As expected, supervising the explicit decision structure helps, improving over plain SFT on AdvBench for both backbones. However, decision-SFT still falls far behind our framework trained with RL, especially on the harder human-crafted MHJ attacks. The reason lies in when the model decides to refuse rather than in how it follows through: once decision-SFT commits to REFUSE, its reply is as reliably safe as that of the RL model, but it commits to REFUSE far less often on disguised payload turns. Imitating labeled decisions thus teaches the model the format, whereas rewarding the decision at the payload turn teaches it when to refuse. Both components are therefore necessary: the explicit decision is what the reply follows, and RL is what makes that decision correct.

Revision: We have added this comparison to Section 3.4 (paragraph "Is RL necessary?") and to the new Appendix E.6 (Table 8).

Dear Reviewer dfti,

As the discussion period closes soon, we want to kindly follow up on our response above. We hope our point-by-point explanations and experiments regarding self-verified scores, utility evaluation, robustness to adaptive attackers and necessity of RL address your concerns. We are happy to clarify any remaining questions. If our reply has resolved your concerns, we would be grateful if you could consider updating your score.

Thank you again for your time and feedback.

Reviewer_ySG7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present ShallowDefend, a framework for defending LLMs against multi-turn jailbreaks. The core idea builds on the observation that safety alignment is shallow, with safety-related behavior concentrated in the first few response tokens. ShallowDefend uses this concentration as a trainable control point through an Assess → Decide → Respond pipeline: the model assesses user intent, explicitly commits to a safety decision, and generates a response conditioned on that decision. Training uses GRPO with rewards assigned at the final conversation turn. The self-verified variant checks whether the decision matches the conversation’s known harmfulness label, requiring no external judge, while an alternative variant scores the response with an external reward model. Evaluated on Llama-3.1-8B and Qwen3-8B across safety, utility, and helpfulness benchmarks, the framework shows strong multi-turn defense and largely preserved general capabilities, although increased over-refusal indicates a cost to helpfulness.

Critical strengths
  1. The decision ablations and intervention experiments are a nice effort to explain why the structure works.

  2. The self-verified reward is another appealing aspect of the work. The same Decision that guides the response also provides a training signal through comparison with known conversation labels, without an external judge. This is an elegant reuse of the design, and the strong safety results across multiple benchmarks make it promising.

Critical weaknesses
  1. The safety gains are clear, but the cost in over-refusal makes the practical benefit less convincing. In Table 1, the self-reward variant has the highest XSB over-refusal on both backbones. Although MT-Bench improves over the base model on Qwen3, the rejection of benign requests remains a concern. Appendix B’s trade-off analysis also excludes XSB from its helpfulness aggregate. These results make the description of the cost as “modest” difficult to justify. Comparisons against strong baselines at matched over-refusal rates would help clarify the practical advantage.

  2. Generalization across model sizes remains unclear. Table 1 evaluates Llama-3.1-8B and Qwen3-8B, providing evidence across model families but only at one scale. Appendix A.1 also attributes the ability to follow an explicit decision to the model’s existing instruction-following capability. This raises the question of whether smaller models can reliably perform the assessment and decision steps, and whether the framework provides comparable additional benefits for larger models.

Questions for authors
  1. Appendix A.3 provides some label-conditional results for the single-turn sampling probe. For the self-verified model under standard multi-turn evaluation, how often are REFUSE outputs judged unsafe, and how does this compare with COMPLY outputs? Reporting the counts and unsafe rates for each label would help distinguish decision errors from failures to follow the decision.

  2. Could you clarify how the token positions used for KL measurement in Figure 4 correspond to the Decision position? Appendix C appears to use a shared continuation generated by the base model for KL, but Decision positions from separate framework-generated outputs. How are these positions aligned to support the interpretation of the curve across the assessment, decision, and response stages?

  3. How might the framework behave against adaptive attackers who know the defense and iteratively modify their user inputs? I understand that this is outside the current threat model, but a discussion of likely failure modes would help clarify the framework’s practical limits.

8 comments

We are delighted to find that the reviewer finds our core idea of harnessing shallow alignment to be creative, and recognizes our self-verified reward design to be appealing. We also thank the reviewer for the feedback, which helped us revise our paper to make it stronger. We hope the following additional experiments and clarifications can address the reviewer's remaining concerns:

1. Clarification on cost in helpfulness

We would like to clarify that our helpfulness evaluation targets two aspects: (1) multi-turn conversational quality (MT-Bench), and (2) over-refusal on harmful-looking but benign requests (XSTest, XSB). In the original Appendix B, we included only MT-Bench and XSTest so that the helpfulness axis reflects both quality and over-refusal in equal measure. Including all three would bias the axis heavily toward over-refusal as it would count the same aspect twice. We therefore believe that taking one benchmark per aspect is the most balanced choice. Note that XSB is still fully reported in Table 1.

That being said, we plot an additional figure where Helpfulness is now the mean of three terms, each on a 0–100 scale: [(100 − XSTest) + (100 − XSB) + 10·MT-Bench] / 3, and the conclusion is unchanged. Every baseline that substantially improves safety incurs a comparable helpfulness cost, so this cost is inherent to effective multi-turn defense rather than specific to our approach, and at a similar cost our variants achieve clearly higher safety. On Qwen3-8B, our external-reward variant is both safer and more helpful than every high-safety baseline, and our self-verified variant outperforms the safest baseline (MTSA) on both axes.

Revision: Appendix B now explains why the helpfulness axis takes one benchmark per aspect, and adds Figure 8, which repeats the trade-off plot with XSB included. Section 3.2 points to both versions.

2. More experiments with adaptive attackers

Please see Point 1 of the General Response for the new experiments with four live adaptive attackers, including one that knows the defense, together with the corresponding revisions.

3. Agreement between self-verified results and external evaluation

Please see Point 2 of the General Response for the new label–reply analysis over all 3,738 held-out safety dialogues, together with the corresponding revisions.

4. Generalization across model sizes

Please see Point 3 of the General Response for the new results at 3B and 14B, together with the corresponding revisions.

5. Explanation on KL token positions

We clarify the computation of Figure 4, whose setup is detailed in Appendix C ("KL-divergence setup", lines 965–981 of the revised manuscript). Following the per-token KL protocol of Qi et al. [1], the Base model first greedily generates its final reply y for each held-out dialogue, and this same y is reused for all models. At each token position i, every model receives the identical context (the dialogue plus the first i−1 tokens of y), and we compute the exact KL divergence between two models' next-token distributions, averaged over dialogues. The x-axis thus indexes token positions in y: Aligned ‖ Unaligned isolates the effect of training, Unaligned ‖ Base the effect of the prompt, and Aligned ‖ Base both. The shaded band marks where the Decision line typically falls in the Aligned model's own outputs.

The Decision position (shaded band) is measured separately. Since y is a Base reply and contains no Intent/Decision lines, we obtain the Decision position from the Aligned model's own greedy outputs on the same dialogues, recording the token index at which the Decision line appears. The band therefore serves as a positional reference, indicating where in a response training reshapes the model's predictions. Training relocates its influence from a fragile refusal prefix to the deliberation preceding the decision, and the interventional probes in Appendix A further confirm the causal role of the committed decision. The causal role of the committed decision is further established by the interventional probes in Appendix A (forced decisions, post-decision prefilling, and decoding-parameter sampling), all of which operate directly on the framework's own outputs.

Revision: We have made this computation explicit in Section 4.1 and in the caption of Figure 4, and expanded the KL-divergence setup in Appendix C to describe the three comparisons, the teacher-forced continuation, and the role of the shaded band.

[1] Safety Alignment Should Be Made More Than Just a Few Tokens Deep. ICLR 2025.

Dear Reviewer ySG7,

As the discussion period closes soon, we want to kindly follow up on our response above. We hope our point-by-point explanations and experiments regarding helpfulness metrics, generalization of self-verified rewards, decision position in KL measurement and adaptive attackers address your concerns. We are happy to clarify any remaining questions. If our reply has resolved your concerns, we would be grateful if you could consider updating your score.

Thank you again for your time and feedback.

I have one more question about Figure 4.

As clarified in your response, KL is computed on Base-generated continuations without Intent or Decision lines, while the shaded band comes from separate Aligned outputs. The band therefore does not mark a Decision emitted in the continuation used to compute KL. I am not sure Figure 4 alone supports the statements that divergence is spread over the assessment and decision stages, or that the distributions converge “once the decision is emitted.” Could you qualify these statements to distinguish what the curve directly shows from the stage-level interpretation?

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Comment (Part I)

We thank all reviewers for their constructive feedback. We are glad that all reviewers find our core idea of turning shallow alignment from a vulnerability into the mechanism of defense innovative, and recognize the effectiveness of our defenders. All changes in the revised manuscript are highlighted in blue. Below we address some concerns shared by reviewers, the remaining are answered in the individual responses.

1. More experiments with adaptive attackers

We agree with the reviewers that adaptive attackers, which observe the defender's replies and may even know the defense, provide a stronger test of robustness. We therefore evaluate our defenders and the strongest baselines against four live attackers that interact with the victim model and adapt turn by turn. Specifically, we use Crescendo with a gpt-4o attacker, together with a decision-aware variant in which the attacker is told that the target writes an Intent/Decision preamble before every reply and is instructed to steer that decision toward COMPLY, which is exactly the "knows the defense" setting raised by reviewer ySG7. We further use SEMA-auto, the interactive version of our training-time attacker, and X-Teaming with its official implementation, using a gpt-4.1 planner/attacker and a gpt-4o judge as in the original paper, with 5 strategies per behavior. All running on the held-out HarmBench behaviors and all victims face identical attackers and behaviors. Our defenders are served behind a thin wrapper that keeps their own Intent/Decision lines in the conversation history, as in deployment, while the attacker sees only the reply. We report SAFE % (↑):

Llama3.1-8B

VictimCrescendoDecision-aware CrescendoSEMA-autoX-Teaming
Base27.028.353.54.0
MUSE-D35.833.380.516.0
X-Boundary86.880.597.556.0
Ours, self-verified98.199.4100.088.0
Ours, external reward91.889.998.798.0

Qwen3-8B

VictimCrescendoDecision-aware CrescendoSEMA-autoX-Teaming
Base26.430.837.10.0
MUSE-D28.330.837.70.0
X-Boundary31.430.841.52.0
Ours, self-verified81.870.498.794.0
Ours, external reward62.357.993.776.0

Across all four live attackers, our defenders remain robust where the baselines largely collapse. X-Teaming, the strongest attacker, breaks nearly every behavior on the baselines but rarely succeeds against our defenders, showing that baseline robustness on static benchmarks does not transfer to live attacks, while ours does. Knowing the defense helps the attacker little: even where the decision-aware attacker lowers our safety, our defenders remain far safer than every baseline, since flipping the decision requires first defeating the intent assessment, which is as hard as any jailbreak. Crescendo on Qwen3 is the hardest setting, where all baselines fall to the base-model level while ours retain substantial robustness. The residual failures, mainly decision errors on borderline intents, are complementary to output-level safeguards: adding Llama-Guard-3 further improves robustness on human-crafted attacks at negligible over-refusal cost.

Revision: New Section 3.3 (Table 2) and Appendix E.5. The related work now discusses X-Teaming, and the ethics statement no longer lists adaptive adversaries as an open limitation.

AuthorsOpenReview ↗

General Comment (Part II)

2. Agreement between self-verified results and external evaluation

We would like to clarify that Figure 5 already shows that the model's own decision label is positively correlated with every external judge, which suggests that it tracks the actual safety of the reply. Moreover, training effectiveness itself provides direct evidence: the external judges see only the reply and are never used in training, so any label–reply gap would lower the safe rate, yet our defenders lead the field under all six judges. That said, following the reviewers' suggestion, we further break this down to examine decision errors and follow-through failures directly. Over all 3,738 held-out safety dialogues, we pair each committed label with the reply from the same turn: the label is parsed from the turn, and the reply is judged by gpt-5-mini.

BackboneREFUSECOMPLYUnparsedSafe | REFUSEUnsafe | REFUSESafe | COMPLYUnsafe | COMPLY
Llama3.1-8B32075181398.6 % (3161)1.4 % (46)77.0 % (399)23.0 % (119)
Qwen3-8B30286624898.7 % (2988)1.3 % (40)83.8 % (555)16.2 % (107)

The results confirm that the committed decision reliably governs the reply, which is exactly what makes the self-verified reward trustworthy. The reward checks only the committed label, so it is sound as long as the label determines what the user actually receives, and the table shows that it does: once the defender commits to REFUSE, the reply is judged safe in nearly all cases on both backbones. This observational result is consistent with the interventional evidence in Appendix A.1, where forcing a REFUSE decision yields a safe reply in every case, and with the label–judge correlation in Figure 5. Together, these three independent views indicate that rewarding the decision effectively rewards the safety of the reply itself. COMPLY decisions show a higher unsafe rate, but whether a compliant reply to a borderline request counts as a jailbreak is inherently subjective, and external judges themselves often disagree on such cases (Section 4.2).

Revision: New Appendix E.4 (Table 7), summarized in Section 4.2.

3. Generalization across model sizes

We are glad that the reviewers acknowledge that the framework works at 8B size. Since practical deployments typically use models of 8B or larger, where instruction-following ability only grows stronger, the 8B results already serve as a conservative lower bound. Nonetheless, to verify transfer in both directions, we retrained the default self-verified recipe at two new scales: Llama-3.2-3B-Instruct (smaller) and Qwen3-14B (larger), using the same data and hyperparameters with no per-scale tuning. Each model is evaluated in three forms: base, unaligned (framework prompt on untrained weights) and aligned (after RL), with the same judges and evaluation cells as Table 1.

Per-form results. Safety (SAFE % ↑): AdvBench by attack family and MHJ; Helpfulness: XSTest ↓, XSB ↓, MT-Bench ↑.

ModelFormSEMACresc.FITDEchoHISMHJXSTest ↓XSB ↓MT-B ↑
Llama-3.2-3Bbase13.562.593.376.983.047.78.418.86.04
unaligned16.464.490.484.690.050.311.633.85.03
aligned98.198.1100.099.0100.054.610.433.54.71
Qwen3-14Bbase12.555.888.573.176.021.25.215.06.94
unaligned19.265.499.084.691.043.910.830.67.41
aligned95.294.2100.0100.0100.076.522.850.07.29

The results follow the same trend as the 8B models in Table 1. At both scales, the framework prompt alone already improves safety over the base model, and training with the self-verified reward brings a substantially larger gain. The improvement is therefore robust to model size. In particular, the 3B results show that a smaller model can reliably carry out the assessment and decision steps once trained. Meanwhile, the helpfulness cost stays moderate and in line with 8B: at 3B, most of the change is already present with the prompt alone, and at 14B conversational quality is preserved.

Revision: Section 3.2 and new Appendix E.7 (Tables 9 and 10).

AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the AC and all reviewers for their time and constructive engagement. We are glad that all reviewers recognize our core idea of turning shallow alignment from a vulnerability into the mechanism of defense, and appreciate the strengths built on it: a judge-free self-verified reward (ySG7, dfti), strong safety gains across model families (dfti, exHZ), comprehensive evaluation (dfti), and informative ablations and mechanistic analyses (ySG7, exHZ). Below we summarize how we addressed the main concerns and the corresponding revisions to the manuscript. All changes are highlighted in blue in the updated manuscript.

  • Robustness to adaptive attackers and comparison with guard models. We evaluated our defenders against four live attackers, including X-Teaming and a decision-aware attacker that knows the defense, and against external guard models. Our defenders remain robust where the strongest baselines largely collapse, and guard models complement rather than replace them (Section 3.3; Appendices E.5, E.8).

  • Reliability of the self-verified reward. We added a label–reply analysis over all 3,738 held-out dialogues and explained the structural safeguard against reward hacking. Once the defender commits to REFUSE, the reply is judged safe in over 98% of cases, and the partial-gradient option further ensures that the reward cannot teach the model to answer against its decision (Section 4.2; Appendix E.4).

  • Generalization across model sizes. We retrained the same recipe on Llama-3.2-3B and Qwen3-14B without per-scale tuning, reproducing the 8B trend at a comparable helpfulness cost (Section 3.2; Appendix E.7).

  • Role of the decision and necessity of RL. We added a matched decision-supervised SFT baseline, which falls far short of RL, and revised the KL analysis to separate what the curves directly show from our interpretation, with the decision's causal role supported by the probes in Appendix A (Sections 3.4, 4.1; Appendices C, E.6).

  • Clarification on utility and helpfulness metrics. We re-evaluated utility with the framework active and extended the trade-off analysis to include XSB. Reasoning is preserved, most of the instruction-following change stems from the deliberation format, and baselines that substantially improve safety incur a comparable helpfulness cost, while ours achieve clearly higher safety at a similar cost (Appendices B, E.1).

Outcome of the discussion. Reviewers dfti and exHZ replied that our responses addressed all their concerns, and both raised their scores. Reviewer ySG7 raised no further concerns about our earlier responses and asked a few additional follow-up questions, each of which we answered in detail and addressed in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.