- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents ShallowDefend, an RL training framework that requires a defender LLM to explicitly verbalize the user's intent and commit to a safety decision (COMPLY, REFUSE, or SAFE-REDIRECT) before generating a response in multi-turn conversations.
The policy is optimized using rewards computed at the final assistant turn. The authors explore two reward formulations: a static, self-verified reward and an LLM-as-judge reward. They also ablate different components of the framework and investigate which token positions receive policy-gradient updates, including an experiment that masks out the response tokens.
With LLM-as-judge rewards, ShallowDefend improves average safety on Llama-3.1-8B-Instruct from 59.9% to 92.9%, and on Qwen3-8B from 46.1% to 87.6%. The self-verified reward achieves 93.4% safety on Qwen3-8B. The method largely preserves general benchmark performance but noticeably increases over-refusal.
The ablations suggest that the explicit safety decision is a crucial component. Without it, the model may generate unsafe responses even after correctly identifying malicious intent. The partial-gradient experiment further suggests that directly optimizing response tokens is not necessary to achieve substantial safety improvements, as the generated response generally follows the preceding safety decision.
Critical strengths
-
Addresses multi-turn jailbreaks, which remain among the most effective attacks against safety-aligned LLMs.
-
Provides informative ablations and mechanistic analyses that support the proposed connection to shallow safety alignment.
-
Demonstrates effectiveness on both reasoning and non-reasoning LLMs, while largely preserving general benchmark performance.
Critical weaknesses
-
Utility evaluation bypasses the Assess → Decide → Respond framework. The general utility benchmarks are evaluated in completion format, where the proposed framework is not activated. However, in deployment, the defender cannot know a priori whether a conversation will be benign or adversarial, so the framework must remain active. It is therefore unclear whether the reported utility is preserved under realistic deployment conditions.
-
The method substantially increases over-refusal. Moreover, Appendix B's helpfulness metric excludes the XSB benchmark, despite reporting it in the main results. Including XSB changes the conclusions of the safety–helpfulness trade-off analysis, particularly the claimed Pareto dominance over competing methods.
-
The evaluation is limited to unrealistic static multi-turn jailbreaks and compares only against internal defense baselines. The authors do not evaluate against adaptive jailbreaks like X-teaming, where automated attackers adjust their strategies based on the defender's responses. Such attacks better reflect realistic adversarial interactions. The paper also lacks comparisons with external defenses, such as Llama Guard and Qwen3Guard, making it difficult to assess the advantages of integrating safety decisions into the defender rather than using a separate guard model.
-
The self-verified reward is potentially vulnerable to reward hacking. It only checks whether the generated decision matches the conversation's ground-truth label, without verifying whether the actual response is consistent with that decision. Although the authors provide empirical evidence that this mismatch is not a significant problem for Llama-3.1-8B and Qwen3-8B, it remains unclear whether the same behavior would hold for larger, more capable models that might learn to exploit this reward structure.
Questions for authors
Please see weaknesses.
6 comments
3. Comparison with external guard models
We would like to clarify that our method operates on the model side, shaping how the defender itself reasons and responds, whereas external guard models act as filters around a model. The two are therefore complementary rather than direct competitors. Nonetheless, to quantify how the two approaches compare and interact, we evaluate Llama-Guard-3-8B and Qwen3Guard-Gen-4B as input and output filters, both on the base models and on top of our self-verified defenders: the guard classifies every user turn and every reply, and any flagged turn or reply is replaced by a fixed refusal, while the answering model is otherwise unchanged. We use the same suites and judge panel as Table 1 (the guard used as a filter is excluded from that row's judge consensus, and MHJ is judged by gpt-5-mini as in Table 1). We report SAFE % and helpfulness:
| Backbone | Model | AdvBench ↑ | HarmBench ↑ | MHJ ↑ | XSTest ↓ | XSB ↓ | MT-Bench ↑ |
|---|---|---|---|---|---|---|---|
| Llama3.1-8B | Base | 54.5 | 62.5 | 38.0 | 10.0 | 30.0 | 6.20 |
| Base + Llama-Guard-3 | 58.7 | 66.0 | 67.4 | 11.2 | 37.6 | 6.09 | |
| Base + Qwen3Guard | 66.0 | 72.8 | 44.5 | 10.8 | 38.5 | 6.09 | |
| Ours (self-verified) | 98.8 | 99.5 | 58.5 | 20.4 | 54.4 | 5.86 | |
| Ours + Llama-Guard-3 | 99.6 | 99.1 | 83.1 | 20.8 | 54.7 | 5.77 | |
| Ours + Qwen3Guard | 99.6 | 99.7 | 67.0 | 20.8 | 55.0 | 5.72 | |
| Qwen3-8B | Base | 49.2 | 52.5 | 22.3 | 7.2 | 15.6 | 6.08 |
| Base + Llama-Guard-3 | 62.2 | 70.0 | 69.1 | 8.8 | 30.9 | 6.05 | |
| Base + Qwen3Guard | 71.1 | 77.9 | 39.1 | 9.2 | 32.9 | 6.17 | |
| Ours (self-verified) | 97.1 | 97.6 | 75.2 | 26.4 | 56.8 | 7.36 | |
| Ours + Llama-Guard-3 | 95.0 | 96.7 | 87.7 | 28.4 | 57.4 | 7.20 | |
| Ours + Qwen3Guard | 98.1 | 97.9 | 81.6 | 27.2 | 57.1 | 7.25 |
The results confirm both points. On their own, guard models are no substitute for our defender: they improve the base model only modestly on AdvBench and HarmBench and fail precisely where attacks are gradual, since each decomposed turn appears harmless to a per-turn filter, so their lower over-refusal comes with little multi-turn robustness. Combined with our defender, however, they are clearly complementary. Adding a guard on top preserves our defender's robustness on AdvBench and HarmBench while substantially improving MHJ, where output filtering is particularly effective, and it does so at almost no additional over-refusal or helpfulness cost, because our defender already refuses most of what the guard would flag.
Revision: We have added this comparison to the new Appendix E.8 (Table 11), which also recommends deploying our defender together with an output-level guard in practice, and referenced it from Section 3.3 and Appendix E.5.
4. More experiments with adaptive attackers
Please see Point 1 of the General Response for the new experiments with four live adaptive attackers, including one that knows the defense, together with the corresponding revisions.
5. Agreement between self-verified results and external evaluation
Please see Point 2 of the General Response for the new label–reply analysis over all 3,738 held-out safety dialogues, together with the corresponding revisions.
6. Generalization across model sizes
Please see Point 3 of the General Response for the new results at 3B and 14B, together with the corresponding revisions.
Dear Reviewer exHZ,
As the discussion period closes soon, we want to kindly follow up on our response above. We hope our point-by-point explanations and experiments regarding utility evaluation, helpfulness metrics, generalization to adaptive attackers and self-verified rewards address your concerns. We are happy to clarify any remaining questions. If our reply has resolved your concerns, we would be grateful if you could consider updating your score.
Thank you again for your time and feedback.
We thank the reviewer for recognizing our model as among the most effective safety-aligned LLMs, and for acknowledging our evaluation as well as its generalization performance. We also thank the reviewer for the feedback, which helped us revise our paper to make it stronger. We hope the following additional experiments and clarifications can address the reviewer's remaining concerns:
1. Clarification on utility evaluation setting
We would like to clarify that our utility evaluation protocol is, in fact, one of the strengths of our paper, reflecting a deliberately rigorous evaluation. Capability benchmarks are commonly evaluated in completion format, which bypasses the chat template and, with it, the safety behavior that alignment modifies. Such evaluations are therefore largely insensitive to safety training and can overstate how well utility is preserved. Had we only followed this practice, our defenders would appear to match the base model (Table 3), and we could have claimed safety gains at no utility cost. Instead, we explicitly highlight this limitation (Figure 2) and complement it with a helpfulness evaluation in chat format with the framework active, covering both conversational quality (MT-Bench) and over-refusal (XSTest, XSB). In short, we follow the standard protocol identically for all methods, and point out its limitation precisely so that it is not taken as the sole evidence of preserved capability.
That said, to further confirm that capabilities are preserved with the framework active, we re-run two suites from Table 3 in the deployment setting: GSM8K for reasoning, and IFEval for instruction following, the axis most sensitive to chat behavior. Each benchmark is queried through the chat template; for the unaligned and aligned models the framework prompt is active and the Intent/Decision preamble is stripped before answer extraction, exactly as a user would receive the reply. GSM8K is evaluated zero-shot with chain-of-thought, and IFEval with its standard prompt-level strict and loose accuracies. Since the chat template unlocks instruction-tuned behavior, absolute scores are higher than in Table 3, so comparisons should be made within the table below, where all rows share the same chat setting.
The results confirm that capabilities are largely preserved in the deployment setting. Reasoning is essentially unaffected, and most of the change in instruction following already appears in the unaligned model, indicating that it stems from the deliberation format rather than from training; our self-verified defender stays within three points of its unaligned counterpart on both backbones. Regarding over-refusal, including XSB in the trade-off leaves the trend unchanged (Appendix B, Figure 8): baselines that substantially improve safety incur a comparable helpfulness cost, while ours achieve clearly higher safety at a similar cost.
Revision: We clarify the utility evaluation protocol in Section 3.2 and add the deployment-setting results to Appendix E.1 (Table 4).
2. Clarification on helpfulness cost
We would like to clarify that our helpfulness evaluation targets two aspects: multi-turn conversational quality (MT-Bench) and over-refusal on harmful-looking but benign requests (XSTest, XSB). The original Appendix B took one benchmark per aspect (MT-Bench and XSTest) so that the helpfulness axis weighs quality and over-refusal equally. Including all three would count over-refusal twice and bias the axis toward it. XSB remains fully reported in Table 1.
To address the concern directly, we add Figure 8 to Appendix B, where Helpfulness is
[(100 − XSTest) + (100 − XSB) + 10·MT-Bench] / 3. The trend is unchanged. Baselines that substantially improve safety while still falling short of ours (X-Boundary and SFT on Llama3.1-8B; MTSA, MUSE-D and DPO on Qwen3-8B) incur a comparable helpfulness cost, so this cost is inherent to effective multi-turn defense rather than specific to our approach, and at a similar cost our variants achieve clearly higher safety on both backbones. On Qwen3-8B, our external-reward variant is both safer and more helpful than every high-safety baseline (DPO, MTSA, MUSE-D), and our self-verified variant outperforms MTSA, the safest baseline, on both axes.Revision: Appendix B now explains why the helpfulness axis takes one benchmark per aspect, and adds Figure 8, which repeats the trade-off plot with XSB included. Section 3.2 points to both versions.