- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RAGuard, a risk-adaptive guardrail that replaces binary allow-or-block moderation with three-way intervention. Clearly harmful prompts are rejected, ambiguous-risk prompts are sanitized, and clearly benign prompts are semantically enhanced. Experiments across safety benchmarks and jailbreak attacks show strong safety performance, low benign rejection, and improved downstream response quality.
Critical strengths
- Risk-adaptive formulation beyond binary filtering. The paper addresses an important limitation of conventional allow-or-block guardrails by introducing different interventions for harmful, ambiguous-risk, and benign prompts. This is practically meaningful because it directly targets the safety–utility trade-off near the decision boundary. 23777.pdf
- Strong and broad empirical evaluation. RAGuard is evaluated across multiple safety benchmarks, jailbreak attacks, benign rejection, response utility, and efficiency, with strong overall results. This breadth makes the proposed system more convincing as a deployable guardrail rather than a narrow classifier improvement.
Critical weaknesses
- The core benefit of explicit three-way routing is not sufficiently isolated. The paper does not compare against a strong uncertainty-triggered or continuous-risk rewriting baseline, so it remains unclear whether the gains come specifically from the proposed three-region formulation or simply from adding conditional rewriting. This directly weakens the paper’s main methodological claim.
- The security evaluation is not sufficiently adaptive or cross-model. Jailbreak robustness is tested on existing attacks with only one victim model, while attacks targeting the router or sanitization expert are not evaluated. This limits confidence that the reported robustness would hold under stronger, adaptive deployment settings.
Questions for authors
- How does RAGuard compare with uncertainty-triggered rewriting using a conventional safety classifier?
- How robust is RAGuard to adaptive attacks targeting the router or sanitization expert?
- Does the reported jailbreak robustness generalize to multiple victim LLMs?
- Can the authors provide human evaluation of semantic/intent preservation after sanitization?
- Can the authors provide multi-annotator agreement for the three risk-region labels?
2 comments
Reply to Q1: Comparison with Uncertainty-Triggered Rewriting
We extended our preliminary investigation to 2,210 WildJailbreak requests, comparing RAGuard with two WildGuard-based score-threshold rewriting strategies (0.05/0.95). All methods used the same downstream model and shared rewriting experts where applicable.
RAGuard achieved 39.5% benign safe task completion, outperforming WildGuard + SEE/RSE (34.3%) and WildGuard + Pass-through/RSE (28.1%). Despite directly rejecting substantially fewer harmful requests (50.9% vs. 90.4%), RAGuard and WildGuard + SEE/RSE each produced 9 GPT-4o-identified unsafe responses among 2,000 harmful requests.
Furthermore, introducing SEE under the same WildGuard routing increased benign safe task completion from 28.1% to 34.3%. These findings demonstrate the value of explicit risk-region modeling and differentiated expert intervention compared with fixed score-threshold rewriting strategies.
We will incorporate the full-scale comparison into Section 6.3 and Appendix B.1.4.
Reply to Q2: Robustness Against Adaptive Attacks
We appreciate the reviewer's suggestion. To investigate potential attack surfaces relevant to adaptive adversaries, we further analyzed 110 ActorAttack requests and their execution traces. Among them, 41 were routed to SEE and 59 to RSE. We identified 33 human-confirmed harmful responses, including 11 from SEE and 22 from RSE.
Our case analysis reveals two potential attack mechanisms: (1) Router-targeted evasion, where malicious intentions are disguised within seemingly benign contexts to avoid sanitization (e.g., actor-063); and (2) RSE-targeted evasion, where harmful objectives remain embedded in rewritten prompts (e.g., actor-042).
These findings identify concrete attack surfaces that adaptive adversaries could exploit and highlight the importance of accurate risk routing and effective sanitization. While these attacks were not specifically optimized against RAGuard, the analysis provides useful evidence for understanding its behavior under sophisticated contextual attacks. We will incorporate these findings and discuss adaptive threat scenarios in the revised appendix.
Reply to Q3: Cross-Model Generalization
To evaluate cross-model generalization, we conducted additional experiments on 1,004 requests across ten jailbreak attacks, using three victim LLMs: GPT-6 Luna, Claude Haiku 5.5, and Qwen 3.8 Flash. All models received the same RAGuard-processed inputs, with 478 requests directly intercepted and 526 forwarded for downstream generation. Responses were evaluated using GPT-4o under consistent settings.
The resulting average ASRs were 1.20%, 3.21%, and 3.75%, respectively, with all three models achieving an average ASR below 4%. These results provide additional evidence that RAGuard maintains effective jailbreak protection across different victim LLM families, supporting the generalizability of its model-independent guardrail architecture.
We will incorporate the cross-model evaluation and per-attack results into the revised appendix.
We thank the reviewer for the valuable feedback. Our responses are provided below.
Reply to Q1: Comparison with Uncertainty-Triggered Rewriting
Concern: The original evaluation lacks a comparison with uncertainty-triggered rewriting using conventional safety classifiers.
Response: We implemented two WildGuard-based alternatives: WildGuard + Pass-through/RSE and WildGuard + SEE/RSE. Both use harmfulness-score thresholds (0.05/0.95): low-risk requests are passed through or enhanced, intermediate-risk requests are sanitized by RSE, and high-risk requests are rejected. The same rewriting experts and downstream model are reused where applicable.
Preliminary Results: We evaluated 20 WildJailbreak requests (10 benign, 10 harmful) from four routing-disagreement categories. RAGuard produced 0/10 unsafe responses, compared with 3/10 for both WildGuard-based alternatives. All three approaches directly rejected 5/10 harmful requests, suggesting that intervention selection matters beyond rejection frequency. For benign requests, safe task completion was 5/10 for RAGuard, 5/10 for WildGuard + SEE/RSE, and 4/10 for WildGuard + Pass-through/RSE. RAGuard directly rejected five benign requests, revealing a safety–utility trade-off.
Limitations and Future Work: These findings suggest that explicitly learned risk-region routing may improve intervention selection over fixed score thresholds. However, the small disagreement-focused sample and exploratory thresholds do not establish overall superiority. We plan to extend the comparison to the full 2,210-request WildJailbreak benchmark and investigate threshold sensitivity.
Manuscript Revisions: We will incorporate this comparison into Section 6.3 and Appendix B.1.4.
Reply to Q2 & Q3: Adaptive Attack Robustness and Cross-Model Generalization
Response: We appreciate the suggestions regarding adaptive attacks targeting the Router and RSE, and generalization across victim LLMs. We plan additional experiments over the next few days to assess RAGuard's robustness against adaptive adversaries and its cross-model defense effectiveness.
Reply to Q4: Human Evaluation of Semantic and Intent Preservation After Sanitization
Concern: The original evaluation relies on embedding-based similarity and lacks human evidence of legitimate-intent preservation.
Response: We conducted a human evaluation on 50 requests randomly sampled from 125 benign-labeled WildJailbreak requests routed to RSE. DeepSeek-V3.2 generated responses to both original and RSE-rewritten prompts. Two independent annotators evaluated responses against the original user request, blinded to rewriting conditions and automated judgments.
Results: Human consensus evaluation showed that 43/50 (86%) RSE-mediated responses strictly fulfilled the original legitimate task, compared with 45/50 (90%) for direct responses. Allowing minor omissions, both conditions achieved 49/50 (98%) core safe task completion. All responses were safe, with no downstream refusals. These results provide human evidence that RSE largely preserves legitimate intent, while the slightly lower strict completion rate indicates possible information loss.
Manuscript Revisions: We will add the evaluation protocol, criteria, results, and limitations to Section 6.2 and Appendix B.5.
Reply to Q5: Multi-Annotator Agreement for Three Risk-Region Labels
Concern: The manuscript lacks independent multi-annotator agreement measurements for the three risk-region labels, particularly ambiguous-risk requests.
Response: We conducted a two-annotator blind evaluation on 60 held-out requests (15 benign, 30 ambiguous-risk, 15 harmful), emphasizing challenging boundary cases. Both annotators independently applied the same labeling guidelines without access to original labels, automated predictions, or each other's judgments.
Results: The annotators agreed on 56/60 requests (93.3%), achieving a Cohen's κ of 0.884, indicating high inter-annotator agreement. Among the 30 originally ambiguous-risk requests, 5 were labeled ambiguous-risk by both annotators, while 23 received consistent alternative labels (15 benign and 8 harmful).
Manuscript Revisions: We will add the blind annotation protocol, agreement statistics, and reference-label discrepancy analysis to Appendix A.1.2.