acceptodds
Under review as a conference paper at ICLR 2027

From Allow-or-Block to Risk-Adaptive Moderation: A Multi-Expert Safety Guardrail for Large Language Model Services

Abstract

External safety guardrails are increasingly used to protect deployed large language model (LLM) services from harmful and adversarial prompts. Existing guardrails commonly rely on binary filtering, forcing each input into an allow-or-block decision. While effective for risk-unambiguous prompts, this mechanism is less reliable for ambiguous-risk prompts, such as dual-use queries and adversarial jailbreaks, often allowing unsafe content to reach downstream LLMs or incorrectly rejecting benign requests. To address this challenge, we propose RAGuard, a risk-adaptive guardrail that replaces binary filtering with risk-aware routing and specialized mediation modules built on a shared lightweight LLM backbone. Specifically, RAGuard routes each prompt, using intermediate backbone representations, to a safety refusal expert, a risk sanitization expert, or a semantic enhancement expert according to its risk level. The safety refusal expert deterministically rejects clearly harmful prompts. The risk sanitization expert rewrites ambiguous-risk prompts by suppressing unsafe operational details while preserving legitimate intent. The semantic enhancement expert normalizes and enriches clearly benign prompts to improve prompt clarity and downstream response quality. By replacing binary filtering with risk-specific expert adaptation, RAGuard shifts guardrails from simple allow-or-block decisions to risk-aware prompt mediation. Extensive experiments under a guardrail-oriented security–efficiency–utility evaluation framework show that RAGuard achieves strong safety-intervention performance, low hard-rejection rates on benign inputs, and improved downstream response quality, with guardrail-side computational costs evaluated separately.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Safety, Safety Guardrails, Jailbreak Defense, Risk-Adaptive Moderation, Prompt Sanitization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FtwDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAGuard, a risk-adaptive guardrail that replaces binary allow-or-block moderation with three-way intervention. Clearly harmful prompts are rejected, ambiguous-risk prompts are sanitized, and clearly benign prompts are semantically enhanced. Experiments across safety benchmarks and jailbreak attacks show strong safety performance, low benign rejection, and improved downstream response quality.

Critical strengths
  1. Risk-adaptive formulation beyond binary filtering. The paper addresses an important limitation of conventional allow-or-block guardrails by introducing different interventions for harmful, ambiguous-risk, and benign prompts. This is practically meaningful because it directly targets the safety–utility trade-off near the decision boundary. 23777.pdf
  2. Strong and broad empirical evaluation. RAGuard is evaluated across multiple safety benchmarks, jailbreak attacks, benign rejection, response utility, and efficiency, with strong overall results. This breadth makes the proposed system more convincing as a deployable guardrail rather than a narrow classifier improvement.
Critical weaknesses
  1. The core benefit of explicit three-way routing is not sufficiently isolated. The paper does not compare against a strong uncertainty-triggered or continuous-risk rewriting baseline, so it remains unclear whether the gains come specifically from the proposed three-region formulation or simply from adding conditional rewriting. This directly weakens the paper’s main methodological claim.
  2. The security evaluation is not sufficiently adaptive or cross-model. Jailbreak robustness is tested on existing attacks with only one victim model, while attacks targeting the router or sanitization expert are not evaluated. This limits confidence that the reported robustness would hold under stronger, adaptive deployment settings.
Questions for authors
  1. How does RAGuard compare with uncertainty-triggered rewriting using a conventional safety classifier?
  2. How robust is RAGuard to adaptive attacks targeting the router or sanitization expert?
  3. Does the reported jailbreak robustness generalize to multiple victim LLMs?
  4. Can the authors provide human evaluation of semantic/intent preservation after sanitization?
  5. Can the authors provide multi-annotator agreement for the three risk-region labels?
2 comments

We thank the reviewer for the valuable feedback. Our responses are provided below.

Reply to Q1: Comparison with Uncertainty-Triggered Rewriting

Concern: The original evaluation lacks a comparison with uncertainty-triggered rewriting using conventional safety classifiers.

Response: We implemented two WildGuard-based alternatives: WildGuard + Pass-through/RSE and WildGuard + SEE/RSE. Both use harmfulness-score thresholds (0.05/0.95): low-risk requests are passed through or enhanced, intermediate-risk requests are sanitized by RSE, and high-risk requests are rejected. The same rewriting experts and downstream model are reused where applicable.

Preliminary Results: We evaluated 20 WildJailbreak requests (10 benign, 10 harmful) from four routing-disagreement categories. RAGuard produced 0/10 unsafe responses, compared with 3/10 for both WildGuard-based alternatives. All three approaches directly rejected 5/10 harmful requests, suggesting that intervention selection matters beyond rejection frequency. For benign requests, safe task completion was 5/10 for RAGuard, 5/10 for WildGuard + SEE/RSE, and 4/10 for WildGuard + Pass-through/RSE. RAGuard directly rejected five benign requests, revealing a safety–utility trade-off.

Limitations and Future Work: These findings suggest that explicitly learned risk-region routing may improve intervention selection over fixed score thresholds. However, the small disagreement-focused sample and exploratory thresholds do not establish overall superiority. We plan to extend the comparison to the full 2,210-request WildJailbreak benchmark and investigate threshold sensitivity.

Manuscript Revisions: We will incorporate this comparison into Section 6.3 and Appendix B.1.4.

Reply to Q2 & Q3: Adaptive Attack Robustness and Cross-Model Generalization

Response: We appreciate the suggestions regarding adaptive attacks targeting the Router and RSE, and generalization across victim LLMs. We plan additional experiments over the next few days to assess RAGuard's robustness against adaptive adversaries and its cross-model defense effectiveness.

Reply to Q4: Human Evaluation of Semantic and Intent Preservation After Sanitization

Concern: The original evaluation relies on embedding-based similarity and lacks human evidence of legitimate-intent preservation.

Response: We conducted a human evaluation on 50 requests randomly sampled from 125 benign-labeled WildJailbreak requests routed to RSE. DeepSeek-V3.2 generated responses to both original and RSE-rewritten prompts. Two independent annotators evaluated responses against the original user request, blinded to rewriting conditions and automated judgments.

Results: Human consensus evaluation showed that 43/50 (86%) RSE-mediated responses strictly fulfilled the original legitimate task, compared with 45/50 (90%) for direct responses. Allowing minor omissions, both conditions achieved 49/50 (98%) core safe task completion. All responses were safe, with no downstream refusals. These results provide human evidence that RSE largely preserves legitimate intent, while the slightly lower strict completion rate indicates possible information loss.

Manuscript Revisions: We will add the evaluation protocol, criteria, results, and limitations to Section 6.2 and Appendix B.5.

Reply to Q5: Multi-Annotator Agreement for Three Risk-Region Labels

Concern: The manuscript lacks independent multi-annotator agreement measurements for the three risk-region labels, particularly ambiguous-risk requests.

Response: We conducted a two-annotator blind evaluation on 60 held-out requests (15 benign, 30 ambiguous-risk, 15 harmful), emphasizing challenging boundary cases. Both annotators independently applied the same labeling guidelines without access to original labels, automated predictions, or each other's judgments.

Results: The annotators agreed on 56/60 requests (93.3%), achieving a Cohen's κ of 0.884, indicating high inter-annotator agreement. Among the 30 originally ambiguous-risk requests, 5 were labeled ambiguous-risk by both annotators, while 23 received consistent alternative labels (15 benign and 8 harmful).

Manuscript Revisions: We will add the blind annotation protocol, agreement statistics, and reference-label discrepancy analysis to Appendix A.1.2.

Reply to Q1: Comparison with Uncertainty-Triggered Rewriting

We extended our preliminary investigation to 2,210 WildJailbreak requests, comparing RAGuard with two WildGuard-based score-threshold rewriting strategies (0.05/0.95). All methods used the same downstream model and shared rewriting experts where applicable.

RAGuard achieved 39.5% benign safe task completion, outperforming WildGuard + SEE/RSE (34.3%) and WildGuard + Pass-through/RSE (28.1%). Despite directly rejecting substantially fewer harmful requests (50.9% vs. 90.4%), RAGuard and WildGuard + SEE/RSE each produced 9 GPT-4o-identified unsafe responses among 2,000 harmful requests.

Furthermore, introducing SEE under the same WildGuard routing increased benign safe task completion from 28.1% to 34.3%. These findings demonstrate the value of explicit risk-region modeling and differentiated expert intervention compared with fixed score-threshold rewriting strategies.

We will incorporate the full-scale comparison into Section 6.3 and Appendix B.1.4.

Reply to Q2: Robustness Against Adaptive Attacks

We appreciate the reviewer's suggestion. To investigate potential attack surfaces relevant to adaptive adversaries, we further analyzed 110 ActorAttack requests and their execution traces. Among them, 41 were routed to SEE and 59 to RSE. We identified 33 human-confirmed harmful responses, including 11 from SEE and 22 from RSE.

Our case analysis reveals two potential attack mechanisms: (1) Router-targeted evasion, where malicious intentions are disguised within seemingly benign contexts to avoid sanitization (e.g., actor-063); and (2) RSE-targeted evasion, where harmful objectives remain embedded in rewritten prompts (e.g., actor-042).

These findings identify concrete attack surfaces that adaptive adversaries could exploit and highlight the importance of accurate risk routing and effective sanitization. While these attacks were not specifically optimized against RAGuard, the analysis provides useful evidence for understanding its behavior under sophisticated contextual attacks. We will incorporate these findings and discuss adaptive threat scenarios in the revised appendix.

Reply to Q3: Cross-Model Generalization

To evaluate cross-model generalization, we conducted additional experiments on 1,004 requests across ten jailbreak attacks, using three victim LLMs: GPT-6 Luna, Claude Haiku 5.5, and Qwen 3.8 Flash. All models received the same RAGuard-processed inputs, with 478 requests directly intercepted and 526 forwarded for downstream generation. Responses were evaluated using GPT-4o under consistent settings.

The resulting average ASRs were 1.20%, 3.21%, and 3.75%, respectively, with all three models achieving an average ASR below 4%. These results provide additional evidence that RAGuard maintains effective jailbreak protection across different victim LLM families, supporting the generalizability of its model-independent guardrail architecture.

We will incorporate the cross-model evaluation and per-attack results into the revised appendix.

Reviewer_94JROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RAGuard uses a shared Llama-3.1-8B backbone to route requests to refusal, risk sanitization, or semantic enhancement. Harmful requests are refused, ambiguous requests are rewritten, and benign requests are enhanced before being passed to a downstream model. The evaluation includes moderation benchmarks, ten jailbreak attack settings, benign utility tests, ablations, and overhead measurements. The system reports strong aggregate intervention results, fewer direct refusals, and a mean attack success rate of 5.77% in the evaluated downstream setup.

Critical strengths

RAGuard implements a way to rewrite mixed-risk requests instead of immediately refusing them, and the shared-backbone system achieves a mean ASR of 5.77% across ten fixed attack settings. These results support the feasibility of routing and rewriting as a moderation policy.

Critical weaknesses
  1. BHR counts direct refusals, while the no-RSE ablation sends ambiguous requests directly to refusal. Some of the reduction in BHR therefore follows from changing the route itself. What I need to know is whether those requests eventually receive safe answers that preserve their legitimate intent. Embedding similarity does not answer this, and the utility gains on clearly benign SEE inputs concern a different group of requests. The main claimed benefit of RSE is thus not directly measured.
  2. The expert-removal ablations change whether rewriting happens at all or whether a request is refused. They do not tell us whether the proposed expert decomposition is better than a classifier followed by a rewriter, or a unified rewriting policy. Without a matched comparison, the gains could come from adding rewriting rather than from the particular architecture.
  3. The guardrail alone takes approximately 18.84 seconds on benign inputs and 14.42 seconds on ambiguous inputs, before downstream inference. This is a substantial delay for a moderation step. The small percentage overhead uses another generative 8B invocation as its reference, and the memory increment assumes that a comparable model is already resident. Longer rewritten prompts also add downstream work. These measurements do not yet show that the benefit justifies the cost of adding RAGuard to a service instead of using a simpler moderation system.
Questions for authors
  1. For requests routed to RSE, how often does the final response safely fulfill the original permissible intent? Please use the same requests as in the BHR analysis and count downstream refusals and loss of legitimate intent as failures.
  2. Do you have results for a classifier-plus-rewriter or a unified rewriting policy with the same backbone, training data, and resource budget? How much does the expert decomposition contribute once both systems can rewrite requests?
  3. How do full-service latency and cost per safely fulfilled request compare with a simpler moderation baseline, including downstream generation and the longer rewritten prompts?
  4. Appendix A.1.2 reports that one reviewer checked 500 prompts and found no inconsistent labels. Was that reviewer shown the automatically assigned labels? Is there evidence that independent reviewers agree on which borderline requests belong in the ambiguous category?
  5. ActorAttack has an ASR of 37.28%, compared with the overall average of 5.77%. Which routes do the failed cases take? Do failures mainly come from harmful requests being classified as benign, unsuccessful sanitization, or the downstream model's response to an apparently safe rewrite?
4 comments

We appreciate the reviewer's careful assessment. We address each question below.

Reply to Q1: Downstream safety and original-task completion after RSE.

Using the WildJailbreak benign-request pool from our BHR evaluation, we identified 125 requests routed to RSE. DeepSeek-V3.2 answered each original prompt and its RSE rewrite; judgments were made against the original legitimate task, counting refusals and unmet requirements as failures. On all 125 pairs, GPT-4o assessed strict safe completion at 101/125 (80.8%) after RSE, versus 109/125 (87.2%) without rewriting. Two annotators independently assessed 50 randomly selected pairs, followed by adjudication: strict safe completion was 43/50 (86%) after RSE versus 45/50 (90%) directly; core safe completion was 49/50 (98%) in both, with 50/50 safe responses and no downstream refusals. Thus, RSE preserved most legitimate tasks but slightly reduced strict completion. We added the evaluation to Sec. 6.2 and Appendix B.5.

Reply to Q2: Classifier-plus-rewriter baselines and expert decomposition.

To investigate the contribution of our architecture, we conducted a preliminary comparison on 20 WildJailbreak requests (10 benign and 10 harmful), sampled from four routing-disagreement groups. We compared RAGuard with WildGuard + SEE/RSE and WildGuard + Pass-through/RSE, using harmfulness thresholds of 0.05/0.95, shared rewriting experts, and the same downstream model.

For harmful requests, RAGuard produced 0/10 unsafe responses, compared with 3/10 for both baselines, although all three methods directly rejected 5/10 harmful requests. This suggests that accurate intervention selection, rather than rejection frequency alone, plays an important role in downstream safety. For benign requests, safe task completion was 5/10, 5/10, and 4/10, respectively.

We are extending this comparison to the full 2,210-request WildJailbreak benchmark to further evaluate the architecture's effectiveness. We will incorporate the comparison into Sec. 6.3 and Appendix B.1.

Reply to Q3: Full-service latency and cost.

We agree that end-to-end latency (including downstream generation and rewritten-prompt length) and cost per safely fulfilled request are important. We have not yet completed a matched comparison with simpler moderation baselines due to time and computational constraints, and plan to investigate it over the next few days.

Reply to Q4: Independence and agreement of risk labels.

In the original 500-prompt audit, the reviewer examined the assigned labels against our annotation guidelines. To further strengthen the evaluation, we conducted an additional independent, two-annotator blind study on 60 held-out prompts (15 benign, 30 ambiguous-risk, and 15 harmful), emphasizing challenging boundary cases. Both annotators independently applied the same labeling guidelines without access to reference labels, detector predictions, or each other's judgments.

The annotators achieved 93.3% agreement (56/60), with Cohen's κ = 0.884, demonstrating strong consistency in their independent risk assessments. Notably, they agreed on 28/30 originally ambiguous-risk samples, including 5 jointly labeled ambiguous and 23 jointly assigned to other risk categories.

We will incorporate the annotation protocol and detailed agreement results into Appendix A.1.2.

Reply to Q5: ActorAttack failures.

To investigate the relatively high ASR of ActorAttack (37.28%), we conducted a supplementary trace-level analysis of 110 sampled requests from the original 600 ActorAttack prompts. Among them, 41 were routed to SEE, 59 to RSE, and 10 had no recorded expert outputs. Replaying the 100 available expert outputs through DeepSeek-V4-Pro identified 33 human-confirmed harmful responses, including 11 from SEE (33.3%) and 22 from RSE (66.7%).

Our analysis identifies two key failure mechanisms: (1) Risk misclassification, where contextually disguised requests enter SEE without sanitization (e.g., actor-063); and (2) Incomplete risk sanitization, where harmful objectives or operational details persist after RSE rewriting (e.g., actor-042). These mechanisms can interact, highlighting how sophisticated contextual attacks challenge multiple stages of the defense pipeline. We will incorporate the failure-path distribution and representative case analyses into the revised appendix.

Reply to Q2: Classifier-plus-Rewriter Baselines and Expert Decomposition

We extended our preliminary comparison to 2,210 WildJailbreak requests, evaluating RAGuard against WildGuard + SEE/RSE and WildGuard + Pass-through/RSE with shared experts and the same downstream model.

RAGuard achieved 39.5% benign safe task completion, compared with 34.3% and 28.1% for the baselines. Importantly, it directly rejected substantially fewer harmful requests (50.9% vs. 90.4%) while maintaining the same observed unsafe-response count as WildGuard + SEE/RSE (9/2,000). This highlights the value of explicit three-class routing for differentiated intervention selection, rather than relying solely on harmfulness-score thresholds.

Moreover, fixing the WildGuard router and replacing direct pass-through with SEE increased benign task completion from 28.1% to 34.3%, providing additional evidence for the contribution of specialized semantic enhancement.

Together, these results support the complementary roles of risk-adaptive routing and expert-based processing. We will incorporate this full-scale comparison into Section 6.3 and Appendix B.1.

Reply to Q3: Full-Service Latency and Cost per Safely Fulfilled Request

We thank the reviewer for highlighting the importance of end-to-end deployment costs. To address this concern, we conducted a controlled evaluation on 50 paired benign WildJailbreak requests, comparing RAGuard with WildGuard + Pass-through/RSE under identical hardware (NVIDIA A40), concurrency (1), and downstream model (DeepSeek-V4.1-Flash-1017). The measurements include guardrail processing, downstream generation, and unsuccessful requests. Final responses were anonymously evaluated for safety and legitimate task completion.

Results. RAGuard safely completed 34/50 requests (68%), compared with 30/50 (60%) for the simpler baseline. Its average end-to-end latency was 17.91 s versus 10.59 s, representing an additional 7.32 s (+69.1%). Guardrail processing accounted for most of this difference (13.30 s vs. 5.92 s).

Amortizing the resources consumed by all 50 requests over successful completions, the end-to-end latency per safely fulfilled request was 26.34 s versus 17.65 s. The corresponding downstream input/output token consumption was 312/1,557 versus 271/1,171, representing a 29.6% increase in total downstream tokens per successful completion.

These results explicitly quantify the deployment trade-off: RAGuard improves safe task completion at the cost of additional latency and token consumption. We will add this controlled comparison to Section 6 and the computational-efficiency appendix. We acknowledge that the additional computational and token overhead remains a limitation of the current RAGuard framework. In future work, we will investigate more efficient routing and lightweight prompt mediation strategies to further reduce resource consumption while preserving safety and utility.

Thank you for the additional experiments, and apologies for the late reply. The new results address the missing baseline, utility, and cost evaluations. However, both blind annotators reassigned 23 of 30 originally ambiguous examples, leaving the original labels’ reliability unclear. The higher latency and token use also leave the benefit over simpler methods uncertain, especially since the cost study uses the weaker baseline. It would help to clarify the label discrepancies and compare costs against WildGuard + SEE/RSE under the same conditions. These concerns remain central to my assessment, so I am keeping my rating at 2.

We sincerely thank the reviewer for the constructive follow-up. We have conducted two additional evaluations directly addressing the remaining concerns regarding reference-label reliability and cost comparisons with the stronger WildGuard + SEE/RSE baseline.

1. Reference-Label Reliability: Additional Random-Sampling Validation

We clarify that our previous 30 ambiguous-risk samples were deliberately selected using a detector-stratified protocol and were primarily drawn from cases where detector predictions were contradictory. This evaluation was designed to examine challenging risk-boundary cases rather than estimate overall label reliability.

To directly investigate the representativeness of the observed discrepancies, we conducted a new blind validation on 150 independently sampled requests (40 benign, 70 ambiguous-risk, and 40 harmful) from the held-out Router test set, excluding all previously evaluated samples. Unlike the earlier study, samples were randomly selected within each reference-label category without conditioning on detector predictions, confidence scores, or perceived difficulty. Three blinded assessments were obtained using the original annotation guidelines, and majority-vote labels were compared with the original references.

Original LabelMajority Agreement
Benign (B)39/40 (97.5%)
Ambiguous-risk (A)57/70 (81.4%)
Harmful (H)37/40 (92.5%)

Overall, 133/150 (88.7%) original labels agreed with the majority assessments. Importantly, 81.4% of the randomly sampled ambiguous-risk labels were retained. Although the two evaluations use different aggregation procedures, these results provide additional evidence that the earlier discrepancies should not be generalized to the full ambiguous-risk population.

2. Full-Service Cost: Comparison with the Stronger WildGuard + SEE/RSE Baseline

Following the reviewer's suggestion, we extended our end-to-end evaluation to WildGuard + SEE/RSE, which supports the same semantic enhancement and sanitization experts as RAGuard.

All three methods were evaluated on the same 50 benign WildJailbreak requests, under identical hardware (NVIDIA A40), concurrency (1), downstream model (DeepSeek-V4.1-Flash-1017), and generation settings. Safe completion requires both response safety and fulfillment of the original legitimate task.

MetricRAGuardWG + Pass/RSEWG + SEE/RSE
Safe completion68%60%66%
Mean end-to-end latency17.91 s10.59 s26.87 s
Mean guardrail latency13.30 s5.92 s20.46 s
End-to-end time per safe completion26.34 s17.65 s40.71 s
Downstream tokens per safe completion (input/output)312/1,557271/1,171384/1,445

Compared with WildGuard + SEE/RSE, RAGuard achieves comparable safe task completion (68% vs. 66%) while reducing:

  • Mean end-to-end latency by 33.3% (26.87 s → 17.91 s).
  • Mean guardrail latency by 35.0% (20.46 s → 13.30 s).
  • End-to-end time per safely fulfilled request by 35.3% (40.71 s → 26.34 s).
  • Downstream input tokens per safe completion by 18.8% (384 → 312), with comparable total downstream token consumption.

These findings demonstrate that RAGuard does not rely on more expensive rewriting to achieve its observed utility. Instead, learned risk-adaptive routing enables more efficient intervention selection than a conventional safety-score-based policy using the same experts.

Overall, the additional random-sampling validation provides stronger evidence for the reliability of our original three-class annotations, while the matched comparison with WildGuard + SEE/RSE demonstrates the practical efficiency benefits of risk-adaptive routing beyond simply introducing rewriting modules.

We appreciate the reviewer's suggestions, which have helped us substantially strengthen the empirical evaluation of RAGuard.

Reviewer_PaisOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAGuard, a risk-adaptive safety guardrail that routes user prompts to three paths: refusal, risk sanitization, and semantic enhancement. Experiments show promising results in safety defense, over-rejection mitigation, and downstream response quality.

Critical strengths
  1. Important practical objective. Handling requests that contain legitimate needs alongside potential safety risks is an important problem for deployed LLM services. Introducing a sanitization path provides a plausible way to serve some requests that a binary guardrail would otherwise reject.

  2. Promising empirical results. The paper evaluates safety-intervention prediction, downstream jailbreak success, benign hard rejection, response quality, and computational overhead. The reported improvements over the evaluated guardrails provide useful initial evidence for the potential of prompt mediation.

Critical weaknesses
  1. Insufficient evaluation of risk sanitization. The paper does not directly demonstrate whether sanitizing ambiguous-risk requests produces responses that are both safe and useful with respect to the original legitimate request. HIR counts sanitization as an intervention without measuring whether the rewrite successfully mitigates risk, while low rejection rates and semantic similarity do not necessarily imply preservation of legitimate user intent. Three-class routing performance is also not reported, making it difficult to assess whether the appropriate intervention is selected.

  2. Limited evidence for the proposed architecture’s advantages. The paper lacks comparisons with simpler alternatives, such as binary classification combined with prompt rewriting. Moreover, the ablation studies change the processing policy when removing components. In Table 5 and Appendix B.1.1, the configuration without the router uses ground-truth labels for dispatch and sends clearly harmful prompts through sanitization instead of rejecting them early. The increased latency therefore partly reflects additional generation. In Table 6 and Appendix B.1.2, removing the sanitization expert directly rejects every ambiguous-risk request, making an increase in hard rejection an expected consequence of the policy change.

Questions for authors
  1. Can the authors compare RAGuard with simpler rewriting-based baselines and provide more controlled ablations to establish the benefits of its architecture?

  2. Can the authors provide direct evaluations of both safety and original-task completion for sanitized ambiguous-risk requests, along with three-class routing performance?

2 comments

We sincerely appreciate your time and effort in reviewing our manuscript. Below are our responses to your valuable comments.

Reply to Q1: Comparison with Simpler Rewriting Baselines and Controlled Ablations

Concern: The original experiments do not sufficiently isolate the benefits of risk-adaptive routing from those of simply introducing prompt rewriting.

Response: We thank the reviewer for this valuable suggestion. As an initial exploration, we conducted a small-scale comparison on 20 WildJailbreak requests (10 benign and 10 harmful), sampled from four routing-disagreement categories. We compared RAGuard against two simpler alternatives: WildGuard + SEE/RSE and WildGuard + Pass-through/RSE, using fixed risk thresholds (0.05 and 0.95), the same rewriting experts where applicable, and the same downstream model.

Preliminary Results. For harmful requests, RAGuard produced 0/10 unsafe responses, compared with 3/10 for both WildGuard-based baselines. Notably, all three methods directly rejected 5/10 harmful requests, suggesting that the safety difference may arise from which requests are assigned to each intervention rather than simply rejecting more requests. For benign requests, RAGuard and WildGuard + SEE/RSE both achieved 5/10 safe task completions, compared with 4/10 for WildGuard + Pass-through/RSE. However, RAGuard directly rejected 5/10 benign requests, whereas neither baseline directly rejected any, highlighting a safety–utility trade-off.

Analysis and Future Evaluation. These preliminary findings suggest that effective risk-conditioned routing contributes to downstream safety beyond merely incorporating rewriting modules. However, the current evaluation focuses on a small, deliberately selected set of routing disagreements, and the WildGuard thresholds are exploratory rather than fully optimized. We plan to extend this comparison to the full 2,210-request WildJailbreak benchmark, systematically evaluating downstream safety, legitimate task completion, and the safety–utility trade-off under different routing strategies.

Reply to Q2: Direct Evaluation of RSE Effectiveness and Three-Class Routing Performance

Concern: The original manuscript lacks direct evaluation of downstream safety and legitimate task completion after sanitization, as well as explicit three-class routing performance.

Response: We thank the reviewer for this suggestion. We have conducted two supplementary evaluations and incorporated the results into the revised manuscript.

(1) Safety and Task Completion. We randomly sampled 50 requests from 125 benign-labeled WildJailbreak prompts routed to RSE. Using the same downstream model (DeepSeek-V3.2), we compared responses to the original and sanitized prompts. Human consensus evaluation showed 100% response safety and 98% core safe task completion in both conditions, with no refusals of legitimate requests. Strict safe completion was 86% after RSE versus 90% without rewriting, indicating that RSE largely preserves legitimate utility, although minor information loss remains.

(2) Three-Class Routing Performance. On 6,432 held-out samples, the Router achieved 85.28% accuracy and 83.27% macro-F1. Recall rates were 95.40%, 64.95%, and 88.11% for clearly benign, ambiguous-risk, and clearly harmful requests, respectively. The lower ambiguous-risk recall highlights the remaining difficulty of identifying borderline-risk inputs.

Manuscript Revisions: We have added the key findings to Section 6.2 and included the complete routing confusion matrix, downstream evaluation protocol, and task-completion results in Appendix B.5.

Reply to Q1: Comparison with Simpler Rewriting Baselines and Controlled Ablations

We extended our preliminary 20-sample investigation to the full 2,210-request WildJailbreak benchmark (210 benign and 2,000 harmful). We compared RAGuard with WildGuard + SEE/RSE and WildGuard + Pass-through/RSE, using fixed thresholds (0.05/0.95), shared rewriting experts, and the same downstream model (DeepSeek-V3.2).

RAGuard achieved the highest benign safe task completion rate (39.5%), compared with 34.3% and 28.1% for the respective baselines. Notably, RAGuard directly rejected only 50.9% of harmful requests, versus 90.4% for both baselines, while producing the same number of GPT-4o-identified unsafe responses as WildGuard + SEE/RSE (9/2,000).

Furthermore, under identical WildGuard routing, introducing SEE improved benign safe task completion from 28.1% to 34.3%. These findings provide further evidence that risk-adaptive intervention selection and specialized expert processing contribute complementary benefits beyond simply introducing prompt rewriting. We will incorporate the full comparison into Section 6.3 and Appendix B.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.