- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies safety-aware multi-LLM routing and proposes SafeRouter, which jointly selects both a target model and a defense strategy for each query. The authors also introduce S²-Bench, containing 1.4M labeled outcomes across 10 models, 16 attack settings, and 16 composite defense configurations. SafeRouter uses risk gating, safety prediction, and utility estimation to route low-risk queries for quality and high-risk queries toward the lowest-cost predicted-safe configuration
Critical strengths
- Important and well-motivated problem formulation. The paper correctly identifies that model routing and safety defense should be optimized jointly rather than independently. This is consequential because defense effectiveness and cost depend on the specific query–model–defense combination.
- Strong benchmark and empirical evaluation. S²-Bench is large and systematically evaluates 1.4M model–defense outcomes. SafeRouter achieves 0.34% ASR while substantially reducing adversarial serving cost, and the paper also evaluates unseen goals/attack methods and statistical significance.
Critical weaknesses
- No adaptive attack against the full routing pipeline. The evaluation holds out existing attack methods but does not test attacks that explicitly optimize against SafeRouter’s risk gate and safety predictor. This limits confidence in the robustness of the system under an informed adversary. The paper explicitly acknowledges this limitation.
- The model pool is still relatively concentrated. Although the evaluation includes ten models, most are from the Qwen3 family, with only two Nemotron models and one code-specialized variant. This leaves some uncertainty about how well the routing policy transfers to more heterogeneous commercial and open-source model pools.
Questions for authors
- How robust is SafeRouter to adaptive attacks targeting the risk gate or safety predictor?
- How sensitive are the results to the composition of the model pool, especially with more heterogeneous model families?
- Could the authors clarify whether the safety–cost advantage remains similar when local routing/embedding compute is included in the end-to-end cost?
# Response to Reviewer 9sxU
We thank the reviewer for recognizing that the intersection of routing and safety is an important and well-motivated problem.
W1 / Q1 — Is SafeRouter robust to attacks that target the router itself?
Even when the attacker targets the risk gate and the safety predictor, SafeRouter is more robust than a single-defense pipeline and comparably robust to the Semantic Router at 4.4× lower cost on S²-Bench. We attack the deployed pipeline with PAIR [12] + DAGR operators [7] (Qwen3-235B attacker, 12 queries/goal, 40 goals). The white-box attacker additionally sees the router's risk score and threshold, the chosen model–defense pair and its predicted safety, and is told that low-risk queries are served undefended.
Knowing the gate gave the attacker no measurable advantage on these 40 goals (11 vs 13 goals): prompts that still carry the harmful goal keep a high risk score, so only 0.47% of black-box and 1.59% of white-box queries reached the undefended low-risk path.
W2 / Q2 — Does SafeRouter work with a more diverse model pool?
SafeRouter generalizes to a more heterogeneous pool without changing the method. We added Gemma-4-26B-A4B, Gemma-4-31B (Google) and MiMo-V2.6-Flash (Xiaomi), fully labelled with the same pipeline, and retrained the router on 13 models. Adversarial ASR stays below 1% in every seed, and the router uses the new models where they help: 15.8% of adversarial traffic goes to them and 69.7% of low-risk benign traffic:
Q3 — Does SafeRouter stay cheaper once routing compute is counted?
Including routing overhead, SafeRouter remains over 3× cheaper than the Semantic Router and Fixed S4. Routing is almost entirely the Qwen3-Embedding-0.6B encoder; the 18-net router adds 0.1–0.4 ms per query. Measured on one RTX 3090 over 1,024 S²-Bench probes (median 171 tokens), routing takes 45–56 GPU-seconds per 1k queries (batch 1 / 32): USD 0.006–0.008 per 1k at the 3090's rental rate (USD 0.50/GPU-h), or at most USD 0.03 per 1k even when billed at an H100 rate (USD 1.99/GPU-h) with the same timing; both rates are RunPod's November 2026 listings [11]. Every learned router (IRT-Router, RouteLLM, CARROT) also embeds each query. SafeRouter's adversarial cost becomes USD 0.099–0.124/1k, versus USD 0.415 for the Semantic Router (excluding its own classifiers) and USD 0.401 for Fixed S4: 3.2–4.2× cheaper.
References
[7] Diversity helps jailbreak large language models. NAACL 2025.
[11] RunPod GPU cloud pricing, https://www.runpod.io/pricing, November 2026: RTX 3090 USD 0.50/h; H100 PCIe USD 1.99/h (Community Cloud).
[12] Jailbreaking black box large language models in twenty queries. SaTML 2025.