- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a selective LLM safety defense framework based on uncertainty resolution. A lightweight linear probe is first used to judge the potential harmfulness of a user request: high-confidence harmful requests are directly refused, high-confidence harmless requests are answered normally, and for uncertain requests, a learnable Intent Token is introduced to generate new hidden representations using a frozen language model, which are then re-evaluated by the same probe.
Critical strengths
The method is lightweight and can be combined with a variety of existing inference-time defense methods. The paper provides fairly comprehensive experiments.
Critical weaknesses
-
The authors attribute the performance gains mainly to the Intent Token's ability to extract harmful intent more effectively, but the current experiments are not sufficient to fully substantiate this explanation. In particular, the additional representation computation, the training on uncertain samples, and changes in the classification decision boundary may jointly affect the final results. Furthermore, although the paper tests different models, the main performance gains are concentrated on Qwen3-8B, while the gains on other models are limited.
-
The security defense mechanism of this system relies heavily on the classification results of the lightweight linear probe, which may introduce additional adversarial robustness risks. In particular, when a harmful request is misclassified as safe by the initial probe with high confidence, the system will directly generate a response without triggering the Intent Token or subsequent safety reasoning. Although the authors evaluate a variety of adaptive jailbreak attacks, these experiments mainly target the system's final response and have not yet adequately examined attacks specifically targeting the probe's classification boundary or the safety routing mechanism.
Questions for authors
The security of RESTATE relies heavily on the classification results of the initial linear probe. If an attacker can push the hidden representation of a harmful request into the high-confidence safe region, the request may bypass all subsequent defense mechanisms. Have the authors considered adaptive attacks specifically targeting the probe's classification boundary or the routing decision?
4 comments
Response to Reviewer WPeY (2/2)
[W2, Q1] Adaptive attacks targeting the safety routing mechanism
Thank you for raising this important security concern. We agree that high-confidence misclassification is a potential vulnerability in selective safety defenses, as it may allow harmful requests to bypass subsequent safeguards.
This is an inherent trade-off in selective defense systems: reducing high-confidence false negatives generally requires escalating more requests to downstream safeguards, increasing computational costs. For instance, Constitutional Classifiers++[1] adopts a two-stage cascade with a lightweight first-stage classifier, allowing more potentially harmful requests to be flagged for further assessment rather than directly refused. Similarly, RESTATE's uncertainty thresholds can be easily adjusted to balance the risk of missed harmful requests against the deliberation rate.
Regarding adaptive jailbreak attack, we clarify that our evaluation adopts a black box threat model, reflecting practical deployment settings where probe weights and classification scores are not exposed to external users [2]. However, prior studies have shown that attackers may infer which safeguard handled a request from observable responses [3, 4]. Therefore, to fully address the reviewer’s concern, we conduct additional experiment by considering an attacker who can observe the routing outcome (direct answer, fixed refusal, or deliberation), without accessing internal classifier information. This provides a realistic, route-aware black box setting for evaluating attacks specifically targeting RESTATE's routing mechanism.
To examine whether an attacker can exploit the confident-allow path, we conducted an additional adaptive attack targeting RESTATE’s routing decision. Specifically, following the staged attack strategy of STACK [3], we selected 39 HarmBench behaviors for which previous PAIR/TAP attacks had successfully jailbroken the undefended Qwen3-8B. RESTATE initially routed 38 of these adversarial requests to fixed refusal and one to deliberation. We then adapted the random search suffix attack [5] to target the routing decision, using up to 9,600 queries per behavior to redirect requests toward the direct-answer path.
We considered two attacker feedback settings: (1) 3-way routing, which reveals the routing outcome (direct answer, fixed refusal, or deliberation), and (2) direct answer indicator, which provides only a binary signal indicating whether the request reaches the direct-answer path.
| Deployment | Attacker feedback | Routing bypasses ↓ |
|---|---|---|
| RESTATE | 3-way routing | 2/39 |
| RESTATE | Direct-answer indicator | 1/39 |
| + Token-confirmed allow | 3-way routing | 0/39 |
| + Token-confirmed allow | Direct-answer indicator | 0/39 |
In table above, a routing bypass indicates that the adversarial request reaches the direct-answer path without triggering refusal or deliberation, rather than a confirmed harmful response. Here, across approximately 360,000 queries per feedback setting, the attacker found routing bypasses for only 1–2 of the 39 behaviors, providing additional evidence of RESTATE’s robustness against adaptive attacks targeting its routing mechanism.
We additionally evaluated a simple variant, token-confirmed allow, which permits direct generation only when the initial probe's confident benign decision is also supported by the intent token assessment. Under the same attacks, no routing bypass was found (0/39). On WildJailbreak, this variant also reduces average ASR from 10.20% to 8.93%, with a modest increase in ORR (7.16%→8.04%) and deliberation rate (14.2%→15.6%).
Overall, these results provide additional evidence of RESTATE's robustness against adaptive attacks targeting the routing decision, while acknowledging that high-confidence misclassification remains a potential failure path. We have included this evaluation in the revised Appendix E.8 (Table 29), together with a discussion of practical safeguards, such as reducing routing-information leakage and monitoring repeated flagged requests.
[1] Cunningham et al. Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks (ICLR, 2026)
[2] Bailey et al. Obfuscated Activations Bypass LLM Latent-Space Defenses (ICLR, 2026)
[3] McKenzie et al. STACK: Adversarial Attacks on LLM Safeguard Pipelines (AAAI, 2026)
[4] Hackett & Garraghan. Behind the Refusal: Determining Guardrail Activation via Behavioral Monitoring (Preprint, 2026)
[5] Andriushchenko et al. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (ICLR, 2025)
We are grateful for your insightful suggestions, which helped us clarify the contributions of uncertainty resolution and further evaluate robustness against attacks targeting the routing mechanism. We hope our additional experiments and analyses satisfactorily address your concerns.
If you have any further questions, please let us know.
Thank you very much,
Authors
Thank you for the careful clarification. Please make sure including these results in the revised version.
Response to Reviewer WPeY
Dear Reviewer WPeY,
We are pleased to hear that our rebuttal addressed your concerns. We will ensure that the additional results presented in Appendix E of our rebuttal are properly integrated into the camera ready, with clearer explanations and discussion. Please let us know if you have any further questions.
Thank you very much.
Sincerely,
Authors
Response to Reviewer WPeY (1/2)
Dear Reviewer WPeY,
We sincerely appreciate your thoughtful review. We carefully considered your concerns regarding (1) the attribution of performance gains and generalization across backbones, and (2) the robustness of the initial probe and routing mechanism against adaptive attacks. Below, we provide additional experimental evidence and clarifications.
[W1] Attribution of gains and cross-backbone generalization
Thank you for raising these important concerns regarding the attribution of performance gains and reduced gain in larger backbones. We appreciate the opportunity to clarify how additional representation computation, uncertainty-focused training, and classification decisions contribute to RESTATE's improvements.
(1) Attribution of performance gains. First, the backbone and Anchor remain frozen throughout intent token training. Thus, improvements do not arise from updating the Anchor's decision boundary; instead, the learned intent token produces an additional representation evaluated by the same fixed classifier.
Our original ablations already examine several contributing factors. Simply appending an untrained token reduces BAcc (85.50→80.26, Table 3), while retraining the Anchor on additional difficult examples yields limited or negative gains (joint: 86.39; sequential: 79.34, Table 2). Moreover, with the same trained token and escalation rate (38.4%), uncertainty-based escalation substantially outperforms random escalation (92.62 vs. 88.41, Table 3), demonstrating the importance of targeting uncertain requests.
To further examine the benefit of separating general classification from targeted reassessment, we additionally evaluated joint training of probe and token.
While joint training also improves classification, separating general harmfulness assessment from uncertainty-focused reassessment achieves a further +5.49pp BAcc improvement. Together with our original ablations, these results suggest that RESTATE's gains are not solely attributable to additional computation or probe retraining, supporting the benefit of learning an additional representation for uncertain requests while keeping the original classifier fixed.
(2) Larger backbone generalization. The original cross-backbone experiments employed a restricted hyperparameter search space, primarily to establish RESTATE's applicability across different models under limited computational budget. To fairly assess its effectiveness, we applied the original search space to each backbone and selected configurations based on a held-out validation set (see Appendix E.4 for details). The resulting configurations achieved average improvements of +2.42pp on Qwen3-32B and +2.70pp on GPT-OSS-20B across BAcc, FNR, and AUC.
Overall, these additional results provide further evidence of RESTATE's effectiveness beyond Qwen3-8B. We have also refined our description of representation refinement to emphasize uncertainty-targeted reassessment, rather than implying that the intent token independently discovers new safety features.