- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper investigates the problem that providing a justification for an LLM answer is costly andn ot always required. To avoid generating unnecessary justifications, it proposes an LLM-based guardrail that checks user input against a list of compliance policies and returns a verdict for each policy. Only if the guardrail finds a potential risk is the reasoning activated.
Critical strengths
S1: The problem is relevant. Reducing the cost of reasoning in cases where it is not needed is practical. S2: The results seem to be competitive.
Critical weaknesses
W1: The novelty with regard to SingGuard seems to lie in the per-policy gate. I find this rather thin. W2: From my understanding, the decision whether to reason or not is dependent on the LLM's assessment of difficulty. This seems fairly dangerous, as this would probably allow adversarial examples or jailbreaks to go through without any reasoning.
Questions for authors
- What is your assessment of the risk regarding jailbreaks?
8 comments
Experimental Results
To recap our results:
- At matched reasoning budget, AdaGuard beats SingGuard's routing by +2.4 / +6.6 per-policy F1 (80.10 vs. 77.68, 64.52 vs. 57.90, ).
- On hard policies the gain is +4.8 / +7.9.
- Per-example, the two are tied on Gemma (88.18 vs. 88.26), which is expected from a per-input gate. AdaGuard leads by +4.2 on Nemotron.
- The gap grows with the number of policies per input: from +0.0 / +2.1 per-policy F1 on inputs with 1–3 policies to +4.0 / +8.5 on inputs with 9 or more.
- Across a sweep of SingGuard's threshold , it approaches AdaGuard only when it reasons on most or all policies (Gemma 79.80 at 100%, Nemotron 63.46 at 76% and 64.22 at ≈100%, vs. AdaGuard's 80.10 / 64.52 at 32% / 36%), i.e. as it approaches always-on reasoning.
- At AdaGuard's measured output length instead of its policy count, AdaGuard still leads by +1.9 / +3.1 per-policy F1.
Due to restrictions on running some third-party model weights in our environment, we could not evaluate SingGuard's own checkpoints. Instead, we reimplemented its hybrid routing rule (its Eq. 7–8) on the AdaGuard backbone. We have the same model's verdict with reasoning-off and reasoning-on for every test input–policy pair. Each selection rule in Table A reasons on as many policies as AdaGuard. The chosen pairs take the reasoning-on verdict and the rest reasoning-off’s, so only the choice of policies differs. SingGuard first decodes one safe/unsafe token for the whole input and computes its normalised confidence . If this confidence reaches a threshold (), it stops and emits , otherwise it reasons over all policies before giving the final verdict. We reproduce this as follows:
- For each policy , we take from its verdict token in
reasoning-off, normalized over Violation/Compliance. Since an input is safe only if it violates none of the matched policies, and , so is the more likely label and , which is Eq. 7 for a binary first decision. We also experimented with a variant that does not depend on the number of policies ( chosen from the most confident violation, or else the least confident compliance), which leads to the same conclusions. - Inputs with take
reasoning-on’s verdicts for all of their policies (SingGuard's slow mode), while the other inputs keep theirreasoning-off’s. - We then rank inputs by , least confident first, and escalate them to reasoning in that order until the number of reasoned policies reaches AdaGuard's (26,407 on Gemma, 29,863 on Nemotron). This becomes equivalent to (Gemma) / (Nemotron).
- Escalated verdicts come from an independent
reasoning-onpass not conditioned on . Therefore, they cannot anchor on the first guess. If anything, this setup favors SingGuard's implementation.
Table A: Matched reasoning budget (per-policy/per-example F1)
| Selection rule | Gemma: reasoned | Gemma: per-policy / per-example F1 | Nemotron: reasoned | Nemotron: per-policy / per-example F1 |
|---|---|---|---|---|
reasoning-off | 0% | 74.50±0.69 / 86.57±0.62 | 0% | 42.23±0.99 / 58.89±1.10 |
reasoning-on | 100% | 79.80±0.75 / 88.24±0.60 | 100% | 64.24±0.90 / 76.62±0.80 |
| SingGuard routing | 32.5% | 77.68±0.64 / 88.26±0.57 | 36.5% | 57.90±0.98 / 72.42±0.93 |
| AdaGuard | 31.8% | 80.10±0.64 / 88.18±0.59 | 36.3% | 64.52±0.93 / 76.66±0.86 |
We can see that at the same budget, AdaGuard beats SingGuard's routing by +2.4 (Gemma) / +6.6 (Nemotron) per-policy F1 (), and it is on par with reasoning on every policy. SingGuard decides per input and reasons on every policy of an uncertain input, and a policy that is uncertain inside a confidently unsafe input is consequently skipped, whereas AdaGuard decides per policy.
We also show that no threshold closes the gap (Table B). Lowering SingGuard's confidence threshold makes it reason on more inputs, but its per-policy F1 rises slowly. On the other hand, ends up sending (almost) every input to reasoning if tuned for accuracy (100% of policies on Gemma and 99.9% on Nemotron, for 79.80 / 64.22 per-policy F1). AdaGuard gets the same accuracy while reasoning on a third of the policies.
Table B: SingGuard's routing threshold sweeps. The table shows share of pairs reasoned · per-policy / per-example F1. An input is reasoned when its confidence is smaller than .
| Gemma: SingGuard routing | Nemotron: SingGuard routing | |
|---|---|---|
| 0.9 | 3.4% · 75.01 / 86.04 | 8.2% · 46.95 / 66.93 |
| 0.99 | 7.0% · 75.48 / 86.55 | 16.3% · 50.93 / 68.88 |
| 0.999 | 11.5% · 76.01 / 87.11 | 23.1% · 53.74 / 70.30 |
| 16.4% · 76.48 / 87.49 | 29.2% · 55.79 / 71.23 | |
| 27.7% · 77.29 / 88.08 | 43.3% · 59.91 / 73.32 | |
| 42.0% · 78.03 / 88.04 | 76.4% · 63.46 / 76.66 | |
| 59.4% · 78.21 / 87.91 | 98.5% · 64.09 / 76.99 | |
| 79.9% · 78.41 / 87.86 | 99.3% · 64.10 / 76.96 | |
| tuned for per-policy F1 (2-fold) | 100.0% · 79.80 / 87.73 | 99.9% · 64.22 / 76.97 |
| AdaGuard (no threshold) | 31.8% · 80.10 / 88.18 | 36.3% · 64.52 / 76.66 |
Also, the gain is concentrated on hard policies, as expected (Table C). AdaGuard reasons on 55% (Gemma) / 63% (Nemotron) of the policies the teacher-assigned hard labels and only 22% / 25% of the easy ones. On hard policies it beats SingGuard's routing by +4.8 / +7.9 per-policy F1, and it matches reasoning-on every policy on Nemotron.
Table C: per-policy F1 on teacher-labelled easy and hard policies
| Gemma: easy | Gemma: hard | Nemotron: easy | Nemotron: hard | |
|---|---|---|---|---|
reasoning-off | 87.65 ± 0.62 | 52.23 ± 1.34 | 55.31 ± 1.25 | 19.65 ± 1.27 |
reasoning-on | 88.60 ± 0.82 | 65.38 ± 1.24 | 76.70 ± 0.98 | 44.39 ± 1.46 |
| SingGuard routing | 89.27 ± 0.59 | 58.70 ± 1.26 | 70.63 ± 1.09 | 36.91 ± 1.51 |
| AdaGuard | 90.02 ± 0.56 | 63.50 ± 1.24 | 76.95 ± 1.00 | 44.81 ± 1.43 |
| AdaGuard: share of these policies reasoned on | 21.6% | 55.4% | 24.8% | 63.0% |
Finally, the advantage of per-policy decisions grows with the number of policies per input, exactly where granularity should matter (Table D).
Table D: By number of policies per input (per-policy F1)
| Policies per input | 1–3 | 4–8 | 9+ |
|---|---|---|---|
| Gemma4-E4B: AdaGuard per-policy F1 | 81.4 ± 1.7 | 80.1 ± 0.9 | 79.7 ± 1.1 |
| Share of policies AdaGuard reasons on | 48.7% | 38.5% | 24.5% |
| AdaGuard − SingGuard routing | +0.0 ± 1.4 | +1.7 ± 0.8 | +4.0 ± 1.0 |
AdaGuard − reasoning-on | +0.4 ± 1.4 | −0.3 ± 0.7 | +1.0 ± 1.3 |
| Nemotron-Nano-4B: AdaGuard per-policy F1 | 68.4 ± 2.0 | 65.3 ± 1.3 | 62.4 ± 1.5 |
| Share of policies AdaGuard reasons on | 50.5% | 43.8% | 28.9% |
| AdaGuard − SingGuard routing | +2.1 ± 1.8 | +6.4 ± 1.2 | +8.5 ± 1.6 |
AdaGuard − reasoning-on | +0.1 ± 1.6 | +0.7 ± 1.0 | −0.2 ± 1.3 |
On inputs with 9+ policies, AdaGuard reasons on only a quarter of them and stays close to reasoning-on on all of them, while SingGuard's routing falls 4.0 / 8.5 points behind.
Other Selection Baselines: Our response to Reviewer 6wrg (Section B) compares further rules at the same budget: random selection, random selection forced inside AdaGuard's own generation, a per-policy confidence gate on the verdict-token probabilities, AdaGuard's own difficulty token used as a confidence gate, and baselines matched on output length rather than on the number of policies reasoned on. AdaGuard matches or beats all of them. The only rule that matches it is the confidence gate. However, its drawback is that it needs token probabilities from an extra reasoning-off pass, and an extreme threshold, found by sweeping on the test set, to spend AdaGuard's budget.
To conclude, we also establish that the reward is what makes this selection possible. We extended the reward ablation of Figure 3 from epoch 1 to epochs 2 and 3 (Table E).
Table E: Reward ablation across epochs. Gemma4-E4B, held-out dynamic-policy set; per-policy F1 / share of policies reasoned on.
| Reward variant | Epoch 1 (step 80) | Epoch 2 (step 144) | Epoch 3 (step 208) |
|---|---|---|---|
| CF & Quality (AdaGuard's reward) | 80.0 ± 0.6 / 37% | 80.3 ± 0.7 / 38% | 80.8 ± 0.6 / 39% |
| CF Only (no quality judge) | 78.1 ± 0.8 / 25% | 80.4 ± 0.6 / 34% | 81.0 ± 0.6 / 36% |
| Quality Only (no counterfactual term) | 80.4 ± 0.7 / 99.6% | 80.0 ± 0.7 / 99.6% | 80.3 ± 0.8 / 99.9% |
| Vanilla (neither) | 80.0 ± 0.7 / 99.4% | 80.6 ± 0.7 / 99.5% | 80.9 ± 0.7 / 99.8% |
We thank the reviewer for their helpful comments and feedback, and we are glad that they find the problem relevant and practical, especially in deployment settings where it is desirable to reduce the overall latency costs (including from reasoning where it is not needed) while maintaining the performance of the guardrail, and that our results are competitive. We address both weaknesses and the question with new experiments on both backbones (Gemma4-E4B / Nemotron-Nano-4B) over the full held-out test set, and on an external benchmark of human-written business rules (DynaBench). Every number on our test set now carries a 95% confidence interval from a cluster bootstrap over test inputs, and comparisons are paired for statistical significance.
For simplicity, we will refer to AdaGuard’s
autoreasoningmode as simply AdaGuard throughout the following sections.High-Level Summary of Our Response
CF-Corr) is what teaches the model when to reason. At a similar reasoning budget, AdaGuard's choice of policies beats SingGuard's routing, reimplemented on our models, by +2.4 / +6.6 per-policy F1 (p<0.001), and the gap grows with the number of policies per input. Without the counterfactual reward, the model simply loses this selective reasoning ability.autoreasoning, very close toreasoning-on(92.3) and second among all other guardrails/models only to WildGuard (95.3), whose training data was generated with the same attack method as WildJailbreak. Jailbreaks are part of AdaGuard's training, and it reasons on hard policies far more often than on easy ones.autoreasoningwhile reasoning on only a fifth of the rules against 68.7 inreasoning-on, while DynaGuard 8B (with CoT) which was trained on DynaBench’s training split, scores 73.1. With DynaBench and WildJailbreak added to the updated Table 1, AdaGuard inautoreasoninghas the highest average across all six static benchmarks (91.0), and over all eight benchmarks (87.8).We hope these answers address the reviewer's concerns, and if they do, that the reviewer will consider raising the score. We are also happy to run any further comparison the reviewer would find useful, and look forward to any feedback.
A) Novelty with Regard to SingGuard
We have experimented with additional baselines, including SingGuard, and we will add all results to the updated manuscript. There are three main differences between AdaGuard and SingGuard:
reasonig-onmode.CF-Corr) samples areasoning-offverdict for each reasoned block and adjusts the reward based on whether reasoning fixed, maintained or broke the verdict. Without it, the learned reasoning rate rises to almost 100% (Figure 3). We will add this comparison to the updated PDF and listCF-Corras an explicit contribution.