acceptodds
Under review as a conference paper at ICLR 2027

Refusal-Gated Decoding: Preserving Refusal Behavior Under High-Temperature Sampling

Abstract

Recent advances in truncation-based sampling techniques have helped mitigate drawbacks of high-temperature sampling such as neural text degeneration, thereby enabling greater diversity in LLM outputs without sacrificing coherence. However, increasing the entropy of the token probability distribution via high temperatures has also been shown to weaken model guardrails by reducing the model's refusal response in the presence of harmful prompts. Existing solutions for maintaining the refusal behavior of LLMs either replace the model's own refusal decision with a separate safety classifier or alter its output distribution for every prompt. To address this gap, we propose refusal-gated decoding (RGD): an efficient sequential decoding approach which preserves a model's greedy decoding refusal response at high temperatures and samples all other prompts from its exact direct high-temperature distribution, while incurring minimal additional latency. RGD runs a short greedy probe that reuses the prompt's KV cache and exits as soon as it becomes incompatible with a learned set of refusal prefixes; it returns the greedy response if the probe remains compatible and otherwise discards the probe and samples from the original prompt. Across seven models and three benchmark datasets at , RGD raises greedy-refusal preservation from 91.9% under direct sampling to 98.3% on average while adding only 2.2–4.3% to the median per-request latency of non-refusals across temperatures. Unlike prompt-screening baselines which route many greedy non-refusals to greedy decoding, RGD keeps at least 98.1% of greedy non-refusals on unchanged high-temperature sampling, thereby preserving the model's natural high-temperature sampling behavior. We also propose a residual-stream variant of our method which lowers this latency overhead to at most 0.5% without sacrificing prompt routing accuracy. Our work shows that unlocking greater diversity via high-temperature sampling need not erode a model's refusal behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Refusal, safety, jailbreak, sampling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qmDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a lightweight decoding strategy to preserve the refusal of LLMs for harmful queries in high-temperature sampling setting, which is quite important and necessary for many domains. The idea is simple and depends on a probe that can detect the refusal early by inspecting the first tokens. If early refusal is not detected, let the model sample from high temp. othervise use the greedy response. The authors test their methods on 3 datasets and 7 models.

Critical strengths
  • The paper's argument is simple, and the presentation is clear and easy to follow, with comprehensive results. Given the major weakness below, it keeps me from giving a strong reject.
Critical weaknesses
  • The whole argument of the paper depends on the observation that the refusal tokens should be in the first few tokens of the model, as observed by Qi et al.. However, this observation is not a fundamental rule of refusal and is only a post-training artifact. We observed this because the refusal post-training data looked like this, and in many cases today, the model first reasons about whether to refuse, then decides to refuse. So the proposed idea wouldn't work in those cases, which is quite a limitation. Also, the model can start without refusing and do self-correction during generation. Those cases are ignored by the authors.

  • The prompt classifiers are actually quite common and effective. It's sometimes more effective than the model's judgment itself e.g. greedy decoding. Please see: Constitutional Classifiers, https://arxiv.org/pdf/2501.18837. So, I believe the paper relies on a moderately. outdated observation and uses a simple trick to convert that observation into compute efficiency, but this tradeoff is questionable.

Questions for authors

I would like to see the author's responses to my criticisms above.

7 comments

Thank you for recognizing the simplicity of our arguments, the clarity of our presentation, and the comprehensiveness of our results. We greatly appreciate your time and efforts to review our paper and have responded to each of your concerns below.

Early-refusal tokens are a post-training artifact; reasoning models decide after thinking

W1: The whole argument of the paper depends on the observation that the refusal tokens should be in the first few tokens of the model…in many cases today, the model first reasons about whether to refuse, then decides to refuse. So the proposed idea wouldn't work in those cases, which is quite a limitation.

Thank you for raising this point regarding text-only gating on thinking traces. Despite this limitation, RGD-residual is actually well-suited for reasoning models because its residual readout predicts post-thinking greedy refusal from the last prompt token before any reasoning trace is generated. Furthermore, RGD-residual can be used with the prefix check applied only to answer-channel tokens (as opposed to reasoning traces) on prompts for which the readout classifier is uncertain.

To test this, we conducted new experiments on Qwen3.5-9B with thinking mode enabled across three variants: Text gate, thinking trace (prefixes re-calibrated on thinking traces), Text gate, answer channel (an ablation applying the published prefix gate to the post-thinking answer channel), and RGD-residual (a thinking-mode-calibrated last-prompt-token readout, AUROC 0.986 [0.981, 0.991], plus the answer-channel gate). We generated a greedy response plus 5 sampled responses at T=1.0 and T=2.0 for 1,000 prompts (4,096-token budget; 896 resolved greedy anchors—408 refusals, 488 non-refusals—with 104 unterminated thinking traces excluded) and replayed each variant's routing on these recorded outputs. Results are provided below for T=2.0 (RGD-residual results are similar for T=1.0). Metrics over the 896 resolved prompts are as in Table 1, except that non-refusal cost is the extra tokens per greedy non-refusal / the token ratio to direct sampling, counted from the recorded outputs.

Qwen3.5-9B, thinking mode, T=2Refusal pres.Non-refusal pres.DisagreementNon-refusal cost
Direct sampling90.6%––0 / 1.00x
Text gate, thinking trace91.6%99.2%38.7%3 / 1.00x
Text gate, answer channel97.8%99.0%5.2%1,126 / 1.44x
RGD-residual (readout + answer-channel gate)97.5%99.4%3.8%194 / 1.08x

You are correct that refusal tokens rarely appear at the start of the reasoning trace: gating the thinking trace adds only +1.0 pp over direct sampling, whereas gating the answer channel boosts refusal preservation to 97.8% at high token cost (1.44x). RGD-residual achieves nearly the same refusal preservation (97.5%) at 1.08x non-refusal cost because the pre-thinking readout sends easy-to-classify non-refusals (406 of 488) directly to sampling without decoding the greedy thinking trace.

We appreciate you raising this concern about reasoning models and will update our paper to include this new thinking-mode analysis and clarify that RGD-residual is the recommended variant when thinking mode is enabled.

Self-correction during generation

W1: The model can start without refusing and do self-correction during generation. Those cases are ignored by the authors.

We already analyzed this exact comply-then-refuse (CTR) scenario in Appendix I. Specifically, we used two LLM annotators to label 766 greedy refusals that RGD or RGD-residual routed to sampling, with an author adjudicating their 48 disagreements. We found that CTR patterns are rare, occurring in only 31 of the 8,512 greedy refusals across the seven models (0.36%), which constitutes 4.3% of RGD's runtime misses and 0.13 pp of its pooled miss rate. Moreover, RGD-residual's high-side readout bypass recovers 5 of those 31 CTR prompts directly from the prompt's refusal representation (with its low-side bypass adding no CTR misses). Conversely, in the reverse self-correction case (refuse-then-comply), if a greedy response opens with a refusal prefix and later transitions into a safe answer, RGD continues the greedy stream to completion and returns that exact greedy response unchanged without safety loss or restart overhead. While we already include the CTR analysis in Appendix I of our submitted manuscript and reference it in Section 5.1, we will make the cross-reference in the main text more prominent.

Prompt classifiers are common and effective

W2: The prompt classifiers are actually quite common and effective. It's sometimes more effective than the model's judgment itself e.g. greedy decoding. Please see: Constitutional Classifiers, https://arxiv.org/pdf/2501.18837.

While we agree that prompt classifiers are quite common and effective, their objective differs from ours in that they aim to replace the model’s refusal policy. In contrast, RGD aims to preserve the model’s high-temperature sampling distribution for prompts that it would otherwise not refuse under greedy decoding while maintaining its refusal policy as temperature increases.

Our main experimental results (Table 1) show how prompt classifiers change the model’s high-temperature sampling behavior on prompts that it should otherwise answer under greedy decoding. On JBB, Shieldstral routes only 47.4–59.7% of greedy non-refusals to sampling, far below that of RGD (98.7–100%). LlamaGuard-4 does better than Shieldstral in terms of greedy non-refusal preservation on JBB (86.5–93.5%) but still underperforms RGD in both this metric and greedy refusal preservation. Furthermore, Appendix G (Table 18) of our paper evaluates all methods directly against dataset ground-truth harm/benign labels (expected_behavior) rather than the model's greedy anchor at T=2.0. On refuse-labeled prompts, RGD and RGD-residual achieve a higher refusal rate than LlamaGuard-4 in all 21 of 21 model–dataset cells, Shieldstral in 17 and 18 of 21 cells, and TPC in 18 of 21 cells. External prompt classifiers also require a separate model to be deployed alongside the target model, while RGD does not.

None of this is to diminish the role or utility of prompt classifiers. In cases where a model’s refusal response needs to be strengthened beyond its baseline level, they could be used in conjunction with RGD to further improve safety. We tested this by stacking each classifier with RGD, routing the model to greedy decoding if either RGD or the prompt classifier predicts a refusal. Pooled over seven models, the share of greedy refusals sent to sampling falls from 32.6% to 5.0% for LlamaGuard-4, from 9.5% to 2.2% for Shieldstral and from 11.3% to 2.2% for TPC. Thus, RGD catches 76–85% of each classifier's misses. Stacking classifiers with RGD keeps 0.5–0.8 pp fewer non-refusals on sampling than the classifier alone and 5.5–17.5 pp fewer than RGD alone.

Regarding Constitutional Classifiers (Sharma et al., 2025), we already cite and describe this work in Section 2 (Safety classifiers, p. 3) of our submitted manuscript. Because it is a closed Anthropic system with no publicly released weights or code, an empirical comparison is not possible, and its mechanism targets a different point in the compute–safety space: it trains a separate safeguard model on constitution-generated synthetic data and runs a streaming output classifier across every generated token (incurring continuous external-model compute throughout generation), whereas RGD requires no separate model and checks only the first 1.62–3.11 greedy tokens on average for non-refusals (or zero probe tokens for most non-refusals under RGD-residual's prefill bypass). We will expand our discussion of Constitutional Classifiers in Section 2 and add the new analysis on combining RGD with prompt classifiers to our updated manuscript.

Outdated observation; questionable trade-off

W2: I believe the paper relies on a moderately outdated observation and uses a simple trick to convert that observation into compute efficiency, but this tradeoff is questionable

The observation of Qi et al. still holds for models released in 2026 (Qwen3.5, Qwen3.8 and the Gemma-4 family), with the most recent being released 3 months ago. On these models, the first few greedy tokens usually reveal the refusal decision: non-refusal probes exit after 1.62–3.11 tokens on average, and RGD's refusal-prefix check routes only 2.69–9.77% of greedy refusals to sampling (Table 2). Based on this, we disagree that the observation is outdated and would greatly appreciate your reconsideration in light of these results.

Regarding the compute efficiency trade-off of our approach, RGD sacrifices at most 0.31 pp of refusal preservation relative to naive greedy (i.e., decoding a full greedy sequence to check for refusal) while cutting the median non-refusal latency from nearly 2x to 1.022–1.043x. Relative to prompt classifiers, RGD preserves 98.7–100% of greedy non-refusals on JBB, against 86.5–93.5% for LlamaGuard-4 and only 22.7–59.7% for Shieldstral and TPC. RGD achieves this without requiring deployment of a separate classifier alongside the target model. To the best of our knowledge, no other existing approach balances refusal & non-refusal preservation at high temperatures as well as RGD at its minimal level of incremental overhead.

I thank the authors for their rebuttal. However, their proposed method does not address a research question but rather combines multiple well-known techniques (e.g., prompt classifiers, such as residual classifiers) and obvious ideas (e.g., string matching over a few tokens) in a very straightforward way. As is evident from the results on reasoning models, most of the gains come from the prompt classifier applied at the beginning. If the authors had been the first to discover the observation made by Qi et al., this type of contribution could have been part of a research paper. However, in its current form, the work is primarily engineering rather than research. I maintain my original score. In fact, the rebuttal has strengthened my position, as the newly reported results show that most of the gains come from the prompt classifiers.

Reviewer_GLLiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RGD checks a short greedy response to decide whether to keep generating greedily or restart with high-temperature sampling. RGD-residual adds a classifier based on the model’s internal representations to reduce this extra work. The experiments measure refusal preservation and latency across several models.

Critical strengths
  1. The idea is simple and useful. It helps preserve refusals at high temperatures while letting most non-refusal prompts continue with normal sampling.

  2. Early stopping and cache reuse keep the extra cost low. The experiments cover several models and include repeated sampling and an analysis of missed refusals.

Critical weaknesses
  1. The method makes a small change, so its value depends largely on how well it balances routing accuracy and speed. The comparisons do not yet make this advantage clear.

  2. I would like to see a comparison with the refusal classifier alone. TPC’s training labels and calibration also need clarification to make the comparison easier to judge.

  3. Most latency results for the residual version use precomputed features. It is unclear whether the overhead stays as low when the full method runs across different models.

Questions for authors
  1. At a similar miss rate, how much does the text check help compared with using the classifier alone?

  2. Was TPC trained to predict harmful prompts or the model’s refusals? How does this affect the comparison?

  3. When do checks usually stop for non-refusal prompts, and what causes most of the extra delay?

3 comments

Thank you for recognizing that RGD meets a practical objective with a simple procedure and that the text-only version needs no activation hook. We greatly appreciate your time and efforts to review our paper and have responded to each of your concerns below.

Technical contribution

W1: RGD's text-only method makes a modest technical change through early stopping with calibrated refusal prefixes, cache reuse, and switching between greedy and sampled decoding. Restarting the unchanged sampler preserves the sampling-branch distribution, a property also available to other prompt-level routers. The main contribution is therefore the routing–cost trade-off.

While it is true that other prompt-level routers preserve the sampling distribution for prompts which they route to temperature-based sampling, they deviate from the original model’s refusal routing distribution much more significantly than RGD and RGD-residual. Harm-label routers (Shieldstral, TPC) keep only 22.7–59.7% of JBB greedy non-refusals on sampling, whereas RGD and RGD-residual preserve 98.7-100% of JBB greedy non-refusals. Thus, our approach better preserves the natural high-temperature sampling behavior of the original model for prompts which it does not refuse under greedy decoding. Our text-only RGD method is also simpler in terms of needing no extra classifier, second model or activation hook, with easy implementation via stock vLLM prefix caching. We will update the discussion of our contributions in the introduction to better emphasize these advantages of our approach.

Residual classifier-only router baseline

W2: For the residual extension, a comparison with its own refusal-trained readout alone is missing; high AUROC cannot establish the text gate's added value at the relevant operating point.

Thank you for this suggestion. We analyzed a readout-only router using the unchanged RGD-residual readout classifier, where s(p) >= theta routes responses to greedy decoding and everything else to temperature-based sampling. We fixed theta on the 500-prompt validation split before evaluation (the maximum validation retention at the published RGD-residual's validation miss rate).

The table below provides results of the classifier-only router on our main evaluation suite consisting of 7 models and 2,650 prompts. We define a Miss as a greedy refusal routed to sampling. Retention refers to a greedy non-refusal which is correctly routed to sampling. Differences are reported as the readout-only classifier minus RGD-residual (in pp). The last column is a diagnostic that uses test labels: the readout-only threshold is moved until its miss rate equals RGD-residual's (to within 0.3 pp).

ModelRGD-residual: miss / retentionReadout only: miss / retentionΔ miss; Δ retentionΔ retention at equal miss
Qwen2.5-7B14.3% / 98.8%13.8% / 98.3%-0.5 [-2.4, 1.5]; -0.5 [-1.1, 0.2]-0.3 [-0.9, 0.3]
Qwen3.5-9B5.0% / 98.7%5.2% / 98.4%+0.2 [-1.0, 1.3]; -0.4 [-1.1, 0.4]-0.4 [-1.1, 0.4]
Qwen3.8-27B5.8% / 98.7%9.2% / 98.0%+3.4 [2.2, 4.7]; -0.8 [-1.4, -0.2]-3.2 [-4.3, -2.2]
Gemma-4-E2B6.2% / 98.7%8.8% / 97.2%+2.5 [1.0, 4.0]; -1.5 [-2.4, -0.6]-3.6 [-4.7, -2.5]
Gemma-4-E4B6.4% / 99.2%6.4% / 94.3%+0.0 [-1.3, 1.2]; -4.9 [-6.2, -3.7]-4.9 [-6.2, -3.7]
Gemma-4-12B3.2% / 99.3%3.8% / 96.5%+0.6 [-0.3, 1.5]; -2.8 [-3.7, -1.9]-3.9 [-4.9, -2.8]
Llama-3.1-8B3.3% / 99.9%5.0% / 97.5%+1.6 [0.5, 2.8]; -2.5 [-3.3, -1.8]-4.0 [-5.0, -3.1]
All seven6.3% / 99.1%7.4% / 97.2%+1.2 [0.6, 1.7]; -1.9 [-2.2, -1.5]-2.9 [-3.3, -2.5]

These results show that the readout-only classifier is never significantly better than RGD-residual in terms of either misses or retentions. It ties RGD-residual on Qwen2.5 and Qwen3.5 while being significantly worse on at least one metric for the other five models. High global AUROC (0.992–0.996) is driven by easy prompts in the extreme tails (s(p) <= eta_- and s(p) >= eta_+), where readout-only and RGD-residual make the exact same single-pass decision. The two methods differ exclusively in the uncertain band eta_- < s(p) < eta_+ (10–21% of prompts across the six two-sided models), where refusal and borderline non-refusal representations overlap: matching RGD-residual's low miss rate forces a scalar readout-only threshold down into that overlap region (theta = 0.22–0.64 on the five non-tied models), sweeping 2.8–5.7% of non-refusals into greedy decoding, whereas RGD-residual checks the first few greedy tokens on just those uncertain prompts before routing. Thus, AUROC does not make the text gate in RGD-residual redundant.

Residual classifier-only router baseline (continued)

Q1: At a comparable rate of routing greedy refusals to sampling, does the hybrid retain more greedy non-refusals for sampling than the readout alone?

Yes. We adjusted the readout-only threshold on the test suite until its miss rate equaled RGD-residual's (to within 0.3 pp). At that equal miss rate, the hybrid keeps 3.2–4.9 pp more greedy non-refusals on sampling on Qwen3.8, the three Gemma-4 models and Llama (theta must drop to 0.22–0.64 to match RGD-residual's miss rate on those five models). Qwen2.5 and Qwen3.5 exhibit a smaller difference of 0.3–0.4 pp. Pooled over the seven models, at nearly equal miss rates (6.2% for readout only, 6.3% for RGD-residual), RGD-residual keeps 2.9 pp more greedy non-refusals on sampling (95% CI [2.5, 3.3]).

Additional out of distribution evaluation. The readout classifier was trained on vanilla WildJailbreak prompts, so we further evaluated all routers on 1,603 harmful and 1,639 benign WildJailbreak adversarial prompts at T=2.0 with 10 seeds and three models (Qwen2.5-7B, Qwen3.5-9B, Gemma-4-E4B). Pooled over the three models:

Adversarial prompts, T=2.0Readout onlyRGDRGD-residual
Harmful: refusal preservation96.4%96.0%96.3%
Harmful: non-refusals kept on sampling91.2%98.8%97.6%
Benign: refusals routed to sampling31.4%23.4%22.0%
Benign: non-refusals kept on sampling96.2%97.8%98.2%

On harmful adversarial prompts, the readout-only classifier ties RGD-residual on preservation but keeps 6.3 pp fewer non-refusals on sampling. On benign prompts, RGD-residual routes 9.4 pp fewer refusals to sampling and keeps 2.0 pp more non-refusals.

We will add the readout-only baseline to Table 1 and these additional comparisons to Appendix A.

TPC labels and calibration

W3 / Q2: Was the local TPC baseline trained to predict prompt harmfulness, as in the original task, or the target model's greedy refusal? Since these labels need not coincide, how should its sampling-retention gap be interpreted as evidence for RGD's routing advantage?

As in Oldfield et al., we trained TPC on prompt-harm labels (WildJailbreak expected_behavior) and will update Appendix C to clarify this. Training TPC on a model’s greedy refusal labels represents an adaptation of the original method. To investigate the effectiveness of this, we evaluated four different variants of TPC which alter the label used for training (prompt harmfulness labels or a model’s own WildGuard greedy-refusal labels) and how the routing is done (score-only as in TPC, or using the RGD-residual hybrid method with a text gate plus two-sided bypass). All four TPC variants were calibrated the same way as the readout-only router using the maximum validation retention at RGD-residual's validation miss rate.

Router (seven models pooled)Refusals routed to samplingNon-refusals kept on samplingMinus RGD-residual (pp): miss; retentionJBB non-refusals kept (per-model range)
RGD-residual (linear readout)6.3%99.1%–98.7–100.0%
TPC as in our paper (harm labels, threshold 0.5)11.3%82.0%+5.0 [3.8, 6.3]; -17.0 [-18.6, -15.5]22.7–39.5%
TPC recalibrated (harm labels, score only)7.2%64.7%+0.9 [0.0, 1.8]; -34.4 [-35.8, -33.0]0.0–54.2%
TPC + RGD's hybrid form6.7%91.6%+0.4 [-0.2, 1.0]; -7.5 [-8.4, -6.6]46.8–100.0%
TPC + RGD's routing target6.5%95.3%+0.3 [-0.3, 0.9]; -3.7 [-4.2, -3.2]85.5–98.7%
TPC + both5.9%98.0%-0.4 [-0.8, 0.0]; -1.1 [-1.4, -0.8]96.1–100.0%
Reference: readout only (no text gate)7.4%97.2%+1.2 [0.6, 1.7]; -1.9 [-2.2, -1.5]92.1–100.0%

The above results show that RGD-residual still beats every evaluated TPC variant. Pooled across models, it keeps 1.1–34.4 pp more non-refusals on sampling than each variant, and none of them misses significantly fewer refusals. Per model, it keeps significantly more non-refusals in 33 of the 35 model–variant pairs; the other two are ties on Llama. On JBB it keeps 98.7–100%, versus 85.5–98.7% for refusal-label TPC. Beyond calibrating TPC on the model’s greedy refusal labels, its performance can be further improved by adopting RGD’s hybrid design (i.e., including the text gate). However, even with both the greedy refusal labels and hybrid design, RGD-residual still preserves 1.1 pp more non-refusals [0.8, 1.4] than TPC's order-5 polynomial at a tied pooled miss rate, outperforming on six of the seven models (Llama was a tie).

These new results demonstrate how RGD's advantage over harm-label classifiers comes from its routing target and its hybrid form. They also show how our hybrid approach is flexible and can perform well when integrated with other classifiers such as TPC. We will include this analysis in Appendix A and update our discussion of RGD’s performance advantage in the main text accordingly.

Residual efficiency from precomputed features and one integrated run

W4: The residual efficiency claims draw on six evaluations with precomputed features and one integrated Llama result, limiting conclusions about end-to-end deployment cost across models.

To address your concern, we implemented the residual readout directly into vLLM. In this new implementation, the worker reads the last prompt token's activation during prefill and routes before the first generated token without relying on any precomputed features. We re-ran evaluations for six models using the new implementation (existing Llama-3.1-8B results in our paper already used a fused low-side implementation). The table below provides the fused implementation's median request latency ratio relative to direct sampling in the same engine (ten-seed means; ranges are over T=1.0/1.5/2.0).

ModelReadout onlyRGD-residual
Qwen2.5-7B1.003–1.004x1.004–1.006x
Qwen3.5-9B1.003x1.017–1.022x
Qwen3.8-27B1.004–1.006x1.025–1.027x
Gemma-4-E2B1.08–1.12x1.04–1.06x
Gemma-4-E4B1.034–1.035x1.007–1.008x
Gemma-4-12B1.041–1.048x1.015–1.020x
Llama-3.1-8B–1.002x (low-side bypass)

Note that readout-only has a higher median latency ratio than two-sided RGD-residual on all three Gemma-4 models despite decoding zero probe tokens. This is because in batches with zero greedy-masked requests, readout-only runs at 1.001–1.007x across all six models, confirming that the live prefill activation readout itself adds negligible overhead (<0.7%). However, vLLM's custom LogitsProcessor greedy mask slows every decode step of any batch containing at least one greedy-routed request (1.03–1.18x per step). Because readout-only misroutes more Gemma-4 non-refusals to greedy (2.8–5.7% vs. 0.7–1.3% for RGD-residual), 57–81% of Gemma-4 batches contain a masked request and cross the p50 median step, whereas fewer than 45% of Qwen batches do (keeping Qwen readout-only p50 at 1.003–1.006x and mean latency ratios at 1.014–1.033x, vs. 1.014–1.056x for RGD-residual).

These results show that our hybrid approach remains efficient when implemented in vLLM without the need for precomputed features. We will add these new latency ratios to Appendix A and will also make our vLLM implementation publicly available upon publication of our paper. We will also revise the abstract's claim that RGD-residual's latency overhead is at most 0.5%, which was measured with precomputed features, to report the fused implementation's median latency ratio of 1.002–1.06x across all seven models.

Probe exits within the c=3 gate

Q3: The text-only method reports low probe overhead. Do the existing traces show how often non-refusal probes exit within the initial c=3 gate versus later, and which cases account for most of the remaining overhead?

Yes, our existing streaming RGD traces record each probe's tokens, stop reason and timestamps. An analysis of these traces for the non-refusal cohort from our main results table (7 models, 10 seeds, T = 1.0/1.5/2.0) yielded the following findings:

  • Exits. 95.7–99.2% of probes exit inside the c=3 gate, 80.9–95.5% at the first token. Another 0.3–2.7% exit a few tokens later (4.0–6.4 tokens on average).
  • Long probes. 0.2–1.6% stay refusal-compatible to EOS or the 128-token budget and are served greedily. Most are non-refusals that open like a refusal ("I'm sorry to hear…", "As an AI…"). They constitute 18–60% of all probe tokens but add almost no latency, because the probe is the returned response.
  • Latency. The overhead is one extra round trip before the first sampled token (60–232 ms) because a first-token probe takes as long as the direct request's own time to first token. Each further probe token adds 16–44 ms. Gate exits account for at least 93% of the mean overhead.

We will update Appendix A to include this analysis and an exit-position histogram (share of requests and of probe tokens, per model).

Reviewer_GVv7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RGD, which decodes greedily until the output stops matching a learned set of refusal prefixes. If the probe stays compatible, it returns the greedy refusal; otherwise it discards the probe and samples at high temperature from the cached prompt. RGD-residual adds a logistic probe on the last prompt token's residual activation to skip the gate when confident.

Critical strengths

Released prompts follow the unchanged sampling distribution, because routing depends only on the prompt. The evaluation is broad (7 models, 3 datasets, 3 temperatures, 10 seeds) and careful, with a judge panel validated against human labels (App. D).

Critical weaknesses
  1. The closest control is missing. The residual probe predicts greedy refusal with AUROC 0.992-0.996, so a classifier-only router trained on the same labels might match RGD without probe tokens. TPC does not answer this: its training labels are unstated, and it sends most JBB non-refusals to greedy. Please add this router at matched operating points.
  2. Leakage is reported per sample, not per prompt. The prefix gate misses up to 20.3% of greedy refusals (Qwen2.5, Table 2), and routing is fixed per prompt, so those prompts are sampled on every request. Under RGD, 1.5-11.9% of refusal-cohort prompts still produce a non-refusal within ten samples at T=2.0T = 2.0, and 9.6% for Qwen2.5 at T=1.0T = 1.0 (Table 7). Given the paper's own Best-of-N framing, this belongs in the main results
Questions for authors

Section 5.2 calls 0.72-1.40% per-sample harmful transitions "very low risk". What is the rate per prompt over ten samples?

2 comments

Thank you for recognizing that RGD leaves the sampling distribution of every released prompt unchanged and that our evaluation is broad. We greatly appreciate your time and efforts to review our paper and have responded to each of your concerns below.

Residual classifier-only router baseline

W1: The closest control is missing. The residual probe predicts greedy refusal with AUROC 0.992-0.996, so a classifier-only router trained on the same labels might match RGD without probe tokens. TPC does not answer this: its training labels are unstated, and it sends most JBB non-refusals to greedy. Please add this router at matched operating points.

Thank you for this suggestion. We analyzed a readout-only router using the unchanged RGD-residual readout classifier, where s(p) >= theta routes responses to greedy decoding and everything else to temperature-based sampling. We fixed theta on the 500-prompt validation split before evaluation (the maximum validation retention at the published RGD-residual's validation miss rate).

(a) Main suite. The table below provides results of the classifier-only router on our main evaluation suite consisting of 7 models and 2,650 prompts. We define a Miss as a greedy refusal routed to sampling. Retention refers to a greedy non-refusal which is correctly routed to sampling. Differences are reported as the readout-only classifier minus RGD-residual (in pp). The last column is a diagnostic that uses test labels: the readout-only threshold is moved until its miss rate equals RGD-residual's (to within 0.3 pp).

ModelRGD-residual: miss / retentionReadout only: miss / retentionΔ miss; Δ retentionΔ retention at equal miss
Qwen2.5-7B14.3% / 98.8%13.8% / 98.3%-0.5 [-2.4, 1.5]; -0.5 [-1.1, 0.2]-0.3 [-0.9, 0.3]
Qwen3.5-9B5.0% / 98.7%5.2% / 98.4%+0.2 [-1.0, 1.3]; -0.4 [-1.1, 0.4]-0.4 [-1.1, 0.4]
Qwen3.8-27B5.8% / 98.7%9.2% / 98.0%+3.4 [2.2, 4.7]; -0.8 [-1.4, -0.2]-3.2 [-4.3, -2.2]
Gemma-4-E2B6.2% / 98.7%8.8% / 97.2%+2.5 [1.0, 4.0]; -1.5 [-2.4, -0.6]-3.6 [-4.7, -2.5]
Gemma-4-E4B6.4% / 99.2%6.4% / 94.3%+0.0 [-1.3, 1.2]; -4.9 [-6.2, -3.7]-4.9 [-6.2, -3.7]
Gemma-4-12B3.2% / 99.3%3.8% / 96.5%+0.6 [-0.3, 1.5]; -2.8 [-3.7, -1.9]-3.9 [-4.9, -2.8]
Llama-3.1-8B3.3% / 99.9%5.0% / 97.5%+1.6 [0.5, 2.8]; -2.5 [-3.3, -1.8]-4.0 [-5.0, -3.1]
All seven6.3% / 99.1%7.4% / 97.2%+1.2 [0.6, 1.7]; -1.9 [-2.2, -1.5]-2.9 [-3.3, -2.5]

These results show that the readout-only classifier is never significantly better than RGD-residual in terms of either misses or retentions. It ties RGD-residual on Qwen2.5 and Qwen3.5 while being significantly worse on at least one metric for the other five models. High global AUROC (0.992–0.996) is driven by easy prompts in the extreme tails (s(p) <= eta_- and s(p) >= eta_+), where readout-only and RGD-residual make the exact same single-pass decision. The two methods differ exclusively in the uncertain band eta_- < s(p) < eta_+ (10–21% of prompts across the six two-sided models), where refusal and borderline non-refusal representations overlap: matching RGD-residual's low miss rate forces a scalar readout-only threshold down into that overlap region (theta = 0.22–0.64 on the five non-tied models), sweeping 2.8–5.7% of non-refusals into greedy decoding, whereas RGD-residual checks the first few greedy tokens on just those uncertain prompts before routing. Thus, AUROC does not make the text gate in RGD-residual redundant.

(b) Out of distribution. The readout classifier was trained on vanilla WildJailbreak prompts, so we further evaluated all routers on 1,603 harmful and 1,639 benign WildJailbreak adversarial prompts at T=2.0 with 10 seeds and three models (Qwen2.5-7B, Qwen3.5-9B, Gemma-4-E4B). Pooled over the three models:

Adversarial prompts, T=2.0Readout onlyRGDRGD-residual
Harmful: refusal preservation96.4%96.0%96.3%
Harmful: non-refusals kept on sampling91.2%98.8%97.6%
Benign: refusals routed to sampling31.4%23.4%22.0%
Benign: non-refusals kept on sampling96.2%97.8%98.2%

On harmful adversarial prompts, the readout-only classifier ties RGD-residual on preservation but keeps 6.3 pp fewer non-refusals on sampling. On benign prompts, RGD-residual routes 9.4 pp fewer refusals to sampling and keeps 2.0 pp more non-refusals.

Residual classifier-only router baseline (continued)

(c) Cost (see details in our reply to GLLi, W4). When implemented directly in vLLM, the readout-only classifier is not uniformly cheaper: it achieves a median latency ratio of 1.003–1.006x for the Qwen models and 1.034–1.12x for the Gemma-4 models, versus 1.002–1.06x for RGD-residual across all seven models. Although the live prefill readout itself adds <0.7% in batches without greedy-routed requests, readout-only misroutes more Gemma-4 non-refusals to greedy (2.8–5.7% vs. 0.7–1.3%), triggering per-step LogitsProcessor greedy masking in more than half of batches and raising median latency.

TPC labels. As in the original task of Oldfield et al., we trained TPC on prompt-harm labels (WildJailbreak expected_behavior) and will update Appendix C to clarify this. See additional new results for TPC in our reply to GLLi, W3/Q2.

Leakage per prompt

W2: Leakage is reported per sample, not per prompt…Given the paper's own Best-of-N framing, this belongs in the main results

Thank you for this observation. We will move the prompt-level any-of-10 metric into our main results (Table 1). At T=2.0, the share of refusal-cohort prompts with at least one coherent non-refusal in ten samples is 14.50–38.17% for direct sampling, 1.52–11.92% for RGD and 2.88–11.20% for RGD-residual. At T=1.0, direct sampling is at 3.15–17.16%. Thus, RGD and RGD-residual still offer significant improvements relative to direct sampling on this metric.

We conducted an additional analysis to further quantify the benefits of RGD and RGD-residual over direct sampling in the any-of-N metric as N increases. The table below provides the share of refusal-cohort prompts with at least one coherent non-refusal among N samples (averaged over all N-seed subsets) for N = 1, 5, and 10 at T=2.0, pooled over seven models:

NDirect samplingRGDRGD-residual
17.4%2.2%2.3%
518.5%4.6%4.6%
1023.6%5.4%5.3%

These results show that the advantage of RGD and RGD-residual grows with N. RGD minus direct is -5.2 pp [-5.6, -4.8] at N=1, -18.2 pp [-19.5, -17.1] at N=10, and is significant on every model at N=10 (-13.0 to -32.3 pp). From N=5 to N=10, direct sampling gains 5.1 pp while RGD gains only 0.8 pp, because fixed per-prompt routing locks caught refusals (79.7–97.5% for RGD; 85.7–96.8% for RGD-residual) to greedy across all seeds, capping Best-of-N leakage at the detector miss rate. On Qwen2.5, where 206 of 225 text-gate misses open with soft deflection or pushback (Table 20), RGD-residual's high-side bypass cuts the miss rate from 20.3% to 14.3% (reducing any-of-10 leakage from 9.6% to 8.6% at T=1.0 and 11.9% to 11.2% at T=2.0). On the four batch-invariant models with matched output banks (Qwen2.5, Gemma-4-E2B, Gemma-4-12B, Llama), readout-only leaks 1.5 pp [0.9, 2.1] more than RGD-residual at N=10. We will update our manuscript to include this new analysis.

Per-prompt harmful-transition rate over ten samples

Q1: Section 5.2 calls 0.72-1.40% per-sample harmful transitions "very low risk". What is the rate per prompt over ten samples?

On a per-prompt basis at T=2.0, 2.05–5.01% of the prompts whose greedy response is a harmless non-refusal (in all ten seeds) receive at least one harmful direct sample in ten (3.36% [2.78, 3.92] when pooled across models; paired per-sample rate 0.46–1.40%, vs. Table 3's seed-specific 0.72–1.40% denominator). At T=1.0, this drops to 1.45–3.63% (2.35% [1.90, 2.81] pooled). Notably, 3.36% is well below the 9.2% expected if the 0.96% pooled per-sample failures occurred independently across prompts, because harmful samples cluster on a small subset of borderline prompts (2.86 per affected prompt on average), where output moderation remains complementary. We will update Section 5.2 to include these per-prompt harmful transition rates and revise the “very low risk” statement to reflect the greater risk posed in the any-of-N sampling setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.