acceptodds
Under review as a conference paper at ICLR 2027

Watch and Correct: Efficient Streaming Intervention for LLM Alignment with Distribution-Free Guarantees

Abstract

Advances in large language models (LLMs) are enabling increasingly capable interactive applications, placing safety and responsiveness at the center of deployment. Despite progress in alignment, these models can still produce harmful content, motivating inference-time safeguards that detect and correct unsafe generations. However, repeated correction consumes computation and delays delivery, with both its effectiveness and cost depending on the generation and guardrail models. The unresolved deployment challenge is to determine the safety attainable within a limited repair budget and the intervention cost required to meet a prescribed risk tolerance. We introduce Watch and Correct (WaC), a framework for risk-constrained generation that combines statistical calibration of repair budgets with efficient streaming correction. WaC identifies budgeted intervention policies that meet a target harm rate with finite-sample, distribution-free guarantees, and characterizes the corrective effort they incur across model configurations. This connects safety requirements to concrete choices of generation models, guardrail models, and repair budgets. To make these policies practical for interactive use, WaC integrates feedback-guided repair with asynchronous verification and computation reuse, reducing redundant work and interruptions to response delivery. Experiments demonstrate that WaC reduces token lag by approximately half compared with state-of-art intervention methods, improving the responsiveness of guarded generation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language models, decoding-time, alignment, output-control, statistical guarantees, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vg3dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of balancing safety guarantees and inference efficiency in LLM generation with corrective interventions. Motivated by the computational overhead and latency introduced by repeated safety verification and regeneration, this paper proposes Watch and Correct (WaC), a framework that combines risk-calibrated repair budgets with asynchronous streaming correction. It employs a Learn-Then-Test procedure to provide finite-sample, distribution-free guarantees on final-response harm rates, while using feedback-guided repair and KV-cache reuse to reduce redundant computation. Experiments on Qwen2.5-7B and its uncensored variant demonstrate lower harmful-response rates and approximately 50% lower token lag compared with existing intervention methods.

Critical strengths

The paper addresses a practically important problem by jointly considering safety guarantees, repair budgets, and inference latency in LLM generation. The integration of risk calibration with asynchronous streaming correction provides a useful framework for deploying safety interventions under resource constraints.

Critical weaknesses
  1. The conceptual novelty of the proposed framework appears somewhat limited, as several key components, including statistical risk calibration, streaming safety intervention, asynchronous verification, and KV-cache reuse, have been explored in prior studies. Although integrating these techniques into a unified framework is practically useful, the paper does not sufficiently establish what new insights or technical capabilities are enabled by their combination. More targeted ablation studies would help clarify the necessity and individual contributions of the proposed components.

  2. The experimental evaluation is insufficient to fully support the proposed design. The paper lacks comprehensive ablation studies to demonstrate the necessity and effectiveness of its key components. Furthermore, there are several inconsistencies in the experimental settings, such as the harm labeler, response quality evaluation, and repair budget configurations described in the main text and appendix. These issues make it difficult to reliably assess the reported improvements.

Questions for authors

Could the authors provide more comprehensive ablation studies to clarify the individual contributions of its key components, particularly asynchronous verification, KV-cache reuse, and feedback-guided correction?

4 comments

We sincerely thank the reviewer for their comments. We also appreciate the opportunity to further clarify the individual contributions of WaC and provide additional experimental evidence supporting our design choices.

Q1: Discussion on the novelty and individual contributions of WaC's components.

While we agree that some components of WaC have been explored individually in prior work, our contribution lies in integrating them into a unified framework for efficient safety intervention with statistical guarantees. Importantly, each component serves a distinct purpose and provides improvements to WaC, as we explain below.

(i) Statistical risk calibration The statistical risk calibration framework jointly calibrates the intervention threshold and repair budget, providing statistical guarantees for the chosen harmful-response rate using minimum budget. Without it, the method would be heuristic.

(ii) Streaming safety intervention. Instead of waiting for the entire response to be generated and then verifying its safety, WaC performs verification and correction incrementally. Since harmful content typically constitutes only a small portion of the generated response, streaming intervention allows WaC to regenerate only the harmful portions rather than the entire response, substantially reducing unnecessary computation.

Latency comparison of WaC with and without streaming.

VariantLag (tok)Time (s)
WaC27.37.21
w/o streaming311.212.36

(iii) Asynchronous verification. Unlike conventional streaming approaches, where the generator pauses until the verifier approves the preceding output, WaC allows generation and verification to proceed in parallel. Combined with the observation that harmful content generally occupies only a small portion of the response, this significantly reduces user-perceived latency, as the generator only needs to pause occasionally to correct harmful segments. To our knowledge, WaC is also the first to introduce asynchronous verification specifically for streaming safety intervention.

Latency comparison over 700 prompts. Lower is better.

MethodTTFT (s)Lag (tok)AWT (tok)
CARE (Synchronous)1.3541.752.2
WaC (Asynchronous)0.7520.939.1

(iv) KV-cache reuse. WaC reuses the cached attention states of previously approved sentences, avoiding redundant computation when regenerating rejected portions. Since rollbacks can occur at multiple locations throughout a response, preserving the approved prefix avoids repeated attention computations and substantially reduces the matrix multiplication operations required during correction.

Compute comparison of WaC with and without KV-cache reuse.

VariantCompute (TFLOPs)
WaC42.2
w/o KV-cache reuse169.4

We have added this discussion in Section 3.3.

Q2: Clarification on the experimental ablations and reported settings.

Ablations are addressed in our response to comment 1 above. For the seeming inconsistencies, we address them as follows.

  (i) The harm labeller is GPT-6-Luna everywhere, which uses the classification prompt of HarmBench. Hence, HarmBench in Appendix D.2 refers to the prompt not the model.

  (ii) The unguarded generator for the quality reference is GPT-4o-mini. We understand that the unguarded generator used in the experiments as a reference and GPT-4o-mini in the quality evaluations were probably conflated. We clarify this in the updated version.

  (iii) The budget is calibrated using LTT under a grid of candidate budgets. Hence, there is no fixed budget used without a calibration step. The T=5 is a typo that we fix, and we have fixed related typos in the updated paper.

We hope this answers the reviewer's questions. We hope they would reconsider their evaluation in light of our answers, and we meanwhile remain open to providing any further required clarifications.

AuthorsOpenReview ↗

Follow-up on our responses

We are grateful to the reviewer for the thoughtful feedback and for recognising that WaC addresses a practically important problem. The reviewer valued that it considers safety guarantees, repair budgets and latency together. The reviewer also noted that combining risk calibration with asynchronous streaming correction gives a useful framework for deploying safety interventions under resource constraints.

The reviewer saw only minor or fixable concerns about soundness, and we addressed them:

  • We clarified the role of each component of WaC in a new discussion (Section 3.3), in support of our contribution.
  • We reported ablations showing what each component contributes. Streaming cuts the lag from 311 to 27 tokens, and reusing the key–value cache cuts compute about fourfold.
  • We resolved the inconsistencies in the experimental settings: the harm labeller, the quality evaluation and the repair budget.

These changes improved how the paper positions its contribution and how clearly it reads, thanks to your comments. We would appreciate it if you could let us know whether our responses answer your questions, or point us to anything that remains so we can address it.

I appreciate the detailed responses and additional experiments. The new ablations and clarifications address several of my concerns regarding the individual components and experimental settings. However, my main concerns remain. First, we still can't tell whether the safety gains are driven by successful corrections or just abstaining more often. Second, the formal objective doesn't actually back up the claim of cost-aware calibration. Finally, the overall approach still comes across as a fairly standard combination of known techniques. Therefore, I am inclined to maintain my original score.

We thank the reviewer for engaging with our rebuttal, and we are glad our clarifications addressed several of the concerns. We appreciate the chance to clarify the remaining points.

1. Correction vs abstention.

We thank the reviewer for this important question. We respectfully refer the reviewer to our response to Reviewer Kg7r's Q1, and summarise it here:

We provide an additional analysis in Table 3 (Appendix F), which reports the harmful outputs successfully corrected through feedback-guided regeneration within the allocated repair budget, on the prompts that never abstained. On this set of prompts, WaC corrects substantially more outputs within the available budget, with a successful correction rate 2.1 times higher than that of CARE/CD on average over the evaluated settings. These results highlight that WaC's safety improvements are not solely attributable to its abstention mechanism, as feedback-guided correction also plays an important role and outperforms the correction of CARE and CD. Moreover, making use of abstention after calibrating using Algorithm 2 of the paper keeps the harmful-response rate below the target level with high probability, and is our contribution, not used in prior work.

The % of responses deemed safe (↑) for WaC, CARE and CD on the same prompts: those with a harmful unguarded answer on which the generator does not abstain.

BenchmarkGeneratorJudgeWaCCARECD
BeaverTails-Eval.UncensoredHarmBench58.558.567.967.975.575.5
BeaverTails-Eval.UncensoredLlama-3.1-8B35.635.624.424.424.424.4
BeaverTails-Eval.UncensoredMistral-7B75.475.437.737.776.876.8
AdvBenchUncensoredHarmBench88.288.282.482.485.385.3
AdvBenchUncensoredLlama-3.1-8B70.770.713.313.316.016.0
AdvBenchUncensoredMistral-7B100.0100.024.424.473.373.3

The results of this table complements our existing results in Tables 1 and 2, where WaC generally achieves substantially lower harmful-response rates on the uncensored generator while operating with smaller repair budgets than CARE/CD. Together, these findings provide further insight into the effectiveness of WaC's correction mechanism.

2. The cost-aware calibration claim.

We thank the reviewer for pointing this out. Any configuration in the admissible set A\mathcal{A} constructed by the calibration procedure (Algorithm 2) satisfies the harm-control guarantee, as Proposition 1 holds simultaneously for every pair in A\mathcal{A}. As also noted in our responses to Reviewer Kg7r's Q4 and Reviewer WkCP's Q1, and as stated in the Introduction, we select "the lowest observed-cost certified configuration compatible with their response-quality and refusal requirements." Therefore, our selection rule is indeed cost-aware.

3. WaC as a standard combination of known techniques.

We appreciate this perspective and agree that WaC builds on known tools, which we see as a strength, as it keeps WaC simple to adopt and guardrail-agnostic. The novelty lies in what the combination enables, which none of these tools provides alone:

  • Risk calibration → a certified repair budget. Existing intervention methods fix the number of retries by hand. WaC calibrates the threshold and the budget jointly, so a provider knows what correction effort a target risk level requires (Section 1).
  • Asynchronous verification → a guarantee that survives streaming. Overlapping verification with generation would normally break a calibration guarantee. WaC's rollback rules preserve it (Remark 1, Section 3.3), while halving the lag relative to CARE.

To our knowledge, no one has combined these techniques in this particular way to achieve the results and guarantees presented in the paper. We respectfully suggest that the value of a method lies in its effectiveness and the new insights we draw from it (how it combines different perspectives) rather than its complexity.

We thank the reviewer for their time to provide a constructive discussion and hope that these comments address their raised questions.

Reviewer_WkCPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WaC, which grades each completed sentence in parallel with generation, regenerates rejected sentences with feedback under a response-wide budget, and abstains when the budget runs out. It calibrates the threshold and budget jointly with Learn-Then-Test and compares against CARE and contrastive decoding.

Critical strengths

Proposition 1 is a valid, if standard, LTT guarantee, and Algorithm 3 labels abstained outputs rather than assuming refusals are safe. Overlapping grading with generation helps: against CARE, lag falls from 41.7 to 20.9 tokens and time-to-first-token from 1.35 to 0.75 s (Fig. 2a).

Critical weaknesses
  1. The certification evidence is circular. Figure 3's "certified level" is the smallest grid β\beta above WaC's test harm rate (p. 16), so every WaC bar sits below its line by construction. Appendix E's claim that β=0.02\beta = 0.02 is certified with the Mistral grader conflicts with Appendix F, where nothing below about 0.024 can be certified (p. 17).
  2. Specifications and results disagree. The final harm labeler is GPT-6-luna (Table 1) or HarmBench "for all runs" (App. D.2); the quality reference is GPT-4o-mini (Table 1) or the unguarded generator (App. D.5); the budget is 5 (App. D.4) or 1-4 (Table 1); feedback includes the grader's explanation (Eq. 2) or only the rejected attempt and σ(γ)\sigma(\gamma) (App. D.4). Figure 3 omits the AdvBench CD runs as "not yet available", yet Table 2 reports them.
Questions for authors
  1. Calibration uses BeaverTails prompts and evaluation uses BeaverTails-Evaluation (p. 17). How is Proposition 1's i.i.d. condition met?
  2. What measured costs support the "least observed-cost" selection (p. 2), given Eq. (11) minimizes abstention?
3 comments

We thank the reviewer for their comments. We also appreciate the opportunity to clarify the concerns raised and address each point below.

W1: Clarification on whether the certification evidence is circular.

We respectfully disagree that the certification evidence is circular. Our certification follows directly from Proposition 1, which provides finite-sample guarantees on the harmful-response rate through the Learn-Then-Test framework. Figure 3 is intended to empirically illustrate the risk-control results. It is therefore expected that the harmful-response rates of WaC fall below the corresponding risk levels (β\beta), as these are the levels being illustrated.

W2: Clarification on the seeming inconsistencies between the specifications and the results.

The mentioned seeming inconsistencies are addressed as follows:

  (i) The harm labeller is GPT-6-Luna everywhere, which uses the classification prompt of HarmBench. Hence, HarmBench in Appendix D.2 refers to the prompt not the model.

  (ii) The unguarded generator for the quality reference is GPT-4o-mini. We understand that the unguarded generator used in the experiments as a reference and GPT-4o-mini in the quality evaluations were probably conflated. We clarify this in the updated version.

  (iii) The budget is calibrated using LTT under a grid of candidate budgets. Hence, there is no fixed budget used without a calibration step. The T=5 is a typo that we fix, and we have fixed related typos in the updated paper.

  (iv) σ(γ)\sigma(\gamma) is not part of what is given to the generator; it is the approval rule as in eq. (4). All the rejected attempts include the judge's explanation (when the judge gives one; the HarmBench classifier only returns a verdict) as specified in Definition 1, eq. (2) and demonstrated in Figures 6-11 in the appendix, which show real examples of generator-judge runs.

  (v) Figure 3 is now updated with Table 2 results and moved to the main text as we find its results important to show in the paper. This also addresses comment 1.

All clarifications are highlighted in blue in the updated version.

Q1: Clarification on the i.i.d. condition and the least observed-cost selection.

There are two points in the question.

  (i) BeaverTails-Evaluation is a subset of BeaverTails, and hence the i.i.d. assumption still holds, supported by the empirical results showing that the held-out risk always falls below β\beta (as the reviewer noted in point 1.).

  (ii) There is no inconsistency between eq. (11) and line 66 in the introduction. The least observed cost is under the refusal requirements as we note in line 66

Under a fixed risk requirement, they can select the lowest observed-cost certified configuration compatible with their response-quality and refusal requirements.

Hence, the selection rule chooses higher response quality first (i.e., lower refusal rate), then lowest cost (i.e., lower budget).

We hope these clarifications convince the reviewer to reconsider their evaluation. We remain open to providing any further clarifications.

AuthorsOpenReview ↗

Follow-up on our responses

Thank you again for the careful reading of our paper and for the respectful and precise questions. We appreciate that the reviewer found Proposition 1 a valid guarantee. We are also glad the reviewer noted that our calibration labels abstained outputs rather than assuming that refusals are safe. Finally, the reviewer recognised that overlapping grading with generation halves the lag and nearly halves the time to first token compared with CARE.

The reviewer judged our soundness concerns to be minor or fixable, and the revision fixes them:

  • Figure 3 now shows the level β\beta that each deployed pair was certified at on the calibration set, before any test prompt is seen, and it is now in the main text.
  • The experimental details are now consistent throughout: the harm labeller, the quality reference, the calibrated budget and the feedback passed to the generator.
  • We corrected the rounded certified level in the appendix and clarified the i.i.d. condition and the selection rule.

Your remarks made the paper more precise and easier to verify, and we thank you for them. We would be grateful if you could let us know whether these changes settle your concerns, or which points you would like us to clarify further.

Reviewer_Kg7rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Watch and Correct (WaC), an inference-time safety alignment framework for LLMs. The main idea is to allow the generator to continue producing text while a separate grader checks previously generated sentences in parallel. When a sentence is flagged as unsafe, the system rolls back and regenerates it using feedback from the grader, subject to a response-wide repair budget. If the budget is exhausted, the system abstains from further generation.

The paper also uses a learn-then-test framework to jointly calibrate the safety threshold and repair budget, providing finite-sample, distribution-free guarantees on the final harmful response rate under i.i.d. assumptions. To reduce latency and redundant computation, WaC combines asynchronous verification with KV-cache reuse.

The authors evaluate WaC on BeaverTails and AdvBench using different generator and grader models, comparing it against CARE and contrastive decoding. The experiments report lower harmful response rates in several settings and substantial improvements in streaming latency, including approximately halved token lag.

Critical strengths
  1. Practical and well-motivated system design. I like the idea of allowing the generator to continue producing text while the grader verifies previous sentences, releasing only approved content to the user. This avoids unnecessary pauses without exposing unverified outputs. The combination of rollback, feedback-guided regeneration, KV-cache reuse, and abstention when the repair budget is exhausted is also practically appealing. I think this is a sensible approach to balancing safety and responsiveness in interactive LLM applications.

  2. Promising empirical results across several relevant dimensions. I appreciate that the paper evaluates not only harmful response rates but also response quality and latency, using different generators and graders. In particular, the large reductions in harmful response rates for the uncensored generator and the approximately halved token lag are encouraging. The attempt to calibrate repair budgets under explicit safety requirements also makes the framework useful for thinking about practical deployment decisions, instead of evaluating safety in isolation.

Critical weaknesses
  1. Limited applicability of the safety guarantees under realistic deployment settings. The finite-sample safety guarantee relies on i.i.d. calibration and test prompts. I am concerned about its applicability to real-world LLM safety. In particular, users may repeatedly modify their prompts based on previous responses or refusals to find ways around the guardrail. These adaptive interactions violate the i.i.d. assumption, and the calibrated harm rate may no longer be reliable. This seems particularly relevant given that the paper aims to provide safety guarantees for practical deployment. I would like to better understand how robust WaC is to adaptive prompting or distribution shifts.

  2. Insufficient mechanism analysis to explain the safety and efficiency improvements. The empirical results are promising, but several mechanisms are changed simultaneously, making it difficult to understand what drives the improvements. In particular, WaC abstains when the repair budget is exhausted, whereas CARE/CD continue generating. How much of the reduction in harmful response rate comes from this different stopping rule rather than a more effective correction mechanism? I also think the latency improvement deserves further analysis, since asynchronous verification may be less effective when the grader is substantially slower than the generator. The paper emphasizes computational efficiency, but lower latency does not necessarily imply lower computational cost, especially when the generator and grader run on separate GPUs. These comparisons would help establish how general and practically meaningful the reported improvements are.

Questions for authors
  1. What actually drives the improvements in safety?

WaC achieves much lower harmful response rates than CARE/CD in some settings, which is impressive. However, WaC abstains when the repair budget is exhausted, while CARE/CD continue generating. How much of the improvement comes from better correction, and how much comes from this different stopping rule? Could the authors compare these methods under the same abstention rule and repair budget? I think this would help clarify whether WaC's correction mechanism itself is more effective.

  1. How robust are the latency improvements to different generator and grader speeds?

I like the idea of asynchronous verification, but its efficiency seems to depend on the grader being sufficiently fast. What happens if the grader is much slower than the generator, especially since the generator can only run three sentences ahead? Would WaC still provide substantial latency improvements? A sensitivity analysis varying their relative speeds, ideally also reporting computational cost, would be helpful in understanding when the approach is most practical.

  1. How meaningful are the safety guarantees under adaptive prompting?

The theoretical guarantee relies on i.i.d. calibration and test prompts. But in real-world LLM safety settings, users may repeatedly modify their prompts based on previous responses or refusals to find ways around the grader. How should we interpret the safety guarantee in such settings? Have the authors considered evaluating WaC under adaptive jailbreak attempts or distribution shifts? I think this is important given the paper's emphasis on safety guarantees for deployment.

  1. How does the proposed configuration selection relate to computational cost?

The Introduction emphasizes selecting the lowest-cost certified configuration, but Algorithm 2 (Eq. 11) selects the configuration with the lowest empirical abstention rate. Why is abstention rate used as the selection objective instead of computational cost? These seem like different objectives, since reducing abstention may require more regeneration and computation. Could the authors clarify how the framework supports the claimed cost-efficient configuration selection?

5 comments

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. We also appreciate the reviewer's recognition of the practical motivation behind WaC and its promising safety and latency improvements. We address the concerns below.

Q1: Discussion on the contribution of feedback-guided correction to safety improvements.

Thank you for raising this important question. We agree that pinpointing the contribution of successful correction compared to the baselines helps better understand the effectiveness of WaC.

To investigate this, we provide an additional analysis in Table 3 (Appendix F), which reports the successfully addressed harmful outputs corrected through feedback-guided regeneration within the allocated repair budget, of the prompts that never abstained.

Interestingly, we observe that mostly WaC corrects substantially more outputs within the available budget in that set of prompts that does not abstain. The successful correction rate is 2.1 times higher than that of CARE/CD on average over the evaluated settings. These results highlight that WaC's safety improvements are not solely attributable to its abstention mechanism; feedback-guided correction also plays an important role, and outperforms the correction of CARE and CD. It should also be noted that making use of abstention after calibrating using Algorithm 2 of the paper is our contribution and is not used in prior work.

The % of responses deemed safe (↑) for WaC, CARE and CD on the same prompts: those with a harmful unguarded answer on which the generator does not abstain.

BenchmarkGeneratorJudgeWaCCARECD
BeaverTails-Eval.UncensoredHarmBench58.558.567.967.975.575.5
BeaverTails-Eval.UncensoredLlama-3.1-8B35.635.624.424.424.424.4
BeaverTails-Eval.UncensoredMistral-7B75.475.437.737.776.876.8
AdvBenchUncensoredHarmBench88.288.282.482.485.385.3
AdvBenchUncensoredLlama-3.1-8B70.770.713.313.316.016.0
AdvBenchUncensoredMistral-7B100.0100.024.424.473.373.3

The results of Table 3 (Appendix F) complements our existing results in Tables 1 and 2, where WaC generally achieves substantially lower harmful-response rates on the uncensored generator while operating with smaller repair budgets than CARE/CD. Together, these findings provide further insight into the effectiveness of WaC's correction mechanism.

Q2: Discussion on the relative speeds of the generator and grader.

Thank you for raising this practical consideration. While we agree that the efficiency gains depend on the verifier's speed, we argue that assuming a verifier much larger than the generator is impractical in real-world deployments. In practice, safety verification is intended to prevent a high-quality (and hence larger) generator from producing harmful content. It is therefore reasonable to assume that the verifier is practically smaller than the generator. Otherwise, one would simply use the verifier as the generator and vice versa.

Moreover, model size alone does not determine whether asynchronous verification pays off; what matters is the time of a verification relative to the time of generating a sentence. The generator produces each sentence through many autoregressive decoding steps, whereas the safety decision is read from the verifier's first verdict-token logits, and with a cached context the verifier only encodes the newly completed sentence. Verification can therefore keep pace even when the verifier is comparable in size to, or larger than, the generator, as in our latency experiments, where the 13B HarmBench judge guards the 7B generator. Should verification fall behind, the generator pauses at the look-ahead limit, so that only approved content is ever released. We have added this discussion in Remark 2 to clarify this practical consideration.

Q3: Discussion on the i.i.d. assumption and robustness under distribution shifts and adaptive prompting.

Thank you for raising this important question. We agree that understanding the behavior of safety calibration under distribution shifts and adaptive prompting is valuable for practical deployment.

While our Proposition 1’s finite-sample guarantee has not been formally extended to arbitrary distribution shifts or adaptive attacks, we conduct two additional experiments to investigate WaC's empirical robustness in these settings.

First, robustness under distribution shifts. We evaluate WaC against two paraphrasing-based attacks on BeaverTails-Evaluation:

  1. Explicit paraphrasing: The original harmful request is reworded while preserving its explicit harmful intent.

  2. Implicit paraphrasing: The request is reformulated to describe the harmful intent indirectly, without explicitly naming the harmful action.

Harmful response rate (%) of the unguarded generator (naive) and of WaC (configuration of Table 1) on BeaverTails-Evaluation and on its explicit and implicit rewordings by DeepSeek-V4-Pro; β\beta is the certified level. Labels by hh against the original request.

GeneratorJudgeβ\betaOriginal — NaiveOriginal — WaCExplicit — NaiveExplicit — WaCImplicit — NaiveImplicit — WaC
Qwen2.5-7BHarmBench0.060.065.35.32.72.79.19.13.03.010.410.44.14.1
Qwen2.5-7BMistral-7B0.030.035.95.91.71.79.19.11.91.910.410.42.62.6
Qwen2.5-7BLlama-3.1-8B0.090.096.06.04.64.69.19.15.35.310.410.47.17.1
UncensoredHarmBench0.10.144.444.46.16.147.647.67.67.645.745.78.48.4
UncensoredMistral-7B0.060.0644.044.05.05.047.647.67.07.045.745.78.48.4
UncensoredLlama-3.1-8B0.450.4544.444.412.912.947.647.614.014.045.745.716.616.6

As shown in the new Table 4 (Appendix F), the observed harmful-response rates remain generally within the prescribed risk levels under both paraphrasing strategies. These results provide additional empirical evidence that WaC can maintain effective harm control under the evaluated prompt distribution shifts, even though such shifts are not covered by the formal i.i.d. guarantee.

Second, robustness under adaptive prompting. We conduct an additional experiment in which DeepSeek-V4.1-Flash acts as an adaptive user, modifying its prompts in response to previous refusals to elicit harmful completions. The results in Table 5 (Appendix F) show that WaC continues to maintain harmful-response rates within the target risk level in this evaluated setting. We also provide representative examples of the adapted prompts in Table 6, illustrating how the attacker modifies its requests across interactions.

Adaptive prompting on 100 random BeaverTails-Evaluation prompts. DeepSeek-V4.1-Flash rewrites refused requests up to 55 times, each in a new WaC conversation. Harm (%) before the attack and of the final answer (WaC failed), where the attacker is satisfied or gives up. Rewritten: share of prompts rewritten at least once; Rewrites: mean number of rewrites over all prompts (0 when the first answer satisfies the attacker), and Per rewritten: the average rewrites over the rewritten prompts only.

GeneratorJudgeβ\betaBefore (%)Rewritten (%)RewritesPer rewrittenWaC failed (%)
Qwen2.5-7BHarmBench0.060.065.05.064643.013.014.704.704.04.0
UncensoredHarmBench0.10.18.08.050502.352.354.704.709.09.0

These findings are encouraging, as they demonstrate empirical robustness beyond the original calibration setting, while preserving the distinction between observed performance and formal statistical guarantees. A more extensive investigation of adaptive jailbreak strategies and theoretical guarantees under non-i.i.d. prompting is an interesting direction for future work.

Reviewer_ogZhOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

na

Critical strengths

na

Critical weaknesses

na

Questions for authors

na

1 comment

Dear Reviewer ogZh,

Thank you for your time and effort. We appreciate your positive evaluation of our work and hope that the overall discussion in the rebuttal has improved your confidence in our manuscript.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Key Updates

Dear Area Chair,

Thank you for the time and effort you have put into overseeing the review of our submission. We are grateful to all reviewers for their constructive feedback and for recognising the main strengths of our work: a "practical and well-motivated system design" that "avoids unnecessary pauses without exposing unverified outputs" (Kg7r), addressing "a practically important problem by jointly considering safety guarantees, repair budgets, and inference latency" (vg3d), with a valid Learn-Then-Test guarantee (WkCP). All three reviewers rated their soundness concerns as "minor or fixable".

During the rebuttal period, we addressed all raised concerns and updated the manuscript accordingly (changes highlighted in blue). We are confident that our responses and revisions answer every question the reviewers raised, and they have made the paper meaningfully better. Our main updates are:

  1. Do the safety gains come from correction or only from abstention? (Kg7r, vg3d). To isolate correction, we looked only at prompts whose unguarded answer is harmful and on which the generator does not abstain, so abstention cannot explain the result. There, WaC successfully corrects 2.1 times more outputs than CARE and CD on average (Table 3). Abstention itself is part of our contribution, since the calibration (Algorithm 2) selects when to abstain so that the harm rate stays below the target level.

  2. Does the guarantee hold beyond the i.i.d. setting? (Kg7r, WkCP). We empirically show that WaC is robust to paraphrasing and adaptive prompt attacks: under explicit and implicit paraphrasing, without recalibration, WaC stays below the certified level in most settings while the unguarded harm rate nearly doubles (Table 4). Against an adaptive attacker that rewrites each refused request up to five times, the final harm rate stays at its level before the attack and below the certified level (4% against β=0.06\beta = 0.06 and 9% against β=0.10\beta = 0.10; Table 5).

  3. How the certified level is obtained, and the experimental setup (WkCP, vg3d). Figure 3, now in the main text, shows the level certified on the calibration set before any test prompt is seen, and every WaC harm rate falls below it. To make the setup easier to verify and understand, all experimental details are now stated together in the main text, matching the appendix.

  4. What each component contributes, and what it costs (vg3d, Kg7r). New ablations (Section 3.3) clearly justify the complementary roles of the proposed design: without streaming, the lag grows from 27 to 311 tokens, and without key-value cache reuse, the compute grows about fourfold. Running verification in parallel keeps the guarantee intact, since it leaves the response distribution unchanged (Remark 1), and it keeps pace even with a larger verifier, as our 13B judge does with a 7B generator, because what matters is how long a verification takes relative to generating a sentence (Remark 2). As for the cost, Proposition 1 holds for every certified configuration at once, so the selection rule can safely take the one with the fewest refusals and then the smallest repair budget, i.e., the "lowest observed cost that meets the response-quality requirement" stated in the Introduction.

We are grateful that Reviewer vg3d engaged with our rebuttal and noted that "the new ablations and clarifications address several of my concerns regarding the individual components and experimental settings". The points they kept open, correction versus abstention and cost-aware selection, were already answered in our rebuttal (items 1 and 4 above), so in our follow-up reply we restated those answers and addressed novelty.

We believe these revisions resolve the concerns raised and make our contributions clear. We thank you again for your time and consideration.

Kind regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.