- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of balancing safety guarantees and inference efficiency in LLM generation with corrective interventions. Motivated by the computational overhead and latency introduced by repeated safety verification and regeneration, this paper proposes Watch and Correct (WaC), a framework that combines risk-calibrated repair budgets with asynchronous streaming correction. It employs a Learn-Then-Test procedure to provide finite-sample, distribution-free guarantees on final-response harm rates, while using feedback-guided repair and KV-cache reuse to reduce redundant computation. Experiments on Qwen2.5-7B and its uncensored variant demonstrate lower harmful-response rates and approximately 50% lower token lag compared with existing intervention methods.
Critical strengths
The paper addresses a practically important problem by jointly considering safety guarantees, repair budgets, and inference latency in LLM generation. The integration of risk calibration with asynchronous streaming correction provides a useful framework for deploying safety interventions under resource constraints.
Critical weaknesses
-
The conceptual novelty of the proposed framework appears somewhat limited, as several key components, including statistical risk calibration, streaming safety intervention, asynchronous verification, and KV-cache reuse, have been explored in prior studies. Although integrating these techniques into a unified framework is practically useful, the paper does not sufficiently establish what new insights or technical capabilities are enabled by their combination. More targeted ablation studies would help clarify the necessity and individual contributions of the proposed components.
-
The experimental evaluation is insufficient to fully support the proposed design. The paper lacks comprehensive ablation studies to demonstrate the necessity and effectiveness of its key components. Furthermore, there are several inconsistencies in the experimental settings, such as the harm labeler, response quality evaluation, and repair budget configurations described in the main text and appendix. These issues make it difficult to reliably assess the reported improvements.
Questions for authors
Could the authors provide more comprehensive ablation studies to clarify the individual contributions of its key components, particularly asynchronous verification, KV-cache reuse, and feedback-guided correction?
4 comments
Follow-up on our responses
We are grateful to the reviewer for the thoughtful feedback and for recognising that WaC addresses a practically important problem. The reviewer valued that it considers safety guarantees, repair budgets and latency together. The reviewer also noted that combining risk calibration with asynchronous streaming correction gives a useful framework for deploying safety interventions under resource constraints.
The reviewer saw only minor or fixable concerns about soundness, and we addressed them:
- We clarified the role of each component of WaC in a new discussion (Section 3.3), in support of our contribution.
- We reported ablations showing what each component contributes. Streaming cuts the lag from 311 to 27 tokens, and reusing the key–value cache cuts compute about fourfold.
- We resolved the inconsistencies in the experimental settings: the harm labeller, the quality evaluation and the repair budget.
These changes improved how the paper positions its contribution and how clearly it reads, thanks to your comments. We would appreciate it if you could let us know whether our responses answer your questions, or point us to anything that remains so we can address it.
I appreciate the detailed responses and additional experiments. The new ablations and clarifications address several of my concerns regarding the individual components and experimental settings. However, my main concerns remain. First, we still can't tell whether the safety gains are driven by successful corrections or just abstaining more often. Second, the formal objective doesn't actually back up the claim of cost-aware calibration. Finally, the overall approach still comes across as a fairly standard combination of known techniques. Therefore, I am inclined to maintain my original score.
We thank the reviewer for engaging with our rebuttal, and we are glad our clarifications addressed several of the concerns. We appreciate the chance to clarify the remaining points.
1. Correction vs abstention.
We thank the reviewer for this important question. We respectfully refer the reviewer to our response to Reviewer Kg7r's Q1, and summarise it here:
We provide an additional analysis in Table 3 (Appendix F), which reports the harmful outputs successfully corrected through feedback-guided regeneration within the allocated repair budget, on the prompts that never abstained. On this set of prompts, WaC corrects substantially more outputs within the available budget, with a successful correction rate 2.1 times higher than that of CARE/CD on average over the evaluated settings. These results highlight that WaC's safety improvements are not solely attributable to its abstention mechanism, as feedback-guided correction also plays an important role and outperforms the correction of CARE and CD. Moreover, making use of abstention after calibrating using Algorithm 2 of the paper keeps the harmful-response rate below the target level with high probability, and is our contribution, not used in prior work.
The % of responses deemed safe (↑) for WaC, CARE and CD on the same prompts: those with a harmful unguarded answer on which the generator does not abstain.
| Benchmark | Generator | Judge | WaC | CARE | CD |
|---|---|---|---|---|---|
| BeaverTails-Eval. | Uncensored | HarmBench | |||
| BeaverTails-Eval. | Uncensored | Llama-3.1-8B | |||
| BeaverTails-Eval. | Uncensored | Mistral-7B | |||
| AdvBench | Uncensored | HarmBench | |||
| AdvBench | Uncensored | Llama-3.1-8B | |||
| AdvBench | Uncensored | Mistral-7B |
The results of this table complements our existing results in Tables 1 and 2, where WaC generally achieves substantially lower harmful-response rates on the uncensored generator while operating with smaller repair budgets than CARE/CD. Together, these findings provide further insight into the effectiveness of WaC's correction mechanism.
2. The cost-aware calibration claim.
We thank the reviewer for pointing this out. Any configuration in the admissible set constructed by the calibration procedure (Algorithm 2) satisfies the harm-control guarantee, as Proposition 1 holds simultaneously for every pair in . As also noted in our responses to Reviewer Kg7r's Q4 and Reviewer WkCP's Q1, and as stated in the Introduction, we select "the lowest observed-cost certified configuration compatible with their response-quality and refusal requirements." Therefore, our selection rule is indeed cost-aware.
3. WaC as a standard combination of known techniques.
We appreciate this perspective and agree that WaC builds on known tools, which we see as a strength, as it keeps WaC simple to adopt and guardrail-agnostic. The novelty lies in what the combination enables, which none of these tools provides alone:
- Risk calibration → a certified repair budget. Existing intervention methods fix the number of retries by hand. WaC calibrates the threshold and the budget jointly, so a provider knows what correction effort a target risk level requires (Section 1).
- Asynchronous verification → a guarantee that survives streaming. Overlapping verification with generation would normally break a calibration guarantee. WaC's rollback rules preserve it (Remark 1, Section 3.3), while halving the lag relative to CARE.
To our knowledge, no one has combined these techniques in this particular way to achieve the results and guarantees presented in the paper. We respectfully suggest that the value of a method lies in its effectiveness and the new insights we draw from it (how it combines different perspectives) rather than its complexity.
We thank the reviewer for their time to provide a constructive discussion and hope that these comments address their raised questions.
We sincerely thank the reviewer for their comments. We also appreciate the opportunity to further clarify the individual contributions of WaC and provide additional experimental evidence supporting our design choices.
While we agree that some components of WaC have been explored individually in prior work, our contribution lies in integrating them into a unified framework for efficient safety intervention with statistical guarantees. Importantly, each component serves a distinct purpose and provides improvements to WaC, as we explain below.
(i) Statistical risk calibration The statistical risk calibration framework jointly calibrates the intervention threshold and repair budget, providing statistical guarantees for the chosen harmful-response rate using minimum budget. Without it, the method would be heuristic.
(ii) Streaming safety intervention. Instead of waiting for the entire response to be generated and then verifying its safety, WaC performs verification and correction incrementally. Since harmful content typically constitutes only a small portion of the generated response, streaming intervention allows WaC to regenerate only the harmful portions rather than the entire response, substantially reducing unnecessary computation.
Latency comparison of WaC with and without streaming.
(iii) Asynchronous verification. Unlike conventional streaming approaches, where the generator pauses until the verifier approves the preceding output, WaC allows generation and verification to proceed in parallel. Combined with the observation that harmful content generally occupies only a small portion of the response, this significantly reduces user-perceived latency, as the generator only needs to pause occasionally to correct harmful segments. To our knowledge, WaC is also the first to introduce asynchronous verification specifically for streaming safety intervention.
Latency comparison over 700 prompts. Lower is better.
(iv) KV-cache reuse. WaC reuses the cached attention states of previously approved sentences, avoiding redundant computation when regenerating rejected portions. Since rollbacks can occur at multiple locations throughout a response, preserving the approved prefix avoids repeated attention computations and substantially reduces the matrix multiplication operations required during correction.
Compute comparison of WaC with and without KV-cache reuse.
We have added this discussion in Section 3.3.
Ablations are addressed in our response to comment 1 above. For the seeming inconsistencies, we address them as follows.
(i) The harm labeller is GPT-6-Luna everywhere, which uses the classification prompt of HarmBench. Hence, HarmBench in Appendix D.2 refers to the prompt not the model.
(ii) The unguarded generator for the quality reference is GPT-4o-mini. We understand that the unguarded generator used in the experiments as a reference and GPT-4o-mini in the quality evaluations were probably conflated. We clarify this in the updated version.
(iii) The budget is calibrated using LTT under a grid of candidate budgets. Hence, there is no fixed budget used without a calibration step. The T=5 is a typo that we fix, and we have fixed related typos in the updated paper.
We hope this answers the reviewer's questions. We hope they would reconsider their evaluation in light of our answers, and we meanwhile remain open to providing any further required clarifications.