- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the sampling bottleneck of randomized smoothing for certifying LLM safety against token-level jailbreak perturbations. It proposes drawing a shared per-sample corruption rate from a Beta distribution, derives certificates for masking, uniform substitution, and deletion, and proves improved polynomial rather than exponential sampling dependence in relevant regimes. Experiments on harmful-content detection compare against fixed-rate smoothing across several LLM detectors and attack scopes.
Critical strengths
The certificate derivations appear internally consistent. In particular, the masking result follows directly from the shared-rate joint-masking probability , and the comparison between and Q∼Beta(a,1) is correctly established in Theorems 2–3. My concern is instead with the empirical claim that the method retains a favorable benign-utility trade-off, which is not evaluated at the decision level of the certified smoothed detector.
The paper identifies and resolves a concrete statistical limitation of fixed-rate smoothing. Theorems 2 and 3 show, at matched evidence (S,N) and expected corruption rate, that fixed-rate masking has a radius scaling logarithmically in the effective confidence quantity, whereas the Beta-mixture construction produces a linear dependence for . This is a clean and consequential observation, rather than merely another corruption heuristic.
The paper develops certificates beyond the simple masking case. Theorem 1 and Appendix B derive grouped fractional-knapsack certificates for uniform substitution and edit-distance deletion, while Theorem 9 gives a nontrivial sufficient sampling result for randomized uniform substitution. The experiments then test both suffix-only and full-input settings, and honestly show in Table 4 that randomized deletion can underperform CERT-ED on unrepeated full inputs.
Critical weaknesses
The claimed benign-detection trade-off is not established for the actual smoothed decision rule. The paper fixes an extremely low target-label threshold, , in Section 4.1, so a benign request is declared unsafe by the smoothed detector whenever its estimated unsafe probability exceeds this very small value. Yet Table 7 reports only the percentage of individual corrupted benign samples classified unsafe, not the fraction of benign requests for which the aggregated smoothed probability exceeds , nor the resulting false-rejection rate. The reported per-sample unsafe rates are themselves large, between 18.81% and 28.35% for masking and substitution, which makes it impossible to infer that the certified detector has acceptable benign behavior at the chosen threshold. This matters because the conclusion claims a “favorable trade-off between robustness and detection performance,” but the presented metric is not the decision metric used in Equation 1 and the certificate. The authors should report request-level smoothed false-positive rates, together with how τ was selected on held-out data, and ideally provide harmful-detection and benign-acceptance results as a threshold trade-off.
Questions for authors
Does diffusion-based purification receive the realized sampled corruption rate Q as an input or conditioning signal, or is it applied identically to samples drawn at all rates? This affects whether the observed gains can be interpreted purely as a change in smoothing geometry, rather than as an interaction between the heavy-tailed corruption distribution and purification behavior.
Thank you for the detailed response and additional experiments. The new request-level false-positive rates clarify benign utility, and the matched-FPR comparisons establish substantial certification gains in the protected-prefix setting. The gains at low false-positive rates for full-input certification remain uncertain. Overall, the new evidence addresses enough of my original concerns for me to raise my rating to 3 (Weak acceptance).