- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Prompt repair can help recover benign requests blocked by safety mechanisms but does not guarantee robustness to similar edits or ensure the safety of generated responses. This paper introduces SCORE (Smoothed Certified Release for Prompt Repair), a framework that selectively repairs, certifies, or abstains, distinguishing between center certification, robustness, and safety of responses. It utilizes likelihood-ratio transfer envelopes to maximize outcomes over small candidate sets. During testing on 606 unsafe prompts, only a few could be certified, with no repaired prompts certified at a Hamming radius of one. Although a policy for releasing benign prompts is noted, it fails to meet recovery and safety targets consistently. Analyses indicate limited neighborhood coverage, suggesting the challenges of smoothing are specific rather than universal. SCORE serves as an audit mechanism for input-side certification rather than a comprehensive safety solution.
Critical strengths
- The theoretical contributions are strong.
- The empirical evaluation is extensive.
Critical weaknesses
- There is no runtime or scalability evidence.
- The central empirical object could be an artifact of a strange design choice.
Questions for authors
-
I am curious about the certification time. It is important in real-world deployment. This is not reported in the experimental evaluation.
-
Can this approach also be applied in VLM?
2 comments
Thank You and Follow-up on Our Responses
Thank you for your time and thoughtful review. We hope our responses and revisions have satisfactorily addressed your concerns. We remain happy to clarify any outstanding points.
Reviewer bfSL: Runtime, design dependence, and VLM applicability
Thank you for recognizing the theory and evaluation breadth. Your questions prompted a pipeline runtime profile and a controlled comparison of repair objectives, directly testing the paper's practical significance.
W1/Q1. What is the certification cost?
For six development inputs with repair candidates, median center-release pipeline latency is 536.46 seconds (8.94 minutes). Gate inference accounts for a median 99.02% of pipeline time. Appendix L.4 gives the profile, and Section 7 discusses its implications.
We use one A100 80 GB, warm fp16 gates, batch size 32, and 10,000 samples per gate. We select 12 development inputs by length quantiles without conditioning on repair success and measure each three times. Six searches find candidates. The other six terminate at a median of 1.40 seconds. Only one input passes the full release rule in all repeats. The reported latency therefore describes processing found candidates, with no claim about deployment traffic.
The profile includes initial gating, beam search, support construction, sampling, both center certificates, and automated fidelity filtering. It excludes initialization, optional Hamming extension, and downstream generation. Both gates use fp16 in this diagnostic, whereas the confirmatory evaluation uses 4-bit WildGuard.
Table 4 separately measures the exact transfer envelope: 0.3312 seconds for a 76-token input at radius two. For uniform supports, radius-two outer enumeration has O(n²) cases independent of support size. Gate sampling remains a separate computational cost.
A paired experiment on three repaired inputs places the gates on separate GPUs. Median inference latency falls from 502.77 to 321.31 seconds, with a median paired speedup of 1.576×. Sample hashes, ordered predictions, unsafe counts, and confidence bounds agree. Additional hardware reduces latency without reducing total computation. Gate inference is the main bottleneck. Production latency and broader kernel/radius scaling remain unmeasured.
W2. Could the coverage gap depend on the repair design?
Yes. We test that dependence directly in Appendix L.1. The matched arms use identical fixed supports, replacement distributions, q = 0.4, a three-edit limit, automated fidelity criteria, and screening budgets. Both prioritize fidelity-valid, hard-accepted candidates. One then minimizes edits, while the other first minimizes the screening unsafe-probability upper bound. Each selected candidate receives 10,000 fresh certification samples.
We select 24 development inputs by length quantiles among historical fp16 rejects. The 4-bit gate rejects 21: 10 benign and 11 unsafe-source. All 21 remain in the route denominators.
The unsafe positives at the two radii are the same input. This comparison shows that optimizing the repair objective can change certificate coverage. It also increases unsafe-source releases, which must be distinguished from harmful generated outputs. The bounded development experiment supports objective dependence without establishing statistical significance or held-out effectiveness of an improved beam policy.
Zero coverage is specific to the evaluated policies and kernels. The development kernel, semantic-support, and objective studies yield sparse nonzero coverage. Together, these results identify factors affecting the center-to-neighborhood gap without implying a universal impossibility for smoothing.
Q2. Does the construction apply to VLMs?
Thank you for highlighting this valuable direction. With the image fixed, our construction certifies text perturbations of a multimodal gate under the same fixed-support assumptions, conditional on that image. Joint image/text robustness requires a joint threat model and transfer construction. Appendix L.5 describes this extension. Our experiments cover text-only gates. In future work, we will explore VLM certification through fixed-image text perturbations and then investigate joint image/text robustness.
Final manuscript commitment. We will make the measured pipeline cost prominent in Section 7 and distinguish it from transfer-envelope timing. We will retain the stage profile and paired GPU results in Appendix L.4, connect the objective comparison to the scope of the coverage finding, and state the fixed-image assumption alongside the VLM extension.