acceptodds
Under review as a conference paper at ICLR 2027

SCORE: Smoothed Certified Release and the Center-to-Neighborhood Gap in Prompt Repair

Abstract

Prompt repair can recover benign requests rejected by an input safety gate, but accepting a repaired prompt does not establish robustness to nearby edits or safety of the generated response. We introduce SCORE (Smoothed Certified Release for Prompt Repair), a selective framework that repairs, certifies, or abstains while explicitly separating center certification, Hamming-neighborhood robustness, and downstream response safety. For fixed, positive categorical smoothing supports, we specialize a likelihood-ratio transfer envelope and maximize it exactly over small candidate-set Hamming balls. We also identify when exact transfer improves a product bound and when finite sampling prevents certification at a given budget. Among 606 held-out unsafe prompts routed to repair, branch-and-bound and beam proposal policies certify 215 and 66 centers, respectively, but neither certifies a repaired prompt at Hamming radius one. A separate frozen dual-gate policy releases 34 of 313 rejected benign prompts after automated fidelity filtering, yet certifies none of these routed prompts at radius one and misses its predeclared joint recovery, risk, and fidelity targets. Development kernel and semantic-support analyses find sparse nonzero neighborhood coverage, so the observed collapse is not a universal claim about smoothing. Finally, two direct Gemma 3 adaptive runs yield 13/224 and 9/224 fresh input-certified prompts with outputs judged harmful by both automated judge families. SCORE is therefore a route-aware audit of what input-side certification does and does not guarantee, not an end-to-end safety certificate.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
certified robustness, randomized smoothing, prompt repair, language model safety, selective prediction, adversarial robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bfSLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Prompt repair can help recover benign requests blocked by safety mechanisms but does not guarantee robustness to similar edits or ensure the safety of generated responses. This paper introduces SCORE (Smoothed Certified Release for Prompt Repair), a framework that selectively repairs, certifies, or abstains, distinguishing between center certification, robustness, and safety of responses. It utilizes likelihood-ratio transfer envelopes to maximize outcomes over small candidate sets. During testing on 606 unsafe prompts, only a few could be certified, with no repaired prompts certified at a Hamming radius of one. Although a policy for releasing benign prompts is noted, it fails to meet recovery and safety targets consistently. Analyses indicate limited neighborhood coverage, suggesting the challenges of smoothing are specific rather than universal. SCORE serves as an audit mechanism for input-side certification rather than a comprehensive safety solution.

Critical strengths
  • The theoretical contributions are strong.
  • The empirical evaluation is extensive.
Critical weaknesses
  • There is no runtime or scalability evidence.
  • The central empirical object could be an artifact of a strange design choice.
Questions for authors
  1. I am curious about the certification time. It is important in real-world deployment. This is not reported in the experimental evaluation.

  2. Can this approach also be applied in VLM?

2 comments
AuthorsOpenReview ↗

Reviewer bfSL: Runtime, design dependence, and VLM applicability

Thank you for recognizing the theory and evaluation breadth. Your questions prompted a pipeline runtime profile and a controlled comparison of repair objectives, directly testing the paper's practical significance.

W1/Q1. What is the certification cost?

For six development inputs with repair candidates, median center-release pipeline latency is 536.46 seconds (8.94 minutes). Gate inference accounts for a median 99.02% of pipeline time. Appendix L.4 gives the profile, and Section 7 discusses its implications.

We use one A100 80 GB, warm fp16 gates, batch size 32, and 10,000 samples per gate. We select 12 development inputs by length quantiles without conditioning on repair success and measure each three times. Six searches find candidates. The other six terminate at a median of 1.40 seconds. Only one input passes the full release rule in all repeats. The reported latency therefore describes processing found candidates, with no claim about deployment traffic.

The profile includes initial gating, beam search, support construction, sampling, both center certificates, and automated fidelity filtering. It excludes initialization, optional Hamming extension, and downstream generation. Both gates use fp16 in this diagnostic, whereas the confirmatory evaluation uses 4-bit WildGuard.

Table 4 separately measures the exact transfer envelope: 0.3312 seconds for a 76-token input at radius two. For uniform supports, radius-two outer enumeration has O(n²) cases independent of support size. Gate sampling remains a separate computational cost.

A paired experiment on three repaired inputs places the gates on separate GPUs. Median inference latency falls from 502.77 to 321.31 seconds, with a median paired speedup of 1.576×. Sample hashes, ordered predictions, unsafe counts, and confidence bounds agree. Additional hardware reduces latency without reducing total computation. Gate inference is the main bottleneck. Production latency and broader kernel/radius scaling remain unmeasured.

W2. Could the coverage gap depend on the repair design?

Yes. We test that dependence directly in Appendix L.1. The matched arms use identical fixed supports, replacement distributions, q = 0.4, a three-edit limit, automated fidelity criteria, and screening budgets. Both prioritize fidelity-valid, hard-accepted candidates. One then minimizes edits, while the other first minimizes the screening unsafe-probability upper bound. Each selected candidate receives 10,000 fresh certification samples.

We select 24 development inputs by length quantiles among historical fp16 rejects. The 4-bit gate rejects 21: 10 benign and 11 unsafe-source. All 21 remain in the route denominators.

Source labelCenter: hard → margin-awareRadius oneRadius two
Benign6/10 → 9/100/10 → 1/100/10 → 0/10
Unsafe7/11 → 9/110/11 → 1/110/11 → 1/11

The unsafe positives at the two radii are the same input. This comparison shows that optimizing the repair objective can change certificate coverage. It also increases unsafe-source releases, which must be distinguished from harmful generated outputs. The bounded development experiment supports objective dependence without establishing statistical significance or held-out effectiveness of an improved beam policy.

Zero coverage is specific to the evaluated policies and kernels. The development kernel, semantic-support, and objective studies yield sparse nonzero coverage. Together, these results identify factors affecting the center-to-neighborhood gap without implying a universal impossibility for smoothing.

Q2. Does the construction apply to VLMs?

Thank you for highlighting this valuable direction. With the image fixed, our construction certifies text perturbations of a multimodal gate under the same fixed-support assumptions, conditional on that image. Joint image/text robustness requires a joint threat model and transfer construction. Appendix L.5 describes this extension. Our experiments cover text-only gates. In future work, we will explore VLM certification through fixed-image text perturbations and then investigate joint image/text robustness.

Final manuscript commitment. We will make the measured pipeline cost prominent in Section 7 and distinguish it from transfer-envelope timing. We will retain the stage profile and paired GPU results in Appendix L.4, connect the objective comparison to the scope of the coverage finding, and state the fixed-image assumption alongside the VLM extension.

AuthorsOpenReview ↗

Thank You and Follow-up on Our Responses

Thank you for your time and thoughtful review. We hope our responses and revisions have satisfactorily addressed your concerns. We remain happy to clarify any outstanding points.

Reviewer_xXZTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCORE, a randomized smoothing-based framework for certifying prompt repair in large language models. It introduces a likelihood-ratio-based method to certify the robustness of repaired prompts against token substitutions within a Hamming neighborhood. Extensive experiments show that while many repaired prompts can obtain center-level certification, almost none achieve neighborhood robustness. Furthermore, adaptive attacks demonstrate that input-side certification does not guarantee the safety of downstream LLM-generated responses.

Critical strengths
  1. The paper carefully distinguishes different levels of safety certification and highlights an important gap between input-side robustness and downstream generation safety.

  2. The evaluation covers multiple benchmarks, safety gates, downstream models, and adaptive attacks, providing a comprehensive assessment of the proposed framework.

Critical weaknesses
  1. The theoretical analysis primarily builds upon established randomized smoothing and Neyman–Pearson likelihood-ratio techniques. Although the paper derives an exact transfer envelope for Hamming neighborhoods, the underlying theoretical principles are not fundamentally new. The main contribution appears to be adapting existing certification techniques to the prompt repair setting, rather than introducing a substantially new certification methodology.

  2. The proposed certification framework only guarantees the stability of the smoothed input safety classifier under a restricted set of token substitutions. It does not guarantee that the original safety classifier will accept all neighboring prompts or that the downstream LLM will generate safe responses. Moreover, the certification is restricted to small Hamming radii and predefined token candidate sets, which may not adequately represent realistic semantic or adaptive adversarial attacks.

Questions for authors
  1. Could the authors further clarify the key theoretical differences between the proposed likelihood-ratio transfer envelope and existing randomized smoothing certification methods?

  2. The proposed framework achieves zero neighborhood certification on the repair-routed test prompts. Have the authors explored whether incorporating the certification margin directly into the prompt repair optimization could improve robustness, rather than performing repair and certification separately?

  3. Since the experiments demonstrate that input-side certification does not guarantee safe LLM responses, could the proposed framework be extended to incorporate output-level safety certification? What would be the main theoretical or computational challenges?

4 comments
AuthorsOpenReview ↗

Reviewer xXZT: Part 1/2: Contribution and significance

Thank you for your careful assessment of the theory and its scope. Your suggestion to incorporate certification margins into repair provides a particularly useful test of the mechanism behind the coverage gap. We also appreciate your recognition of the evaluation's breadth and the importance of separating input certification from output safety.

SCORE addresses a concrete question: how much of a prompt-repair system's center-certified coverage survives when the guarantee must hold throughout a declared neighborhood? Our contribution combines an exact repair-neighborhood computation, a characterization of its certification thresholds, and a controlled empirical diagnosis of the coverage gap. The additional experiment in Part 2 directly tests your proposal to optimize certification margins during repair.

W1/Q1. What does SCORE add to existing smoothing theory?

The statistical foundation is established Neyman-Pearson likelihood-ratio ordering. SCORE specializes it to fixed, position-specific repair supports and determines the worst transfer over the entire declared Hamming ball. Section 4 and Appendix B describe the construction, and Appendix G compares its scope with existing certification methods.

The contribution has three concrete components:

  1. Exact neighborhood computation. Each changed position reduces to three categories: the original token, the replacement token, and the remaining support. Convolution combines their probability masses. Maximizing over positions and distinct replacement masses then yields the exact transfer envelope over the candidate-set ball, without enumerating complete token sequences. The supports and replacement distributions stay fixed across compared centers. For uniform supports, the radius-two outer search has O(n²) cases, independent of vocabulary-support size.
  2. A boundary that explains when exact transfer matters. Corollary 4.3 identifies when the exact envelope and the product-maximum bound give identical binary certification decisions, and when exact transfer can certify additional cases. For q = 0.4 and K = 32, the required unsafe-probability upper bounds are:
CertificateRequired upper bound
Center< 50%
Radius one, exact or product< 1.0204%
Radius two, exact< 0.2704%
Radius two, product< 0.02082%

Thus a center certificate can be far from a neighborhood certificate. At radius two, the audit contains two certified method-prompt rows under exact transfer and zero under the product bound (Table 13). These rows belong to the full-pool audit. The repair-route counts remain zero.

  1. A separation of sampling feasibility from achieved margin. With N = 10,000 and alpha = 0.001, the smallest Clopper-Pearson upper bound is 0.06905%. Exact certification is therefore feasible at both radii, while radius-two product certification is impossible at this budget even with zero unsafe samples (Proposition 4.4). Table 14 then isolates the empirical bottleneck: replacing confidence bounds with observed unsafe frequencies leaves 280/281 radius-one failures and all 281 radius-two failures uncertified among center-certified repair-route rows. This diagnostic points to insufficient observed margin, rather than sampling uncertainty, as the dominant explanation in these rows.

Why this matters. The resulting analysis tells us when a tighter bound can change a release decision, when more samples are necessary, and when the observed repair margin itself is inadequate. The held-out evaluation quantifies the consequence: 215 and 66 center certificates among 606 routed inputs, but zero radius-one certificates for either policy. A separate frozen dual-gate policy recovers 34/313 rejected benign prompts, certifies none at radius one, and misses all three predefined utility-risk-fidelity criteria (Sections 6.1 and 6.3). The development kernel and objective studies identify settings with nonzero neighborhood coverage, establishing the scope of these findings.

The envelope is tight over randomized tests and valid for deterministic gate events, whose indivisible probability atoms may prevent attainment. A failed certificate leaves robustness unresolved because failure of the bound alone cannot establish an adversarial label change. These distinctions are explicit in Section 4.

Final manuscript commitment. We will add the numerical threshold table above to Section 4 and explain how to use it to distinguish sampling limits from insufficient observed margins. We will sharpen the contribution statement around exact repair-neighborhood maximization and its empirical implications, with explicit attribution of the established Neyman-Pearson foundation.

AuthorsOpenReview ↗

Reviewer xXZT: Part 2/2: Repair objectives and guarantee scope

Q2. Can optimizing certification margins improve coverage?

Yes, in the matched development comparison. Margin-aware search produces two radius-one certified releases among 21 routed inputs, compared with zero under hard-acceptance ranking (Appendix L.1, Table 30).

Both arms start from the original input and use identical fixed supports and replacement weights, uniform supports capped at 32, q = 0.4, at most three substitutions, and the same automated fidelity filter. Both prioritize fidelity-valid, hard-accepted candidates. The baseline then minimizes edit count, while the margin-aware arm first minimizes the screening upper bound on unsafe probability. Under this fixed uniform kernel, that bound is a monotone proxy for the transfer certificate.

Each arm evaluates eight candidates in each of three rounds, using 128 screening samples per candidate. Each selected candidate receives a separate final certificate from 10,000 fresh samples at alpha = 0.001. We select 24 development inputs by length quantiles among historical fp16 gate rejects, without conditioning on repair success. The 4-bit gate rejects 21 inputs. We retain all 21 in the denominators, including failed searches and release checks.

Source labelObjectiveCenterRadius oneRadius two
BenignHard acceptance6/100/100/10
BenignMargin-aware9/101/100/10
UnsafeHard acceptance7/110/110/11
UnsafeMargin-aware9/111/111/11

The unsafe-source positives at radii one and two are the same input. The comparison shows that the repair objective can change neighborhood coverage under matched budgets. It also increases center releases in both source groups, so better certification margins must be evaluated alongside utility and risk. Unsafe-source labels alone do not establish harmful generated outputs.

This controlled development experiment shows that the objective affects coverage in the evaluated sample. Its 21 inputs and bounded local search do not establish statistical significance, held-out effectiveness, or the performance of a margin-aware version of the historical beam policy. Section 7 uses it to motivate certification-aware repair while retaining the original policy-specific conclusions.

W2. What does the restricted threat model establish?

SCORE certifies the smoothed input gate throughout a fixed-support Hamming neighborhood. The hard gate's labels and the downstream model's outputs are separate objects. Small radii and predefined supports make exact transfer tractable and define the scope of the guarantee. We then measure how much of the repair route receives it.

The adaptive suffix experiment examines the next interface: whether a freshly center-certified input can elicit an output judged harmful. Its suffixes lie outside the certified Hamming threat set. The experiment tests the release policy's input-to-output boundary and makes no claim to refute the neighborhood theorem. Section 3 and Table 1 distinguish these guarantees before the results.

Q3. Can the framework certify downstream outputs?

The transfer theorem applies to a bounded output event. Fix a generator GG, decoding rule, and output judge JJ, and define

h(z)=Pr⁡R[J(G(z,R))=harmful].h(z)=\Pr_R[J(G(z,R))=\mathrm{harmful}].

Because hh lies in [0,1][0,1], Theorem 4.1 transfers an upper bound on its expectation under QuQ_u to an upper bound under QvQ_v. Estimating the starting bound requires fresh independent input-noise and generation samples after candidate selection. Because every sample requires generation and judging, certifying a small harmful-output probability can be substantially more expensive than certifying an input gate.

The resulting statement concerns the specified smoothed, judge-defined pipeline. Safety at every unperturbed input and agreement with human judgments require additional assumptions. Appendix L.5 presents this mathematical extension. Implementation and empirical evaluation remain future work.

Final manuscript commitment. We will bring the matched objective comparison into the main-text discussion, retain its full protocol in Appendix L.1, and label it as development evidence. We will explain the fixed-support threat model in plain language and separate the proposed output-level extension from experimentally evaluated capabilities.

We would welcome your thoughts on whether these clarifications and the matched repair-objective experiment address your main concerns. Would further information, analysis, or a targeted experiment help you assess the contribution? If a question remains, please let us know which evidence would be most useful. We would be glad to discuss what we can provide during the discussion period.

Thanks for the rebuttal, I will raise my score.

Thank you for reconsidering your assessment and for the constructive discussion. Your questions helped us clarify the theoretical contribution, test the role of the repair objective and sharpen the scope of our guarantees. We appreciate your time and thoughtful feedback.

Reviewer_8RUeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SCORE, which takes a prompt wrongly blocked by a safety filter, edits a few tokens so the filter accepts it, and releases the edit only if a randomized-smoothing check certifies that the filter's decision is stable. It gives a tight bound for extending that certificate from the edited prompt to all prompts one or two token substitutions away. In experiments, many edited prompts are certified on their own, but none stay certified once nearby substitutions are included, and the full system misses all three of its own preset targets for recovering benign prompts. Attacks can also obtain certified prompts that still produce harmful outputs, so the authors present SCORE as an audit of what input-side certificates guarantee, not as a safety guarantee.

Critical strengths

I specifically looked for test-set exposure. The paper does far better than usual.

Route HCRR divides by every prompt sent to repair, including failed searches and abstentions.

Extremely like the negative results presented.

Critical weaknesses

The paper is dense and acronym-heavy, which made reading difficult.

No code or per-example data are released to make reproduction easy -- -- the author promised to upload code

Questions for authors

Have you checked for near-duplicate behaviors across AdvBench, JailbreakBench, and StrongREJECT between the development rows and the 629-prompt test pool, and across the several pools

-answered my question

In the adaptive study, is the target response used for the final "harmful by both" endpoint regenerated independently after selection, or is it the same response that was selected for being judged harmful?

-answered my question

How were the fidelity thresholds (0.8 / 0.8 / 0.35) chosen? -answered my question

Why use base Llama-2-7B in raw completion mode as a primary target? Would the RQ3 conclusions change if only safety-tuned chat targets were reported?

  • answered my question
2 comments
AuthorsOpenReview ↗

Reviewer 8RUe: Overlap, adaptive evaluation, and reproducibility

Thank you for examining test-set exposure and adaptive selection so carefully. Your questions sharpen the study's validity checks, and we appreciate your recognition of the route denominators and the value of reporting negative results.

Q1. Development/test overlap.

We checked normalized exact matches, lexical Jaccard similarity, and MPNet cosine similarity against eligible development texts. The unsafe pool contains 17 exact matches, of which 16 enter repair. After exclusion, branch-and-bound center coverage changes from 215/606 (35.48%) to 209/590 (35.42%), and beam coverage becomes 66/590. Both policies retain zero radius-one and radius-two certificates.

The broader exclusions give the following counts (Table 31). Every similarity row also excludes exact matches.

ExclusionRoutedBranch-and-bound centersBeam centers
Exact59020966
Jaccard ≥ 0.8057920766
Cosine ≥ 0.8041512746
Cosine ≥ 0.8547015455
Cosine ≥ 0.9052117657

Neighborhood coverage remains zero in every row.

The unified selective-recovery pool has no normalized exact development/test matches. Cosine exclusions at 0.80/0.85/0.90 yield benign recovery of 30/268, 30/295, and 31/310, and unsafe-source release of 12/568, 13/632, and 14/669. The original counts are 34/313 and 15/684. A conservative exclusion against all recorded development pools gives 28/284 benign recovery and 13/625 unsafe-source release (Appendix L.2). The tested exclusions preserve failure to meet the joint utility-risk-fidelity target.

These checks establish robustness to the tested overlap exclusions. Behavioral disjointness remains a stronger property: MPNet has a 384-token context limit, and embedding similarity is an imperfect measure of behavioral equivalence.

Q2. Is the harmful response independently regenerated?

We reuse the attack-selected response and refresh only the final input-certification samples. In the direct Gemma runs, target decoding is greedy. As specified in Appendix C.3, selection uses the primary judge's harmfulness label. The secondary judge is used only in the reported conjunction.

Each reported success is therefore a witnessed pair: a freshly certified input and its selected output judged harmful by both judge families. Fresh certification is independent of search and screening. This endpoint demonstrates that input certification can coexist with an output judged harmful. It does not estimate harmfulness under independent response regeneration. The two attack seeds share the goal population and are reported separately.

Q3. Why the 0.80/0.80/0.35 fidelity thresholds?

These are operational cutoffs: minimum cosine and bidirectional-NLI scores of 0.80, and maximum edit ratio of 0.35. We recorded them during development before the confirmatory test and applied the same frozen rule to all 2,419 test decisions. Appendix L.3 distinguishes this recorded continuity from external preregistration.

Table 33 tests six one-at-a-time alternatives plus the original setting on development data, using q = 0.1 and 1,000 samples per gate. Benign recovery stays at 7/81 throughout. Unsafe-source release stays at 2/171 except when cosine is relaxed to 0.75, giving 3/171. This supports local stability of the measured outcomes. The thresholds remain operational filters, with no claim of optimality or calibrated human intent preservation.

Q4. Does the downstream conclusion depend on base Llama 2?

The conclusion also holds for Gemma 3 12B IT with WildGuard as the input gate. Two direct adaptive runs yield 13/224 and 9/224 freshly center-certified inputs whose selected outputs are judged harmful by both judge families. Section 6.4 leads with these chat-model results. The Gemma results establish the finding on a safety-tuned chat model under the evaluated attack, while base Llama 2 illustrates how outcomes vary across targets.

Presentation, reproducibility, and final manuscript commitments.

We will summarize overlap sensitivity alongside the main coverage results, state response reuse and fresh certification together in Section 6.4, and retain Gemma as the lead adaptive result. We will explain the fidelity thresholds as operational choices and retain all sensitivity settings in Appendix L.3. To improve readability, we will reduce acronym use and define the three evaluated objects before introducing notation. Appendices B and C will retain the detailed algorithms and protocols. Executable code and per-example records are absent from the current submission. The final manuscript will state artifact availability explicitly so readers can distinguish documented procedures from independently reproducible experiments.

AuthorsOpenReview ↗

Thank You and Follow-up on Our Responses

Thank you for your time and thoughtful review. We hope our responses and revisions have satisfactorily addressed your concerns. We remain happy to clarify any outstanding points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.