acceptodds
Under review as a conference paper at ICLR 2027

Risk-Certified Prompt Optimization for LLM-as-a-Judge Guardrails

Abstract

LLM-as-a-judge guardrails are often controlled primarily through system prompts, yet standard prompt optimization methods target average performance rather than explicit safety constraints. This paper studies system-prompt selection as a constrained optimization problem: maximize benign pass rate while keeping the missed-attack rate below a user-specified level with finite-sample statistical control. The framework combines LLM-based judging, automatic prompt generation, and distribution-free risk certification. This work instantiates it in two procedures: CPS and CRISP. CPS generates a fixed pool of prompts from training data, then certifies candidates on a held-out evaluation set using simultaneous upper confidence bounds and selects the certified prompt with the highest benign utility. CRISP instead integrates risk feedback into an adaptive search, using hard-example-guided rewriting and a reusable-holdout interface to steer generation toward the feasible utility frontier. Experiments on four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure show that risk-certified selection consistently respects the target missed-attack level on held-out tests, while utility improves as the permitted risk is relaxed. CRISP is especially useful when fixed-pool search fails to produce certifiable high-utility candidates. Code is available at https://anonymous.4open.science/r/guardrails

Primary area:
optimization
Keywords:
LLM guardrails, LLM-as-a-judge, distribution-free risk control, automatic prompt optimization, conformal prediction, adaptive data analysis, reusable holdout, prompt injection, AI safety, constrained optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ga4oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how to transform a general-purpose LLM into a guardrail that balances safety and utility by optimizing the System Prompt, without requiring any fine-tuning of the LLM's parameters. This method first generates a fixed set of candidate prompts, then filters them to satisfy safety constraints using a statistical upper confidence bound, and finally selects the prompt with the highest benign utility. Subsequently, the method incorporates risk-certification feedback into an iterative prompt search process, enabling the model to proactively revise prompts that fail to meet safety requirements.

Critical strengths
  1. This paper is well organized and clearly written.

  2. The motivation of the paper is attractive.

  3. The experiments are extensive and helpful to validate the effectiveness of the model.

Critical weaknesses
  1. The components of the method are similar to existing work, please present the novelty of the proposed technique better for understanding.

  2. Risk certification holds only under the IID assumption and cannot guarantee adversarial robustness.

  3. The optimized prompt may have learned spurious patterns present in the dataset.

Questions for authors

Please refer to the above weaknesses.

2 comments

We thank the reviewer for the positive assessment of the motivation and the experiments.


"W1. The components of the method are similar to existing work, please present the novelty of the proposed technique better for understanding."

CPS combines an existing optimizer with LTT/PRC (Zollo et al., ICLR 2024). We present it to expose the limits of this direct approach; its failure mode motivates CRISP (CPS returns an empty certified set in 4 of 12 prompt-injection settings).

The contributions are:

  • Problem formulation: Hosted LLM judges often expose no score or logprobs, so the system prompt is the only control surface. Risk control becomes a search over a discrete, open-ended prompt space rather than threshold calibration.
  • Method: LTT and PRC certify a candidate family fixed before the calibration data are seen. Guess-and-Check is a validity mechanism, not a search method. CRISP is a prompt optimization process that uses risk certificates signals to steer prompt search into risk-certified regions. This helps find certified prompts even if the required risk level (α) is low. To our knowledge, this is the first LTT-style certificate where hypotheses are generated adaptively from calibration feedback.

The table below shows that at a tight risk level (α=0.10), every simpler combination of existing pieces fails to certify. Setting: α=0.10, gpt-5.4-mini judge; entries show test risk / utility. "–" means no certified prompt. We will add this to the paper.

MethodBIPIADeepset
PRC on 12 published hand-written prompts––
Risk-aware GEPA, then LTT––
CPS––
CRISP0.01 / 0.780.04 / 0.94

"W2. Risk certification holds only under the IID assumption and cannot guarantee adversarial robustness."

We agree that the certificate is stated under i.i.d. calibration. We add three things.

Formally:

Following robust validation and conformal prediction beyond exchangeability (Cauchois et al., 2024; Barber et al., 2023), suppose the deployment malicious distribution satisfies TV(π1,π1′)≤ηTV(\pi_1, \pi_1') \le η, where TV(P,Q)=sup⁡A∣P(A)−Q(A)∣TV(P, Q) = \sup_A |P(A) - Q(A)| is the total variation distance. Then ∣Rπ1′(p)−Rπ1(p)∣≤η|R_{\pi_1'}(p) - R_{\pi_1}(p)| \le η for every prompt pp, since the missed-attack loss is bounded in [0,1][0, 1]. Hence running CPS or CRISP at risk level α - η certifies α under any η-bounded shift.

If the shift is only in the label distribution (i.e., π(y)\pi(y) changes but π(x∣y)\pi(x\mid y) does not), the guarantee is unchanged, since RR and UU are class-conditional. We will add this as a corollary in App. B and state in Limitations that η must be specified.

Empirically, under distribution shift:

We took the CRISP prompts certified at α=0.20 on BIPIA and Deepset and measured their test missed-attack rate on two other prompt-injection benchmarks, without re-certification: Qualifire (Qualifire, 2025; Ivry & Nahum, 2025) and LLMail-Inject (Abdelnabi et al., 2025).

Certified on ↓ / tested on →BIPIADeepsetQualifireLLMail
BIPIA0.140.1450.0330.072
Deepset0.0370.050.0020.010

Risk stays below α=0.20 in every pair, suggesting the certified prompts are not overfit to the calibration attack distribution.

Empirically, under adaptive attacks:

Figure 3 already reports Best-of-N jailbreaking on BIPIA. We also ran two adaptive attacks, Crescendo and PAIR, each with up to 5 turns, on 30 malicious Deepset seed prompts. Judge: gpt-4o-mini; entries are attack success rates (lower is better).

Crescendo is multi-turn attack shows the guardrail the full dialogue. PAIR is iterative attack.

MethodCrescendoPAIR
MIPROv20.630.60
ProTeGi0.230.40
GEPA0.130.23
CRISP (α=0.10)0.000.13

CRISP yields the most robust prompt under both attacks. We report this as a robustness diagnostic, not a guarantee, and will keep adaptive robustness as an explicit limitation. It is worth mentioning that certifying guardrails against adaptive attacks remains an open research problem beyond restricted threat models.


"W3. The optimized prompt may have learned spurious patterns present in the dataset."

We thank the reviewer for this point. We address this concern with the following evidence:

  • The prompts follow a general rule rather than dataset-specific patterns. Each prompt's core rule is to block prompt injection, i.e., hidden instructions. Details such as truncated text or odd tables (App C) are examples showing the judge what hidden instructions look like, as in few-shot prompting.
  • No sign of overfitting. Calibration and test performance closely agree (Tables 5–6, App A.7), and all 50 independent BIPIA re-splits satisfy Rtest≤αR_{test} \le α (Table 7, App A.8).
  • Transfer to other datasets. Deepset, Qualifire and LLMail-Inject do not share BIPIA's formatting. Prompts certified on BIPIA keep risk well below α on all three (0.145, 0.033 and 0.072 at α = 0.20) and retain benign utility on Deepset (0.974).

We hope these clarifications address the reviewer's concerns.

Thank you for the follow-up. Part of the evidence for your four points was posted in our replies to the other reviewers, so we gather it here.

Novelty. LTT and PRC require the prompt family to be fixed before calibration. CRISP generates prompts from certification feedback, and Prop. 2 proves the certificate remains valid for every certified prompt, regardless of the number of rounds. To our knowledge, no prior LTT-style result covers adaptively generated hypotheses.

Contribution of each component (Deepset, gpt-5.4-mini judge, α = 0.20, T = 10):

  • Single population: one pool ranked by the penalized training score , instead of the two populations.
  • ReduceRisk only: both rewrites each round go to the lowest-risk uncertified prompt.
  • ImproveUtility only: both rewrites go to the highest-utility prompt, certified or not.
  • Random parents: both populations and branches, parents drawn uniformly.
VariantTest risk / utilityCertified / pool
CRISP0.091 / 0.98710/19
Single population0.036 / 0.8833/21
ReduceRisk only0.055 / 0.94816/21
ImproveUtility only–0/21
Random parents0.091 / 0.96110/18
  • Both objectives matter: ImproveUtility-only certifies nothing (0/21). ReduceRisk-only certifies the most candidates (16/21), but reduces utility.
  • Two-population split matters most: The single-population variant certifies only 3/21 candidates. We hypothesize penalized training risk poorly separates certifiable candidates, so the search cannot specialize.
  • Parent selection matters less: Random selection matches risk but reduces utility. Most gains come from the population split and complementary rewrite branches.

Moreover, replacing Guess-and-Check by direct reuse of the calibration set (BIPIA, α = 0.10) returns a prompt with test risk 0.113, above the target.

Fair comparisons. All methods share the same seed prompt, rewriter, judge, splits and filtered test set. We added three controls: CPS with a matched budget (N = 41), risk-aware GEPA and ProTeGi followed by LTT, and PRC on 12 published prompts. At α ≥ 0.20, budget-matched CPS matches CRISP, and we report this. At α = 0.10 (gpt-5.4-mini judge), none of these baselines certifies and CRISP does.

Realistic scenarios. Prompts certified on BIPIA and Deepset transfer without re-certification to LLMail-Inject, a public adaptive challenge in which participants attacked an LLM email assistant (missed-attack rates 0.072 and 0.010), and to Qualifire (0.033 and 0.002). We also report Best-of-N, Crescendo and PAIR attacks, a comparison with deployed classifiers (PIGuard, Prompt Guard, Model Armor), and 50 independent re-splits.

If a specific baseline or scenario is missing, we would be glad to run it.

Reviewer_gZmvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper formulates LLM guardrail system-prompt design as risk-certified prompt optimization: maximize benign utility UU subject to missed-attack rate R≤αR \leq \alpha with probability at least 1−δ1-\delta. It proposes CPS, a fixed-pool Learn-then-Test procedure with Clopper--Pearson risk bounds, and CRISP, a risk-feedback-driven rewrite loop using Guess-and-Check reusable holdout. Experiments cover prompt injection, system-prompt leakage, and sensitive-information disclosure, with α∈{0.10,0.20,0.30}\alpha \in \{0.10,0.20,0.30\} and δ=0.10\delta=0.10. CPS uses twenty candidates; CRISP uses 20 rounds and at most 41 candidates. The key validation is that CPS returns an empty certified set in 4 of 12 prompt-injection settings, whereas CRISP certifies in all 12; on 50 BIPIA re-splits, certified splits keep test risk below a threshold.

Critical strengths

1.The main strength is that the proposed CRISP addressed a concrete failure mode of fixed-pool certification. CPS returns an empty certified set in 4 of 12 prompt-injection settings, whereas CRISP certifies a prompt in all 12 by feeding risk certificates back into generation. This is consequential because it shows certification alone cannot recover from an inadequate candidate pool, and that adaptive hypothesis generation can remain statistically valid via reusable holdout.

2.The 50 BIPIA re-splits further support that every certified split keeps test risk below the target in practice. Together, these results support a narrow but real methodological contribution: extending LTT-style certification from a fixed candidate family to a sequential, feedback-driven prompt search.

Critical weaknesses

1.CPS is closely related to Prompt Risk Control (Zollo et al., ICLR 2024), which already selects prompts using rigorous risk upper bounds; no direct comparison is provided, so CPS's incremental contribution is unclear. For CRISP, the gain over CPS may be confounded by candidate generation and search budget: CPS uses N=20, while CRISP can evaluate up to 2T+1=41 candidates. No matched CPS–CRISP control isolates closed-loop feedback from differences in search strategy and budget.

2.There is also no ablation of Guess-and-Check against naive repeated use of the calibration set, so its empirical benefits and costs are not directly demonstrated.

Questions for authors

1 How does CPS empirically compare with Prompt Risk Control (Zollo et al., ICLR 2024) on the same guardrail tasks, especially in certifiability and benign utility?

2 Under an equal candidate/rewrite budget, does CRISP still outperform CPS, or is the gain mainly due to evaluating more candidates?

3 If Guess-and-Check is replaced by naive repeated evaluation on the calibration set, how do empirical risk violations and search efficiency change as adaptive evaluations increase?

1 comment

We thank the reviewer for the positive assessment. We ran the requested experiments.


"W1. Q1. How does CPS empirically compare with Prompt Risk Control (Zollo et al., ICLR 2024) on the same guardrail tasks, especially in certifiability and benign utility?"

We agree that CPS is an instance of Prompt Risk Control (PRC). PRC certifies a fixed, given prompt candidate set with LTT and a family-wise error correction, and it does not say how the candidates are produced. CPS uses the same certification step and adds one generation step: GEPA is run on bootstrap samples of DtrainD_{train}.

We present CPS to expose the limitations of this direct approach, and its failure mode is what motivates CRISP (CPS returns an empty certified set in 4 of 12 prompt-injection settings).

For comparison, we ran PRC on a pool of 12 published hand-written prompts for prompt injection from Rebuff, OpenMates, NeMo Guardrails and three papers [1–3].

PRC certifies only at α=0.30, and on BIPIA its utility is lower (0.57 vs. 0.89 for CRISP). CPS gets certification at α=0.20. Only CRISP certifies at α=0.10.

Results are reported as test risk / utility; "–" indicates that no prompt was certified.

SettingCRISPCPSPRC
BIPIA,α=0.100.01 / 0.78--
BIPIA,α=0.200.14 / 0.85--
BIPIA,α=0.300.18 / 0.890.17 / 0.860.18 / 0.57
Deepset,α=0.100.04 / 0.94--
Deepset,α=0.200.05 / 0.990.09 / 0.99-
Deepset,α=0.300.09 / 1.000.15 / 1.000.15 / 1.00

"W1. Q2. Under an equal candidate/rewrite budget, does CRISP still outperform CPS, or is the gain mainly due to evaluating more candidates?"

We reran CPS with N=41 GEPA candidates. We use gpt-5.4-mini as the judge. Results are reported as test risk / utility; "–" indicates that no prompt was certified.

SettingCPS, N=20CPS, N=41CRISP
BIPIA, α=0.10––0.01 / 0.78
BIPIA, α=0.20–0.14 / 0.880.14 / 0.85
BIPIA, α=0.300.17 / 0.860.23 / 0.940.18 / 0.89
Deepset, α=0.10––0.04 / 0.94
Deepset, α=0.200.09 / 0.990.09 / 0.990.05 / 0.99
Deepset, α=0.300.15 / 1.000.11 / 0.990.09 / 1.00

The results depend on the target risk level α. At α ≥ 0.20, increasing CPS to 41 candidates allows it to certify prompts with utility matches or slightly exceeds CRISP. However, at α=0.10, CPS still fails to certify any of its 41 candidates on either dataset, while CRISP succeeds on both. This is because CRISP uses risk-certificate signals to guide prompt optimization into risk-certified regions. This helps find certified prompts even when the required risk level (α) is low. Running risk-aware prompt optimization on the training set and certifying afterwards also does not help at low α (see our response to Q1 of Reviewer fBVH).


"W2. Q3. If Guess-and-Check is replaced by naive repeated evaluation on the calibration set, how do empirical risk violations and search efficiency change as adaptive evaluations increase?"

We evaluated a naive CRISP variant that replaces Guess-and-Check (GnC) with direct evaluation on DcertmalD_{cert}^{mal} at every round. The search observes each candidate's exact calibration risk and accepts prompts based on their observed calibration risk, without accounting for adaptive reuse.

Setting: dataset BIPIA, risk level α=0.10. Entries are risk / utility of the prompt each variant would return after T rounds. Neither variant certifies the seed (T=0).

TCRISP calibCRISP testNaive calibNaive test
10.014 / 0.2940.008 / 0.3450.079 / 0.8050.056 / 0.828
20.020 / 0.2990.024 / 0.2840.079 / 0.8050.056 / 0.828
50.023 / 0.5080.008 / 0.5950.079 / 0.8050.056 / 0.828
100.020 / 0.5330.024 / 0.6290.090 / 0.8210.113 / 0.836
200.042 / 0.7660.048 / 0.7590.090 / 0.8210.113 / 0.836

Naive CRISP violates the risk constraint. Up to T=5, the naive prompt's calibration risk overestimates its test risk (0.079 vs. 0.056). At T=10, naive switches to a prompt that looks feasible on DcertD_{cert} (0.090) but has test risk 0.113 > α. This is consistent with overfitting to calibration noise as more adaptively generated candidates are evaluated on the same data. With GnC, calibration and test risk stay under α, but this comes at the cost of utility.


Planned revisions. In the revised version, we will:

  • clarify that CPS is an instance of PRC with an added generation step, and present it as the direct baseline that motivates CRISP;
  • add the PRC comparison on hand-written prompts (Q1);
  • add the budget-matched CPS (N=41) comparison (Q2);
  • add the naive-vs-GnC ablation, including a discussion of the utility cost of GnC (Q3).

We hope these additional experiments address the reviewer's concerns.


[1] Jacob et al. (2025). PromptShield: Deployable Detection for Prompt Injection Attacks.

[2] Le et al. (2026). Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models.

[3] Liu (2026). Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents.

Reviewer_fBVHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a risk-certified prompt optimization framework for LLM-as-a-judge guardrails, aiming to maximize benign query pass rates while keeping missed-attack rates below a user-specified threshold. It introduces two methods: CPS, which generates a fixed pool of candidate prompts and selects the best risk-certified one, and CRISP, which iteratively refines prompts using risk feedback while maintaining statistical guarantees through a reusable-holdout mechanism.

Critical strengths
  1. The paper addresses a practically relevant problem by formulating LLM guardrail prompt optimization as a constrained optimization task. Explicitly controlling the missed-attack rate while maximizing benign utility provides a meaningful safety–utility trade-off compared with conventional prompt optimization methods that primarily optimize average performance.

  2. The proposed CRISP framework integrates risk feedback into iterative prompt optimization while accounting for the statistical challenges of adaptively reusing calibration data. Although the underlying statistical tools are established, their integration into adaptive guardrail prompt optimization is practically valuable.

Critical weaknesses
  1. The proposed methods largely build upon existing prompt optimization and statistical risk control techniques. CPS combines an existing prompt optimizer with Learn-then-Test certification, while CRISP incorporates the established Guess-and-Check reusable-holdout mechanism into iterative prompt rewriting. Although the integration is practically meaningful, the methodological contributions beyond existing approaches are not sufficiently clear.

  2. The theoretical risk guarantees rely on the assumption that calibration and deployment data follow the same distribution. However, real-world prompt injection attacks are often adaptive and may exhibit substantial distribution shifts. Although the paper includes a Best-of-N robustness evaluation, these adversarial settings are not covered by the theoretical guarantees, limiting their practical applicability.

Questions for authors
  1. Could the authors clarify the advantages of CRISP over a simpler approach that uses risk-aware prompt optimization on the training set, followed by independent risk certification? Would such an approach achieve comparable performance without requiring the more complex reusable-holdout mechanism?

  2. How much does each component of CRISP contribute to its overall performance? In particular, could the authors provide ablation studies on the two-population search strategy and the separate Improve Utility and Reduce Risk branches?

  3. The theoretical guarantees assume that calibration and deployment samples follow the same distribution. How sensitive is the proposed approach to changes in attack distributions or the emergence of previously unseen attack strategies? Could the authors evaluate risk control under cross-dataset transfer or distribution shifts?

2 comments

We thank the reviewer for the careful reading.


"W1. The proposed methods largely build upon existing prompt optimization and statistical risk control techniques ... the methodological contributions beyond existing approaches are not sufficiently clear."

We agree that CPS combines an existing optimizer with LTT/PRC. We present it to expose the limitations of this direct approach, and its failure mode is what motivates CRISP (CPS returns an empty certified set in 4 of 12 prompt-injection settings).

The contributions are:

  • Problem formulation: without an exposed score, the prompt is the only control surface for risk control.
  • Method: Guess-and-Check is a validity mechanism, not a search method. CRISP is risk certified prompt optimization process that uses risk certificate signals to guide prompt search into risk-certified regions. This helps find certified prompts even when the risk level (α) is low (Table 1, response to Q1). To our knowledge, this is the first LTT-style certificate where hypotheses are generated adaptively from calibration feedback.

We will clarify this in the paper.


"Q1. Could the authors clarify the advantages of CRISP over a simpler approach that uses risk-aware prompt optimization on the training set, followed by independent risk certification?"

We ran GEPA and ProTeGi on DtrainD_{train} using the risk-aware score U^−10max⁡(0,R^−α)\hat{U}-10\max(0,\hat{R}-α), then applied LTT to all generated prompts on DcertD_{cert}.

Setting: Deepset, gpt-5.4-mini judge; entries show test risk / utility. "–" means no certified prompt.

Methodα=0.10α=0.20α=0.30
GEPA + LTT––0.20 / 1.00
ProTeGi + LTT–0.02 / 0.870.09 / 1.00
CRISP0.04 / 0.940.05 / 0.990.09 / 1.00

Train-then-certify works at α=0.30, but tighter levels yield no certified prompt. Re-running the optimizer with different parameters reuses DcertD_{cert} and invalidates the certificate. This is what makes CRISP useful.


"Q2. How much does each component of CRISP contribute to its overall performance?"

Setting: [Deepset, gpt-5.4-mini judge, α = 0.2, T=10]. Variants:

  • Single population: one pool ranked by the penalized training score U^−10max⁡(0,R^−α)\hat{U}-10\max(0,\hat{R}-α), instead of the two populations.
  • ReduceRisk only: both rewrites each round go to the lowest-risk uncertified prompt.
  • ImproveUtility only: both rewrites go to the highest-utility prompt, certified or not.
  • Random parents: both populations and branches, parents drawn uniformly.
VariantTest risk / utilityCertified / pool
CRISP (T=10T=10)0.091 / 0.98710/19
Single population0.036 / 0.8833/21
ReduceRisk only0.055 / 0.94816/2
ImproveUtility only–0/21
Random parents0.091 /0.96110/18
  • Both objectives matter: ImproveUtility-only certifies nothing (0/21). ReduceRisk-only certifies the most candidates (16/21), but reduces utility.
  • Two-population split matters most: The single-population variant certifies only 3/21 candidates. We hypothesize penalized training risk poorly separates certifiable candidates, so the search cannot specialize.
  • Parent selection matters less: Random selection matches risk but reduces utility. Most gains come from the population split and complementary rewrite branches.

"Q3. Could the authors evaluate risk control under cross-dataset transfer or distribution shifts?"

"W2. The theoretical risk guarantees rely on the assumption that calibration and deployment data follow the same distribution"

We agree that the certificate is stated under i.i.d. calibration. We add two things.

Formally:

Following robust validation and conformal prediction beyond exchangeability (Cauchois et al., 2024; Barber et al., 2023), if the deployment malicious distribution satisfies TV(π1,π1′)≤ηTV(\pi_1, \pi_1') \le η, then ∣Rπ1′(p)−Rπ1(p)∣≤η|R_{\pi_1'}(p) - R_{\pi_1}(p)| \le η for every prompt pp, since the missed-attack loss is bounded. Hence, running CPS or CRISP at risk level α−ηα - η certifies α under any η-bounded shift.

If the shift is only in the label distribution, the guarantee is unchanged, since RR and UU are class-conditional. We will add this as a new corollary in Appendix B and state in Limitations that η must be specified, and that adaptive adversaries remain out of scope; certifying guardrails against them remains an open research problem beyond restricted threat models.

TVTV is the total variation distance, TV(P,Q)=sup⁡A∣P(A)−Q(A)∣TV(P, Q) = \sup_A |P(A) - Q(A)|.

Empirically:

We took the CRISP prompts certified at α=0.20 on BIPIA and Deepset and measured their test missed-attack rate, without re-certification, on two other prompt-injection benchmarks: Qualifire (Qualifire, 2025; Ivry & Nahum, 2025) and LLMail-Inject (Abdelnabi et al., 2025):

Certified on ↓ / tested on →BIPIADeepsetQualifireLLMail
BIPIA0.140.1450.0330.072
Deepset0.0370.050.0020.010

Risk stays below α=0.20 in every pair, which shows that the certified prompts are not overfit to the calibration attack distribution.

Thank you for following up. Our earlier reply spread the contribution across several answers, so we state it here in one place.

The contribution is a guarantee, not a combination. LTT and PRC certify a prompt family fixed before the calibration data are seen. In practice, guardrail prompts are written adaptively: evaluate, inspect failures, rewrite, evaluate again. Every such round reuses the calibration set and voids the certificate. Prop. 2 is, to our knowledge, the first LTT-style result that stays valid in this loop. It holds simultaneously for all certified prompts and regardless of how many rounds are run or how the search stops (Remarks B.1–B.2). It required adapting Guess-and-Check to one-sided Clopper–Pearson certificates (Lemma B.2) and keeping the benign holdout untouched, so the utility guarantee survives the search.

Existing pieces do not reach the same result. With the gpt-5.4-mini judge at α = 0.10, PRC on published prompts, risk-aware GEPA + LTT, and CPS with a matched budget of 41 candidates certify nothing on BIPIA or Deepset. CRISP certifies on both. Guess-and-Check alone is not a search method: without the certificate-driven two-population split, 3/21 candidates certify instead of 10/19.

In the revision, Prop. 2 become the main result. If a prior method already provides this guarantee, we would be grateful for the reference and will compare against it.

Comments on the paper

1 comment

We thank the reviewers and Area Chair. We summarize key points and revisions below.


The Contribution Is a Guarantee Under Adaptive Search (Reviewers fBVH, gZmv, ga4o)

We clarify that the contribution is Proposition 2, not a combination of existing tools. LTT and PRC certify a prompt family that is fixed before the calibration data are seen. Guardrail prompts are written the opposite way: evaluate, inspect failures, rewrite, evaluate again. Every such round reuses the calibration set and voids the certificate. CRISP generates each new prompt from earlier certification outcomes, and Proposition 2 proves that every certified prompt still has risk at most α\alpha with probability at least 1−δR1 - \delta_R, for any number of rounds and any stopping rule. To the best of our knowledge, this is the first LTT-style certificate for hypotheses generated adaptively from calibration feedback; if a prior method provides it, we would be grateful for the reference. We have revised the Abstract, Introduction, and Related Work to state Proposition 2 as the main result.


Clarifying Misunderstandings

Role of CPS and Relation to PRC (Reviewer gZmv, ga4o, fBVH): We agree that CPS is Prompt Risk Control (Zollo et al., 2024) with an added generation step, and we no longer claim it as a contribution. It is now presented as the fixed-pool baseline whose failure mode motivates CRISP: it returns an empty certified set in 4 of 12 prompt-injection settings.

Feedback, Not Budget (Reviewer gZmv, fBVH, ga4o): The gain does not come from evaluating more candidates. With the gpt-5.4-mini judge at α=0.10\alpha = 0.10, PRC on 12 published prompts, risk-aware GEPA followed by LTT, and CPS with a matched budget of 41 candidates certify nothing on BIPIA or Deepset, while CRISP certifies on both. At α≥0.20\alpha \geq 0.20, budget-matched CPS matches or slightly exceeds CRISP in utility, and we report this. Guess-and-Check alone is a validity mechanism, not a search method: without the certificate-driven two-population split, 3 of 21 candidates certify instead of 10 of 19.

i.i.d. Assumption and Adaptive Attacks (Reviewer fBVH, ga4o): The certificate is stated under i.i.d. calibration, as Section 6 already says. We now show that it degrades gracefully: under any shift of the malicious distribution bounded by η\eta in total variation, running CPS or CRISP at level α−η\alpha - \eta certifies α\alpha, and label shift alone leaves the guarantee unchanged. Adaptive adversaries remain outside the guarantee, so Best-of-N, Crescendo, and PAIR are reported as robustness diagnostics, not certified claims.

Spurious Patterns (Reviewer ga4o): The core rule of each selected prompt is to block hidden instructions; cues such as truncated text act as examples, as in few-shot prompting. Calibration and test performance agree (Tables 5–6), every certified BIPIA re-split meets the target (Table 7), and prompts certified on BIPIA keep risk below α\alpha on three datasets with different formatting.


Additional Materials

Certification Baselines (Reviewer gZmv, fBVH): We ran PRC on 12 published hand-written prompts, CPS with N=41N = 41 candidates, and risk-aware GEPA and ProTeGi followed by LTT. The detailed setup and results are provided in Appendices D.1–D.2.

Component and Reusable-Holdout Ablations (Reviewer fBVH, gZmv, ga4o): We ablated the two-population split, the two rewrite branches, and parent selection, and we replaced Guess-and-Check with direct reuse of the calibration set, which returns a BIPIA prompt with test risk 0.113 at α=0.10\alpha = 0.10. The detailed setup and results are reported in Appendices D.3–D.4.

Bounded Distribution Shift (Reviewer fBVH, ga4o): We added a corollary covering η\eta-bounded shift and label shift. The statement and proof are given in Appendix E, and Section 6 is updated accordingly.

Cross-Dataset Transfer (Reviewer fBVH, ga4o): We evaluated prompts certified on BIPIA and Deepset, without re-certification, on Qualifire and LLMail-Inject; the missed-attack rate stays below α=0.20\alpha = 0.20 in every pair. The detailed setup and results are presented in Appendix D.5.

Adaptive Attacks (Reviewer ga4o): We ran Crescendo and PAIR with up to 5 turns against all prompt-optimization methods; CRISP yields the most robust prompt under both. The detailed setup and results are given in Appendix D.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.