acceptodds
Under review as a conference paper at ICLR 2027

TRAP: Auditing Target Risk After Prompt-Side Acceptance

Abstract

Prompt-side acceptance is decided before a response exists, but downstream harm also depends on the model that answers. We test whether the downstream risk meaning of a frozen prompt-side release decision survives response-target substitution. A prospective recovery policy releases the same 109 of 150 source-unsafe OR-Bench prompts to four targets, including a base-model stress target. Harmful-release mass ranges from 0.67% to 64.00%; the Mistral versus Qwen2.5 gap is 26.0 percentage points, corroborated by independent grading and a blinded human audit. A distinct direct-screen policy replicates target dependence on 1,000 sealed unsafe prompts, with a prespecified 24.3-point Ministral versus gpt-oss gap (97.5% CI [20.4,28.2]). Modern targets under the recovery policy vary less. To test the boundary of this result, we identify an input-guard policy that satisfies a prespecified Qwen3 risk-coverage contract using guards disjoint from the response evaluators. On a separate sealed population of 2,000 unsafe and 150 legitimate prompts, harmful-release mass is at most 1.10% for Qwen3 and Gemma-3 under each evaluator. Under Gemma-3 substitution, neither the predeclared 5% risk-violation criterion nor the three-point paired-margin criterion is met. Target dependence therefore need not violate a qualified contract; downstream assurance requires validation under the declared deployment contract.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language model safety, prompt-side release, target substitution, selective release, risk calibration, safety evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yj6GOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

A prompt-side safety filter decides whether to let a request through before any model has answered it. The paper asks whether "the filter let this through" means the same level of risk no matter which model answers. If it doesn't, a filter validated with one model can't simply be reused in front of another. To test this, the authors hold the filter and the set of prompts it released fixed, send the same prompts to several models with the same settings and judges, and compare how often each model gives a harmful answer. Under a policy that rewrites rejected prompts and releases them, and under a loose direct input screen, harm rates differ a lot between models. But for the one filter tuned to meet a stated risk target, swapping Qwen3 for Gemma-3 still meets the target. The authors conclude that safety claims should cover the filter and the model together, and be re-checked when the model changes.

Critical strengths
  • The comparison is cleanly controlled. The same prompts, decoding settings and judges are used for every model, and the tests are fixed in advance, so differences can be attributed to the answering model.
  • The blinded human audit is reported honestly, including where it disagrees with the automated judges.
Critical weaknesses
  • The paper needs a much clearer motivation, starting with who would use this. It isn't clear which deployments put a prompt-side recovery policy in front of interchangeable models, and the paper doesn't name any. Teams deploying a filter also usually tune and validate it on the model they deploy, and re-run evals when the model changes. Prior work already evaluates guards per model (Wang et al., 2026) and builds guards tailored to each model (Liang and Liao, 2026), and the paper cites both. It would help to say who the intended user is and which decision this work changes for them.
  • The setup leans on prompt-side filtering where I'd expect output-side checks to do most of the work. A frontier model's own refusal policy is usually already well tuned to the over-refusal boundary. Classifiers are most useful as a backstop that blocks harmful outputs when the model's refusal policy is wrong on a given case, which needs to see the output. Classifiers that see both the input and the output (transcript classifiers) are close to the state of the art for harm prevention, for example constitutional classifiers for CBRN misuse (Sharma et al., 2025). Not building on this is a major gap. Standard harm benchmarks such as HarmBench (Mazeika et al., 2024) also judge harm with a classifier that sees both the request and the response. The paper's own judges, Llama Guard 3 and WildGuard, can already classify responses, so the components for an output check are already in the pipeline. A prompt-only filter can't act as this backstop, so it is unsurprising that its risk depends on the answering model. The largest differences also come from setups unlikely to be deployed: the recovery policy re-releases 109 of 150 unsafe prompts, and the direct screen lets through 615 of 1,000 (which the authors note is not a recommended setting). The one filter that meets its risk target still meets it after the model swap (Sec 5.4).
Questions for authors
  1. Who do you expect to use a prompt-side recovery policy like this, and is there a deployed system that works this way?
  2. A prompt-only filter never sees the response, so it is expected that harm depends on which model answers. In practice, an output classifier is what catches harmful responses when the model's own refusals are wrong. Why not include one? I would expect an output classifier to remove most of the differences between models you report.
  3. Can you show a case where a filter that passes a safety bar (for example, under 5% harmful answers) with one model fails that same bar when the model is swapped, ideally with an output classifier in the system?
  4. Your calibration shows no threshold that is both safe and useful, and the intent audit covers 47 prompts on one model. Why build the main analysis on this recovery policy, and how often does a rewrite change what the user actually wanted?

To change my score, I would want to see: a clear deployment setting where this design is used, a system that includes an output classifier, and a filter that passes a safety bar with one model and fails it with another. I think this needs more than a rebuttal can provide.

2 comments

Thank you for specifying the evidence that would change your assessment. Section 5 and Table 2 now report output-classifier controls, with full measurement boundaries in Appendix K. We clarify the intended validation decision but do not claim a production deployment of the exact recovery algorithm or a fresh qualified-to-violating substitution.

1. Intended user, deployment, and prior work

The intended user is an engineer deciding whether an existing validation supports a response-backend replacement while retaining safety-layer settings. We clarified this decision without asserting that our exact recovery algorithm is a production defense.

A concrete documented managed-service architecture is Amazon Bedrock ApplyGuardrail: input and output checks are decoupled from foundation-model invocation and use an identified guard version. This supports the architecture-level motivation. It is not an audited customer deployment, a reproduction of Bedrock safeguards, or evidence that a production customer uses our six-rewrite recovery policy. We cannot supply that literal deployment case.

We agree that prior guardrail system evaluation and target-specific guard design already motivate model-aware validation. We do not claim a first demonstration of that principle. The revised paper preserves its narrower measurement question and removes an unsupported categorical comparison.

2. Output classifier

Your expectation is substantially supported. We use a frozen LG4 transcript gate, while LG3/WildGuard and StrongREJECT measure outcomes separately:

ArchitectureQwen2.5 harmful/150Mistral harmful/150Qwen2.5/Mistral delivered
No guards564150/150
Output only216147/98
Input only847109/109
Input + output413105/72

The recovery-policy gap falls from 26 to 6 points, with legitimate delivery falling from 67/100 to 64/100 for both legacy targets. Delivery is not useful fulfillment.

The residual continuous StrongREJECT difference is .0607 [.0301,.0944], but this is instrument-specific and not a 5% binary-risk guarantee. The modern human result is an important boundary: LG4 blocks six of ten harmful Gemma responses and none of five harmful Qwen3 responses, giving 4/150 versus 5/150 delivered harms. It also blocks eight human-safe Gemma responses. We do not claim that all output classifiers leave a gap or that this experiment reproduces Constitutional Classifiers.

3. Reference-qualified to violating substitution

We have not demonstrated the requested fresh qualified-to-violating substitution. The original Qwen3-to-Gemma test remains null under the declared 5% risk and three-point margin criteria. The new output-gate controls and 32B comparison do not retrospectively create reference qualification.

We do not select a safety bar from observed target gaps, compare a continuous score with a binary harm threshold, or treat point estimates on opposite sides of a threshold as a prospective qualification result. This condition remains unmet, and the revised manuscript does not claim otherwise.

4. Unqualified recovery and intent preservation

The recovery policy is an audit instrument, not evidence of successful safe-and-useful repair. Its failed joint calibration remains in the main results.

The semantic audit contains 98 triples; 47 originals receive majority-legitimate labels. Among those 47, 44 rewrites preserve intent exactly, two have minor shifts, one changes intent, and 46 are useful. This does not validate intent preservation for the 109 unsafe releases. Existing pilot/development semantic pairs do not exactly match those 109 source/rewrite pairs and cannot fill that gap.

An intent change limits faithful-recovery and deployment claims. It does not remove the controlled comparison between targets receiving the exact same realized rewrite, but we do not extend the legitimate semantic-audit result to unsafe requests.

In summary, we provide direct output-classifier evidence and a more concrete validation decision, including results that support your concerns. Literal deployment of the recovery algorithm and fresh qualified crossing remain unestablished. The minimal revision adds evidence and corrects scope without replacing the original study.

Thanks for adding the output classifier controls and for being upfront about what they show, so I'm raising my score from 1 to 2. I still lean reject because the controls mostly confirm that output classification closes the gap, and there is still no example of a filter that meets a safety bar with one model and then fails it after a model swap.

Reviewer_NRCEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TRAP, a framework for examining whether prompt-side safety decisions remain reliable when the downstream target model changes. By fixing the prompt-release policy and evaluating identical released prompts across different LLMs, the authors measure target-dependent harmful-response risks and assess the transferability of prompt-level risk scores. Experiments across multiple models, datasets, and release policies show that target substitution can substantially change downstream risk, although such changes do not necessarily violate prespecified safety contracts.

Critical strengths
  1. The paper addresses a practically relevant problem by examining the relationship between input safety filtering and downstream response risks when the target model changes. This is relevant to LLM deployments that replace or update their underlying models.

  2. The experimental design is relatively rigorous. The authors fix the complete prompt-side release policy and the set of released prompts, enabling paired comparisons across target models. They also evaluate multiple datasets and release policies, supported by prespecified hypotheses, statistical tests, and independent human annotations.

  3. The paper provides a useful analysis of the boundaries of safety guarantees. Beyond demonstrating risk differences after target substitution, it examines whether such differences necessarily violate prespecified risk and coverage requirements. The inclusion of a setting where model substitution remains within the declared safety contract supports a more nuanced conclusion.

Critical weaknesses
  1. The novelty of the empirical findings is limited. Although the frozen release-policy design provides a controlled evaluation, the main observation that identical prompts induce different safety behaviors across LLMs is already well established. The additional insights beyond existing cross-model safety evaluations remain unclear.

  2. The role of model-specific refusal behavior is insufficiently characterized. The direct-screen analysis suggests that much of the observed risk difference is driven by refusal tendencies. A more systematic decomposition of refusals, harmful content generation, and generation failures across experiments would strengthen the conclusions.

  3. The evaluation has limited deployment generalizability. Experiments focus on 7B–20B open-weight models in single-turn settings, without evaluating closed-source APIs, multi-turn interactions, or tool-using systems. This limits the applicability of the findings to more complex LLM deployments.

Questions for authors
  1. Compared with existing cross-model safety evaluation and guardrail research, what specific new insights does TRAP reveal through its frozen release-policy evaluation? Can these findings lead to more concrete recommendations for designing safety systems?

  2. Could the authors provide a more systematic decomposition of the changes in harmful-release mass across the main experiments, separately quantifying the contributions of refusal behavior, harmful content generation, and generation or evaluation failures? To what extent do these factors explain the observed cross-model differences?

  3. Could the authors further examine whether TRAP's findings generalize to closed-source API models, larger models, or multi-turn agent workflows? Would similar patterns of downstream risk variation persist in systems involving tool use or more complex interaction histories?

2 comments

Thank you for the constructive assessment. We retain the original study and add a compact layer-control comparison in Section 5 and Table 2, with protocols in Appendix K and a supporting larger-model comparison in Appendix M. The revision adds evidence and clarifies scope rather than introducing a new defense or research question.

1. Incremental insight and practical implications

We agree that cross-model safety differences and model-aware validation are established. The controls distinguish original target behavior, changes under a fixed input policy, and delivered outcomes after response moderation:

ArchitectureQwen2.5 harmful/150Mistral harmful/150Qwen2.5/Mistral delivered
No guards564150/150
Output only216147/98
Input only847109/109
Input + output413105/72

The outcome union is separate from the LG4 output gate. The input-policy gap falls from 26 to 6 points after moderation. The modern human comparison changes direction, so the evidence does not support universal persistence of the gap.

We retain the overlap with guardrail system evaluations and model-specific guards. We adopt, rather than claim to invent, the allow-and-harm event. The scoped contribution is the joint audit of a fixed release set, its score, layer-specific outcomes, and explicit qualification criteria.

For an engineer replacing a backend, the controls motivate checking delivered responses after the actual output layer, using paired requests to isolate the substitution, measuring outcomes separately from the authorizing gate, and distinguishing delivery from usefulness. A risk difference and a contract violation require different evidence. These principles are not all new individually; the study measures their different consequences.

2. Refusal, harmful content, and failures

The direct-screen primary endpoint includes unresolved generations. Its 1,000-prompt accounting is:

OutcomeMinistralgpt-oss
Input-withheld385385
Generation unresolved0117
Evaluation unresolved after generation00
Parsed harmful under union3688
Parsed nonharmful under union247490
Total1,0001,000

Thus 125 conservative gpt-oss harms comprise eight flagged complete responses plus 117 unresolved generations. We keep this primary evaluation separate from the fixed retry sensitivity, whose contrast is 28.9 points.

WildGuard records refusal on 38/615 generated Ministral responses and 488/498 complete gpt-oss responses. One Ministral response has both refusal and harmful labels; refusal and harm are not mutually exclusive. The human audit has only ten mutually non-refusal pairs, tied at 8/10 harmful. These data characterize the boundary but do not identify causal mediation or establish equivalence beyond refusal.

3. Larger models and deployment generalization

We completed an unchanged-policy Qwen3-14B/32B comparison on the same 109 released unsafe and 67 released legitimate prompts. Under native templates, disabled thinking, and the fixed 256-token greedy contract, automated union harmful counts are 3/150 versus 5/150 before output moderation and 3/150 for both afterwards. Post-moderation labels agree case-by-case; that is not population equivalence.

The ungated difference is 1.33 points with a pointwise 95% source-group bootstrap interval [0,3.95]. StrongREJECT has the opposite direction, so we do not infer a general size-dependent safety ordering. Of the 176 32B responses, 80 reach the generation limit without EOS, including 24 unsafe and 56 legitimate responses. They remain in the evaluation, whose scope is that token budget. There are no matching 32B human labels.

Appendix M reports this supporting comparison. It extends model-size coverage, but not to hosted APIs, multi-turn histories, tool execution, or adaptive use. It neither identifies size as a causal mechanism nor demonstrates a qualified crossing. The manuscript retains those limitations.

Thank you for the additional refusal and failure analysis and the Qwen3-32B experiment. These additions help clarify the reported risk differences and broaden the model-size coverage. However, my concern about the incremental novelty beyond existing cross-model safety and guardrail evaluations remains. I therefore maintain my rating of 2.

Reviewer_NyozOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether a prompt-side safety decision (a guard or rewrite-and-release policy that decides, before generation, which prompts get through) keeps its risk meaning when the downstream response model is swapped. The authors freeze a release policy and its released prompt set, then send those identical prompts to several targets with the same judges. The main estimand is harmful-release mass: the fraction of the whole unsafe population that is both released and answered harmfully.

Critical strengths
  • The estimand seems correct. Using the full source population as the denominator, rather than the conditional harm rate among released prompts, properly penalizes release and credits abstention. Counting unresolved generations as harmful is a defensible conservative choice, and they report a sensitivity analysis for it.
  • Negative results are reported honestly. These include the failed calibration grid, the failed target-aware prediction gate, and the failed supportive replication gate.
Critical weaknesses
  • The filter lets most unsafe prompts through (109 of 113 selected ones). So the harm numbers mostly reflect how each target handles unsafe prompts, not anything about the filter. The obvious control is to skip the filter, send every prompt to each target, and see if the targets rank the same way. If they do, the filter adds nothing to the story, and the finding is just “some models refuse more than others,” which is already known.
  • The paper’s whole point is that a safety measurement made with one model may not hold for another. But the judges have the same problem. In the authors’ own human check, the judges caught 7 of 10 truly harmful Gemma answers and 0 of 5 truly harmful Qwen3 answers. So some differences between targets could come from the judges missing harm on one model more than another. It also means the claim “Qwen3’s harm is under 1%” can’t be trusted, since the judge missed every harmful Qwen3 answer the humans found.
  • RQ3 asks whether swapping Qwen3 for Gemma-3 breaks the 5% safety limit. But harm for both was around 1% or less, so nothing could get near 5%. Passing that test says the limit is loose, not that swapping models is safe. Gemma actually had 13 harmful answers where Qwen3 had 1, so the models do differ here too. The test was just too blunt to see it.
  • RQ2 asks whether a prompt risk score built using Qwen2.5 still works on other targets. But the score barely works even on Qwen2.5. It was trained on only 17 harmful examples and blocks more legitimate prompts than unsafe ones. Some of its learned weights point the wrong way: when a guard says a prompt is safe, the score predicts more harm. If a tool doesn’t work at home, showing it fails elsewhere isn’t informative.
  • LlamaGuard-3 and WildGuard help pick which prompts get released, and then they also judge the answers. If these guards have a blind spot, the filter may favor prompts that hit it, and the same blind spot then hides harm in the answers. How much that hides can vary by target, which muddies the target comparison. The authors use separate models for filtering and judging in their RQ3 test, but not in the main experiments.
Questions for authors

Please address my weaknesses.

2 comments

Thank you for identifying the missing controls and measurement limitations. The minimal revision retains the original prospective results and qualification null. Section 5 and Table 2 add the no-filter/output-moderation controls; Appendices K and L give the measurement and reference-score diagnostics. We also correct the conditional-risk domain, bootstrap-percentile wording, and description of the shared-prompt leave-one-target-out evaluation, without changing the original primary counts or thresholds.

1. No-filter control and known target ordering

We agree that target ordering alone does not establish an effect specific to the filter. The full no-filter control uses all 150 enrolled source-unsafe prompts, including those withheld by recovery:

TargetOriginal prompts, no input filterRecovery policy
Qwen2.55/1508/150
Mistral64/15047/150
Llama-31/1501/150
Llama-2 base131/15096/150

The legacy ordering predates recovery. For Mistral versus Qwen2.5, the gap is 39.33 percentage points on originals and 26.00 points under recovery. We do not present this ordering as newly created by the policy. The study concerns the measured risk of the fixed release decision and its interaction with the target and output layer. This is a narrower contribution than discovering that models refuse differently.

The recovery mask releases 109 of 150 unsafe source requests, or 109 of 113 selected candidates. We retain both denominators. Its failed joint risk-and-utility calibration remains visible; we do not recommend it as a deployable safety defense.

2. Judge misses and the meaning of low measured risk

The human census supports your concern. Among 109 released responses per modern target, the LG3/WildGuard union detects 0/5 human-harmful Qwen3 responses and 7/10 Gemma responses. The output gate also misses all five Qwen3 harms. Different checkpoints do not guarantee independent measurement errors.

Appendix K now reports both detections and false positives. As descriptive sensitivities, StrongREJECT at threshold .25 detects 5/5 and 9/10, but also flags 20/104 and 14/99 human-safe responses. At .50, detections fall to 3/5 and 3/10. We report these as alternative measurement tradeoffs, not proof that another grader eliminates the problem or as historically prespecified endpoints.

After LG4 gating, human-harmful delivered counts are Qwen3 5/150 and Gemma 4/150, rather than 5/150 and 10/150. This is not evidence of population equivalence or a statistically established reversal. It limits any blanket claim that the ungated modern contrast survives output moderation.

The revised qualified-policy subsection explicitly states that its bounds are evaluator-defined, not a validated human-harm rate on that separate test population. We cannot extrapolate the census miss rates to the 2,000-prompt qualification test. Matching human labels are unavailable there.

3. The 5% bar and the qualification null

We agree that this experiment does not show a qualified policy becoming violating. On 2,000 unsafe test prompts, Qwen3/Gemma counts are 1/13 under LG3 and 10/22 under WildGuard. A measured difference exists, but neither the declared 5% violation criterion nor the three-point paired-margin criterion is triggered.

The null answers a contract question, not whether the targets have identical risks or whether substitution is generally safe. The positive statement is limited to the specified evaluator-risk/coverage region and tested pair. We keep the original bar, denominator, and null rather than selecting a threshold after observing the differences. Coverage of 139/150 legitimate prompts is also distinct from strict safe-useful fulfillment, which is 51.33% and 38.67%.

4. Weak reference score

We agree that the historical head is weak at its reference target: it has only 17 harmful development examples, and Qwen2.5 AUROC .631 has a 95% interval of [.474,.782]. The revised main text states that this is not evidence about transport failure of a well-calibrated reference predictor. Correlated guard features and sparse events also preclude a causal interpretation of coefficient signs.

To test whether weak reference discrimination alone explains the concern, we ran a separate reference-only diagnostic on the existing 530-prompt panel in 516 source groups. Five group folds exclude each evaluation prompt's source group from its training data. Standardization is training-only, logistic regularization is fixed at C=1, and only Qwen3 labels enter fitting. The same pooled out-of-fold scores yield:

Outcome targetAUROCPointwise 95% group-bootstrap interval
Qwen3 reference.750[.685,.811]
Gemma-3.629[.564,.689]
gpt-oss.650[.589,.708]
Ministral.747[.678,.812]

Reference-minus-target differences are .121 [.059,.185] for Gemma, .099 [.025,.169] for gpt-oss, and .003 [-.063,.069] for Ministral. Thus useful reference discrimination need not imply equal discrimination elsewhere, but degradation is not universal.

This diagnostic is retrospective cross-fitting, not a single prospectively frozen score or risk qualification. Its bootstrap is conditional on fitted folds and does not include retraining uncertainty. It does not replace the historical head, and AUROC does not establish calibration. Appendix L gives these distinctions.

5. Shared selection/evaluation checkpoints

This remains a limitation of the historical recovery study. Cross-fitting does not remove shared-checkpoint selection bias. We use a separate StrongREJECT grader and existing blinded human evidence to triangulate the principal legacy contrast, while explicitly reporting target-correlated disagreements.

After output gating, the legacy full-source mean StrongREJECT difference is .0607, with a pointwise 95% source-group bootstrap interval [.0301,.0944]. This is a continuous sensitivity endpoint, not a binary 5% safety claim. The historical human subset for this particular legacy contrast contains 18 sampled pairs, not complete human coverage. We do not transfer its uncertainty or labels to a different frame.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.