- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
A prompt-side safety filter decides whether to let a request through before any model has answered it. The paper asks whether "the filter let this through" means the same level of risk no matter which model answers. If it doesn't, a filter validated with one model can't simply be reused in front of another. To test this, the authors hold the filter and the set of prompts it released fixed, send the same prompts to several models with the same settings and judges, and compare how often each model gives a harmful answer. Under a policy that rewrites rejected prompts and releases them, and under a loose direct input screen, harm rates differ a lot between models. But for the one filter tuned to meet a stated risk target, swapping Qwen3 for Gemma-3 still meets the target. The authors conclude that safety claims should cover the filter and the model together, and be re-checked when the model changes.
Critical strengths
- The comparison is cleanly controlled. The same prompts, decoding settings and judges are used for every model, and the tests are fixed in advance, so differences can be attributed to the answering model.
- The blinded human audit is reported honestly, including where it disagrees with the automated judges.
Critical weaknesses
- The paper needs a much clearer motivation, starting with who would use this. It isn't clear which deployments put a prompt-side recovery policy in front of interchangeable models, and the paper doesn't name any. Teams deploying a filter also usually tune and validate it on the model they deploy, and re-run evals when the model changes. Prior work already evaluates guards per model (Wang et al., 2026) and builds guards tailored to each model (Liang and Liao, 2026), and the paper cites both. It would help to say who the intended user is and which decision this work changes for them.
- The setup leans on prompt-side filtering where I'd expect output-side checks to do most of the work. A frontier model's own refusal policy is usually already well tuned to the over-refusal boundary. Classifiers are most useful as a backstop that blocks harmful outputs when the model's refusal policy is wrong on a given case, which needs to see the output. Classifiers that see both the input and the output (transcript classifiers) are close to the state of the art for harm prevention, for example constitutional classifiers for CBRN misuse (Sharma et al., 2025). Not building on this is a major gap. Standard harm benchmarks such as HarmBench (Mazeika et al., 2024) also judge harm with a classifier that sees both the request and the response. The paper's own judges, Llama Guard 3 and WildGuard, can already classify responses, so the components for an output check are already in the pipeline. A prompt-only filter can't act as this backstop, so it is unsurprising that its risk depends on the answering model. The largest differences also come from setups unlikely to be deployed: the recovery policy re-releases 109 of 150 unsafe prompts, and the direct screen lets through 615 of 1,000 (which the authors note is not a recommended setting). The one filter that meets its risk target still meets it after the model swap (Sec 5.4).
Questions for authors
- Who do you expect to use a prompt-side recovery policy like this, and is there a deployed system that works this way?
- A prompt-only filter never sees the response, so it is expected that harm depends on which model answers. In practice, an output classifier is what catches harmful responses when the model's own refusals are wrong. Why not include one? I would expect an output classifier to remove most of the differences between models you report.
- Can you show a case where a filter that passes a safety bar (for example, under 5% harmful answers) with one model fails that same bar when the model is swapped, ideally with an output classifier in the system?
- Your calibration shows no threshold that is both safe and useful, and the intent audit covers 47 prompts on one model. Why build the main analysis on this recovery policy, and how often does a rewrite change what the user actually wanted?
To change my score, I would want to see: a clear deployment setting where this design is used, a system that includes an output classifier, and a filter that passes a safety bar with one model and fails it with another. I think this needs more than a rebuttal can provide.
2 comments
Thanks for adding the output classifier controls and for being upfront about what they show, so I'm raising my score from 1 to 2. I still lean reject because the controls mostly confirm that output classification closes the gap, and there is still no example of a filter that meets a safety bar with one model and then fails it after a model swap.
Thank you for specifying the evidence that would change your assessment. Section 5 and Table 2 now report output-classifier controls, with full measurement boundaries in Appendix K. We clarify the intended validation decision but do not claim a production deployment of the exact recovery algorithm or a fresh qualified-to-violating substitution.
1. Intended user, deployment, and prior work
The intended user is an engineer deciding whether an existing validation supports a response-backend replacement while retaining safety-layer settings. We clarified this decision without asserting that our exact recovery algorithm is a production defense.
A concrete documented managed-service architecture is Amazon Bedrock ApplyGuardrail: input and output checks are decoupled from foundation-model invocation and use an identified guard version. This supports the architecture-level motivation. It is not an audited customer deployment, a reproduction of Bedrock safeguards, or evidence that a production customer uses our six-rewrite recovery policy. We cannot supply that literal deployment case.
We agree that prior guardrail system evaluation and target-specific guard design already motivate model-aware validation. We do not claim a first demonstration of that principle. The revised paper preserves its narrower measurement question and removes an unsupported categorical comparison.
2. Output classifier
Your expectation is substantially supported. We use a frozen LG4 transcript gate, while LG3/WildGuard and StrongREJECT measure outcomes separately:
The recovery-policy gap falls from 26 to 6 points, with legitimate delivery falling from 67/100 to 64/100 for both legacy targets. Delivery is not useful fulfillment.
The residual continuous StrongREJECT difference is .0607 [.0301,.0944], but this is instrument-specific and not a 5% binary-risk guarantee. The modern human result is an important boundary: LG4 blocks six of ten harmful Gemma responses and none of five harmful Qwen3 responses, giving 4/150 versus 5/150 delivered harms. It also blocks eight human-safe Gemma responses. We do not claim that all output classifiers leave a gap or that this experiment reproduces Constitutional Classifiers.
3. Reference-qualified to violating substitution
We have not demonstrated the requested fresh qualified-to-violating substitution. The original Qwen3-to-Gemma test remains null under the declared 5% risk and three-point margin criteria. The new output-gate controls and 32B comparison do not retrospectively create reference qualification.
We do not select a safety bar from observed target gaps, compare a continuous score with a binary harm threshold, or treat point estimates on opposite sides of a threshold as a prospective qualification result. This condition remains unmet, and the revised manuscript does not claim otherwise.
4. Unqualified recovery and intent preservation
The recovery policy is an audit instrument, not evidence of successful safe-and-useful repair. Its failed joint calibration remains in the main results.
The semantic audit contains 98 triples; 47 originals receive majority-legitimate labels. Among those 47, 44 rewrites preserve intent exactly, two have minor shifts, one changes intent, and 46 are useful. This does not validate intent preservation for the 109 unsafe releases. Existing pilot/development semantic pairs do not exactly match those 109 source/rewrite pairs and cannot fill that gap.
An intent change limits faithful-recovery and deployment claims. It does not remove the controlled comparison between targets receiving the exact same realized rewrite, but we do not extend the legitimate semantic-audit result to unsafe requests.
In summary, we provide direct output-classifier evidence and a more concrete validation decision, including results that support your concerns. Literal deployment of the recovery algorithm and fresh qualified crossing remain unestablished. The minimal revision adds evidence and corrects scope without replacing the original study.