- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GuardAlign, which uses offline guardrail judgments to guide safety preference alignment. GuardAlign assumes independently labeled unsafe prompts with safe/unsafe response pairs and guardrails provide auxiliary prompt risk evidence. Its GuardPO objective extends SimPO with an example-dependent margin. The detection term emphasizes guardrail misses and weak evidence, while a detached, helpfulness-gated term emphasizes response pairs the current model still ranks incorrectly. Four guardrails provide training annotations, and deployment uses only ShieldGemma-9B before the trained backend, without additional inference stages beyond the input-guardrail pipeline.
The experimental results show that GuardAlign achieves a mean ASR of 2.42% across four backbones and six safety test sets, versus 7.23% for the original LLM with an input guardrail and 4.96% for G-SimPO. Direct adaptive attacks without a guardrail yield 23.75% versus 31.00% for G-SimPO. Component ablations and BBH evaluations provide further evidence, although the latter also show capability losses relative to the original models.
Critical strengths
-
A simple training objective that uses guardrail evidence. GuardPO extends SimPO with an example-dependent margin while keeping the original safety preference labels and all training pairs. This provides a practical way to focus training on guardrail misses and weak evidence without adding inference stages to the existing pipeline.
-
The improvements also appear without an inference-time guardrail. Under direct adaptive attacks, GuardPO achieves a mean ASR of 23.75%, compared with 31.00% for G-SimPO under matched data and training configurations. This supports an improvement in the backend itself, beyond protection from the input guardrail.
Critical weaknesses
-
The choice of deployment guardrail is not justified. The paper uses only ShieldGemma-9B for input filtering, although Qwen3Guard and WildGuard are already used for training annotations. It does not compare their detection performance on the evaluation sets or explain why ShieldGemma was selected. The results therefore do not establish whether GuardPO remains useful with a stronger guardrail, or whether replacing ShieldGemma could provide similar protection without backend training. This leaves the practical benefit of the method unclear.
-
The experiments do not sufficiently isolate the benefit of guardrail evidence. The G-SimPO comparison and component ablations show benefits from the proposed objective, but do not distinguish useful guardrail evidence from a larger margin or general emphasis on difficult examples. The benefit of combining four guardrails with equal weights is also not tested. The paper includes a hyperparameter search, but lacks controlled sensitivity analysis and repeated-run results. These gaps make it difficult to assess whether the gains depend on informative guardrail evidence and remain stable across parameter choices.
Questions for authors
-
Clarification — deployment guardrail. Why was ShieldGemma-9B selected instead of Qwen3Guard or WildGuard? How do their false-negative and false-positive rates compare on the evaluation distribution?
-
Clarification — training data and baselines. How is PKU-SafeRLHF-30K processed to satisfy the assumption of verified unsafe prompts with safe/unsafe response pairs? Where do the prompt safety labels and independent helpfulness preferences come from? How, if at all, do G-DPO and G-SimPO use the guardrail annotations in training?
-
Clarification — experimental consistency. Why do the full GuardPO results on Gemma differ between Tables 1 and 2? How do the Stage-I ranks and selected candidates in Table 5 follow the ASR-first ranking rule and the results in Table 4, particularly for S1-11 and S1-24? Also, §4.1 uses a learning rate of 10⁻⁵, while Appendix B uses 10⁻⁶. Is this difference intentional, and which setting was used for the final results?
-
Additional validation — deployment benefit. Does GuardPO still improve over G-SimPO when Qwen3Guard or WildGuard is deployed? How does ShieldGemma with GuardPO compare with these guardrails paired with the original backend? ASR, benign refusal rates, and latency would help determine whether backend training provides a practical benefit over changing the guardrail.
-
Additional validation — guardrail evidence and robustness. Does GuardPO outperform a tuned constant-margin baseline and a variant with shuffled guardrail evidence? Does using four guardrails improve over using a single guardrail? Sensitivity analysis and repeated runs would help establish whether the gains are stable. These comparisons would directly address whether the proposed evidence and design choices are necessary.