acceptodds
Withdrawn from ICLR 2027

GuardAlign: Guardrail-Aware Safety Alignment for Modern LLM Pipelines

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

External guardrails increasingly complement aligned LLMs against jailbreaks, yet guardrails and backend models are typically optimized independently. Consequently, an unsafe prompt missed by an input guardrail reaches a backend never explicitly aligned to the guardrail's coverage gaps. We introduce Guardrail-Aware Safety Alignment (GuardAlign), a paradigm that retains the full safety preference dataset and augments each example with offline decisions and normalized unsafe-confidence evidence from multiple input guardrails. Within GuardAlign, Guardrail-Aware Preference Optimization (GuardPO) is a reference-free objective that replaces a static margin with an example-specific target margin. Its Detection Penalty emphasizes guardrail misses and low-confidence detections, while its Stubbornness Penalty targets current-policy misrankings only when safety and helpfulness agree. Across four LLM backbones and six safety test sets, every GuardAlign variant attains a lower overall mean pipeline attack success rate (ASR) than Pre-Guardrail, Post-Guardrail, and Self-Refine; GuardPO performs best at 2.42%, versus 7.23% for the strongest of these baselines. Without an inference-time guardrail, GuardPO also reduces adaptive-attack ASR to 23.75%, compared with 31.00% for G-SimPO and 43.75% for G-DPO. It further incurs substantially less online overhead than output filtering and iterative refinement, while retaining the most BIG-Bench Hard utility among the evaluated guardrail-aware objectives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, LLM Fine-tuning, Preference Optimization, Jailbreak Defense

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L7NHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GuardAlign, which uses offline guardrail judgments to guide safety preference alignment. GuardAlign assumes independently labeled unsafe prompts with safe/unsafe response pairs and guardrails provide auxiliary prompt risk evidence. Its GuardPO objective extends SimPO with an example-dependent margin. The detection term emphasizes guardrail misses and weak evidence, while a detached, helpfulness-gated term emphasizes response pairs the current model still ranks incorrectly. Four guardrails provide training annotations, and deployment uses only ShieldGemma-9B before the trained backend, without additional inference stages beyond the input-guardrail pipeline.

The experimental results show that GuardAlign achieves a mean ASR of 2.42% across four backbones and six safety test sets, versus 7.23% for the original LLM with an input guardrail and 4.96% for G-SimPO. Direct adaptive attacks without a guardrail yield 23.75% versus 31.00% for G-SimPO. Component ablations and BBH evaluations provide further evidence, although the latter also show capability losses relative to the original models.

Critical strengths
  1. A simple training objective that uses guardrail evidence. GuardPO extends SimPO with an example-dependent margin while keeping the original safety preference labels and all training pairs. This provides a practical way to focus training on guardrail misses and weak evidence without adding inference stages to the existing pipeline.

  2. The improvements also appear without an inference-time guardrail. Under direct adaptive attacks, GuardPO achieves a mean ASR of 23.75%, compared with 31.00% for G-SimPO under matched data and training configurations. This supports an improvement in the backend itself, beyond protection from the input guardrail.

Critical weaknesses
  1. The choice of deployment guardrail is not justified. The paper uses only ShieldGemma-9B for input filtering, although Qwen3Guard and WildGuard are already used for training annotations. It does not compare their detection performance on the evaluation sets or explain why ShieldGemma was selected. The results therefore do not establish whether GuardPO remains useful with a stronger guardrail, or whether replacing ShieldGemma could provide similar protection without backend training. This leaves the practical benefit of the method unclear.

  2. The experiments do not sufficiently isolate the benefit of guardrail evidence. The G-SimPO comparison and component ablations show benefits from the proposed objective, but do not distinguish useful guardrail evidence from a larger margin or general emphasis on difficult examples. The benefit of combining four guardrails with equal weights is also not tested. The paper includes a hyperparameter search, but lacks controlled sensitivity analysis and repeated-run results. These gaps make it difficult to assess whether the gains depend on informative guardrail evidence and remain stable across parameter choices.

Questions for authors
  1. Clarification — deployment guardrail. Why was ShieldGemma-9B selected instead of Qwen3Guard or WildGuard? How do their false-negative and false-positive rates compare on the evaluation distribution?

  2. Clarification — training data and baselines. How is PKU-SafeRLHF-30K processed to satisfy the assumption of verified unsafe prompts with safe/unsafe response pairs? Where do the prompt safety labels and independent helpfulness preferences come from? How, if at all, do G-DPO and G-SimPO use the guardrail annotations in training?

  3. Clarification — experimental consistency. Why do the full GuardPO results on Gemma differ between Tables 1 and 2? How do the Stage-I ranks and selected candidates in Table 5 follow the ASR-first ranking rule and the results in Table 4, particularly for S1-11 and S1-24? Also, §4.1 uses a learning rate of 10⁻⁵, while Appendix B uses 10⁻⁶. Is this difference intentional, and which setting was used for the final results?

  4. Additional validation — deployment benefit. Does GuardPO still improve over G-SimPO when Qwen3Guard or WildGuard is deployed? How does ShieldGemma with GuardPO compare with these guardrails paired with the original backend? ASR, benign refusal rates, and latency would help determine whether backend training provides a practical benefit over changing the guardrail.

  5. Additional validation — guardrail evidence and robustness. Does GuardPO outperform a tuned constant-margin baseline and a variant with shuffled guardrail evidence? Does using four guardrails improve over using a single guardrail? Sensitivity analysis and repeated runs would help establish whether the gains are stable. These comparisons would directly address whether the proposed evidence and design choices are necessary.

Reviewer_q7PGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how a backend model can reduce harmful responses when an input guardrail misses an unsafe prompt. It proposes GuardAlign and GuardPO, which adapt the preference-optimization margin based on multiple guardrails’ decisions and confidence scores, as well as the policy’s current ranking of safe and unsafe responses.

Critical strengths

The paper targets a practical systems problem: when an upstream guardrail misses, the backend remains the last opportunity to prevent a harmful response. The evaluation goes beyond overall safety to include adaptive attacks without an inference-time guardrail, BBH utility, and component ablations. This gives a broader picture of safety, robustness, and utility trade-offs than a single benchmark would.

Critical weaknesses
  1. The experiments do not clearly separate the benefit of guardrail signals from the benefit of fine-tuning on safety preferences.

  2. The ablations are limited to Gemma, Table 2 has relatively coarse ASR increments, and the paper reports no across-seed variation or confidence intervals.

Questions for authors
  1. Could you report a standard SimPO baseline trained with the same data and budget, and compare it with GuardPO on the guardrail-miss subset? Such a comparison would make me more confident that the gains come from guardrail-aware training and would make me consider raising my score.

  2. The main results and adaptive-attack results appear to come from a single training run and a single attack run. Could you report results across multiple random seeds and provide statistical significance tests?

  3. The paper notes that safety guardrails may miss unsafe prompts. Have you measured the miss rate of each guardrail separately? It would also be helpful to report GuardPO’s performance on prompts missed by only some guardrails and on prompts correctly detected by all guardrails but with low confidence.

Reviewer_ZoFSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GuardAlign, a training paradigm that uses offline decisions and normalized unsafe-confidence evidence from multiple input guardrails as auxiliary supervision for backend LLM safety alignment. Within GuardAlign, the paper introduces GuardPO, a reference-free preference objective that replaces SimPO’s static target margin with an example-specific margin. The margin combines a prompt-level Detection Penalty, which increases pressure on prompts missed by any guardrail or detected by all guardrails with low aggregate unsafe confidence, and a response-level Stubbornness Penalty, which activates when the current policy misranks the unsafe response above the safe response and safety/helpfulness preferences agree.

Critical strengths
  1. The core idea is practical and well motivated: guardrail false negatives expose residual risk, and guardrail behavior can be converted into an offline training signal without changing the deployed inference architecture.
  2. The experimental matrix is broad: four model families, six safety test sets, an adaptive attack, ablations of both penalties and the all-correct confidence branch, and BBH utility.
Critical weaknesses
  1. Table 1 reports GuardPO on Gemma with IJP 2.5, AutoDAN 4.0, AutoDAN-Turbo 9.0, and SafeRLHF 0.5, giving a six-set average of 2.75%. Table 2 reports “Full GuardPO” on Gemma with IJP 1.92, AutoDAN 3.85, AutoDAN-Turbo 5.77, and SafeRLHF 0.0, giving an average of 1.92%. These are not the same results.
  2. Section 4.1 states that fine-tuning uses a 10^-5 learning rate, while Appendix B states that hyperparameter selection uses 10^-6.
Questions for authors
  1. How sensitive is GuardPO to the deployed guardrail? What happens if the inference-time guardrail is not one of the four annotation guardrails, or if multiple guardrails are deployed?
  2. How is the helpfulness agreement indicator h obtained, and how reliable is it? What fraction of training examples have h = 0, and how does the Stubbornness Penalty behave on those examples?
  3. Can the authors report the distributions of C_G(x) and C_S(x,y), and a sensitivity analysis for τ_G, α, and the standardization statistics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.