acceptodds
Under review as a conference paper at ICLR 2027

Improving Safety Alignment and Health Safe Completion via Query-Specific Safety Rubrics in OPSD

Abstract

Safety alignment methods have improved resistance to harmful requests, but can still exhibit harmful compliance or unnecessary refusal of benign requests; their ability to provide useful assistance on borderline health queries also remains underexplored (health safe completion). One recent method, OPSA, applies on-policy self-distillation (OPSD) with hard-refusal-oriented teacher privileged information (PI) for harmful queries; in our evaluation, it improves safety but substantially increases over-refusal and reduces health safe completion. Relaxing this PI improves helpfulness but raises harmful compliance, showing that label-level instructions cannot reliably distinguish allowed assistance from prohibited content for each query. We therefore propose Query-Specific Safety Rubrics in OPSD (QSSR-OPSD), which selects and merges rules spanning truthfulness, harmlessness, and helpfulness into general principles, then operationalizes them as query-specific rubrics specifying content to include and avoid. From a probabilistic view of the response distribution, we show that operationalizing general principles into rubrics shifts probability mass toward a set of safe and useful responses when the rubrics reduce penalties on this set more than on other responses, which helps explain our observations both before and after QSSR-OPSD training. Across four backbones, QSSR-OPSD achieves the highest Composite Safety Score (CSS) and health Safe Completion Rate (SCR) in our main results. On Qwen3-8B, it improves CSS from 85.34% to 92.07% and SCR from 36.06% to 74.79% over OPSA, while reducing over-refusal and attack success.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, Health Safe Completion, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bytuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to effectively apply the training paradigm of on-policy self-distillation (OPSD) for safety alignment while preserving utility by reducing over-refusals. The authors claim the existing OPSD safety alignment methods (such as OPSA) condition a frozen teacher model on coarse, label-level privileged information (a hard refusal for harmful queries and a direct-answer instruction for benign ones), resulting in increased over-refusal rate (ORR) and degradation in safe completion rate (SCR) on borderline health requests. The paper proposes Query-Specific Safety Rubrics in OPSD (QSSR-OPSD) to address these failure modes, using query-specific safety rubrics built dynamically for each query from 100 RuleAdapter rules by GPT-5.4. The teacher then sees a meta instruction, a label-conditioned safety statement, and the rubrics, while the student sees only the query. The authors argue, via a KL-regularized penalty minimization framework, that operationalizing principles into rubrics shifts probability mass toward safe, useful responses primarily by suppressing unsafe completions. Across four models and various alignment methods, QSSR-OPSD achieves the highest composite safety score (CSS) and SCR while preserving general reasoning performance.

Critical strengths
  1. The paper offers a comprehensive, strong evaluation and shows clear improvement on the important safety-alignment trade-off. The authors consider harmful compliance, over-refusal, and safe completion together rather than in isolation. The evaluation covers four models with different sizes and model families across diverse safety datasets, which makes the results more reliable.

  2. The paper provides useful evidence on how privileged information (PI) design can affect OPSD's effect on safety alignment. The ablation of PI and the comparison of rubric-conditioned SFT and OPSD may give practical guidance to future works applying OPSD to safety alignment.

Critical weaknesses
  1. The measurement of utility is limited. Helpfulness is assessed through a single health benchmark via SCR. On general benign queries, ORR records only whether the model refused, so a vague or incorrect answer counts as success. On harmful queries, the label policy asks for detailed safe alternatives, but the trained student is not evaluated on whether it provides them. Evaluating answer quality on general benign and harmful queries would directly test the paper’s main claim.

  2. QSSR-OPSD receives supervision that other OPSD-based baselines do not. It adds per-query guidance generated by GPT-5.4 across the training set, so part of its advantages may come from distilling a stronger model’s judgment rather than from the method itself. The self-generated rubric ablation (Table 10) is encouraging but is reported only for Qwen3-8B, so it remains unclear how much the gains depend on the external model.

Questions for authors
  1. Can you evaluate answer quality on general benign queries and on harmful queries outside health? This would directly test whether the student provides useful safe alternatives.

  2. Can you report self-generated rubric results on the other backbones? How much does the performance depend on a strong external model like GPT-5.4?

Reviewer_rwSZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the trade-off among harmful compliance (ASR), over-refusal (ORR), and safe completion on borderline health queries (SCR). The proposed QSSR-OPSD uses GPT-5.4 to select rules from the RuleAdapter pool for each training query, merge them into query-conditioned principles, and operationalize these into at most six rubrics with must-include and must-not-include criteria. The rubrics, together with a label-conditioned safety statement, form the teacher's PI, and the student is trained with OPSA's mixed-KL objective. Experiments cover four backbones, two training-data sources, safety/refusal benchmarks, a health benchmark, and reasoning tasks.

Critical strengths
  1. Moving from hard refusals to safe completions without reopening jailbreak risk is a critical problem.
  2. According to the experiments, QSSR-OPSD achieves the best CSS and SCR on all four backbones (Table 1), and the trade-offs are acknowledged honestly. The method changes only the teacher context, so it plugs easily into existing OPSD pipelines.
Critical weaknesses
  1. The novelty is incremental. Rubrics as OPSD privileged information (RGSD, RuPI), safety OPSD with PI (OPSA), and the rule pool (RuleAdapter) are all existing methods, the contribution is mainly their combination for safety.
  2. The query-specific rubrics do not drive the health safe-completion gains. In Table 3, replacing generic principles with rubrics lowers ASR (0.1163→0.0379) but raises ORR (0.0726→0.1207) and lowers SCR (0.7591→0.7480). Most of the SCR gain over OPSA is already obtained with the meta instruction alone, and S(x,ℓ)S(x,\ell) and principles add the rest. Rubrics mainly trade benign helpfulness for attack robustness.
  3. The comparison with OPSA and the aggregate metric favor the method. OPSA's hard-refusal PI hurts ORR and SCR by design, so it is a weak baseline on exactly these metrics.
  4. The pilot PI was refined "based on failure modes observed ... on the evaluation datasets", so the design was selected using test performance. In addition, SafeChain prompts are drawn from WildJailbreak, whose evaluation split supplies 2,000 of the 3,253 harmful test prompts. Although the splits are nominally disjoint, no held-out validation set or deduplication is described. Please clarify how the final PI was selected and whether near-duplicates were removed.
  5. Health safe completion is claimed as one of main contributions, but it rests on a single benchmark and a single primary judge.
Questions for authors

See Weakness.

Reviewer_wS3EOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This model investigates using on-policy self-distillation for alignment on healthy questions. Specifically, OPSA by itself has problems where it can sharply reduce harmful compliance but cause heavy over-refusal and unhelpful answers to borderline health-related questions. The authors argue that using a single instruction for each label is too coarse, and instead propose QSSR OPSD, which gives the hidden teacher a custom safety rubric instead for every training query used. These are built using GPT-5.4 from a pool of established safety rules that specify what the answer (a safe answer) should include, such as useful, helpful, safe alternatives, and what it shouldn't include, such as dangerous details and fake information. This student never sees the rubric and learns the behavior by standard on policy distillation. The authors tested 4 models (Qwen3 4B, 8B, 14B, and Llama 3.1 8B and this method achieves the best overall safety score and health-safe competition while dropping overrefusal and keeping reasoning abilities intact.

Critical strengths
  • There are decently strong results showing that QSSR OPSD is the best in terms of composite safety score and health-safe completion rate on all the backbones, all while reducing over-refusal and keeping reasoning ability
  • There's a wide range of safety datasets used (8 safety datasets), and there are different judges used, 3 decoding seeds, 3 training seeds, and a second judge check on health-safe evaluation.
  • Health analysis is fine-grained and understandable. The paired rescue and regression rates and helpfulness shifts relative to the base show a clear and rich picture of the method's strengths in this domain
  • The ablations are well done and disambiguate the effects of the hidden prompt and also show the trade-off between harmful compliance and over-refusal.
Critical weaknesses
  • The novelty is limited. The training recipe is largely unchanged from OPSA. The rubric itself doesn't count towards a strong contribution to alignment training, but it has been done in RGSD and RuPI (https://arxiv.org/abs/2606.12507, https://arxiv.org/abs/2608.02948). The contribution is mainly a recombination of existing methods, with a specific domain of health as an application/focus area of analysis
  • Most of the gains over OPSA seem to come from replacing hard refusal with a soft safety policy and not from the rubrics themselves. Going directly from principles to rubrics seems to add only 0.015 to the safety score, lowers health-safe completion slightly, and nearly doubles over refusal. There should be an OPSA baseline using the same soft policy as another ablation to check this concern
  • Hidden prompt revisions are in the pilot and the choice of epochs were informed by the test set/benchmark's performance which risks leakage.
  • The label policy closely mirrors the health benchmark's grading, which also can be a sign of leakage and inflate the results gains
  • The theory section (prop. 1 and prop. 2) are weak. prop. 1 is a standard result and prop.2 assumes its conclusion through a condition that is completely unverified. In fact, part of the paper's own diagnostic results contradicts that condition since over-refusals gain likelihood under rubrics.
  • Over-refusal remains about five times the base model’s rate (0.12 vs 0.026), which the equally weighted composite score obscures.
  • Health claims have very narrow scope since it relies only on one benchmark of 1000 queries graded by an LLM judge and no human verification of rubric quality or response quality. For a paper with "health" in the title and as a main contribution, this seems very weak.
Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.