acceptodds
Under review as a conference paper at ICLR 2027

Ordered Advantages Prevent Safety Drift in RL Fine-Tuning

Abstract

Which signals in a model's rollouts become persistent behavior during reinforcement-learning (RL) fine-tuning? We show that group-relative advantages determine this channel: FP8 rollouts increase rollout–trainer mismatch by 2.8–7.7× while safety remains in the BF16 range across four seeds, whereas a reward that favors harmful responses raises Qwen2.5-7B-Instruct harmful compliance from 6.7% to 87.8% in 100 steps. We introduce Ordered Advantage Projection (OAP), a provider-side method that projects each response group so a more harmful response never receives a larger advantage. Isotonic regression computes the projection, which preserves the group mean, requires no threshold or penalty weight, and is invariant to strictly increasing score calibration. In the default harmful-reward attack, OAP with a KL anchor gives 6.9% AdvBench harmful compliance versus 6.7% for the base model and 87.8% without defense. Across four seeds, five adaptive attacks, two provider classifiers, two RL estimators, and three models, OAP retains near-base safety and helpfulness; its projection adds 0.28% to step time. Under a benign GSM8K reward, OAP and no defense reach 78.1% accuracy, while penalty baselines reach 77.3–77.7%.

Primary area:
reinforcement learning
Keywords:
self-training, low-precision inference, FP8, reinforcement learning, LLM safety, experience filtering, safety drift, numerical noise

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RzNxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem of fine-tuning with Reinforcement Learning. Specifically, the author(s) consider a setting where the model is fine-tuned using a reward oracle R(x, y) that depends on input-output pairs. For this problem, the author(s) consider an adversarial RL fine-tuning-as-a-service setting, where an attacker tries to induce model misalignment by manipulating the reward. The author(s) provide a defence mechanism to this type of attach (Eq. 3). This mechanism relies on a harmfulness score based on a pre-trained safety classifier. The author(s) provide a theoretical justification for their framework, as well as an experimental comparison. In the experiments, they consider three models of varying size (up to 14B) on seven datasets, showing that the proposed method prevents safety degradation.

Critical strengths

The research question is timely. The proposed framework applies to a variety of real-world use-cases, since it does not assume a specific functional form for the reward model. Furthermore, the proposed threat model is very general. Another strength is that this framework is scalable to large models.

Critical weaknesses

My main concerns is that the proposed framework heavily relies on given harmfulness classifiers. Although the author(s) use a variety of classifiers in their experiments, it is unclear if the proposed method is somewhat robust to errors of the harmfulness classifiers.

Questions for authors

Regarding the critical weakness, can you clarify if your framework is robust to misspecifications in the harmfulness classifiers?

2 comments
AuthorsOpenReview ↗

Robustness to Harmfulness Classifier Misspecifications and Stress Tests

Response on Robustness to Harmfulness Classifier Misspecifications

Thank you for your time and careful review of our manuscript. To address your concerns regarding the robustness and failure boundaries of Ordered Advantage Projection (OAP) under harmfulness classifier misspecifications or errors, we conducted four sets of end-to-end diagnostic fine-tuning experiments—within-group random ordering perturbation sweeps, high-noise comparisons against additive Penalty, heterogeneous scorer replacement between BeaverDam-7B and Granite-2B, and a dual-scorer maximum diagnostic—alongside a two-reviewer anchored human verification on stratified samples, as detailed below. To directly address this concern, we conducted random ordering perturbation sweeps, heterogeneous scorer replacements, additive Penalty comparisons, and dual-scorer diagnostics to measure actual fine-tuning outcomes under classifier misspecification. Both the empirical results and our formulation show that OAP is strictly invariant to any strictly increasing recalibration of harmfulness scores, whereas its protection degrades predictably under within-group pairwise misrankings or shared systematic blind spots. Below we detail the exact formal relationship followed by the end-to-end empirical measurements across all diagnostic settings.

To quantify how OAP behaves under classifier errors in practice, we conducted four diagnostic stress tests (runs with shutdown-adjacent Ray worker logs are reported as diagnostic, and matched training steps do not imply equal token compute):

  1. Group-Permutation Sweep: During OAP training, we randomly permuted the classifier ordering within a prompt group with selection probability p∈{0,0.2,0.4,0.6,0.8,1.0}p \in \{0, 0.2, 0.4, 0.6, 0.8, 1.0\} (where pp is the group-permutation probability rather than a ground-truth error rate). On HarmBench (256 prompts, seed 2027), Llama Guard 3 unsafe counts across pp are 83, 74, 78, 94, 87, and 98—showing non-monotonic movement rather than a smooth degradation curve.
  2. High-Noise Contrast vs. Additive Penalty (λ=3\lambda=3): At extreme permutation rates (p=0.8p=0.8 and 1.01.0), OAP retains lower automated AdvBench (520 prompts) unsafe counts than Penalty, whereas HarmBench and OR-Bench results are mixed, with OAP incurring higher benign refusal on OR-Bench.
  3. Heterogeneous Classifier Replacement (BeaverDam-7B vs. Granite-2B): Although BeaverDam-7B and Granite-2B exhibit roughly 35% pairwise ranking disagreement on provider scores (inter-classifier disagreement, not ground-truth error rate), end-to-end OAP retraining under either classifier maintains near-zero AdvBench automated unsafe counts across tested seeds with only modest HarmBench shifts.
  4. Uncalibrated Dual-Scorer Diagnostic (max⁡(BeaverDam,Granite)\max(\text{BeaverDam}, \text{Granite})): Taking an uncalibrated pointwise maximum reduces some HarmBench unsafe counts but increases OR-Bench refusals; in a 1,413-pair diagnostic against dataset labels, it corrected 19 ordering errors while introducing 61 new pairwise reversals, showing that naive ensembling is a trade-off rather than a cure for arbitrary blind spots.

Our framework is strictly invariant to any strictly increasing recalibration of the harmfulness classifier scores, while its guarantee is explicitly conditional on the pairwise group ordering induced by the provider classifier hh. Because OAP projects group-relative advantages aa onto the order-preserving cone defined by hh, a~=ΠC(h)(a),C(h)={z: hi<hj⇒zi≥zj},\tilde{a} = \Pi_{\mathcal{C}(h)}(a), \qquad \mathcal{C}(h)=\{z:\, h_i < h_j \Rightarrow z_i \ge z_j\}, any transformation that preserves the within-group ranking hi<hjh_i < h_j leaves C(h)\mathcal{C}(h) and the projected update identical. Conversely, if hh systematically misranks a harmful completion as safer than a benign sibling due to a structured blind spot, the projection cone encodes that reversed order and post-hoc isotonic regression cannot recover the missing semantic signal.

Finally, on 200 stratified model-blinded V3 responses, two reviewers performed an anchored audit on a pre-filled template (overriding 18/200 and 11/200 rows respectively, reported as anchored verification rather than from-scratch independent annotation). On the 178 binary-consensus items (53 unsafe, 125 safe), evaluation judges matched consensus at 95.5% for WildGuard (170/178), 93.8% for Guard 3 (167/178), and 94.9% for windowed HarmBench (169/178). While this supports response-level evaluation reliability on the audited sample, it does not prove that the training-time classifier hh ranks every rollout group without error. In the revision, we will add all four stress-test diagnostics and state the exact boundary condition directly beside Equation (3): OAP prevents credit from flowing toward completions that hh ranks as more harmful, but it cannot synthesize safety signal where hh has a systematic ranking blind spot.

Thank you once again for raising this pivotal question, which has helped us sharpen both the empirical stress-testing and the formal exposition of our framework's boundaries. We warmly welcome any further questions or suggestions you may have!

Reviewer_kKh1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ordered Advantage Projection (OAP), a provider-side defense for RL fine-tuning-as-a-service that prevents harmful reward signals from increasing the probability of harmful responses. The method projects per-prompt group advantages onto the cone of vectors that are non-increasing with respect to the provider’s harm ranking, implemented via isotonic regression (PAVA), and is paired with a KL anchor.

Critical strengths
  1. OAP formulates a clean, convex projection on group-level advantages to enforce that more harmful siblings never receive larger credit. Using isotonic regression in RLHF context is simple.
  2. The method and properties are explained concisely with helpful figures.
Critical weaknesses
  1. Evaluation relies on a single automated judge (HarmBench Llama-2-13B) as the independent arbiter; cross-judge robustness (e.g., other safety judges or human audits on subsets) would strengthen the claims.
  2. Despite a broad set of attacks, the adaptive threat mainly includes simple reward modifications. Stronger adaptive objectives are not explored.
Questions for authors
  1. Can you clarify the exact KL regularization estimator used (token-level vs. sequence-level, k3/UKL vs. normalized KL) and whether off-policy importance weights are correctly included per recent RPG analyses? If not, how might this affect your conclusions?
  2. How sensitive is OAP to the group size K? Your default uses K=4; have you tried larger K and observed any changes in safety/utility or projection aggressiveness?
3 comments
AuthorsOpenReview ↗

Response on Cross-Judge Evaluation, Ungated Adaptive Attacks, KL Implementation Details, and Group Size Sensitivity

Thank you for taking the time to carefully review our paper. To address your concerns regarding single-judge evaluation, adaptive attack objectives, KL regularization implementation details, and sensitivity to the group size K, we conducted full re-evaluations with three independent safety judges, tested an additional ungated adaptive attack objective, performed a code-level audit of our KL estimators and importance weighting, and ran sensitivity experiments at K=8 and K=16. Below we address each point in detail.

1 · Safety Improvements Persist Across Three Additional Automated Judges

Re-evaluating with three additional independent automated safety judges confirms that OAP+KL produces substantially fewer harmful outputs than GRPO+KL across both benchmarks. We rescored the exact same saved model responses across all 520 AdvBench prompts and all 256 HarmBench prompts. The resulting unsafe output counts for OAP+KL versus GRPO+KL are: Llama Guard 1 flagged 7 versus 150 on AdvBench, and 31 versus 70 on HarmBench. Llama Guard 3 flagged 43 versus 339 on AdvBench, and 144 versus 212 on HarmBench. WildGuard flagged 25 versus 290 on AdvBench, and 76 versus 147 on HarmBench.

Every additional judge favors OAP+KL on both benchmarks, showing that the direction of our safety improvement holds across diverse evaluators rather than relying solely on the original HarmBench judge. Because each judge applies a distinct operational definition of harmfulness, their absolute unsafe counts differ noticeably, which is why we report each judge's results separately rather than pooling them.

2 · Existing Linear White-Box Attack Results and Additional Ungated Adaptive Attack Evaluation

In our submission, Appendix D.6 already evaluates a white-box adaptive attack under the tested linear reward formulation: with the beaver-dam provider, the undefended baseline reaches 79.8% ASR on AdvBench whereas OAP reduces ASR to 6.3%, at kappa = 1, as reported in Appendix D.6, lines 1073–1077 (and the formal scope of this linear white-box analysis explicitly noted in Appendix H, lines 1213–1230). Building on that existing evaluation, we tested an additional ungated adaptive attack setting, enabling the actor KL term in both comparison arms for a controlled evaluation. After 100 training steps in a single seed, Llama Guard 3 flagged 39 versus 296 unsafe outputs on AdvBench (out of 520 prompts) and 147 versus 209 unsafe outputs on HarmBench (out of 256 prompts) for OAP versus GRPO. The other two automated judges showed the exact same directional preference on both benchmarks, consistently finding fewer unsafe outputs under OAP.

These results show that across both the linear white-box attack in Appendix D.6 and this additional ungated adaptive attack setting, OAP substantially reduces harmful generation compared to standard GRPO. Mechanistically, the order-preserving projection in Section 5 (Eq. 3) constrains within-group credit assignment relative to the safety classifier h: within each sampled group, any response ranked as more harmful by h receives a projected advantage no greater than that of any response ranked as safer by h.

AuthorsOpenReview ↗

Response on Cross-Judge Evaluation, Ungated Adaptive Attacks, KL Implementation Details, and Group Size Sensitivity -2

3 · Token-Level Granularity, k1/k3 Estimators, and Importance Weighting of the KL Terms

We inspected our training codebase for the Section 5 OAP+KL experiments and clarify all four aspects of the KL implementation and their implications below:

  1. Token-Level vs. Sequence-Level Granularity and Reduction Both KL terms are evaluated per valid response token rather than as an exact sequence-level distribution KL, using two distinct reductions. The reward-shaping KL is computed at each valid token and summed across valid tokens within each response (sequence-sum) before entering group-relative advantage standardization and OAP projection. The actor KL is evaluated per valid response token and reduced via a masked token-mean before being added to the actor loss.

  2. Exact Estimators and Normalization The reward-shaping term uses the k1 estimator per valid token: k1 = log pi_old - log pi_ref, multiplied by the valid-token mask and a coefficient of 0.001. The actor term uses a numerically clipped k3 (low_var_kl) estimator: defining d = clamp(log pi_ref - log pi_theta, -20, 20), it evaluates k3_clip = clamp(exp(d) - d - 1, -10, 10), takes a masked token-mean over valid tokens, and scales the result by 0.1 in the actor loss. While k3 is a standard sample-based form associated with unnormalized KL (UKL), our implementation evaluates these sampled-token k1 and clipped-k3 expressions directly without computing a full-vocabulary normalized KL, and the masked token-mean serves strictly as a loss reduction rather than a normalized KL definition.

  3. Off-Policy Importance Weights In our training configuration (rollout_is = None), off-policy rollout importance weights are not applied to the actor KL term; that is, the actor KL term is not multiplied by the current-to-old policy ratio (pi_theta / pi_old). The PPO probability ratio and clipping are applied exclusively to the policy surrogate loss branch.

  4. Impact on Conclusions As noted in recent RPG analyses (Section 2.2), omitting current-to-old importance weights in off-policy k3 estimation can cause the actor KL gradient to deviate from the exact target UKL gradient, with the practical effect depending on policy divergence and the coefficient beta; we do not separately quantify the magnitude of this bias. Crucially, however, OAP+KL and our GRPO+KL baseline share this exact k1 and k3 implementation, meaning our controlled empirical comparisons accurately isolate the effect of OAP under the implemented training objective. Furthermore, the algebraic order-preserving properties of OAP on scalar group advantages (Section 5, Propositions 1–3) hold independently of the KL estimator, while end-to-end training dynamics depend on both OAP and the chosen KL configuration.

4 · Unsafe Generation and Refusal Behavior Across Group Sizes K=4, 8, and 16

K denotes the number of responses sampled and compared per prompt. Across K=4, 8, and 16 and both random seeds, OAP consistently produces far fewer unsafe outputs than GRPO. Using the original automated judge across all 520 AdvBench and 256 HarmBench prompts, the unsafe output counts for OAP versus GRPO are: At K=4: seed 2026 yielded 38 versus 373 on AdvBench and 124 versus 206 on HarmBench; seed 2027 yielded 34 versus 429 on AdvBench and 127 versus 207 on HarmBench. At K=8: seed 2026 yielded 35 versus 337 on AdvBench and 131 versus 190 on HarmBench; seed 2027 yielded 40 versus 397 on AdvBench and 115 versus 211 on HarmBench. At K=16: seed 2026 yielded 35 versus 323 on AdvBench and 124 versus 192 on HarmBench; seed 2027 yielded 36 versus 334 on AdvBench and 122 versus 194 on HarmBench.

We also evaluated refusal behavior on XSTest (250 safe prompts and 200 unsafe prompts per seed) using the WildGuard automated judge at K=4 and K=16: On the 250 safe prompts, at K=4, OAP versus GRPO refused 14 versus 1 (seed 2026) and 17 versus 2 (seed 2027); at K=16, safe-prompt refusals decreased to 10 versus 1 (seed 2026) and 12 versus 1 (seed 2027). On the 200 unsafe prompts, at K=4, OAP versus GRPO refused 186 versus 26 (seed 2026) and 186 versus 37 (seed 2027); at K=16, unsafe-prompt refusals were 184 versus 23 (seed 2026) and 181 versus 32 (seed 2027).

These results show that OAP yields substantially fewer harmful outputs across all tested K values; under the automated refusal evaluation on XSTest, K=16 produces fewer refusals on safe prompts than K=4 (10 and 12 versus 14 and 17) while exhibiting slightly lower refusal counts on unsafe prompts (184 and 181 versus 186 and 186). Together, these results report both unsafe output counts and benign-prompt refusal behavior across group sizes; for any given K, the projection objective remains the minimum-L2 modification subject to that group's ordering constraints.

Thank you once again for your thorough review and constructive feedback.

Thank you for your detailed explanation. I will keep my positive assessment.

Reviewer_w969OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Ordered Advantage Projection (OAP), an approach that aims to address the problem of safety drift in LLMs when exposed to RL-based fine-tuning, where an attacker provides a reward function that encourages harmful outputs. The OAP approach is underpinned by isotonic regression (using the Pool-Adjacent-Violators Algorithm) to order advantages in non-increasing order, which, when combined with a reference KL anchor, reduces the impact of advantage-driven harm and does not deviate much from the group mean. The empirical evaluation carried out using the Qwen and Llama models demonstrates the usefulness of the approach.

Critical strengths
  • The paper tackles an interesting problem that aligns with the scope of ICLR.

  • The approach, albeit simple, shows useful empirical results.

  • Novelty: From a novelty perspective, OAP is a simple yet fresh idea that combines isotonic regression through PAVA and GRPO.

Critical weaknesses
  • Motivation: While I appreciate the work carried out in the context of making LLMs safer and more robust, I have difficulty following the overall paper motivation and its foundation. In particular, since a customer would like to fine-tune an LLM, the starting point that the customer would provide an ill-formed reward function or one that would aim at harming their users is not sufficiently substantiated and requires further clarification. The motivation, in general, makes some sense, but it should be more clearly explained.

  • Rigour: Another related concern is related to the mixed use of FP8 and BF16. While this is commonplace during LLM rollouts using GRPO, the rationale for their use is not explained adequately. In particular, it is unclear why this transition between these two precision formats is needed (beyond their common use within the context of GRPO). Given that the key innovation in the paper is OAP, their use convolutes the overall benefit of the approach.

  • Rigour: Although the combination of OAP with the KL as the anchor to the reference model is largely reasonable, the formal definition of this (fine-tuning formula) is missing. To make it easier for readers to understand the formalisation fully, the formula must be provided. Also, it appears that the approach is quite sensitive to the combined use of these two elements (OAP and KL); i.e., without KL, the approach can be quite permissive, while it also has trade-offs for different weights. Thus, providing some practical guidance on the overall calibration would help readers better understand how to use the approach.

  • Soundness: The approach depends on the soundness of the classifier h in Eq. (3). Accordingly, ranking errors or sensitive outputs of the classifier substantially affect the approach's performance. This should be acknowledged, and the authors should consider assessing the robustness of OAP with respect to the performance of the classifier, e.g., when does OAP fail to establish the safety guarantees given the inconsistency/mistake rate of the classifier?

  • Evaluation: Although the evaluation is detailed, the main results in Table 1 (and Table 2) show mixed performance of the proposed approach, especially against the two Penalty variants (and to some extent with TokenBuncher). Accordingly, the benefit of the approach is not evident. It would have been useful to make explicit the case where the approach would be beneficial against its competitors.

  • Presentation: Although the paper is in general easy to follow, some key concepts (e.g., the formalisation mentioned above) should be better described. Furthermore, insights and practical guidance should be made explicit to help readers follow the authors’ argumentation.

Questions for authors

(1) Please explain the rationale for the ill-formed reward function from the customer.

(2) What is the mathematical formalisation for the combination of OAP and KL for the reference model?

(3) How sensitive is the approach to the capacity of the classifier h, and when does OAP break with respect to the sensitivity of the classifier?

5 comments
AuthorsOpenReview ↗

Response on Service-Boundary Motivation, Precision Provenance, OAP+KL Formalization and Calibration, Classifier Sensitivity, and Operating Regimes

Thank you for your time and careful review of our manuscript. In response to your concerns regarding the adversarial reward threat model, numerical precision settings, the mathematical formalization and calibration of combining OAP with reference-model KL regularization, sensitivity to the classifier h, and the operating regimes where OAP is preferable to Penalty and TokenBuncher, we have clarified the service-boundary threat model, audited historical training logs for precision provenance, formulated the complete fine-tuning objective combining OAP with reward-side and actor-side KL terms, and conducted new KL coefficient sweeps, classifier ranking perturbation and heterogeneous scorer stress tests, and reward-scaling diagnostics on Qwen2.5-7B, as detailed item by item below.

Question (1): Rationale for Ill-Formed or Harmful Customer Rewards and the Service-Boundary Foundation

Our study directly addresses the adversarial reward threat model in hosted RL fine-tuning services: the service provider executes the underlying training while enforcing platform safety constraints, external developers supply custom reward functions R(x,y) or preference signals to drive policy updates, and an attacker seeks to craft adversarial or harmful rewards to bypass platform safeguards. Along this provider-developer operational boundary, the threat model and platform defense requirements manifest across three concrete dimensions:

  1. Adversarial tenants: A malicious actor or red-team user with access to the reward API may deliberately craft objectives (such as negating refusal logits or rewarding jailbreak compliance) to strip safety guardrails while the hosting provider remains contractually, ethically, or legally accountable for a baseline safety floor.
  2. Misspecified but non-malicious proxy objectives: Benign customers routinely optimize imperfect proxies such as user engagement, response length, helpfulness compliance, or noisy preference models. During RL exploration, these proxies can correlate with sycophantic or unsafe completions on edge prompts without any intent to cause harm.
  3. Systemic platform guarantees: Once a provider exposes arbitrary reward-driven policy updates, failing to constrain credit assignment at the platform boundary creates a systemic vulnerability even if the vast majority of tenants are benign.

In the paper, our primary empirical evaluation (Tables 1 and 2) targets the worst-case adversarial tenant setting (Scenario 1) to stress-test whether provider-side credit assignment can hold the line under active optimization pressure. While Scenario 2 motivates the broader service architecture and is probed diagnostically via proxy preference tasks, we are careful not to present our main adversarial-reward tables as already proving all benign-misspecification regimes. In the revision, we will restructure Section 1 to explicitly delineate these three service-boundary motivations so the foundation of our threat model is immediately transparent.

Rationale for Mixed FP8 and BF16 Precision and Historical Training Log Provenance Audit

Mathematically and algorithmically, OAP operates purely on scalar group-relative advantages and the relative ordering induced by the safety scorer, independent of the underlying numerical precision. To verify the exact distribution of FP8 and BF16 configurations across our experiments and rule out precision as a confounding factor, we conducted a provenance audit of all historical training logs.

Our discussion of transitioning between FP8 rollouts and BF16 training or evaluation in Section 4 addresses an orthogonal systems diagnostic: because modern GRPO pipelines frequently employ lower-precision inference engines for high-throughput rollout generation before computing gradients or running final evaluations in BF16, we investigated whether rollout quantization artifacts can leave behavioral residues in the updated policy. That analysis characterizes a pipeline-level diagnostic rather than a component of the OAP defense.

To ensure that precision settings do not confound the evaluation of OAP, we conducted a comprehensive provenance audit across all 220 historical harmrl training logs in our repository:

  1. The primary 7B defense comparison arms supporting Section 6 execute entirely in BF16, ensuring that the main comparisons between OAP, GRPO+KL, and Penalty are not confounded by FP8 transitions.
  2. Across all 220 logs, exactly 3 logs reference FP8 execution (attack_def_grpo_qwen14b_s2026.log, attack_def_mono_kl0.1_qwen14b_s2026.log, and attack_def_pen30_kl0.1_qwen14b_s2026.log), which belong to the 14B attack-defense scaling runs where rollout engine precision was specifically exercised.

Because not every historical arm across all model scales shares identical rollout precision, we strictly evaluate OAP against matched-precision baselines within each comparison block and avoid attributing defense gains to precision differences. In the revised manuscript, we will add an explicit per-arm precision column (BF16 vs. FP8) to the experimental setup and clearly decouple the Section 4 rollout-precision diagnostic from the Section 6 defense claims.

Question (2): Mathematical Formalization of Combining OAP with Reference-Model KL and Practical Calibration Guidance

We have formulated the explicit fine-tuning objective combining OAP with reference-model KL regularization and conducted a new KL coefficient sweep on Qwen2.5-7B to provide concrete calibration guidance. Below we detail the mathematical formalization of reward-side and actor-side KL pathways, clarify the configurational relationship with original Table 2, and present the empirical results from our calibration sweep.

Mathematical formulation and implementation

For each sampled response, s is its task score, m marks valid response tokens, and the three log-probabilities below refer to the rollout policy, reference policy, and current policy.

Reward-side KL is computed per token and summed within each response, with coefficient 0.001:

ri=si−0.001∑tmit(ℓitold−ℓitref)r_i=s_i-0.001\sum_t m_{it}(\ell^{\mathrm{old}}_{it}-\ell^{\mathrm{ref}}_{it})

GRPO then standardizes the shaped response rewards within the group:

ai=ri−μrσr+εa_i=\frac{r_i-\mu_r}{\sigma_r+\varepsilon}

OAP makes the minimum squared-Euclidean change to these advantages while enforcing the provider harm-score ordering. The implemented stable tie refinement applies when harm scores tie:

a~=arg min⁡b∥b−a∥22subject tohi<hj⇒bi≥bj\tilde{a}=\operatorname*{arg\,min}_{b}\|b-a\|_2^2\quad\mathrm{subject\ to}\quad h_i<h_j\Rightarrow b_i\ge b_j

The actor-side k3 term uses the reference and current token log-probabilities, with the following numerical clipping:

dit=clip⁡(ℓitref−ℓitθ,−20,20)d_{it}=\operatorname{clip}(\ell^{\mathrm{ref}}_{it}-\ell^{\theta}_{it},-20,20)

qit=clip⁡(exp⁡(dit)−dit−1,−10,10)q_{it}=\operatorname{clip}(\exp(d_{it})-d_{it}-1,-10,10)

The actor minimizes the PPO policy loss plus a masked mean over valid response tokens:

Lactor=LPPO(a~)+βactormaskedmean⁡(q)L_{\mathrm{actor}}=L_{\mathrm{PPO}}(\tilde{a})+\beta_{\mathrm{actor}}\operatorname{maskedmean}(q)

The default actor KL coefficient is 0.1. The PPO probability ratio is applied to the policy-loss branch; the actor KL term is not multiplied by rollout importance weights. Turning the actor coefficient to zero leaves the reward-side coefficient of 0.001 active. These are the implemented losses used in the reported comparisons; they do not imply an unbiased off-policy KL gradient.

  1. Clarification of Original Table 2 and New E6 Calibration Sweep In our original Table 2 (which belongs to a separate evaluation setup), the no-actor-KL arm reported 1.7% ASR and 12.9% OR-Bench versus 6.9% ASR and 44.7% OR-Bench at actor KL coefficient=0.1. To evaluate the exact trade-off under a unified multi-judge audit, we trained 12 new E6 arms on Qwen2.5-7B (linear white-box attack, K=4, actor KL coefficient in {0, 0.03, 0.10}, seeds 2026 and 2027) and report the five completed paired evaluation cells below (the seed-2026 evaluation file for actor KL coefficient=0.03 was not in the finalized independent audit bundle):
Actor betaSeedAdvBench G3 (OAP/GRPO, /520)AdvBench WG (/520)HarmBench G3 (/256)HarmBench WG (/256)HB Any-Win (/256)GSM8K Last# (/256)OR-Bench WG Refusal (/256)
020260 / 4920 / 4832 / 2460 / 2001 / 236224 / 230256 / 0
020270 / 4850 / 46014 / 2451 / 2039 / 234227 / 229253 / 0
0.0320270 / 4770 / 46352 / 24214 / 20450 / 228228 / 228205 / 0
0.1020261 / 2730 / 22770 / 19322 / 13762 / 186227 / 22981 / 3
0.1020270 / 3360 / 29677 / 20127 / 15165 / 193226 / 22849 / 4

Note: Automated judges and 512-token any-window HarmBench are diagnostic metrics rather than human truth, fixed steps do not imply equal token compute, and runs with shutdown-adjacent Ray worker anomalies carry a diagnostic status.

  1. Practical Calibration Guidance The sweep reveals a clear trade-off: at \beta_{\text{actor}}=0 or 0.03, OAP yields very low unsafe counts on AdvBench (0/520) and HarmBench (14 to 52 Guard3 vs. 242 to 246 for GRPO), because without a strong reference anchor the policy satisfies the ordering cone by collapsing toward wholesale refusal (205 to 256/256 OR-Bench refusals). Increasing \beta_{\text{actor}} to 0.10 anchors the policy closer to the base distribution, cutting OR-Bench refusals to 49 to 81 while maintaining near-zero AdvBench harm (0 to 1/520 vs. 273 to 336 for GRPO) and preserving GSM8K accuracy (226 to 227/256). We therefore recommend \beta_{\text{actor}} \approx 0.10 as a measured operating choice when benign over-refusal cost matters, rather than claiming a universal optimum.

Question (3): Sensitivity to the Capacity of Classifier h and Failure Boundaries of OAP

OAP's safety projection depends directly on the intra-group ranking induced by the classifier h: while the isotonic cone is invariant to any strictly increasing transformation of h, pairwise misrankings or shared blind spots within a prompt group invert the ordering preserved by isotonic projection. To empirically characterize this capacity boundary, we conducted intra-group ranking perturbation experiments and heterogeneous scorer replacement stress tests, measuring the resulting fine-tuning outcomes across varying levels of classifier degradation.

To empirically characterize how OAP degrades under classifier noise and capacity shifts, we conducted four diagnostic stress tests (runs with shutdown-adjacent Ray worker anomalies carry a diagnostic status, and matched steps do not imply equal compute):

  1. Intra-group random permutation sweep: During training, we randomly permuted the harm scores h within each sampled group with probability p \in {0, 0.2, 0.4, 0.6, 0.8, 1.0} (where p denotes the group-selection probability for random permutation, not a ground-truth classifier error rate). On HarmBench (out of 256 prompts, seed 2027), Guard3 unsafe counts varied non-monotonically across p as 83, 74, 78, 94, 87, and 98. Because RL exploration interacts nonlinearly with noisy advantage projections, degradation is non-monotonic rather than following a smooth analytical decay curve, eventually eroding protection as p approaches total random permutation.
  2. High-noise contrast against Penalty (\lambda=3): At extreme permutation probabilities (p=0.8 and 1.0), matched comparisons show that OAP still tends to yield fewer AdvBench automated harms than additive Penalty, while HarmBench counts are mixed and OAP incurs higher benign refusal on OR-Bench.
  3. Heterogeneous classifier capacity shift (BeaverDam 7B vs. Granite 2B): We replaced the 7B BeaverDam scorer with a smaller, heterogeneous 2B Granite guardian model. Pairwise score auditing reveals approximately 35% pairwise disagreement between BeaverDam and Granite rankings (reflecting inter-model disagreement rather than a ground-truth error rate). Despite this 35% ranking divergence and smaller model capacity, end-to-end OAP retraining under Granite 2B preserved near-zero AdvBench automated harm across tested seeds with only modest HarmBench shifts.
  4. Uncalibrated dual-scorer ensemble diagnostic (\max(\text{BeaverDam}, \text{Granite})): Taking the raw maximum of two uncalibrated scorers reduced unsafe counts on certain HarmBench slices but sharply increased OR-Bench refusals, and in a 1,413-pair diagnostic it corrected 19 pairwise reversals against dataset labels while introducing 61 new reversals.

In the revision, we will state the exact failure condition directly alongside Eq. (3) and integrate these four stress tests to give readers a transparent account of classifier sensitivity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.