- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the problem of fine-tuning with Reinforcement Learning. Specifically, the author(s) consider a setting where the model is fine-tuned using a reward oracle R(x, y) that depends on input-output pairs. For this problem, the author(s) consider an adversarial RL fine-tuning-as-a-service setting, where an attacker tries to induce model misalignment by manipulating the reward. The author(s) provide a defence mechanism to this type of attach (Eq. 3). This mechanism relies on a harmfulness score based on a pre-trained safety classifier. The author(s) provide a theoretical justification for their framework, as well as an experimental comparison. In the experiments, they consider three models of varying size (up to 14B) on seven datasets, showing that the proposed method prevents safety degradation.
Critical strengths
The research question is timely. The proposed framework applies to a variety of real-world use-cases, since it does not assume a specific functional form for the reward model. Furthermore, the proposed threat model is very general. Another strength is that this framework is scalable to large models.
Critical weaknesses
My main concerns is that the proposed framework heavily relies on given harmfulness classifiers. Although the author(s) use a variety of classifiers in their experiments, it is unclear if the proposed method is somewhat robust to errors of the harmfulness classifiers.
Questions for authors
Regarding the critical weakness, can you clarify if your framework is robust to misspecifications in the harmfulness classifiers?
2 comments
Thank you once again for raising this pivotal question, which has helped us sharpen both the empirical stress-testing and the formal exposition of our framework's boundaries. We warmly welcome any further questions or suggestions you may have!
Robustness to Harmfulness Classifier Misspecifications and Stress Tests
Response on Robustness to Harmfulness Classifier Misspecifications
Thank you for your time and careful review of our manuscript. To address your concerns regarding the robustness and failure boundaries of Ordered Advantage Projection (OAP) under harmfulness classifier misspecifications or errors, we conducted four sets of end-to-end diagnostic fine-tuning experiments—within-group random ordering perturbation sweeps, high-noise comparisons against additive Penalty, heterogeneous scorer replacement between BeaverDam-7B and Granite-2B, and a dual-scorer maximum diagnostic—alongside a two-reviewer anchored human verification on stratified samples, as detailed below. To directly address this concern, we conducted random ordering perturbation sweeps, heterogeneous scorer replacements, additive Penalty comparisons, and dual-scorer diagnostics to measure actual fine-tuning outcomes under classifier misspecification. Both the empirical results and our formulation show that OAP is strictly invariant to any strictly increasing recalibration of harmfulness scores, whereas its protection degrades predictably under within-group pairwise misrankings or shared systematic blind spots. Below we detail the exact formal relationship followed by the end-to-end empirical measurements across all diagnostic settings.
To quantify how OAP behaves under classifier errors in practice, we conducted four diagnostic stress tests (runs with shutdown-adjacent Ray worker logs are reported as diagnostic, and matched training steps do not imply equal token compute):
Our framework is strictly invariant to any strictly increasing recalibration of the harmfulness classifier scores, while its guarantee is explicitly conditional on the pairwise group ordering induced by the provider classifier h. Because OAP projects group-relative advantages a onto the order-preserving cone defined by h, a~=ΠC(h)(a),C(h)={z:hi<hj⇒zi≥zj}, any transformation that preserves the within-group ranking hi<hj leaves C(h) and the projected update identical. Conversely, if h systematically misranks a harmful completion as safer than a benign sibling due to a structured blind spot, the projection cone encodes that reversed order and post-hoc isotonic regression cannot recover the missing semantic signal.
Finally, on 200 stratified model-blinded V3 responses, two reviewers performed an anchored audit on a pre-filled template (overriding 18/200 and 11/200 rows respectively, reported as anchored verification rather than from-scratch independent annotation). On the 178 binary-consensus items (53 unsafe, 125 safe), evaluation judges matched consensus at 95.5% for WildGuard (170/178), 93.8% for Guard 3 (167/178), and 94.9% for windowed HarmBench (169/178). While this supports response-level evaluation reliability on the audited sample, it does not prove that the training-time classifier h ranks every rollout group without error. In the revision, we will add all four stress-test diagnostics and state the exact boundary condition directly beside Equation (3): OAP prevents credit from flowing toward completions that h ranks as more harmful, but it cannot synthesize safety signal where h has a systematic ranking blind spot.