- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how preference optimization affects different language model capabilities and how to retain those that degrade during training. It proposes RETAINPO, which uses task-specific monitoring to identify capability loss and allocates a limited retention budget to selected capabilities. The paper evaluates the approach under DPO and SimPO, comparing it with other retention strategies and examining its effects on safety and general capabilities.
Critical strengths
S1: Identifies a practically relevant monitoring failure
Section 3.2 reports rising margins across all capabilities despite declining safety. Appendix K demonstrates the divergence on the same question, weakening the alternative explanation that certificate questions are simply unrepresentative of evaluation questions. Distinguishing failure as a detector from effectiveness as an intervention is a valuable empirical insight beyond proposing another regularizer.
S2: Data roles and endpoint evaluation are clearly organized
Section 4.1 separates sentinels, certificate banks, and evaluation panels. Appendices A/M reserve sealed tests and describe deduplication and generation protocols. Table 9 includes several benchmarks, including XSTest, to assess whether restored safety merely reflects more refusals. Such separation is particularly important for a monitoring-driven method.
S3: Targeted protection is tested directly
The single-capability transfer matrix in Appendix D compares diagonal and off-diagonal effects at a common dose. The safety targeting contrast has a positive lower bound. The authors also acknowledge that code lacks clear positive targeting evidence and that instruction certificates may act as in-distribution supervision. These qualifications make the mechanism analysis more credible.
S4: Baselines and statistical procedures are relatively well documented
Table 1 covers several retention strategies. Appendix G describes paired bootstrap analysis, some simultaneous inference, and a GPU nondeterminism floor. The main text acknowledges that Static suffices when safety is the only degrading capability under DPO, rather than treating dynamic allocation as universally necessary.
Critical weaknesses
W1: The budget comparison is not fully fair
Under SimPO, RETAINPO uses more retention tokens than Static and Uniform: 3.05% versus 1.89% (Section 5.4 and Appendix N). Its better results may therefore come partly from extra training rather than better allocation. Under DPO, Static already performs similarly to RETAINPO. A comparison using the same total retention budget would make the benefit of dynamic allocation clearer.
W2: The online experiments used an earlier, flawed code verifier
Appendices A/H explain that the earlier verifier incorrectly marked some correct code as wrong, causing unnecessary code protection during training. Rescoring the final outputs fixes the reported scores but does not undo those training decisions. Rerunning at least one seed with the corrected verifier would help confirm the results.
W3: Monitoring adds substantial cost
The extra retention tokens are few, but monitoring generates tokens equivalent to 19.5% of the preference tokens and takes about 2.4 A100 hours per run (Appendix F). This cost should be included when assessing efficiency. The paper should report the full cost of monitoring, certificate construction, and training, and compare it with simpler fixed protection.
W4: Better safety comes with remaining trade-offs
On the sealed tests, RETAINPO's HarmBench safety score remains below the reference model: 66.2 versus 77.5. Its compliance on safe XSTest prompts also falls from DPO's 72.3 to 59.2, suggesting more unnecessary refusals (Appendix M, Table 9). These trade-offs should be more prominent in the main text. The truthfulness results are mainly multiple choice, so they do not establish better factual accuracy in open-ended answers.
W5: Detection reliability and generalization need more evidence
The detector repeatedly checks several capabilities, but its threshold does not control false alarms across all checks. Different sentinel-set sizes may also make some capability losses easier to detect than others. A small sensitivity study of thresholds and sentinel sizes would help. Experiments use only one base model, with three seeds that change data order, so broader generalization remains uncertain.
Questions for authors
- Under exactly the same total retention-token quota, does dynamic allocation still outperform Static/Uniform on SimPO?
- How do active sets, actual spending, and final outcomes change when training is rerun with the corrected code verifier?
- What is the end-to-end cost including bank generation, the shared trigger, all detection, and waiting time?
- Would treating XSTest compliance as another capability to protect change the conclusions about safety gains?
- How sensitive are false activations and missed losses to small changes in thresholds, sentinel sizes, and monitoring frequency?
5 comments
Response to Reviewer FHxW (2/2)
Q3. What is the full cost, including monitoring and waiting? (W3)
A3. Appendix F and Table 6 include the following components. Service waiting time is reported here separately from GPU compute:
| Component | DPO | SimPO |
|---|---|---|
| Banks, once and shared | 0.7 A100 h + 0.5 L40S h | Same banks |
| Initial trigger | 0.7–1.1 A100 h per seed, shared | Online from 10% |
| Online detection per run | 1.95–2.66 A100 h, 6–7 calls | 3.83–4.02 A100 h, 9 calls |
| Training, original-run means in Table 6, excluding waits | RetainPO 2.2, Static 2.3, DPO 2.2 H200 h | RetainPO 1.7, Static 1.6 H200 h |
| Waiting for detector service | 23–26 wall-clock h | 15–29 wall-clock h |
Detection generates 17.8–20.4% of preference tokens under DPO and 26.5–27.4% under SimPO. Static avoids continued monitoring and is cheaper under DPO. Under SimPO, the extra compute enables reallocation to later math loss, with the matched-spending benefit reported above. The long waits reflect the shared service queue, not additional GPU computation.
Q4. How should the safety and compliance trade-off be interpreted? (W4)
A4. Section 5.3 now discusses the joint sealed results in Table 12. RetainPO combines SafeRLHF 97.8 with MC2 52.2, versus reference 98.4/36.4 and DPO 92.0/51.9. Higher-beta DPO and BAPO retain strong safety but have MC2 34.2 and 34.3. HarmBench is only partially recovered (66.2 versus reference 77.5), and benign compliance is 59.2 versus DPO's 72.3. We discuss this refusal trade-off explicitly and consistently call the truthfulness measure multiple-choice MC2. These are submitted-run sealed results; corrected runs are separate in Table 8.
Protecting benign compliance would require its own disjoint sentinels and certificates. It is a sensible additional target, but sealed XSTest cannot be reused for training and that intervention has not been evaluated.
Q5. How sensitive is detection, and what generalizes? (W5)
A5. We addressed detection sensitivity and broader evaluation separately. First, to test sensitivity to threshold, cadence and sentinel size, we replayed corrected-verifier DPO scores over thresholds 1.28, 1.645, 1.96 and 2.33. With full sentinel sets, safety is detected in every seed and no other capability is flagged in 150 checks at each threshold. Checking every other boundary gives 0/90 flags, but delays mean first safety detection from update 111 to 127. With random half-sized sets, flags occur in 1.4% of checks over 1,000 draws at 1.645, falling to 0.1% at 2.33 with later detection (Table 7).
Here a flag is measured on a capability without supported held-out loss. These are sensitivity results, not familywise false-alarm guarantees. The pointwise bound in Eq. 2 remains pointwise. Replay measures decisions rather than the trajectories that retraining under another cadence would produce.
Second, to examine whether the loss pattern is specific to the original preference dataset, we evaluated the HH-RLHF diagnostic. The unprotected run shows a different pattern: at 40%, MC2 falls 7.5 points with pointwise interval support, and safety falls 1.6 with simultaneous six-capability support. We then tested protection by running Static and RetainPO from the same 10% checkpoint with the corrected verifier. RetainPO spends 0.81% of preference tokens versus Static's 4.33%, with worst-capability point changes of −3.9 and −4.2 (DPO: −4.7). Static retains more safety but loses more code, and MC2 remains below reference under both. The six-capability table is in our reply to 8xzw.
This comparison therefore adds a second dataset with a different loss and protection pattern. Because it uses one seed and unequal realized spending, the completed three-seed matched-spending SimPO test remains the direct evidence for adaptive allocation.
Follow-up on the completed checks and revisions
Thank you again for the care you have taken in reviewing our work. We hope the completed reruns and comparisons have addressed the evaluation concerns you raised. We would like to summarize how each check bears on those concerns.
First, because rescoring alone cannot undo the verifier's earlier training decisions, we reran both affected DPO seeds with the corrected verifier inside training. The erroneous code activations disappear, while held-out safety stays within 0.8 points of reference. Seed 44's code point estimate remains below reference, so we retain that residual in the comparison rather than describe recovery as complete.
Second, to make the allocation comparison fair at the same retention spending, we matched tokens and summed penalty coefficients per interval. RetainPO has a smaller worst-capability loss than Static and Uniform in all three SimPO seeds, supporting an allocation benefit under the same realized spending schedule.
Finally, we report monitoring cost separately from retention spending and make the safety/compliance trade-off explicit. We also distinguish the detector's measured sensitivity from a simultaneous false-alarm guarantee. We would be glad to discuss any remaining question about these checks or their interpretation.
Response to Reviewer FHxW (1/2)
Thank you for highlighting the distinction between detecting capability loss and intervening on it, and for the careful checks on the verifier.
Both affected DPO seeds have now been rerun with the corrected verifier inside training. The new SimPO comparison also matches realized retention spending, including the summed penalty coefficient.
A1. The three-seed SimPO control matches RetainPO's tokens and summed penalty coefficient in each interval, including abstentions. Matching coefficients accounts for Eq. 6's normalization by active capability count. Static allocates to its frozen safety set, while Uniform divides the allowance over all six capabilities.
Total token differences are at most 0.003%, and coefficients agree to floating-point precision. Uniform's per-interval difference is at most 0.008% due to realizable bank quotas. At 3.05% spending, worst-capability changes are −4.5 for RetainPO, −5.5 for Static and −7.4 for Uniform. RetainPO improves over Static by 0.3/0.5/2.1 points in seeds 42/43/44, and over Uniform by 2.0/2.7/3.9. Figure 6 and Table 14 give the complete results, also summarized in our general response. This tests allocation conditional on the same realized schedule. Static remains sufficient under DPO's stable safety-loss target.
A2. Replay reproduces all 47 original decisions. Correcting the verifier changes three: seed 43 at 50%, and seed 44 at 60% and 90%. DPO seed 42 and all 27 SimPO decisions are unchanged in replay.
We reran both affected seeds with the corrected verifier inside the online detector, from the shared 10% checkpoint with optimizer state. Old and corrected endpoints, and the reference, are evaluated on H200 (Table 8):
The erroneous code activations disappear, spending falls to 0.81% in both runs, and safety stays within 0.8 points of reference. Seed 44's corrected code score remains below reference (49.2 versus 50.8 in the table). Its loss lower bound is −1.0, so it does not meet the supported-loss criterion, although the point degradation remains. Static is 1.2 points below reference in that seed, with lower bound −1.3.
Sealed evaluations are also complete: corrected seeds 43/44 score 96.7/96.7 on SafeRLHF, 67.9/66.7 on HarmBench and 57.2/57.6 on benign XSTest compliance. Table 8 compares them with the original runs and hardware-matched reference, including worst-capability changes. The main table retains submitted runs and Appendix H gives the correction comparison.
We also corrected Figure 5's caption and Appendix N: the SimPO seed-44 code activation remains under the fixed verifier, with supported sentinel loss 0.71. Replay and online reruns are now distinguished throughout.