acceptodds
Under review as a conference paper at ICLR 2027

Not All Forgetting Is Equal: Budgeted Preference Optimization for Retaining Critical Capabilities

Abstract

Preference optimization is known to erode abilities acquired earlier, and existing remedies treat this alignment tax as a single quantity to bound. Tracking six capabilities through full-parameter DPO of Qwen2.5-1.5B-Instruct over three seeds, we find instead that the tax is paid unevenly. Held-out safety falls by up to 5.3 points, whereas instruction following and truthfulness improve and the remaining capabilities hold. As a result, an average over capabilities rises and hides the safety loss. Reference-free SimPO erodes math as well. Moreover, likelihood margins between the model’s own verified and failed safety answers keep rising, so the most readily available retention signal misses the loss. Motivated by these findings, we propose RETAINPO, which separates detecting capability loss from protecting against it. RETAINPO measures a statistically supported loss for each capability on task-native sentinel questions, spends a capped token budget on the capabilities with the largest supported loss and abstains when none is supported. A hinge on the likelihood margins of verified certificates from those capabilities then enters the preference objective. Across three seeds, RETAINPO restores held-out safety from 92.5 to 97.0 percent at a preference accuracy comparable to DPO and keeps its truthfulness gain, whereas every anchoring baseline that retains safety gives that gain up. Under SimPO, it adds math once math loss appears and reaches a smaller worst-capability loss than the tested fixed allocations, although it spends more retention tokens. A protection-transfer matrix further shows that protecting safety raises its held-out safe rate by 5.9 points over DPO and exceeds the average effect of protecting the other five capabilities by 5.0 points. Certificates thus fail as a detector yet work as an intervention, and our findings recast retention during preference optimization as an online allocation problem that intervenes only on measured capability loss.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
preference optimization, capability retention, catastrophic forgetting, alignment tax, adaptive budget allocation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FHxWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how preference optimization affects different language model capabilities and how to retain those that degrade during training. It proposes RETAINPO, which uses task-specific monitoring to identify capability loss and allocates a limited retention budget to selected capabilities. The paper evaluates the approach under DPO and SimPO, comparing it with other retention strategies and examining its effects on safety and general capabilities.

Critical strengths

S1: Identifies a practically relevant monitoring failure

Section 3.2 reports rising margins across all capabilities despite declining safety. Appendix K demonstrates the divergence on the same question, weakening the alternative explanation that certificate questions are simply unrepresentative of evaluation questions. Distinguishing failure as a detector from effectiveness as an intervention is a valuable empirical insight beyond proposing another regularizer.

S2: Data roles and endpoint evaluation are clearly organized

Section 4.1 separates sentinels, certificate banks, and evaluation panels. Appendices A/M reserve sealed tests and describe deduplication and generation protocols. Table 9 includes several benchmarks, including XSTest, to assess whether restored safety merely reflects more refusals. Such separation is particularly important for a monitoring-driven method.

S3: Targeted protection is tested directly

The single-capability transfer matrix in Appendix D compares diagonal and off-diagonal effects at a common dose. The safety targeting contrast has a positive lower bound. The authors also acknowledge that code lacks clear positive targeting evidence and that instruction certificates may act as in-distribution supervision. These qualifications make the mechanism analysis more credible.

S4: Baselines and statistical procedures are relatively well documented

Table 1 covers several retention strategies. Appendix G describes paired bootstrap analysis, some simultaneous inference, and a GPU nondeterminism floor. The main text acknowledges that Static suffices when safety is the only degrading capability under DPO, rather than treating dynamic allocation as universally necessary.

Critical weaknesses

W1: The budget comparison is not fully fair

Under SimPO, RETAINPO uses more retention tokens than Static and Uniform: 3.05% versus 1.89% (Section 5.4 and Appendix N). Its better results may therefore come partly from extra training rather than better allocation. Under DPO, Static already performs similarly to RETAINPO. A comparison using the same total retention budget would make the benefit of dynamic allocation clearer.

W2: The online experiments used an earlier, flawed code verifier

Appendices A/H explain that the earlier verifier incorrectly marked some correct code as wrong, causing unnecessary code protection during training. Rescoring the final outputs fixes the reported scores but does not undo those training decisions. Rerunning at least one seed with the corrected verifier would help confirm the results.

W3: Monitoring adds substantial cost

The extra retention tokens are few, but monitoring generates tokens equivalent to 19.5% of the preference tokens and takes about 2.4 A100 hours per run (Appendix F). This cost should be included when assessing efficiency. The paper should report the full cost of monitoring, certificate construction, and training, and compare it with simpler fixed protection.

W4: Better safety comes with remaining trade-offs

On the sealed tests, RETAINPO's HarmBench safety score remains below the reference model: 66.2 versus 77.5. Its compliance on safe XSTest prompts also falls from DPO's 72.3 to 59.2, suggesting more unnecessary refusals (Appendix M, Table 9). These trade-offs should be more prominent in the main text. The truthfulness results are mainly multiple choice, so they do not establish better factual accuracy in open-ended answers.

W5: Detection reliability and generalization need more evidence

The detector repeatedly checks several capabilities, but its threshold does not control false alarms across all checks. Different sentinel-set sizes may also make some capability losses easier to detect than others. A small sensitivity study of thresholds and sentinel sizes would help. Experiments use only one base model, with three seeds that change data order, so broader generalization remains uncertain.

Questions for authors
  1. Under exactly the same total retention-token quota, does dynamic allocation still outperform Static/Uniform on SimPO?
  2. How do active sets, actual spending, and final outcomes change when training is rerun with the corrected code verifier?
  3. What is the end-to-end cost including bank generation, the shared trigger, all detection, and waiting time?
  4. Would treating XSTest compliance as another capability to protect change the conclusions about safety gains?
  5. How sensitive are false activations and missed losses to small changes in thresholds, sentinel sizes, and monitoring frequency?
5 comments
AuthorsOpenReview ↗

Response to Reviewer FHxW (1/2)

Thank you for highlighting the distinction between detecting capability loss and intervening on it, and for the careful checks on the verifier.

Both affected DPO seeds have now been rerun with the corrected verifier inside training. The new SimPO comparison also matches realized retention spending, including the summed penalty coefficient.

Q1. Does allocation help under exactly the same retention budget? (W1)

A1. The three-seed SimPO control matches RetainPO's tokens and summed penalty coefficient in each interval, including abstentions. Matching coefficients accounts for Eq. 6's normalization by active capability count. Static allocates to its frozen safety set, while Uniform divides the allowance over all six capabilities.

Total token differences are at most 0.003%, and coefficients agree to floating-point precision. Uniform's per-interval difference is at most 0.008% due to realizable bank quotas. At 3.05% spending, worst-capability changes are −4.5 for RetainPO, −5.5 for Static and −7.4 for Uniform. RetainPO improves over Static by 0.3/0.5/2.1 points in seeds 42/43/44, and over Uniform by 2.0/2.7/3.9. Figure 6 and Table 14 give the complete results, also summarized in our general response. This tests allocation conditional on the same realized schedule. Static remains sufficient under DPO's stable safety-loss target.

Q2. What changes after rerunning training with the corrected verifier? (W2)

A2. Replay reproduces all 47 original decisions. Correcting the verifier changes three: seed 43 at 50%, and seed 44 at 60% and 90%. DPO seed 42 and all 27 SimPO decisions are unchanged in replay.

We reran both affected seeds with the corrected verifier inside the online detector, from the shared 10% checkpoint with optimizer state. Old and corrected endpoints, and the reference, are evaluated on H200 (Table 8):

RunSafety/code protected intervalsSpend %Held-out safetyHeld-out code
Reference——97.550.8
Seed 43, original5/11.6297.150.2
Seed 43, corrected3/00.8196.750.7
Seed 44, original3/21.3397.351.0
Seed 44, corrected3/00.8197.549.2

The erroneous code activations disappear, spending falls to 0.81% in both runs, and safety stays within 0.8 points of reference. Seed 44's corrected code score remains below reference (49.2 versus 50.8 in the table). Its loss lower bound is −1.0, so it does not meet the supported-loss criterion, although the point degradation remains. Static is 1.2 points below reference in that seed, with lower bound −1.3.

Sealed evaluations are also complete: corrected seeds 43/44 score 96.7/96.7 on SafeRLHF, 67.9/66.7 on HarmBench and 57.2/57.6 on benign XSTest compliance. Table 8 compares them with the original runs and hardware-matched reference, including worst-capability changes. The main table retains submitted runs and Appendix H gives the correction comparison.

We also corrected Figure 5's caption and Appendix N: the SimPO seed-44 code activation remains under the fixed verifier, with supported sentinel loss 0.71. Replay and online reruns are now distinguished throughout.

AuthorsOpenReview ↗

Response to Reviewer FHxW (2/2)

Q3. What is the full cost, including monitoring and waiting? (W3)

A3. Appendix F and Table 6 include the following components. Service waiting time is reported here separately from GPU compute:

ComponentDPOSimPO
Banks, once and shared0.7 A100 h + 0.5 L40S hSame banks
Initial trigger0.7–1.1 A100 h per seed, sharedOnline from 10%
Online detection per run1.95–2.66 A100 h, 6–7 calls3.83–4.02 A100 h, 9 calls
Training, original-run means in Table 6, excluding waitsRetainPO 2.2, Static 2.3, DPO 2.2 H200 hRetainPO 1.7, Static 1.6 H200 h
Waiting for detector service23–26 wall-clock h15–29 wall-clock h

Detection generates 17.8–20.4% of preference tokens under DPO and 26.5–27.4% under SimPO. Static avoids continued monitoring and is cheaper under DPO. Under SimPO, the extra compute enables reallocation to later math loss, with the matched-spending benefit reported above. The long waits reflect the shared service queue, not additional GPU computation.

Q4. How should the safety and compliance trade-off be interpreted? (W4)

A4. Section 5.3 now discusses the joint sealed results in Table 12. RetainPO combines SafeRLHF 97.8 with MC2 52.2, versus reference 98.4/36.4 and DPO 92.0/51.9. Higher-beta DPO and BAPO retain strong safety but have MC2 34.2 and 34.3. HarmBench is only partially recovered (66.2 versus reference 77.5), and benign compliance is 59.2 versus DPO's 72.3. We discuss this refusal trade-off explicitly and consistently call the truthfulness measure multiple-choice MC2. These are submitted-run sealed results; corrected runs are separate in Table 8.

Protecting benign compliance would require its own disjoint sentinels and certificates. It is a sensible additional target, but sealed XSTest cannot be reused for training and that intervention has not been evaluated.

Q5. How sensitive is detection, and what generalizes? (W5)

A5. We addressed detection sensitivity and broader evaluation separately. First, to test sensitivity to threshold, cadence and sentinel size, we replayed corrected-verifier DPO scores over thresholds 1.28, 1.645, 1.96 and 2.33. With full sentinel sets, safety is detected in every seed and no other capability is flagged in 150 checks at each threshold. Checking every other boundary gives 0/90 flags, but delays mean first safety detection from update 111 to 127. With random half-sized sets, flags occur in 1.4% of checks over 1,000 draws at 1.645, falling to 0.1% at 2.33 with later detection (Table 7).

Here a flag is measured on a capability without supported held-out loss. These are sensitivity results, not familywise false-alarm guarantees. The pointwise bound in Eq. 2 remains pointwise. Replay measures decisions rather than the trajectories that retraining under another cadence would produce.

Second, to examine whether the loss pattern is specific to the original preference dataset, we evaluated the HH-RLHF diagnostic. The unprotected run shows a different pattern: at 40%, MC2 falls 7.5 points with pointwise interval support, and safety falls 1.6 with simultaneous six-capability support. We then tested protection by running Static and RetainPO from the same 10% checkpoint with the corrected verifier. RetainPO spends 0.81% of preference tokens versus Static's 4.33%, with worst-capability point changes of −3.9 and −4.2 (DPO: −4.7). Static retains more safety but loses more code, and MC2 remains below reference under both. The six-capability table is in our reply to 8xzw.

This comparison therefore adds a second dataset with a different loss and protection pattern. Because it uses one seed and unequal realized spending, the completed three-seed matched-spending SimPO test remains the direct evidence for adaptive allocation.

AuthorsOpenReview ↗

Follow-up on the completed checks and revisions

Thank you again for the care you have taken in reviewing our work. We hope the completed reruns and comparisons have addressed the evaluation concerns you raised. We would like to summarize how each check bears on those concerns.

First, because rescoring alone cannot undo the verifier's earlier training decisions, we reran both affected DPO seeds with the corrected verifier inside training. The erroneous code activations disappear, while held-out safety stays within 0.8 points of reference. Seed 44's code point estimate remains below reference, so we retain that residual in the comparison rather than describe recovery as complete.

Second, to make the allocation comparison fair at the same retention spending, we matched tokens and summed penalty coefficients per interval. RetainPO has a smaller worst-capability loss than Static and Uniform in all three SimPO seeds, supporting an allocation benefit under the same realized spending schedule.

Finally, we report monitoring cost separately from retention spending and make the safety/compliance trade-off explicit. We also distinguish the detector's measured sensitivity from a simultaneous false-alarm guarantee. We would be glad to discuss any remaining question about these checks or their interpretation.

Reviewer_8xzwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RetainPO, which monitors individual capabilities during preference optimization and allocates a limited retention budget to those showing degradation. It separates task-based detection from a margin-based retention loss, recovering safety while preserving truthfulness gains in the tested setting. 59450_Not_All_Forgetting_Is_Eq.pdf

Critical strengths
  1. The motivation is clear: average performance can improve while an important capability, such as safety, deteriorates.
  2. The finding that likelihood margins can increase even as safety declines is useful and supports separating detection from intervention.
  3. The analysis is careful, with three seeds, separate monitoring and evaluation sets, and targeted ablations showing which capabilities benefit from protection.
Critical weaknesses
  1. All experiments use Qwen2.5-1.5B and UltraFeedback. It remains unclear whether the findings generalize to larger models and other preference datasets.
  2. The benefit of dynamic allocation is not fully isolated. Static allocation performs similarly under DPO, while RetainPO uses more retention tokens in the SimPO comparison. A comparison with equal token budgets would strengthen the central claim.
  3. Safety recovery comes with increased refusal of harmless requests: XSTest compliance drops from 72.3% under DPO to 59.2%. Safety also remains substantially below the initial model on HarmBench.
  4. Monitoring is costly, adding approximately 2.4 A100 hours per run. The small retention-token budget therefore does not fully reflect the method’s overall overhead.
Questions for authors

See Weaknesses Above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8xzw (1/2)

Q2. Why adapt the allocation? (W2)

A2. RetainPO has the smallest worst-capability loss in every seed of the matched-spending SimPO test. Static and Uniform receive the same retention tokens and summed penalty coefficient as RetainPO in each interval. At 3.05% spending for every arm, RetainPO's worst-capability loss is −4.5 points, versus −5.5 for Static and −7.4 for Uniform. Figure 6 and Table 14 give the full comparison, also reproduced in the general response.

Static preserves 1.1 more safety points but loses 2.8 math points relative to RetainPO. Uniform preserves 0.8 more math points but loses 2.9 safety points. The test isolates where a given spending schedule should go. Under DPO, the loss target stays on safety and Static is sufficient. Under SimPO, math loss appears later, making reallocation useful. The two objectives show when adaptation changes the outcome.

Q3. What happens to safety, benign compliance and truthfulness together? (W3)

A3. The joint table makes the retained MC2 gain visible alongside safety and compliance. These are selected submitted-run sealed results; Table 12 includes all methods, and corrected-verifier runs are separate in Appendix H.

MethodSafeRLHFHarmBenchUnsafe refusedBenign compliedTruthfulQA MC2
Reference98.477.598.052.836.4
DPO92.059.088.572.351.9
DPO, beta=0.197.273.894.563.134.2
BAPO97.979.296.753.534.3
RetainPO97.866.293.259.252.2

RetainPO retains SafeRLHF recovery together with the MC2 gain, whereas the anchoring controls here give up that gain. Those controls are competitive on safety alone. HarmBench remains below reference, and benign compliance falls from DPO's 72.3 to 59.2. Both refusal measures move toward the reference, so part of recovered safety is restored refusal. Section 5.3 now discusses this joint result explicitly, with truthfulness consistently identified as multiple-choice MC2.

AuthorsOpenReview ↗

Response to Reviewer 8xzw (2/2)

Q4. What is the monitoring cost? (W4)

A4. Online detection takes 1.95–2.66 A100 hours per DPO run and 3.83–4.02 per SimPO run. Bank construction is a shared one-time cost of about 0.7 A100 plus 0.5 L40S hours; the DPO trigger costs 0.7–1.1 A100 hours per seed, shared by the relevant arms. DPO training itself takes about 2.2 H200 hours. Appendix F and Table 6 report these components separately from retention tokens.

Static avoids continued monitoring and is cheaper in the tested DPO setting. SimPO is where monitoring discovers a new loss target and enables the matched-allocation benefit. Thus the relevant trade-off is better capability balance for additional monitoring compute. An every-other-boundary replay reduces checks but delays initial safety detection by about 16 of 475 updates; retraining at that cadence has not been tested.

Q1. Does a second preference dataset show the same pattern? (W1)

A1. We evaluated an existing HH-RLHF DPO run of the same model and hyperparameters on all six capabilities:

HH-RLHF, seed 42: change from reference, points10%40%70%100%
Safety−0.2−1.6*†−0.8−1.4*
Code−1.8*−1.8−2.3*−1.5
Math+0.4−0.50.0+0.5
Knowledge−0.3−0.2+0.7+0.4
Instruction following−0.2+0.6+0.6−0.2
Truthfulness, MC2+0.2−7.5*−5.7−4.7

* One-sided 95% question-bootstrap bound below zero. † Also below zero with a bound simultaneous over all six capabilities. Reference and checkpoints are evaluated on H200.

The pattern differs from UltraFeedback, where DPO raises MC2 by 12.9 points at the endpoint (Table 9). Here MC2, code and safety decline at particular checkpoints with pointwise bounds below zero, and only safety at 40% also passes the simultaneous bound, while math, knowledge and instruction following remain within 0.7 points of reference.

HH-RLHF's endpoint KL is 1.9 nats per sequence, versus 22 on UltraFeedback. This adds a distinct case of uneven, dataset-dependent change. Its sentinel trigger selects safety and truthfulness at 20%.

The method comparison on HH-RLHF.

Having observed a different loss pattern on HH-RLHF, we next tested how the protection strategies respond to it. We trained Static and RetainPO from the same 10% checkpoint, with the corrected verifier and the same evaluation. Static protects safety and truthfulness from 20% onward. RetainPO protects both at 20%, truthfulness at 40%, and abstains at the other boundaries. Their endpoint results are:

HH-RLHF, seed 42: change from reference, pointsDPOStaticRetainPO
Safety−1.4*+1.2*−0.6
Code−1.5−3.0*−1.3
Math+0.5+0.1+1.3
Knowledge+0.4+0.3−0.2
Instruction following−0.2−0.7−0.2
Truthfulness, MC2−4.7−4.2*−3.9
Worst point change−4.7−4.2−3.9
Retention tokens / preference tokens—4.33%0.81%

* A one-sided 95% question-bootstrap bound supports the direction of the change from reference for this capability alone. These markers do not test differences between methods. All endpoints are scored on H200.

RetainPO spends less with a slightly better worst-change point estimate. Static retains more safety but loses more code, and MC2 remains below reference under both. This is a single-seed diagnostic with unequal realized spending. The three-seed matched-spending SimPO experiment remains the direct allocation test; the HH result adds a second dataset with a different protection pattern. Generalization across base models remains to be tested.

AuthorsOpenReview ↗

Follow-up on allocation, cost and broader evaluation

We appreciate the time you have taken to review our work. We hope our response has clarified when adaptive allocation helps and what it costs. In particular, we have separated the budget question from the practical trade-offs.

First, to test whether the SimPO advantage comes only from additional retention spending, we matched both tokens and summed penalty coefficients per interval. Worst-capability loss is 4.5 points for RetainPO, versus 5.5 for Static and 7.4 for Uniform, with the advantage in every seed. Extra retention spending therefore does not explain this result. By contrast, Static remains sufficient in the tested DPO setting, where the loss target stays stable.

Matching retention spending does not remove monitoring overhead, so we also report it separately: about 2.0–2.7 A100 hours per DPO run and 3.8–4.0 per SimPO run. We make the safety/compliance trade-off and incomplete HarmBench recovery explicit as well. Finally, the HH-RLHF evaluation checks a second preference dataset and shows a different loss pattern. We present it as a single-seed diagnostic, rather than evidence of generalization across models.

We would welcome any further thoughts on these practical trade-offs.

Reviewer_RrUfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Preference optimization has to pay the alignment tax, where the base model loses some of its capability during training. The existing alignment tax addressing research mitigated such issue in a global manner, without capability-wise analysis. However, the authors have found that the tax is paid unevenly for six capabilities. Moreover, they have found that the likelihood margin approach failed to detect the erosions. To address such limitations, the authors propose RetainPO, which is inspired from continual learning methods, to monitor the preference optimization process into equal intervals on six capabilities, allocate budget to the capabilities suffering erosion, and finally perform capability-specific retention via penalizing the likelihood margin. Empirical results show the efficacy of RetainPO.

Critical strengths

Clear evidence for separating detection from retention.

The authors have clearly shown that the held-out changes of the six-capabilities move in both direction, but the likelihood margin monitor fails to detect the erosions. The justification of using task-native sentinels instead of the likelihood margin is clear.

A well-motivated connection to continual learning.

The design of the RetainPO is well-inspired by the continual learning methods, generating stored examples(certificates) with allocating the retention budget to the important, hardly-eroding capabilities to maximize the retention capability is aligned with the continual learning motivation, which addresses forgetting while allowing learning new capabilities.

Critical weaknesses

The retention-penalty via likelihood margin loss needs further analysis

Figure 1b shows that the likelihood margins for all capabilities have exceeded its original value regardless of their held-out change, which is why we can not use the likelihood margin for erosion detection. However, for the triplets in the certificate, they will show a similar trend since the they have the same structure of (x,y+,y−)(x,y^+, y^-). Consequently, that can mean that fewer certificates could receive nonzero penalties when the policy model is experiencing high erosion. Can the authors plot the fraction of certificates with nonzero Huber loss and the mean retention loss, and the RcR_c? If the mean retention penalty or the non zero fraction indeed shows decreased strength as the training progresses, where as the RcR_c remains high, that could suggest a limitation of using the likelihood margin for retention. I am willing to increase the score if my concern is properly addressed.

Questions for authors
  1. Can the authors report the likelihood margin trend as in Figure 1b, but during the RetainPO? the analysis for the likelihood margin trend on the certificate data would be appreciated too.
  2. Can the authors compare the performance of RetainPO and its variants at a matched total retention-token budget to analyze the stability-plasticity trade-off under the same retention budget?
4 comments
AuthorsOpenReview ↗

Response to Reviewer RrUf

Q1. Does the retention penalty remain effective as margins rise?

A1. To check whether a shrinking active fraction coincides with continuing capability loss, we tracked that fraction alongside the bank-wide hinge, the applied penalty and held-out safety. In the DPO runs, the active fraction falls as safety recovers, while a below-reference tail remains. Figures 7–8 now show the actual RetainPO trajectories.

Across the three original DPO seeds:

QuantityRetainPOStaticUnprotected DPO
Mean monitor-margin change at endpoint, nats+52+54+46
Safety-bank fraction below reference, 30–100% of training8–19%4–19%32–42%
Bank-wide hinge at endpoint1.20.310.9
Held-out safety at endpoint97.097.892.5

Reference safety is 97.3. At the first protected boundary, 32–39% of safety certificates are below reference. One protected interval reduces this to 10–15%, while safety at 40% is 97.7–98.2 across seeds. In seed 42, pausing protection lets the tail rise from 7% to 18%; reactivation brings it back to about 10%. The applied-penalty curve is zero during unprotected intervals even when violations remain in the bank.

The declining active fraction therefore accompanies recovery, rather than continuing safety loss with an inactive penalty. The separate targeting experiment supports the intervention effect: safety protection improves held-out safety by 5.9 points (lower bound 3.1), with a targeting contrast of 5.0 (lower bound 2.8; Appendix D). In the seed-42 diagnostic runs (Table 5), one protected interval leaves 7.7% of safety certificates below reference, against 35.4% under DPO, with a retention gradient at 15.5% of the DPO gradient norm.

Static preserves slightly more safety in these original runs, spending 2.07% versus RetainPO's 1.25%. Figure 7 includes the original DPO seeds affected by the code verifier. Their safety decisions are unchanged in replay; both affected seeds have now been rerun with the corrected verifier inside training, with completed endpoints in Table 8.

Under SimPO, the math tail remains at 41–46% from halfway through training, but protection limits violation size: endpoint bank hinge is 4.3 for RetainPO, 10.1 for Static and 13.7 for unprotected SimPO. Held-out math is 82.1, 78.8 and 78.7 in that original-run evaluation (Figure 8; Table 13).

Q2. Does the benefit remain at equal retention spending?

A2. To test whether allocation helps at equal spending, we matched both retention tokens and summed penalty coefficients per interval across three SimPO seeds. RetainPO's worst-capability loss is −4.5 points, versus Static's −5.5 and Uniform's −7.4, with the advantage in every seed. This comparison isolates allocation under the same realized spending schedule, independently of the hinge analysis. We report the complete table and protocol in our general response and Table 14.

We hope these results address your concern about the effectiveness of the retention penalty as margins rise.

AuthorsOpenReview ↗

Follow-up on the mechanism and matched-spending tests

We would like to thank you again for the time and thought you have given to our work. We hope the additional analyses have helped clarify how the retention penalty behaves as training progresses.

First, to check whether the penalty still acts as margins rise, we added trajectories from the actual DPO training runs. These show the fraction of certificates below reference declining alongside held-out safety recovery. In seed 42, the violating fraction grows when protection pauses and falls after reactivation. The separate targeting control already reported in our response also gives a 5.9-point held-out safety gain over unprotected DPO, with a lower bound of 3.1. Together, these results support effective protection even while the average margin rises.

Second, to distinguish the benefit of allocation from simply spending more, we matched retention tokens and summed penalty coefficients in each interval. RetainPO has a smaller worst-capability loss than both Static and Uniform in all three SimPO seeds. This comparison supports an allocation benefit under the same realized spending schedule.

We would be happy to discuss either check further if it leaves part of your mechanism question unresolved.

I appreciate the authors for addressing my concern.

The rebuttal has justified the use of margin, where it is limited for the detector, but is still effective for retention. As my main concern is resolved, I will raise my score from 2 to 3.

AuthorsOpenReview ↗

Thank you for your encouraging reply

Thank you for taking the time to read our response and for your encouraging follow-up. We’re glad the additional analysis addressed your main concern. Figures 7 and 8 of the revised paper show how likelihood margins and the retention penalty change during training. We also included the matched-budget comparison you suggested in Figure 6 and Table 14. Thank you again for your thoughtful feedback and support. We really appreciate it!

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response

Thank you for the reviews. We have completed the actual-training hinge analysis, the three-seed matched-spending control, and corrected-verifier reruns of both affected DPO seeds. The revised PDF accompanies these responses. In the updated PDF, dark red identifies key revised passages and selected captions.

The shared question: does allocation help at equal spending?

Static and Uniform receive RetainPO's retention tokens and summed penalty coefficient in each interval of the same SimPO seed, allocated to their own capability sets. All arms spend 3.05% of preference tokens. The held-out results are:

Matched spending; H200 evaluationSafetyMathWorst changePreference score
Uniform90.183.3−7.452.3
Static94.179.7−5.551.3
RetainPO93.082.5−4.551.5

RetainPO has the smallest worst-capability loss against both controls in every seed. This isolates allocation given the same spending schedule (Figure 6; Table 14). DPO provides the complementary setting: when safety remains the loss target, fixed protection is sufficient.

The mechanism and verifier checks are also complete.

Figures 7–8 show the hinge during actual training, including the below-reference tail and applied penalty. Appendix H reports completed held-out and sealed evaluations for corrected DPO seeds 43/44. Section 5.3 now reports safety alongside benign compliance, and Appendix F gives the monitoring cost separately from retention-token spending. The HH-RLHF method comparison is also complete. In this single-seed diagnostic, RetainPO spends 0.81% of preference tokens versus Static's 4.33%, with worst-capability point changes of −3.9 and −4.2. The detailed results are in our replies to 8xzw and FHxW.

AuthorsOpenReview ↗

Response Summary: Allocation, Retention Mechanism and Verification

Dear Reviewers,

Thank you for recognizing the evidence that capability loss is uneven and that likelihood margins can miss it. These findings motivated RetainPO's separation of task-based detection from certificate-based protection. In the original DPO experiment, RetainPO raises held-out safety from 92.5% to 97.0% while retaining the multiple-choice truthfulness gain. Our responses focus on the three questions about how this protection works.

First, does allocation help when spending is matched?

This was a shared concern. In the added SimPO control, Static and Uniform receive RetainPO's actual retention tokens and summed penalty coefficients in every interval. The mean worst-capability loss is 4.5 points for RetainPO, versus 5.5 for Static and 7.4 for Uniform, with RetainPO better in each of the three seeds. By replaying the same spending schedule, this comparison isolates where protection is allocated.

Second, does the hinge remain effective as average margins rise?

Following RrUf's question, we measured the active certificate fraction and retention penalty along actual RetainPO training. These traces show that rising average margins coexist with certificates below their reference margins and a nonzero penalty. The fraction below reference falls during protection, rises during a pause, and falls after protection resumes. This supports the original distinction: margins can provide an intervention even when their averages are unsuitable for detecting loss.

Third, do the verifier and cost checks support the reported behavior?

For FHxW's verifier concern, we reran both affected DPO seeds with the corrected online verifier and reported their held-out and sealed evaluations. For 8xzw's questions, we put safety and benign compliance side by side and reported monitoring computation separately from retention-token spending. The second-dataset experiment and its single-seed scope are described in our replies.

We appreciate the questions that led to these targeted checks. If there is a remaining concern about the allocation or mechanism, we would welcome the chance to address it directly.

AuthorsOpenReview ↗

Summary of Major Contributions, Strengths, and Responses to Concerns

Dear AC, SAC, and PCs,

Thank you for your work in coordinating the review process, and to the reviewers for their careful consideration of our paper and responses. During the discussion, RrUf raised their rating from 2 to 3, and FHxW confirmed that their major concerns had been addressed and maintained their weak-accept recommendation. We appreciate this further engagement. Below, we bring together the paper’s contributions, the strengths recognized in the reviews, and the supporting evidence to help with your final assessment.

1. Core Contributions

  • Evidence that preference optimization forgets unevenly. Across six capabilities, improving averages can conceal important losses. Our same-question analysis further shows that likelihood margins can rise while task performance declines. This establishes the value of task-native measurements for detecting capability loss.

  • Adaptive retention of the capabilities that are actually declining. RetainPO separates detection from intervention and allocates protection as losses appear during training. In the original DPO experiments, it improves held-out safety from 92.5% to 97.0% while preserving the gain in multiple-choice truthfulness.

  • A benefit from allocation at equal spending. Under SimPO, matching retention tokens and summed penalty coefficients per interval yields mean worst-capability losses of 4.5 points for RetainPO, 5.5 for Static and 7.4 for Uniform. RetainPO has the smallest loss in every seed (Table 14), supporting the value of adapting where protection is applied.

2. Strengths Recognized by Reviewers

  • A useful distinction between detection and retention. RrUf recognized the evidence for task-native detection and the connection to continual learning. FHxW highlighted the monitoring failure demonstrated by the same-question analysis.

  • Practical importance and careful controls. 8xzw emphasized that improving averages can conceal capability losses and noted the three-seed evaluation, separate monitoring and evaluation sets, and targeted ablations.

3. Responses to Key Concerns

We focused the additional experiments on whether the penalty remains active, whether adaptive allocation itself helps, and whether the verifier correction changes the findings.

ReviewerConcernEvidence in the final manuscript
RrUfCan the retention penalty still work when average margins rise?Actual training traces show nonzero penalties and a response to renewed protection. In DPO seed 42, the fraction below the reference margin rises from about 7% to 18% while protection pauses, then falls to about 10% after protection resumes (Figure 7).
RrUf, 8xzw, FHxWIs the benefit explained by spending more?The per-interval comparison matches both retention tokens and summed penalty coefficients. RetainPO retains its advantage over both fixed allocations in all three seeds (Table 14).
FHxWDo the verifier corrections affect the findings?Both affected DPO seeds were rerun with the corrected verifier and evaluated on held-out and sealed data (Table 8). All SimPO detection decisions remain unchanged. Monitoring and training costs are reported separately.

4. Concluding Remarks

RetainPO connects an empirical finding to an intervention: detect capability-specific loss during training, then direct retention to where it is needed. The original task-level analyses establish the need for this separation, and the training traces and matched-spending experiments support the resulting design. We thank the reviewers for their engagement and the chairs for considering this evidence in the final decision.

The authors of RetainPO

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.