acceptodds
Under review as a conference paper at ICLR 2027

Beyond Uniform Forgetting in Alignment: Characterising, Predicting, and Recovering Retention in Sequential Preference Optimisation

Abstract

Sequential Direct Preference Optimisation (DPO) lets a model acquire new alignment objectives without retraining from scratch, but later training can degrade preferences learned earlier. It remains unclear whether such forgetting is uniform, predictable or recoverable, and how it should be measured. We study these questions across five preference settings (HH-RLHF, HelpSteer2, PKU-SafeRLHF, UltraFeedback and Summarize-Feedback), organised as , and . When measured relative to the original base model, earlier objectives show degradation in some settings and stability or improvement in others. Across 16 retention comparisons, comprising ten two-stage orderings and six three-stage steps, the policy's own chosen-over-rejected ranking is substantially more stable than the reference-relative metrics. For length-normalised accuracy, a decline of more than 2.5 percentage points is ruled out in 13 of 16 comparisons, so large reference-relative declines need not indicate preference reversals. Before Stage 2, the Stage-1 model's margin on the incoming objective matches the direction of later margin retention in nine of ten orderings. The evidence comes from five datasets and describes margin changes rather than the policy's own accuracy. Replaying a small share of earlier data shows nominal improvements in margin retention in seven of nine orderings, while policy accuracy improves in only two.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Direct Preference Optimization, catastrophic forgetting, preference alignment, AI safety, replay-based mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pzVZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies forgetting in sequential DPO. The authors compare reward margins with the model's own preference accuracy across five datasets. They find that reward margins can drop while preference accuracy stays stable. They also propose a Stage-1 compatibility score to predict retention and test data replay to reduce forgetting.

Critical strengths
  1. Useful evaluation perspective. The paper shows that a drop in the DPO reward margin does not always mean the model has reversed its preference. This is useful when evaluating forgetting.

  2. Fairly broad experiments. The paper covers five preference datasets, different training orders, and additional three-stage experiments. It also reports confidence intervals and several diagnostic results.

Critical weaknesses
  1. A continuous reward margin can decrease without changing the sign of a preference ranking. This is expected from the metric definitions. Also, the DPO margin uses summed log-probabilities, while the main policy accuracy is length-normalized. It is unclear how much of the reported gap reflects a new finding rather than these metric differences.

  2. Sequential preference alignment and forgetting have been studied in SPO (Lou et al., 2025), COPR (Zhang et al., 2025), and LifeAlign (Li et al., 2026). Harmon et al. (2026) also studied forgetting at the sample level. This paper adds a useful metric comparison, but the overall contribution beyond these studies seems limited.

  3. Stage-1 compatibility is simply the model's margin on the incoming objective. Its sign matches retention in 9 of 10 orderings, but these come from only five datasets. Leave-one-dataset-out prediction gets 7/10 correct, compared with 6/10 for the majority baseline. The paper also provides no clear theoretical reason why this score should predict retention.

  4. Replay improves policy accuracy in only 2 of 9 orderings and sometimes hurts learning of the new objective. Most results also rely on ranking fixed response pairs rather than generated outputs. This makes the practical benefit less clear.

Questions for authors
  1. Can the authors better explain why the gap between reward margins and policy accuracy is more than an expected result of their definitions? Would the main finding hold under matched length-normalized metrics?

  2. What is the key difference from SPO, COPR, LifeAlign, and the sample-level forgetting analysis of Harmon et al.?

  3. Can the compatibility score be tested on more datasets and models? Does it still work when overlapping training and evaluation pairs are removed?

  4. Can the authors show that replay improves actual generated responses without hurting the new objective? How does it compare with existing continual alignment methods?

6 comments
AuthorsOpenReview ↗

Response to Reviewer pzVZ

Thank you for a clear and thoughtful review, and for finding the evaluation perspective useful. Further experiments and analysis were carried out to answer the remaining concerns and queries, provided below.

W1. A continuous reward margin can decrease without changing the sign of a preference ranking. This is expected from the metric definitions. Also, the DPO margin uses summed log-probabilities, while the main policy accuracy is length-normalized. It is unclear how much of the reported gap reflects a new finding rather than these metric differences.

Answer: Part of the gap does follow from the definitions, and we now report everything under matched, length-normalised metrics so that readers can see exactly which part. The table below uses matched definitions on the clean data.

OrderingNormalised margin change [95% CI]Normalised accuracy change, pp [95% CI]
HH harmless→helpful−0.044 [−0.061, −0.027]−0.7 [−2.1, +0.7]
HH helpful→harmless−0.048 [−0.063, −0.034]−0.9 [−2.0, +0.2]
HS2 coherence→verbosity+0.007 [−0.005, +0.018]−0.3 [−1.9, +1.3]
HS2 verbosity→coherence+0.035 [+0.025, +0.047]+0.9 [−0.4, +2.3]
SF coverage→accuracy−0.203 [−0.308, −0.101]−11.4 [−18.8, −3.8]
UF honesty→IF+0.730 [+0.573, +0.901]+2.3 [+0.7, +3.9]

In HH, the matched margin still falls while ranking accuracy stays stable. Most of this gap comes from where the pairs sit after Stage 1. GIR counts a pair as improved when its margin is above that of the reference model, and about 55% of pairs sit so close to that line that a typical Stage-2 change can push them across it. Only 11–18% sit that close to the model's own decision boundary, which is what ranking accuracy counts. The same small shift therefore lowers GIR much more than ranking accuracy. Shifting every pair by the average change already predicts a GIR drop of 10–14 points, at least as large as the observed 6–8 points. For HelpSteer2, the matched metrics show that the earlier margin decline does not persist under length normalisation, so we have revised that claim. In SF coverage→accuracy, both the margin and the ranking accuracy fall, so the forgetting is real under either definition.

What goes beyond the definitions appears at the level of individual pairs. First, which rankings are lost is not random. In HH, about a third of the rankings newly learned in Stage 1 are lost in Stage 2 (29–38% across both orders and three seeds), compared with only 1–4% of rankings that were already correct at the base model, and newly learned rankings are lost more often than older ones even when both are matched on Stage-1 margin. Because this comparison holds the Stage-1 margin fixed, it cannot follow from the metric definitions. Second, large margin changes and ranking flips are largely different events. The correlation between the size of a pair's margin change and whether it flips is 0.15 or lower in eight of the ten orderings, and the pairs that flip tend to be those that sat close to the decision boundary after Stage 1.

W2. Sequential preference alignment and forgetting have been studied in SPO (Lou et al., 2025), COPR (Zhang et al., 2025), and LifeAlign (Li et al., 2026). Harmon et al. (2026) also studied forgetting at the sample level. This paper adds a useful metric comparison, but the overall contribution beyond these studies seems limited.

Answer: These are important works, and the revision discusses them explicitly. They also help clarify what our analysis adds.

  • How forgetting is measured: SPO evaluates sequential preference alignment primarily with GPT-4 pairwise win rates and alignment scores, with additional reward- and cost-model analyses. COPR evaluates continual learning using task-level normalized Preference Scores and standard continual-learning measures including average accuracy, average incremental accuracy, forgetting measure and backward transfer, alongside reward-based, GPT-4 and human evaluation. LifeAlign similarly reports aggregate Last Performance, Backward Transfer and Average Performance, where task performance is measured with BLEU-4, ROUGE-L and an LLM judge. Harmon et al. move beyond task averages and measure sample-level 1→0 and 0→1 correctness transitions on knowledge benchmarks. In contrast, we follow every held-out preference pair through the base model, Stage 1 and Stage 2, and evaluate it using both the fixed-reference DPO margin and the policy's own ranking.

  • Whether a falling margin means a lost preference: COPR additionally monitors the average preferred–dispreferred probability gap on samples in its memory buffer, but its continual-forgetting measures themselves remain task-level aggregates. We resolve this distinction pair by pair on held-out data and compare it with the reference-relative margin. This shows that a large reference-relative margin change often does not correspond to a reversed policy preference: margin changes and ranking flips are weakly coupled in most settings, and flips concentrate near the Stage-1 decision boundary.

  • Which preferences are lost: Harmon et al. show that aggregate accuracy can hide simultaneous item-level forgetting and backward transfer. We extend this analysis to preference alignment and ask a further question: which kind of preference is being lost? In HH, 29–38% of rankings newly acquired during Stage 1 are subsequently lost, compared with only 1–4% of rankings that were already correct before Stage 1. Crucially, the difference remains after matching on Stage-1 confidence: newly learned pairs retain only 62–71% of their rankings, versus 79–90% for matched already-correct pairs. Thus aggregate retention can hide a systematic concentration of forgetting on what the earlier stage actually learned.

  • Training order and replay: For SPO, the original training dataset is trained harmlessness→helpfulness, while the reversed helpfulness→harmlessness order is used with a modified dataset; thus training order is not reversed while holding the dataset fixed. LifeAlign studies order sensitivity across forward, reverse and random six-task sequences. We instead run both orders on the same data for every objective pair, allowing us to show directly that retention depends not only on which objectives are combined but also on which one is learned first. COPR formulates continual learning with a historical memory buffer and evaluates experience-replay-based variants, while LifeAlign explicitly incorporates a fixed-size rehearsal buffer into its FPO procedure. Our replay experiment adds an equal-compute control: additional training on repeated Stage-2 data does not prevent forgetting, whereas replacing the same amount of extra training with Stage-1 replay restores retention. This isolates the benefit of revisiting earlier data from the benefit of simply training longer.

Together, these results complement optimization-focused continual-alignment methods by resolving what aggregate retention measures can obscure at the level of individual preferences. Aggregate measures can overstate preference loss when reference-relative margins fall without ranking reversals, and can understate it when losses are concentrated on preferences that the earlier stage newly acquired. Our pair-level diagnostic can be applied directly to SPO, COPR, LifeAlign or other continual-alignment methods to determine which preferences are actually retained. The compute-matched replay results further show that simple replay is a strong baseline in the HH setting and motivate equal-compute replay controls when evaluating more complex continual-alignment methods.

W3. Stage-1 compatibility is simply the model's margin on the incoming objective. Its sign matches retention in 9 of 10 orderings, but these come from only five datasets. Leave-one-dataset-out prediction gets 7/10 correct, compared with 6/10 for the majority baseline. The paper also provides no clear theoretical reason why this score should predict retention.

Answer: We tested compatibility on more data and without overlapping pairs, and the results provided a clearer view of what it measures.

On more data, we built eight new orderings from objective pairs never used in developing the measure (UltraFeedback truthfulness × helpfulness, HelpSteer2 complexity × helpfulness, and two CodeUltraFeedback pairs) and froze all predictions before training Stage 2. Compatibility ranked the strength of transfer across these new orderings very well (Spearman 0.93, and 0.95–0.98 for two closely related variants). On a second model (Llama-3.2-3B, full-parameter, all ten orderings), its sign agreement with retention reproduces the 8B pattern exactly (the same 7 of 10 orderings).

Without overlapping pairs, compatibility's agreement with the sign of retention weakens. Sign agreement falls from 9/10 to 5/10 and leave-one-dataset-out accuracy from 7/10 to 1/10, and the Pearson correlation is 0.90. On the clean retrained runs the corresponding figures are 7/10, 4/10 and 0.91.

The results point to an explanation for this, which also addresses the theoretical question. Compatibility measures how training on objective A affects objective B, while retention measures how training on B affects A. The two coincide when this influence is symmetric, as in HH and UltraFeedback, and diverge when it is not, as in PKU and SF accuracy→coverage. Compatibility is therefore a measure of transfer strength between objectives, which is exactly what the held-out test confirms, rather than a forecast of retention. For this reason, we agree that compatibility should not be used as a forecasting tool for training decisions, and the revised paper no longer presents it that way. It is presented as a descriptive measure of transfer strength, and the practical role goes to compute-matched replay, which restores the earlier objective in both margins and ranking accuracy (see W4).

Reviewer_YadGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies retention in sequential DPO across five preference settings and ten two-stage orderings, with additional three-stage experiments on HelpSteer2. It distinguishes changes in reference-relative reward margins and Gap-Improvement Rate from changes in the policy's own chosen-over-rejected ranking. The authors find that ranking accuracy is often more stable than the reference-relative measures. They also investigate Stage-1 compatibility as a diagnostic of subsequent margin retention and use replay to test which forms of retention can be recovered. The main experiments use Llama-3.1-8B-Instruct with LoRA, with a limited second-model replication and supplementary generation-level checks.

Critical strengths

The distinction between reference-relative change and actual ranking reversal is useful. The paper shows why a decline in the reward margin or GIR should not automatically be interpreted as a reversal of the policy's preferences. Reporting both length-normalised and summed policy accuracy also helps expose length-related effects. This is a valuable measurement lesson for studies of forgetting in preference alignment, where conclusions can depend heavily on the metric used.

The analysis is reasonably systematic and includes useful statistical checks. The paper examines multiple preference settings and both training orders, extends the analysis to a third stage, and reports paired confidence intervals and lost/gained counts. The compatibility and replay experiments also distinguish margin-level effects from ranking-level effects. These analyses provide useful evidence about the behaviour of sequential DPO in the tested setup.

Critical weaknesses

W1. Stable net accuracy does not directly establish retention of preferences newly learned in Stage 1.

Policy accuracy only tracks whether a pair's margin remains positive. A small net accuracy change can mask internal shifts where newly acquired rankings are lost while previously incorrect pairs flip to correct. Table 6 already highlights these dynamics: in HH harmless→helpful, 25 pairs are lost and 19 gained (25 lost and 14 gained for helpful→harmless). Thus, bounded net accuracy changes do not prove that Stage-1 specific updates survived Stage 2.

Furthermore, Stage-1 accuracy gains are often modest to begin with (Table 5: 46.6%→48.4% on HH harmless; 51.7%→52.4% on HelpSteer2 coherence). Stable final accuracy may thus reflect base-model knowledge preservation rather than Stage-1 preference retention. Binary accuracy also overlooks margin shrinkage that hasn't crossed zero.

While this doesn't invalidate the observation that reference-relative drops don't imply preference reversals, a three-checkpoint transition analysis (Base → Stage 1 → Stage 2) tracking the subset of pairs gained in Stage 1 and retained in Stage 2, along with continuous margin dynamics for this subgroup, may clarify true preference retention.

W2. The diagnostic contribution has limited practical value beyond the descriptive analysis.

As an empirical analysis, the paper does not require a novel algorithm, but its main diagnostic (compatibility) and mitigation (replay) offer limited actionable guidance. The nine-of-ten sign agreement is positive, but leave-one-dataset-out prediction yields only 7/10 accuracy against a 6/10 majority baseline (Section 4.2). Furthermore, compatibility fails to predict where replay helps, and replay primarily recovers reference-relative margins rather than policy accuracy (Section 4.3).

Consequently, the paper provides a clearer description of the phenomenon than a validated tool for predicting or restoring capabilities. The contribution would be stronger if the diagnostic enabled reliable pre-training decisions or if replay demonstrated tangible behavioral benefits beyond restoring reference-relative metrics. While the measurement insights remain interesting, the practical methodological utility appears limited.

W3. The experimental scope is too narrow to support broad conclusions about sequential preference optimization.

The core experiments rely heavily on Llama-3.1-8B-Instruct, LoRA, and DPO. While Appendix J reproduces the HH-RLHF pattern on Qwen2.5-7B-Instruct, both tested orderings show negative compatibility, leaving positive-compatibility settings, other datasets, and alternative adaptation methods unverified. Additionally, three-stage experiments are restricted to HelpSteer2.

It remains unconfirmed whether policy-ranking stability holds under full-parameter fine-tuning or alternative preference objectives like SimPO and IPO, which alter policy drift and margin evolution. Including at least one broader control (e.g., full-parameter tuning or SimPO) would clarify whether these findings reflect fundamental dynamics of sequential preference learning or artifacts of the LoRA-DPO setup.

W4. Likelihood-based diagnostics are insufficient to establish retention of downstream capabilities.

Evaluating fixed chosen/rejected test pairs measures preference scoring over supplied responses, but does not establish downstream generation quality or task performance. While Appendix K provides a generation-level check, its scope is limited: 200 prompts per ordering, a single sampled output per prompt, a reference-proximity judge, and the exclusion of Summarize-Feedback.

Furthermore, static rankings and generation metrics occasionally diverge. For instance, HelpSteer2 verbosity→coherence shows a 5.5 percentage-point drop in generation alignment (Table 16) despite a 0.8-point increase in length-normalized policy accuracy (Table 1). Validating retention across generation benchmarks (e.g., AlpacaEval 2, Arena-Hard, IFEval) alongside direct evaluations of the target capabilities (e.g., harmlessness or summarization quality) would strengthen these findings.

Questions for authors

Q1. What fraction of newly acquired correct rankings is subsequently lost? Could you separate pairs that were already correct at the base model from pairs that became correct only after Stage 1, then report Stage-2 retention and continuous margin changes for each group? This would complement the existing lost/gained table and distinguish preservation of base rankings from preservation of new learning.

Q2. How closely do the proposed diagnostics track downstream behaviour? Could you evaluate the same three checkpoints on independent generation benchmarks and direct measures of the earlier objective, then compare those changes with policy accuracy and compatibility? The HelpSteer2 result in Appendix K would be particularly useful to examine, as would the Summarize-Feedback orderings omitted from that check.

Q3. Does the main finding persist beyond the current LoRA-DPO setup? A full-parameter fine-tuning comparison or an experiment with another preference loss would help assess the scope of the finding. Extending the second-model replication to at least one positive-compatibility setting would also clarify whether the current result transfers beyond the HH-RLHF cases.

I would be happy to revise my assessment if the authors can provide stronger evidence connecting these diagnostics to newly learned preferences and downstream behaviour.

8 comments
AuthorsOpenReview ↗

Response to Reviewer YadG

Thank you for a thorough and constructive review. Below, we try our best to answer your questions with supporting evidence where necessary.

W1. Stable net accuracy does not directly establish retention of preferences newly learned in Stage 1.

Answer: To further understand this, we followed each test pair of the earlier objective (a prompt with a preferred and a dispreferred response) through the three checkpoints. We then separated newly learned pairs that the base model ranked incorrectly but the Stage-1 model ranked correctly from pairs that were already correct at the base model. To account for differences in Stage-1 margins, we matched each newly learned pair to the already-correct pair with the closest Stage-1 margin.

The results show that stable net accuracy does hide the loss of what Stage 1 newly learned. About a third of the rankings that Stage 1 newly acquired in HH are lost during Stage 2, compared with 1–4% of rankings that were already correct at the base model, in every one of three independently trained seeds (results below).

HH ordering, seed (DPO)Newly learned lostAlready-correct lostMargin change, newly learnedMargin change, already correct
harmless→helpful, 4234% (of 56)3% (of 414)−0.62+0.07
harmless→helpful, 12332% (of 62)3% (of 412)−0.54−0.17
harmless→helpful, 202434% (of 65)4% (of 417)−0.72−0.04
helpful→harmless, 4229% (of 31)2% (of 610)−0.46−0.70
helpful→harmless, 12338% (of 34)1% (of 612)−0.50−0.52
helpful→harmless, 202438% (of 32)2% (of 612)−0.69−0.58

Furthermore, the continuous margins add an interesting nuance. In harmless→helpful, the newly learned pairs also lose far more margin than the others. In helpful→harmless, both groups lose a similar amount of margin, yet the newly learned rankings flip far more often, because they sit closer to the decision boundary. This shows, at the level of individual pairs, that a drop in reference-relative margin and a reversal of the ranking are distinct events.

The pattern is consistent across settings. Against the matched controls, newly learned rankings are kept less often in every forgetting setting we tested, significantly so in most of them, including the second preference loss.

OrderingNewly learned keptMatched already-correct keptFisher p
SF coverage→accuracy (9 runs: 3 splits × 3 seeds)37–68%86–96%<0.05 in 8/9
HH harmless→helpful, seeds 42 / 123 / 202466% / 68% / 66%86% / 82% / 83%0.03 / 0.10 / 0.04
HH helpful→harmless, seeds 42 / 123 / 202471% / 62% / 62%90% / 79% / 81%0.11 / 0.18 / 0.16
HH harmless→helpful, IPO66% (of 215)90%<10−8<10^{-8}
HH helpful→harmless, IPO58% (of 93)84%<0.001
HH harmless→helpful, Llama-3.2-3B LoRA60%91%<0.01
UF IF→honesty84%91%0.52

Two further observations help interpret this. When Stage 2 is replaced by a null stage with randomly swapped labels, far fewer newly learned rankings are lost (PKU: 13/31 kept under real training vs 25/31 under the null, p=0.004; SF: 6/16 vs 13/16, p=0.029). And where the two objectives are compatible (UltraFeedback), there is no detectable loss of newly learned rankings (84% vs 91% kept, p=0.52). We have added this analysis to the paper.

W2. The diagnostic contribution has limited practical value beyond the descriptive analysis.

Answer: Thank you for this important question. We used this point to rethink both parts.

Replay. We designed a cleaner experiment to test whether replay restores only reference-relative margins or also the model's own rankings. The paper's original replay keeps the Stage-2 budget fixed, so it cannot tell whether a benefit comes from the earlier data itself or from how the training time is spent. We therefore ran three branches from the same Stage-1 model on HH, with three independently trained seeds. The first is the original Stage 2. The second is a control that receives 10% extra training on repeated Stage-2 data. The third is replay, which receives the same 10% extra in the form of Stage-1 data. The control and replay therefore have the same compute.

Ordering, seedRetention RAR_A: baseline / controlRetention RAR_A: replay [95% CI]Ranking accuracy on the earlier objective, replay − control (pp) [95% CI]New-objective learning, replay / baseline [95% CI]
harmless→helpful, 42−0.369 / −0.410+0.670 [+0.52, +0.82]+2.5 [+0.9, +4.1]0.94 [0.88, 1.01]
harmless→helpful, 123−0.478 / −0.406+0.836 [+0.68, +0.99]+3.1 [+1.3, +4.9]0.95 [0.88, 1.02]
harmless→helpful, 2024−0.531 / −0.498+0.745 [+0.58, +0.91]+2.1 [+0.4, +3.8]0.84 [0.78, 0.91]
helpful→harmless, 42−0.398 / −0.472+0.092 [+0.01, +0.17]+1.1 [0.0, +2.3]0.99 [0.94, 1.05]
helpful→harmless, 123−0.314 / −0.340+0.090 [+0.02, +0.16]+0.8 [−0.2, +1.8]1.10 [1.03, 1.17]
helpful→harmless, 2024−0.367 / −0.414+0.128 [+0.05, +0.20]+1.5 [+0.5, +2.5]0.91 [0.86, 0.97]

We observed that the compute-matched control forgets just as much as the original Stage 2, while with replay the earlier objective's margin rises rather than falls, in all six runs. So the benefit comes from the earlier data, not from extra training. Most importantly for your concern, replay also improves the policy's own ranking accuracy on the earlier objective, by 2.1–3.1 points in harmless→helpful (confidence intervals above zero in all three seeds) and by 0.8–1.5 points in helpful→harmless. This recovery comes at little cost to the new objective. Replay reaches 91–110% of the baseline's gain on the Stage-2 objective in helpful→harmless and 84–95% in harmless→helpful. We present this as a favourable trade-off, with a modest reduction in new learning in exchange for recovering what would otherwise be forgotten. The recovery is also visible in generated behaviour. In harmless→helpful, replay lowers harmful compliance relative to the compute-matched control in both seeds (StrongREJECT score −0.025 and −0.059), and the HarmBench classifier and Llama Guard 3 agree.

Finally, we checked whether a model partway between Stage 1 and Stage 2 would achieve the same. We interpolated exactly between the Stage-1 and Stage-2 models and found that retention and new learning trade off almost linearly along the way. Moving partway back towards the Stage-1 model therefore buys less forgetting only at the same price in new learning, whereas replay recovers the earlier objective while keeping most of the new learning.

Compatibility. We agree that compatibility should not be used as a forecasting tool for training decisions, and the revised paper no longer presents it that way. What it does capture reliably is how strongly two objectives transfer. On eight new orderings from four objective pairs that were not used during development (UltraFeedback truthfulness × helpfulness, HelpSteer2 complexity × helpfulness, and two CodeUltraFeedback pairs), with predictions frozen before Stage 2 was trained, it ranked the size of transfer with Spearman 0.93 (0.95–0.98 for two closely related variants). The revision therefore presents compatibility as a descriptive measure of transfer strength and gives the practical role to compute-matched replay, which, as shown above, restores the earlier objective in both margins and ranking accuracy. Because replay is now evaluated directly against a compute-matched control, deciding where it helps no longer depends on compatibility.

W3. The experimental scope is too narrow to support broad conclusions about sequential preference optimization.

Answer: We extended our experiments and evaluations to the broader context that was suggested, and the central forgetting result holds in each.

Full-parameter training and a second model, including positive transfer. On Llama-3.2-3B-Instruct, we trained all ten orderings with full-parameter DPO (no LoRA) and four orderings with LoRA using the same recipe as for 8B. HH forgetting and UltraFeedback's positive transfer both reproduce as demonstrated below:

Ordering8B LoRA3B LoRA3B full-parameter
HH harmless→helpful−0.37 [−0.45, −0.28]−0.36 [−0.43, −0.28]−0.11 [−0.16, −0.06]
HH helpful→harmless−0.40 [−0.48, −0.32]−0.31 [−0.37, −0.25]−0.13 [−0.16, −0.10]
UF IF→honesty+6.43 [+5.91, +6.96]+5.31 [+4.90, +5.72]+2.72 [+2.53, +2.92]
UF honesty→IF+8.41 [+7.68, +9.23]+6.18 [+5.60, +6.79]+3.28 [+2.98, +3.59]

The full-parameter effects are smaller because Stage 1 itself learns less at the conservative learning rate we used. However, relative to how much Stage 1 learned, the amount of HH forgetting is very similar (R/acq=-0.31 on 3B vs -0.36 on 8B). PKU harmless→helpful also retains harmlessness with full-parameter training.

A second preference loss. We repeated the HH experiments with IPO in both stages. On the matched (length-normalised) metrics, forgetting appears in both orders and is in fact stronger than with DPO: the length-normalised margin falls by 0.10 and 0.19 (DPO: 0.044 and 0.048), and ranking accuracy on the earlier objective falls by 4.4 and 3.0 points (DPO: 0.7 and 0.9). The loss of newly learned rankings is also visible under IPO (see W1).

Independent seeds. The HH result is consistent across three independently trained seeds (both stages retrained each time), with every confidence interval below zero:

OrderingSeed 42Seed 123Seed 2024
HH harmless→helpful−0.37 [−0.45, −0.28]−0.48 [−0.56, −0.39]−0.53 [−0.62, −0.44]
HH helpful→harmless−0.40 [−0.48, −0.32]−0.31 [−0.38, −0.25]−0.37 [−0.45, −0.29]

Three-stage sequences. We agree that extending the three-stage analysis beyond HelpSteer2 would be valuable, and we thank you for the suggestion. Given the limited time, compute and budget available for this revision, we prioritised the questions that bear most directly on the paper's central claims such as clean data, the changes in newly learned rankings, full-parameter training, a second preference loss, independent seeds and generation-level evaluation. The three-stage experiments therefore remain on HelpSteer2 in this version. We strongly believe that longer sequences across more preference settings are a natural and exciting extension of this work, and the revised paper now describes them explicitly as future work.

Reviewer_ymACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies the preference degradation under sequential DPO. The authors define three metrics to measure the preference change from two perspectives and claim that large reference-relative declines do not necessarily mean that the model has reversed its preferences. They further propose a metric named compatibility to predict the preference change before the stage2 DPO and study whether it helps with replay strategy. The extensive experimental results support the major claim of the paper.

Critical strengths
  1. Through extensive experiments, the paper demonstrates that reference-relative preference changes are not necessarily equivalent to the model's actual preferences. This finding is meaningful for research on sequential DPO, as it motivates the development of metrics that better reflect the model's true preferences and can more effectively guide training.

  2. The paper introduces a novel metric, compatibility, to predict preference changes in Stage 2 and guide the design of replay strategies. Although compatibility-guided replay shows clear improvements only in terms of reference-relative metrics, it still represents a novel and promising approach.

Critical weaknesses

There seems to be a clear issue of data contamination in the experiments. Specifically, the test set from Stage 1 appears in the training set of Stage 2, which substantially undermines the reliability of the experimental conclusions. The authors state that this issue was discovered after the main analysis. Although such overlap during training may be difficult to avoid, ensuring separation of the test sets should still be feasible. Moreover, as shown in Section 4.1.2, “The top 2% of pairs contribute 86.5% of the total margin change.” It remains unclear whether these top-contributing pairs are strongly correlated with the contaminated pairs. The authors do not provide any analysis to investigate this potential relationship.

Questions for authors
  1. As stated in the weakness, whether the top-contributing pairs in margins are strongly correlated with the contaminated pairs?

  2. Using aggregate metrics, including margin and policy accuracy, may obscure substantial preference changes at the sample level, as suggested by Table 6 and Section 4.1.2. Therefore, a more fine-grained sample-level analysis would be valuable. Specifically, for samples exhibiting large margin changes, do they also exhibit policy preference flips? Conversely, for samples where policy preference flips occur, how do their corresponding margins change? If these two phenomena are found to be weakly correlated, this would provide stronger evidence in support of the paper's central claim.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ymAC

We thank the reviewer for the thoughtful review. We have carried out careful experiments to clarify the concerns and address queries.

1. There seems to be a clear issue of data contamination in the experiments. Specifically, the test set from Stage 1 appears in the training set of Stage 2. […] ensuring separation of the test sets should still be feasible.

Answer: We agree that separating the test sets was both feasible and necessary. Following the suggestion, we carried out separate experiments by removing the overlap to understand how much it mattered. All five datasets were rebuilt so that no prompt, near-duplicate prompt or turn of the same conversation is shared between any Stage-1 test set and any training set, and all ten orderings were retrained. We then assessed whether the paper’s central conclusions change and found that they “do not change”.

OrderingSubmitted RClean R [95% CI]Changed ?
HH harmless→helpful−0.21−0.37 [−0.45, −0.28]forgetting holds
HH helpful→harmless−0.51−0.40 [−0.48, −0.32]forgetting holds
UF IF→honesty+6.90+6.43 [+5.91, +6.96]transfer holds
UF honesty→IF+7.79+8.41 [+7.68, +9.23]transfer holds
PKU harmless→helpful+2.86+2.12 [+1.88, +2.38]retention holds
HS2 coherence→verbosity−0.21−0.26 [−0.72, +0.22]same sign
HS2 verbosity→coherence−0.25−0.17 [−0.38, +0.05]same sign
PKU helpful→harmless+0.62−0.07 [−0.31, +0.19]updated (≈ 0)
SF coverage→accuracy+0.28−1.81 [−2.69, −0.95]updated (forgetting)
SF accuracy→coverage+1.71+0.80 [−0.41, +2.01]same sign

The three findings the paper builds on (forgetting in HH, positive transfer in UF and retained harmlessness in PKU) keep their sign and remain clearly different from zero. We further checked that these are not single-run effects. HH forgetting replicates in two additional, independently trained seeds (R=-0.48 and -0.53 for harmless→helpful; -0.31 and -0.37 for helpful→harmless), and the SF coverage→accuracy result is negative across all nine runs of a three-split × three-seed grid. The two orderings whose values changed have been updated in the revised version.

2. whether the top-contributing pairs in margins are strongly correlated with the contaminated pairs?

Answer: Thank you for pointing this out. For the ordering where the figure was 86.5% (PKU helpful→harmless), the answer is yes. 16 of its 20 top-contributing pairs (80%) were overlap pairs, whereas overlap pairs make up only 6.5% of all test pairs in that ordering (p<0.001). In the clean data, the total change for that ordering is close to zero, so a concentration figure is no longer meaningful, and we have updated the statement accordingly. Where overlap was not driving the change, concentration is stable on clean data. The top 2% of pairs account for 9.0% and 10.1% of the change in the two UltraFeedback orderings and 16.1% in PKU harmless→helpful. In short, the overlap mattered in one ordering, PKU helpful→harmless, where it produced the 86.5% figure. In the three orderings above, concentration remains modest.

3. for samples exhibiting large margin changes, do they also exhibit policy preference flips? Conversely, for samples where policy preference flips occur, how do their corresponding margins change? If these two phenomena are found to be weakly correlated, this would provide stronger evidence in support of the paper's central claim.

Answer: Thank you for suggesting this pair-level analysis. We examined both directions on the clean data and found the weak association your question points to in most orderings.

Looking from margins to flips, we first looked at the pairs whose margins changed the most, the top 10% in each ordering. Even these pairs rarely flip their ranking. In seven of the ten orderings, 8% or fewer of them flip. Across all pairs, the size of a pair's margin change is only weakly related to whether it flips. The correlation is 0.15 or lower in eight of the ten orderings.

OrderingFlip rate top 10% of |margin change|Flip rate other pairsSpearman ρ
HH harmless→helpful2.0%5.7%0.11
HH helpful→harmless8.0%2.8%0.11
HS2 coherence→verbosity2.7%2.4%0.15
HS2 verbosity→coherence0.0%2.0%0.03
PKU harmless→helpful2.0%3.2%0.03
PKU helpful→harmless26.0%6.9%0.28
UF IF→honesty7.1%4.0%0.02
UF honesty→IF7.1%4.3%0.07
SF coverage→accuracy63.6%11.7%0.34
SF accuracy→coverage30.0%6.1%0.11

The table uses length-normalised margins, which match how the policy ranks the pair. With the summed DPO reward margins the picture is the same. The correlation is again 0.15 or lower in eight of the ten orderings.

Looking from flips back to margins gives the same picture. The pairs that flip do not have unusually large margin changes. Their median change sits between the 48th and 83rd percentile. What sets them apart is that they were already close to the decision boundary after Stage 1. Their median absolute Stage-1 margin is 0.008–0.33, compared with 0.29–1.09 for the other pairs. In every ordering, the closer a pair was to the boundary, the more likely it was to flip (correlation −0.17 to −0.34). The two orderings with stronger coupling, SF coverage→accuracy and PKU helpful→harmless, are the two orderings in which ranking accuracy on the earlier objective falls significantly.

This pair-level analysis supports our distinction between reference-relative margin changes and ranking reversals. Flips were more closely associated with proximity to the Stage-1 decision boundary than with the size of the subsequent margin change. We have added the analysis to the revised paper.

We hope this has addressed your concerns. Please let us know if any further questions and comments remain. Thank you again.

Thank you for providing additional experiment results. After decontamination, the major conclusion still holds and several statements are updated. These increase the soundness of the paper. The additional pair level analysis provide more evidence to support the main claim. Therefore, I decide to raise my score.

Thank you very much for revisiting the review and for raising the soundness score. We really appreciate that the additional decontaminated experiments and pair-level analysis helped address your main concerns.

If there are any remaining concerns that you feel are not yet fully addressed, we would be very grateful to hear them, and are happy to further clarify during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.