- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies forgetting in sequential DPO. The authors compare reward margins with the model's own preference accuracy across five datasets. They find that reward margins can drop while preference accuracy stays stable. They also propose a Stage-1 compatibility score to predict retention and test data replay to reduce forgetting.
Critical strengths
-
Useful evaluation perspective. The paper shows that a drop in the DPO reward margin does not always mean the model has reversed its preference. This is useful when evaluating forgetting.
-
Fairly broad experiments. The paper covers five preference datasets, different training orders, and additional three-stage experiments. It also reports confidence intervals and several diagnostic results.
Critical weaknesses
-
A continuous reward margin can decrease without changing the sign of a preference ranking. This is expected from the metric definitions. Also, the DPO margin uses summed log-probabilities, while the main policy accuracy is length-normalized. It is unclear how much of the reported gap reflects a new finding rather than these metric differences.
-
Sequential preference alignment and forgetting have been studied in SPO (Lou et al., 2025), COPR (Zhang et al., 2025), and LifeAlign (Li et al., 2026). Harmon et al. (2026) also studied forgetting at the sample level. This paper adds a useful metric comparison, but the overall contribution beyond these studies seems limited.
-
Stage-1 compatibility is simply the model's margin on the incoming objective. Its sign matches retention in 9 of 10 orderings, but these come from only five datasets. Leave-one-dataset-out prediction gets 7/10 correct, compared with 6/10 for the majority baseline. The paper also provides no clear theoretical reason why this score should predict retention.
-
Replay improves policy accuracy in only 2 of 9 orderings and sometimes hurts learning of the new objective. Most results also rely on ranking fixed response pairs rather than generated outputs. This makes the practical benefit less clear.
Questions for authors
-
Can the authors better explain why the gap between reward margins and policy accuracy is more than an expected result of their definitions? Would the main finding hold under matched length-normalized metrics?
-
What is the key difference from SPO, COPR, LifeAlign, and the sample-level forgetting analysis of Harmon et al.?
-
Can the compatibility score be tested on more datasets and models? Does it still work when overlapping training and evaluation pairs are removed?
-
Can the authors show that replay improves actual generated responses without hurting the new objective? How does it compare with existing continual alignment methods?
6 comments
W2. Sequential preference alignment and forgetting have been studied in SPO (Lou et al., 2025), COPR (Zhang et al., 2025), and LifeAlign (Li et al., 2026). Harmon et al. (2026) also studied forgetting at the sample level. This paper adds a useful metric comparison, but the overall contribution beyond these studies seems limited.
Answer: These are important works, and the revision discusses them explicitly. They also help clarify what our analysis adds.
-
How forgetting is measured: SPO evaluates sequential preference alignment primarily with GPT-4 pairwise win rates and alignment scores, with additional reward- and cost-model analyses. COPR evaluates continual learning using task-level normalized Preference Scores and standard continual-learning measures including average accuracy, average incremental accuracy, forgetting measure and backward transfer, alongside reward-based, GPT-4 and human evaluation. LifeAlign similarly reports aggregate Last Performance, Backward Transfer and Average Performance, where task performance is measured with BLEU-4, ROUGE-L and an LLM judge. Harmon et al. move beyond task averages and measure sample-level 1→0 and 0→1 correctness transitions on knowledge benchmarks. In contrast, we follow every held-out preference pair through the base model, Stage 1 and Stage 2, and evaluate it using both the fixed-reference DPO margin and the policy's own ranking.
-
Whether a falling margin means a lost preference: COPR additionally monitors the average preferred–dispreferred probability gap on samples in its memory buffer, but its continual-forgetting measures themselves remain task-level aggregates. We resolve this distinction pair by pair on held-out data and compare it with the reference-relative margin. This shows that a large reference-relative margin change often does not correspond to a reversed policy preference: margin changes and ranking flips are weakly coupled in most settings, and flips concentrate near the Stage-1 decision boundary.
-
Which preferences are lost: Harmon et al. show that aggregate accuracy can hide simultaneous item-level forgetting and backward transfer. We extend this analysis to preference alignment and ask a further question: which kind of preference is being lost? In HH, 29–38% of rankings newly acquired during Stage 1 are subsequently lost, compared with only 1–4% of rankings that were already correct before Stage 1. Crucially, the difference remains after matching on Stage-1 confidence: newly learned pairs retain only 62–71% of their rankings, versus 79–90% for matched already-correct pairs. Thus aggregate retention can hide a systematic concentration of forgetting on what the earlier stage actually learned.
-
Training order and replay: For SPO, the original training dataset is trained harmlessness→helpfulness, while the reversed helpfulness→harmlessness order is used with a modified dataset; thus training order is not reversed while holding the dataset fixed. LifeAlign studies order sensitivity across forward, reverse and random six-task sequences. We instead run both orders on the same data for every objective pair, allowing us to show directly that retention depends not only on which objectives are combined but also on which one is learned first. COPR formulates continual learning with a historical memory buffer and evaluates experience-replay-based variants, while LifeAlign explicitly incorporates a fixed-size rehearsal buffer into its FPO procedure. Our replay experiment adds an equal-compute control: additional training on repeated Stage-2 data does not prevent forgetting, whereas replacing the same amount of extra training with Stage-1 replay restores retention. This isolates the benefit of revisiting earlier data from the benefit of simply training longer.
Together, these results complement optimization-focused continual-alignment methods by resolving what aggregate retention measures can obscure at the level of individual preferences. Aggregate measures can overstate preference loss when reference-relative margins fall without ranking reversals, and can understate it when losses are concentrated on preferences that the earlier stage newly acquired. Our pair-level diagnostic can be applied directly to SPO, COPR, LifeAlign or other continual-alignment methods to determine which preferences are actually retained. The compute-matched replay results further show that simple replay is a strong baseline in the HH setting and motivate equal-compute replay controls when evaluating more complex continual-alignment methods.
W3. Stage-1 compatibility is simply the model's margin on the incoming objective. Its sign matches retention in 9 of 10 orderings, but these come from only five datasets. Leave-one-dataset-out prediction gets 7/10 correct, compared with 6/10 for the majority baseline. The paper also provides no clear theoretical reason why this score should predict retention.
Answer: We tested compatibility on more data and without overlapping pairs, and the results provided a clearer view of what it measures.
On more data, we built eight new orderings from objective pairs never used in developing the measure (UltraFeedback truthfulness × helpfulness, HelpSteer2 complexity × helpfulness, and two CodeUltraFeedback pairs) and froze all predictions before training Stage 2. Compatibility ranked the strength of transfer across these new orderings very well (Spearman 0.93, and 0.95–0.98 for two closely related variants). On a second model (Llama-3.2-3B, full-parameter, all ten orderings), its sign agreement with retention reproduces the 8B pattern exactly (the same 7 of 10 orderings).
Without overlapping pairs, compatibility's agreement with the sign of retention weakens. Sign agreement falls from 9/10 to 5/10 and leave-one-dataset-out accuracy from 7/10 to 1/10, and the Pearson correlation is 0.90. On the clean retrained runs the corresponding figures are 7/10, 4/10 and 0.91.
The results point to an explanation for this, which also addresses the theoretical question. Compatibility measures how training on objective A affects objective B, while retention measures how training on B affects A. The two coincide when this influence is symmetric, as in HH and UltraFeedback, and diverge when it is not, as in PKU and SF accuracy→coverage. Compatibility is therefore a measure of transfer strength between objectives, which is exactly what the held-out test confirms, rather than a forecast of retention. For this reason, we agree that compatibility should not be used as a forecasting tool for training decisions, and the revised paper no longer presents it that way. It is presented as a descriptive measure of transfer strength, and the practical role goes to compute-matched replay, which restores the earlier objective in both margins and ranking accuracy (see W4).
Response to Reviewer pzVZ
Thank you for a clear and thoughtful review, and for finding the evaluation perspective useful. Further experiments and analysis were carried out to answer the remaining concerns and queries, provided below.
Answer: Part of the gap does follow from the definitions, and we now report everything under matched, length-normalised metrics so that readers can see exactly which part. The table below uses matched definitions on the clean data.
In HH, the matched margin still falls while ranking accuracy stays stable. Most of this gap comes from where the pairs sit after Stage 1. GIR counts a pair as improved when its margin is above that of the reference model, and about 55% of pairs sit so close to that line that a typical Stage-2 change can push them across it. Only 11–18% sit that close to the model's own decision boundary, which is what ranking accuracy counts. The same small shift therefore lowers GIR much more than ranking accuracy. Shifting every pair by the average change already predicts a GIR drop of 10–14 points, at least as large as the observed 6–8 points. For HelpSteer2, the matched metrics show that the earlier margin decline does not persist under length normalisation, so we have revised that claim. In SF coverage→accuracy, both the margin and the ranking accuracy fall, so the forgetting is real under either definition.
What goes beyond the definitions appears at the level of individual pairs. First, which rankings are lost is not random. In HH, about a third of the rankings newly learned in Stage 1 are lost in Stage 2 (29–38% across both orders and three seeds), compared with only 1–4% of rankings that were already correct at the base model, and newly learned rankings are lost more often than older ones even when both are matched on Stage-1 margin. Because this comparison holds the Stage-1 margin fixed, it cannot follow from the metric definitions. Second, large margin changes and ranking flips are largely different events. The correlation between the size of a pair's margin change and whether it flips is 0.15 or lower in eight of the ten orderings, and the pairs that flip tend to be those that sat close to the decision boundary after Stage 1.