- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why preference optimization can improve held-out ranking metrics while severely degrading language generation. The authors identify a common-mode offset in the final hidden states, defined as the shared component of the displacement between a trained model and its base checkpoint. They argue that relative ranking objectives provide insufficient constraints on this shared shift because its effects on chosen and rejected responses largely cancel in the ranking margin. Through the output projection, however, the offset can disproportionately increase the logits of particular vocabulary items, creating a vocabulary well and potentially causing generation collapse.
The paper evaluates 17 base-model lineages across two preference datasets and considers both LoRA-based and full-parameter fine-tuning. The authors report collapse signatures in 14 of the 17 lineages under their per-token ranking objective. They support the proposed mechanism through several interventions, including isolating the offset, transplanting it into a base model, and centering the rejected-side loss to prevent the well from forming. They also propose offset regularization that prevents well formation in several model families for up to three times the base training budget while largely preserving held-out ranking performance. Finally, they report a negative correlation between held-out ranking accuracy and the preference of two external reward models for the trained models' generated samples. The paper's central contribution is a representation-level explanation of a failure mode that conventional ranking metrics may not detect.
Critical strengths
(1) The paper does more than observe that hidden-state displacement correlates with degraded generation. The authors show that the common-mode component alone can reconstruct the vocabulary well, that transplanting the learned offset into an untreated base model can induce the corresponding failure, and that norm-matched random directions do not produce the same behavior. They further show that centering the rejected-side loss can prevent well formation and that subtracting the offset's logit contribution can substantially restore generation in a collapsed model. Taken together, these interventions provide compelling evidence that the offset is causally involved in the observed failure mode.
(2) The study covers 17 base-model lineages, five tokenizer families, two datasets, and both adapter-based and full-parameter training. It also examines multiple preference objectives rather than treating all preference optimization methods as equivalent. The proposed offset-control strategy preserves strong held-out ranking performance while preventing well formation across several model families at up to three times the base training budget. This combination of mechanistic analysis, cross-model experiments, and a potential diagnostic or mitigation method makes the work relevant to the development and evaluation of preference-optimized language models.
Critical weaknesses
(1) The paper distinguishes likelihood displacement, vocabulary-well formation, and terminal generation collapse, but its operational criterion counts a run as collapsed when either a vocabulary well forms or sampled generations substantially degenerate. Consequently, the reported result that 14 of 17 model lineages collapse does not necessarily mean that all 14 exhibit comparably severe failures in sampled text. For example, some entries in Table 4 exhibit high well capture but relatively low measured degeneration. Separating well-only cases from runs with clear sampled-generation collapse would make the empirical conclusion easier to interpret.
Moreover, the strongest results concern the authors' per-token ranking objective, whose stated demand is approximately 42 nats per token at the main setting. Textbook DPO remains below the proposed well-formation threshold at three times the base budget, and SimPO initialized from the public SFT checkpoint does not form a well within the tested budget. These differences are scientifically interesting, but they also suggest that the observed failure depends substantially on the objective, its effective margin demand, initialization, and training budget. The paper should be careful not to generalize the strongest collapse findings to preference optimization as a whole.
(2) The reported negative association between ranking accuracy and external reward-model preference is suggestive, but the evaluation uses two public reward models and one generated sample per prompt across 300 held-out UltraFeedback prompts. Without uncertainty estimates, human preference judgments, or analyses controlling for objective and training budget, it is difficult to determine how generally the observed correlation reflects a true trade-off between ranking performance and response quality.
The mitigation experiments also have a bounded scope. The penalty strengths are calibrated on the driving dataset, and the main cross-family prevention results extend to three times the base budget. At six times the budget, the OLMo experiments show that probability mass can move into alternative failure modes, including punctuation, end-of-text tokens, and word-family rings; the residual penalty does not fully resolve every failure mode in every tested lineage. In addition, the proposed early-warning signal is evaluated retrospectively on completed training runs rather than as a prospective monitoring procedure. These limitations do not invalidate the proposed mechanism, but they weaken the broadest claims about robust prevention and practical early detection.
Questions for authors
(1) Could the authors report the number of runs that exhibit (i) vocabulary-well formation without substantial sampled-generation degeneration, (ii) clear sampled-generation collapse, and (iii) diffuse generation degradation? In particular, how many of the 14 out of 17 lineages satisfy the sampled-generation criterion independently of the vocabulary-well criterion? This distinction would materially affect the interpretation of the headline result. For the negative correlation between held-out ranking accuracy and external reward-model preference, have the authors tested whether the relationship persists after controlling for objective type and training budget, or across independent training seeds? Could uncertainty estimates or additional evaluation results clarify the robustness of this relationship, given that the current analysis uses two reward models and one sample per prompt for 300 prompts?
(2) Evaluating additional preference objectives, including IPO, ORPO, and KTO, as well as more instruction-tuned or SFT-initialized models, would help establish the generality of the proposed mechanism. Prospective validation of the offset-based early-warning signal and broader human or model-based quality evaluations would also strengthen the practical implications.
2 comments
Response to Reviewer 1Zkk (2/2): the correlation holds on new prompts, and alarms fixed in advance caught 15 of 17 damaged runs
Continued from part 1 (items 1 to 4).
5. Is the correlation between ranking accuracy and reward robust? (W2, Q1) It holds with more samples per prompt and on new prompts, the main run repeats across seeds, and it now comes with intervals. Across the 17 UltraFeedback runs of Table 7, the runs that rank held-out answers better win less often, with a Spearman correlation of −0.78 (−0.94 to −0.41) under Skywork-Reward-V2 and −0.79 (−0.94 to −0.45) under ArmoRM. Four samples per prompt or 133 unused prompts leave both correlations between −0.76 and −0.80, and two more seeds of the main per-token run of Table 7 win 23.5% and 23.8%, against 24.5%. Controlling for objective family and budget lowers the correlation to −0.63 (−0.96 to 0.38) and −0.28 (−0.92 to 0.55), with intervals that include zero, so the abstract and §8 now say that the trend is weaker after these controls. The same pattern appears within one objective and budget in the authors' text format, in the table of item 2 in part 1, where every learning rate above the official one has higher ranking accuracy and a lower win rate (Appendix C.7).
6. Do other quality evaluations agree with the reward models? (W2, Q2) Yes, two chat-model judges and AlpacaEval 2 do. When Tülu 3 and Olmo-3 Instruct rate each answer alone from 1 to 9, the runs of Table 7 with higher ranking accuracy again get lower ratings, with Spearman correlations of −0.76 and −0.74. AlpacaEval 2 points at the same damaged run. On its 805 prompts, the length-controlled win rate of SimPO at 1e-6 is 3%, below the 9% of its starting model Zephyr SFT, while standard DPO and SimPO with an NLL term at the same learning rate reach 21% and 23%, against 27% for the public SimPO model. The judge is GPT-4o-mini because GPT-4 Turbo is no longer served, so these numbers are not comparable with the public leaderboard. We have no human judgments, and the limitations section says so (Appendix C.7, C.11).
7. Do the alarms work prospectively? (W2, Q2) Yes. We fixed two alarms and wrote down our predictions before 44 new training runs. Together they caught 15 of the 17 runs that ended damaged (88%) and raised 4 false alarms in 24 healthy runs (17%). The offset alarm is the submission's dose, now computed for whichever token the offset raises most at that step, and it fires when this dose passes one. The entropy alarm is new and fires when the model's output entropy on a fixed set of examples reaches 1.5 times its starting value, a threshold chosen on 9 older checkpoints outside the 44 runs.
| final win rate | runs | entropy alarm fires | offset alarm fires | either fires |
|---|---|---|---|---|
| below 40% | 17 | 12 | 4 | 15 |
| 40% to 50% | 3 | 2 | 0 | 2 |
| 50% or above | 24 | 4 | 0 | 4 |
The offset alarm singles out collapse onto one string. It fired in all 3 runs among the 44 that formed a well and in one more damaged run, three-epoch SimPO at 1e-6, where /******/ appears in 78% of the answers without reaching the well criterion. It fired in none of the 24 healthy runs. The entropy alarm caught most of the runs damaged without a well. Acting on an alarm rescued the run. Switching on an NLL term of weight 0.1 when the alarm fired brought the two SimPO runs at 1e-6 from 48% and 38% to 67% and 65%. The alarms fire close to the drop in quality, so the abstract and §8 now present them as detectors. They come 50 to 240 steps before the first evaluation below 40% in 6 of the 15 caught runs, at that evaluation in 8, and 30 steps after it in one.
The alarms caught all 15 damaged runs trained without centering. The two they missed were trained with centering. In both, answers end too early (see item 8) and output entropy falls instead of rising. Neither alarm fired in the Dutch run of item 2, which is outside the 44 because our reward models cannot score Dutch. Its dose reached 0.90 and its output entropy also fell. The counts on the 87 earlier runs stay in the paper, marked as retrospective (Appendix C.3).
8. Does offset control prevent wells beyond the data it was tuned on? (W2) Yes. Centering has no weight to tune, and in the public Dutch failure of item 2 it removes /******/ from all 300 samples. The penalty strengths were set on the driving data and not tuned again, and with them no well forms on any of the four base models of Table 3, which train on that data. The OLMo failures at six times the budget stand as reported, and §7 and §9 now claim offset control only for preventing wells up to three times the base budget. Reward is protected by a different remedy, an NLL term of weight 0.1, which keeps SimPO at 1e-6 at 65% against 48% without it. Centering alone lowers the official SimPO recipe from 68% to 51%, and we traced this cost. Without centering, the offset pushes the end-of-sequence token down harder than any other token. Centering removes that push, and a quarter of the answers then end within two words (Appendix C.9).
Response to Reviewer 1Zkk (1/2): degenerate samples in 13 of the 14 collapsed lineages, and IPO, DPO and SimPO fail at published settings
Thank you for the review. Items 1 to 4 are here and items 5 to 8 in a second comment.
1. How many of the 14 collapsed lineages have broken samples? (W1, Q1) Thirteen. In each, most sampled answers are degenerate, so they meet the sample criterion on their own. The fourteenth,
Qwen2.5-0.5B, has a well only, with one token taking 94% of the probability mass shifted by training and 12% of its samples degenerate. Two more lineages,Gemma-2-2BandPhi-3-mini, form no well within three times the base budget and degrade diffusely at six times. They stay outside the count, andLlama-3.2-3Bshows none of the three. Your three categories therefore hold 1, 13 and 2 lineages. The Table 4 rows with high capture and low measured degeneration areLlama-3.1-8B,OLMo-2-7BandQwen2.5-0.5B. The submission's audit counted words by spaces and missed long token strings without spaces. A re-audit that catches them flags 52% and 73% of the samples of the first two, which raises the count from 11 to 13, and leavesQwen2.5-0.5Bas the well-only case. The abstract and §1 now report this split, and Table 15 lists every lineage at its budget (Appendix C.13).2. Do the findings reach beyond our per-token objective? (W1) Yes. Starting from public SFT models, IPO, standard DPO and SimPO each fail at a published setting, and the offset is involved every time.
zephyr-7b-sft-full, the Mistral-7B SFT model that the SimPO paper starts from (Zephyr SFT below), we run TRL's IPO at τ = 0.01 at both ends of the learning-rate range that paper searched for IPO. In both runs/******/takes over the output, every sampled answer is degenerate, and a reward model prefers the trained model's answer to Zephyr SFT's only 6% of the time. The offset's dose on this string passes one in both, the necessary condition of §6.GEITje-7B-ultra-sftand its data, reproduces the failure that the model's developer reported, with 99% of answers repeating/******/. Centering, the paper's fix that acts only on the offset, removes the string from every sample.Win rate is how often Skywork-Reward-V2 prefers the trained model's answer to its starting model's on 300 held-out prompts, with 95% intervals in parentheses.
In each case the failure sits where the objective pushes harder, at the smallest τ, the smaller β, the larger learning rate and the longer run. Standard DPO at 1e-6 still wins 67%, so we do not extend these failures to preference optimization as a whole. The abstract, §1 and §6 ("Where the failure occurs") state this boundary together with the settings that stay healthy.
3. Do IPO, ORPO and KTO show the mechanism? (Q2) Yes. The offset appears under all three, and a well forms only where its dose passes one (§6). From Zephyr SFT at 1e-6, the offset ends at 11% of the hidden-state norm under ORPO at λ = 0.1, 12% under KTO at β = 0.1, and 15% to 90% under IPO as τ falls from 1.0 to 0.01. The dose passes one only under IPO at τ = 0.01, where the well of item 2 forms, and stays below 0.5 in every other run, where none forms. ORPO, KTO and IPO at τ = 0.5 and 1.0 win 63%, 70%, 60% and 57%, while IPO at τ = 0.1 ends at 15% (11.0 to 19.0) without a well. From the Llama-2 base model, IPO at τ = 0.02 also forms a well (Appendix C.2).
4. Does the mechanism appear from SFT and instruction-tuned starts? (Q2) Yes. From Zephyr SFT, IPO forms a well within one epoch and SimPO forms one after six epochs, with a win rate of 5% (2.3 to 7.0). Of six instruction-tuned starts under our per-token loss, three form wells and three degrade diffusely. From the SimPO authors' Llama-3-8B SFT model, SimPO formed no well even at 12 times the usual training length. In the authors' text format on the full data, the official recipe from that model wins 66% after one epoch, level with the public model's 65%, and 69% after three. The 34% at three epochs in the last row of Table 7 came from our earlier text format and a 15.7k-pair subset, and the caption now says so (Appendix C.4, C.8).