acceptodds
Under review as a conference paper at ICLR 2027

ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization

Abstract

Preference optimization can improve every ranking metric while building a hidden, shared representation shift that the relative loss barely sees, one that can eventually destroy generation. We trace this failure to the common-mode offset, a shared directional shift in the model's hidden states that appears across token positions, prompts, and both chosen and rejected responses. Because it moves chosen and rejected representations together, the relative ranking loss barely reacts to its accumulation. As this offset grows, it projects onto the output layer to form a vocabulary well, elevating a single token's probability until it dominates decoding and causes generation collapse. We demonstrate the causal role of this mechanism through targeted interventions. The isolated offset alone reconstructs the vocabulary well, injecting the learned offset into an untreated base model recreates the collapse where norm-matched random directions do not, and masking this common component from the loss prevents the well from forming. Under the per-token ranking loss we study, the same collapse appears in 14 of 17 base-model lineages and under full-parameter fine-tuning. Moreover, the offset is not merely an artifact of collapse, since it is already present in textbook DPO and SimPO runs that form no well. Textbook DPO keeps it weakly aligned with any single vocabulary direction even with three times our base budget. Trained from base models for that budget, however, full-parameter SimPO drives it into collapse on both Llama-2 and Llama-3.1, and the same kind of offset grows from a public SFT checkpoint under the official SimPO recipe. Across 17 UltraFeedback runs, the better a run ranks held-out responses, the less often two public reward models prefer its samples to the base model's. These findings expose a blind spot in standard ranking metrics and introduce a representation-level signal that can warn of generation collapse before it occurs.

Primary area:
interpretability and explainable AI
Keywords:
preference optimization, direct preference optimization, listwise preference learning, representation geometry, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1ZkkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why preference optimization can improve held-out ranking metrics while severely degrading language generation. The authors identify a common-mode offset in the final hidden states, defined as the shared component of the displacement between a trained model and its base checkpoint. They argue that relative ranking objectives provide insufficient constraints on this shared shift because its effects on chosen and rejected responses largely cancel in the ranking margin. Through the output projection, however, the offset can disproportionately increase the logits of particular vocabulary items, creating a vocabulary well and potentially causing generation collapse.

The paper evaluates 17 base-model lineages across two preference datasets and considers both LoRA-based and full-parameter fine-tuning. The authors report collapse signatures in 14 of the 17 lineages under their per-token ranking objective. They support the proposed mechanism through several interventions, including isolating the offset, transplanting it into a base model, and centering the rejected-side loss to prevent the well from forming. They also propose offset regularization that prevents well formation in several model families for up to three times the base training budget while largely preserving held-out ranking performance. Finally, they report a negative correlation between held-out ranking accuracy and the preference of two external reward models for the trained models' generated samples. The paper's central contribution is a representation-level explanation of a failure mode that conventional ranking metrics may not detect.

Critical strengths

(1) The paper does more than observe that hidden-state displacement correlates with degraded generation. The authors show that the common-mode component alone can reconstruct the vocabulary well, that transplanting the learned offset into an untreated base model can induce the corresponding failure, and that norm-matched random directions do not produce the same behavior. They further show that centering the rejected-side loss can prevent well formation and that subtracting the offset's logit contribution can substantially restore generation in a collapsed model. Taken together, these interventions provide compelling evidence that the offset is causally involved in the observed failure mode.

(2) The study covers 17 base-model lineages, five tokenizer families, two datasets, and both adapter-based and full-parameter training. It also examines multiple preference objectives rather than treating all preference optimization methods as equivalent. The proposed offset-control strategy preserves strong held-out ranking performance while preventing well formation across several model families at up to three times the base training budget. This combination of mechanistic analysis, cross-model experiments, and a potential diagnostic or mitigation method makes the work relevant to the development and evaluation of preference-optimized language models.

Critical weaknesses

(1) The paper distinguishes likelihood displacement, vocabulary-well formation, and terminal generation collapse, but its operational criterion counts a run as collapsed when either a vocabulary well forms or sampled generations substantially degenerate. Consequently, the reported result that 14 of 17 model lineages collapse does not necessarily mean that all 14 exhibit comparably severe failures in sampled text. For example, some entries in Table 4 exhibit high well capture but relatively low measured degeneration. Separating well-only cases from runs with clear sampled-generation collapse would make the empirical conclusion easier to interpret.

Moreover, the strongest results concern the authors' per-token ranking objective, whose stated demand is approximately 42 nats per token at the main setting. Textbook DPO remains below the proposed well-formation threshold at three times the base budget, and SimPO initialized from the public SFT checkpoint does not form a well within the tested budget. These differences are scientifically interesting, but they also suggest that the observed failure depends substantially on the objective, its effective margin demand, initialization, and training budget. The paper should be careful not to generalize the strongest collapse findings to preference optimization as a whole.

(2) The reported negative association between ranking accuracy and external reward-model preference is suggestive, but the evaluation uses two public reward models and one generated sample per prompt across 300 held-out UltraFeedback prompts. Without uncertainty estimates, human preference judgments, or analyses controlling for objective and training budget, it is difficult to determine how generally the observed correlation reflects a true trade-off between ranking performance and response quality.

The mitigation experiments also have a bounded scope. The penalty strengths are calibrated on the driving dataset, and the main cross-family prevention results extend to three times the base budget. At six times the budget, the OLMo experiments show that probability mass can move into alternative failure modes, including punctuation, end-of-text tokens, and word-family rings; the residual penalty does not fully resolve every failure mode in every tested lineage. In addition, the proposed early-warning signal is evaluated retrospectively on completed training runs rather than as a prospective monitoring procedure. These limitations do not invalidate the proposed mechanism, but they weaken the broadest claims about robust prevention and practical early detection.

Questions for authors

(1) Could the authors report the number of runs that exhibit (i) vocabulary-well formation without substantial sampled-generation degeneration, (ii) clear sampled-generation collapse, and (iii) diffuse generation degradation? In particular, how many of the 14 out of 17 lineages satisfy the sampled-generation criterion independently of the vocabulary-well criterion? This distinction would materially affect the interpretation of the headline result. For the negative correlation between held-out ranking accuracy and external reward-model preference, have the authors tested whether the relationship persists after controlling for objective type and training budget, or across independent training seeds? Could uncertainty estimates or additional evaluation results clarify the robustness of this relationship, given that the current analysis uses two reward models and one sample per prompt for 300 prompts?

(2) Evaluating additional preference objectives, including IPO, ORPO, and KTO, as well as more instruction-tuned or SFT-initialized models, would help establish the generality of the proposed mechanism. Prospective validation of the offset-based early-warning signal and broader human or model-based quality evaluations would also strengthen the practical implications.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1Zkk (1/2): degenerate samples in 13 of the 14 collapsed lineages, and IPO, DPO and SimPO fail at published settings

Thank you for the review. Items 1 to 4 are here and items 5 to 8 in a second comment.

1. How many of the 14 collapsed lineages have broken samples? (W1, Q1) Thirteen. In each, most sampled answers are degenerate, so they meet the sample criterion on their own. The fourteenth, Qwen2.5-0.5B, has a well only, with one token taking 94% of the probability mass shifted by training and 12% of its samples degenerate. Two more lineages, Gemma-2-2B and Phi-3-mini, form no well within three times the base budget and degrade diffusely at six times. They stay outside the count, and Llama-3.2-3B shows none of the three. Your three categories therefore hold 1, 13 and 2 lineages. The Table 4 rows with high capture and low measured degeneration are Llama-3.1-8B, OLMo-2-7B and Qwen2.5-0.5B. The submission's audit counted words by spaces and missed long token strings without spaces. A re-audit that catches them flags 52% and 73% of the samples of the first two, which raises the count from 11 to 13, and leaves Qwen2.5-0.5B as the well-only case. The abstract and §1 now report this split, and Table 15 lists every lineage at its budget (Appendix C.13).

2. Do the findings reach beyond our per-token objective? (W1) Yes. Starting from public SFT models, IPO, standard DPO and SimPO each fail at a published setting, and the offset is involved every time.

  1. IPO forms a well within one epoch. From zephyr-7b-sft-full, the Mistral-7B SFT model that the SimPO paper starts from (Zephyr SFT below), we run TRL's IPO at τ = 0.01 at both ends of the learning-rate range that paper searched for IPO. In both runs /******/ takes over the output, every sampled answer is degenerate, and a reward model prefers the trained model's answer to Zephyr SFT's only 6% of the time. The offset's dose on this string passes one in both, the necessary condition of §6.
  2. Standard DPO reproduces a failure reported in public. The Zephyr DPO recipe of the alignment handbook, with β = 0.01, run on the public Dutch SFT model GEITje-7B-ultra-sft and its data, reproduces the failure that the model's developer reported, with 99% of answers repeating /******/. Centering, the paper's fix that acts only on the offset, removes the string from every sample.
  3. SimPO's win rate drops inside its recommended learning rates. From Zephyr SFT, the win rate falls as the learning rate rises, while ranking accuracy keeps improving. Trained for six epochs at 1e-6, the run forms a well. Even after one epoch, adding the offset of the first 1e-6 run to Zephyr SFT drops its win rate to 14%, against 48% for a random direction of the same size.
learning ratewin rate against Zephyr SFTheld-out ranking accuracy
3e-7, official recipe68.0% (62.7 to 73.0)0.667
5e-760.3% (54.8 to 65.7)0.694
8e-753.0% (47.3 to 58.7)0.710
1e-6, first data order48.2% (42.5 to 53.8)0.706
1e-6, second data order37.8% (32.3 to 43.2)0.708

Win rate is how often Skywork-Reward-V2 prefers the trained model's answer to its starting model's on 300 held-out prompts, with 95% intervals in parentheses.

In each case the failure sits where the objective pushes harder, at the smallest τ, the smaller β, the larger learning rate and the longer run. Standard DPO at 1e-6 still wins 67%, so we do not extend these failures to preference optimization as a whole. The abstract, §1 and §6 ("Where the failure occurs") state this boundary together with the settings that stay healthy.

3. Do IPO, ORPO and KTO show the mechanism? (Q2) Yes. The offset appears under all three, and a well forms only where its dose passes one (§6). From Zephyr SFT at 1e-6, the offset ends at 11% of the hidden-state norm under ORPO at λ = 0.1, 12% under KTO at β = 0.1, and 15% to 90% under IPO as τ falls from 1.0 to 0.01. The dose passes one only under IPO at τ = 0.01, where the well of item 2 forms, and stays below 0.5 in every other run, where none forms. ORPO, KTO and IPO at τ = 0.5 and 1.0 win 63%, 70%, 60% and 57%, while IPO at τ = 0.1 ends at 15% (11.0 to 19.0) without a well. From the Llama-2 base model, IPO at τ = 0.02 also forms a well (Appendix C.2).

4. Does the mechanism appear from SFT and instruction-tuned starts? (Q2) Yes. From Zephyr SFT, IPO forms a well within one epoch and SimPO forms one after six epochs, with a win rate of 5% (2.3 to 7.0). Of six instruction-tuned starts under our per-token loss, three form wells and three degrade diffusely. From the SimPO authors' Llama-3-8B SFT model, SimPO formed no well even at 12 times the usual training length. In the authors' text format on the full data, the official recipe from that model wins 66% after one epoch, level with the public model's 65%, and 69% after three. The 34% at three epochs in the last row of Table 7 came from our earlier text format and a 15.7k-pair subset, and the caption now says so (Appendix C.4, C.8).

AuthorsOpenReview ↗

Response to Reviewer 1Zkk (2/2): the correlation holds on new prompts, and alarms fixed in advance caught 15 of 17 damaged runs

Continued from part 1 (items 1 to 4).

5. Is the correlation between ranking accuracy and reward robust? (W2, Q1) It holds with more samples per prompt and on new prompts, the main run repeats across seeds, and it now comes with intervals. Across the 17 UltraFeedback runs of Table 7, the runs that rank held-out answers better win less often, with a Spearman correlation of −0.78 (−0.94 to −0.41) under Skywork-Reward-V2 and −0.79 (−0.94 to −0.45) under ArmoRM. Four samples per prompt or 133 unused prompts leave both correlations between −0.76 and −0.80, and two more seeds of the main per-token run of Table 7 win 23.5% and 23.8%, against 24.5%. Controlling for objective family and budget lowers the correlation to −0.63 (−0.96 to 0.38) and −0.28 (−0.92 to 0.55), with intervals that include zero, so the abstract and §8 now say that the trend is weaker after these controls. The same pattern appears within one objective and budget in the authors' text format, in the table of item 2 in part 1, where every learning rate above the official one has higher ranking accuracy and a lower win rate (Appendix C.7).

6. Do other quality evaluations agree with the reward models? (W2, Q2) Yes, two chat-model judges and AlpacaEval 2 do. When Tülu 3 and Olmo-3 Instruct rate each answer alone from 1 to 9, the runs of Table 7 with higher ranking accuracy again get lower ratings, with Spearman correlations of −0.76 and −0.74. AlpacaEval 2 points at the same damaged run. On its 805 prompts, the length-controlled win rate of SimPO at 1e-6 is 3%, below the 9% of its starting model Zephyr SFT, while standard DPO and SimPO with an NLL term at the same learning rate reach 21% and 23%, against 27% for the public SimPO model. The judge is GPT-4o-mini because GPT-4 Turbo is no longer served, so these numbers are not comparable with the public leaderboard. We have no human judgments, and the limitations section says so (Appendix C.7, C.11).

7. Do the alarms work prospectively? (W2, Q2) Yes. We fixed two alarms and wrote down our predictions before 44 new training runs. Together they caught 15 of the 17 runs that ended damaged (88%) and raised 4 false alarms in 24 healthy runs (17%). The offset alarm is the submission's dose, now computed for whichever token the offset raises most at that step, and it fires when this dose passes one. The entropy alarm is new and fires when the model's output entropy on a fixed set of examples reaches 1.5 times its starting value, a threshold chosen on 9 older checkpoints outside the 44 runs.

final win raterunsentropy alarm firesoffset alarm fireseither fires
below 40%1712415
40% to 50%3202
50% or above24404

The offset alarm singles out collapse onto one string. It fired in all 3 runs among the 44 that formed a well and in one more damaged run, three-epoch SimPO at 1e-6, where /******/ appears in 78% of the answers without reaching the well criterion. It fired in none of the 24 healthy runs. The entropy alarm caught most of the runs damaged without a well. Acting on an alarm rescued the run. Switching on an NLL term of weight 0.1 when the alarm fired brought the two SimPO runs at 1e-6 from 48% and 38% to 67% and 65%. The alarms fire close to the drop in quality, so the abstract and §8 now present them as detectors. They come 50 to 240 steps before the first evaluation below 40% in 6 of the 15 caught runs, at that evaluation in 8, and 30 steps after it in one.

The alarms caught all 15 damaged runs trained without centering. The two they missed were trained with centering. In both, answers end too early (see item 8) and output entropy falls instead of rising. Neither alarm fired in the Dutch run of item 2, which is outside the 44 because our reward models cannot score Dutch. Its dose reached 0.90 and its output entropy also fell. The counts on the 87 earlier runs stay in the paper, marked as retrospective (Appendix C.3).

8. Does offset control prevent wells beyond the data it was tuned on? (W2) Yes. Centering has no weight to tune, and in the public Dutch failure of item 2 it removes /******/ from all 300 samples. The penalty strengths were set on the driving data and not tuned again, and with them no well forms on any of the four base models of Table 3, which train on that data. The OLMo failures at six times the budget stand as reported, and §7 and §9 now claim offset control only for preventing wells up to three times the base budget. Reward is protected by a different remedy, an NLL term of weight 0.1, which keeps SimPO at 1e-6 at 65% against 48% without it. Centering alone lowers the official SimPO recipe from 68% to 51%, and we traced this cost. Without centering, the offset pushes the end-of-sequence token down harder than any other token. Centering removes that push, and a quarter of the answers then end within two words (Appendix C.9).

Reviewer_ZXB9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper attributes generation collapse in preference optimization to a "common-mode offset", a hidden-state shift shared by chosen and rejected responses that ranking losses barely penalize. Through the unembedding, it lifts a single token until it dominates decoding. Causal interventions support this mechanism, and the authors propose centering plus offset penalties to prevent collapse. Experiments cover multiple models, using driving-response preference ranking and general preference optimization on UltraFeedback.

Critical strengths
  1. The common-mode offset is an interesting finding, supported by clear causal interventions with random-direction controls, linking the shared hidden-state shift to generation collapse.
  2. The “demand” view neatly explains why DPO, SimPO, and the per-token loss behave differently, relating their behavior to the per-token margin each objective continues to push for.
Critical weaknesses
  1. The writing is defensive and unconventional, which hurts readability. For example, the introduction states, “This vulnerability is not an artifact of one unusual hyperparameter setting,” which reads as a preemptive rebuttal. Similar phrasing appears throughout the paper, such as “The collapse does not depend on adapters either” (§4). The introduction also ends by contrasting the work with prior studies rather than clearly summarizing its contributions, making the narrative feel argumentative.
  2. The collapse mainly appears under a non-standard high-demand loss; standard DPO and the official SimPO recipe form no well, so the practical relevance is limited.
  3. Table 7 appears to mix runs trained on driving and UltraFeedback data, which weakens the §8 correlation and the comparison with DPO.
Questions for authors
  1. Does the common-mode offset still appear in recent frontier models or in online RL such as GRPO?
  2. Does the §8 correlation hold when only runs trained on UltraFeedback are included?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZXB9: scope beyond our loss, GRPO and recent models, and the correlation on UltraFeedback

Thank you for the review.

W1. Defensive phrasing and the end of the introduction. We rewrote the passages you quote and the others like them, and §1 now ends with a contribution list in place of the paragraph that contrasted the work with prior studies. In §1, "This vulnerability is not an artifact of one unusual hyperparameter setting" is gone, and the sentence after it now reads "Under our main objective, the ranking term of PRO (Song et al., 2024), 14 of 17 base-model lineages collapse, 13 with degenerate samples and one with a well only". In §4, "The collapse does not depend on adapters either" now reads "Full-parameter fine-tuning collapses as well".

W2. Does the collapse matter outside our own loss? Yes. From public SFT models, IPO, standard DPO and SimPO each fail at a published setting, with the offset involved.

  1. IPO forms a well within one epoch. From zephyr-7b-sft-full, the SFT model the SimPO paper starts from (Zephyr SFT below), TRL's IPO at τ = 0.01 fails at both ends of the learning-rate range that paper searched for IPO. /******/ takes over the output, every sample is degenerate, and the win rate is 6%. In both runs the offset's dose on this string passes one, the necessary condition of §6.
  2. Standard DPO reproduces a failure reported in public. The alignment handbook's Zephyr DPO recipe, with β = 0.01, run on the Dutch SFT model GEITje-7B-ultra-sft and its data, reproduces the failure its developer reported, with 99% of answers repeating /******/. Centering, which acts only on the offset, removes the string from every sample.
  3. SimPO's win rate drops inside its recommended learning rates while ranking accuracy keeps improving. After six epochs at 1e-6 it forms a well. Even after one epoch, adding the offset of the first 1e-6 run to Zephyr SFT cuts its win rate to 14%, against 48% for a same-size random vector.
learning ratewin rate vs. Zephyr SFT (95% interval)ranking accuracy
3e-7, official68.0% (62.7-73.0)0.667
5e-760.3% (54.8-65.7)0.694
8e-753.0% (47.3-58.7)0.710
1e-6, order 148.2% (42.5-53.8)0.706
1e-6, order 237.8% (32.3-43.2)0.708

Win rate is the share of 300 held-out prompts where Skywork-Reward-V2 prefers the trained model's answer to its starting model's.

Each failure sits where the objective pushes harder, at the smallest τ, the smaller β, the larger learning rate or the longer run. The abstract, §1 and §6 ("Where the failure occurs") state this boundary together with the settings that stay healthy.

Q1. Does the offset appear in recent models and in online RL such as GRPO? Yes, in both. It appears in all nine public post-training stages we measured, from the recent open releases Tülu 3 and Olmo-3 and the Llama-3 and Mistral SimPO models. We measured open releases because the offset is a shift of hidden states, which closed models do not expose. Four of the nine stages are online RL, three with a GRPO variant and one with PPO. We also trained GRPO from Zephyr SFT, with TRL defaults and ArmoRM as the reward.

learning rateKL termwin rate vs. Zephyr SFToffset norm (% of hidden-state norm)
1e-6, default0.0466.3% (61.0-71.3)11%
1e-6071.0% (65.8-76.0)23%
2e-60.0458.7% (53.0-64.0)17%
3e-60.0443.7% (38.2-49.2)26%
5e-60.0429.0% (24.0-34.0)30%
5e-6017.7% (13.5-22.0)37%

The offset appears in every run and grows with the learning rate. At the default the model improves on Zephyr SFT, so GRPO carries the offset there without harm. Neither the public stages nor our runs form a well, and no token takes more than 9% of the probability mass shifted by training in the public stages or 16% in ours. At three and five times the default learning rate the model ends below Zephyr SFT. Removing the offset at decoding from a model trained at five times the default does not restore it (25% against 29%), so we do not attribute that damage to the offset (App. C.6).

W3 and Q2. Does Table 7 mix datasets, and does the correlation hold on UltraFeedback alone? Every run in Table 7, including the DPO rows, trains on UltraFeedback alone. The caption did not say so and now does. Across these 17 UltraFeedback runs the Spearman correlation of ranking accuracy with win rate is −0.78 (−0.94 to −0.41) under Skywork-Reward-V2 and −0.79 (−0.94 to −0.45) under ArmoRM. It is unchanged with four samples per prompt (−0.80 and −0.79) and on 133 new prompts (−0.76 under both). Controlling for objective family and training budget lowers it to −0.63 and −0.28, and with 17 runs the intervals include zero, so the abstract and §8 now say that the trend is weaker after these controls. Within one objective and budget the pattern stays clear in the W2 table, where every learning rate above the official one has higher ranking accuracy and a lower win rate (App. C.7).

Thank you for the detailed response. The additional experiments address some of my concerns. However, they also suggest that the common-mode offset is often benign in recent models and GRPO, and that collapse requires unusually aggressive optimization. The practical significance of this phenomenon for current frontier post-training therefore remains uncertain.

AuthorsOpenReview ↗

Follow-up Response

Thank you for the follow-up.

These three failures occur within one epoch, inside published hyperparameter ranges and alongside settings that work.

recipepublished rangeworksfailsranking acc. (works → fails)
IPO in TRLτ grid of the SimPO paperτ = 0.5, win rate 60%τ = 0.01, a well, win rate 6%0.65 → 0.70
SimPOlearning rates of its README3e-7, win rate 68%1e-6, win rates 48% and 38%0.67 → 0.71
DPO on the Dutch modelTRL default and alignment-handbook recipeβ = 0.1, /******/ in 0 of 300 answersβ = 0.01, /******/ in 99% of answers0.917 → 0.945

Held-out ranking accuracy is higher at the failing setting in every row, so the usual validation signal points the wrong way (§6, Appendix C.1, C.2, C.12).

The practical use is in the search that comes before a release. The nine public stages we measured, up to Tülu 3 and Olmo-3, show settings kept after tuning, with a dose of at most 0.23, whereas in our runs a well appears only after the dose passes one. During tuning, our GRPO run drops from a 66% win rate at the TRL default to 59% and 44% at 2× and 3× learning rates as its offset grows from 11% to 17% and 26% of the SFT hidden-state norm. For such sweeps, our pre-fixed offset-dose and entropy alarms need no reward model and caught 15 of 17 damaged runs (all 3 wells via dose, 12 via entropy) with 4 false alarms in 24 healthy runs (Appendix C.3, C.6).

Thank you for the clarification and additional experiments. I have raised my rating to weak accept.

Reviewer_KtiiOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a common-mode hidden-state offset that can drive vocabulary collapse during preference optimization. This offset shifts chosen and rejected representations together and can strongly favor a token through the unembedding layer while ranking metrics remain healthy. Decomposition, injection, removal, and training-time interventions support its causal role in the studied collapse cases. Experiments span 17 base-model lineages, two datasets, and adapter and full-parameter training. Offset control prevents vocabulary wells in several tested settings, while retrospective trajectories suggest a potential early-warning signal.

Critical strengths
  • S1. The mechanistic evidence goes beyond correlation. The isolated offset reconstructs the vocabulary well, injecting it into the base model reproduces collapse, and removing its logit contribution suppresses degeneration. Norm-matched random controls strengthen the interpretation that the learned direction, rather than displacement magnitude alone, matters.

  • S2. The paper distinguishes likelihood displacement, vocabulary-well formation, and generation collapse. Comparisons across objectives show that large offsets need not produce wells, while external reward evaluations demonstrate that healthy ranking metrics can coexist with degraded generation. These distinctions make the analysis more informative than simply documenting overoptimization.

Critical weaknesses
  • W1. The practical scope of the collapse mechanism remains insufficiently established. Most collapse evidence comes from base-start training, especially the high-demand per-token objective. Textbook DPO forms no vocabulary well within the tested budgets, and official SFT-start SimPO develops an offset without forming one. Although these boundaries are acknowledged, the evidence does not establish when the mechanism becomes a material risk in standard post-training. This limits the practical reach of the central finding despite the strong interventions in collapsing settings.

  • W2. The proposed early-warning signal lacks independent predictive validation. The retrospective analysis tracks when the eventual well token enters the top eight offset projections, but does not evaluate a fixed alarm rule on separate collapsing and stable runs. Large offsets also occur without collapse, and dose above one is not sufficient. The trajectories therefore motivate monitoring, but do not yet establish how reliably the signal predicts collapse or improves stopping decisions.

Questions for authors
  • Q1. What evidence supports the statement that SFT delays rather than prevents well formation? The official SFT-start SimPO runs form no well within the tested budget. Should this statement remain a hypothesis beyond the observed horizon?

  • Q2. Can an alarm defined without knowing the eventual collapse token distinguish collapsing from stable runs? Please report detection rates, false-positive rates, and lead times using a threshold selected independently of the evaluation runs.

  • Q3. Beyond the demand sweeps already reported, do demand-matched comparisons isolate the effect of the reference model? Please clarify whether the evidence supports demand as a useful predictor or establishes it as the explanation for textbook DPO’s resistance to well formation.

  • Q4. In the SFT-start SimPO setting, does offset control improve external reward quality relative to an otherwise matched run without it? Since the untreated run forms no well, this would clarify whether the intervention also helps with the degradation observed there.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Ktii: failures at published settings, a prospective alarm test, and what each remedy does

Thank you for the review.

W1. Does this matter in standard post-training? Yes. From public SFT models, published recipes fail in three ways, each involving the offset.

  1. IPO forms a well in 1 epoch. From zephyr-7b-sft-full (SimPO's Mistral-7B SFT, Zephyr SFT below), TRL IPO (τ = 0.01) fails at both ends of SimPO's IPO learning-rate (LR) search. /******/ takes over the output, every sample is degenerate, win rate is 6%, and the submission's offset alarm fires (see W2). Win rate is the share of 300 held-out prompts where Skywork-Reward-V2 prefers the trained model to its starting model.
  2. Standard DPO reproduces a public failure. The alignment handbook's Zephyr DPO recipe (β = 0.01) on Dutch SFT GEITje-7B-ultra-sft and its data reproduces its developer's reported failure, with 99% of answers repeating /******/. Centering, acting only on the offset, removes it from every sample.
  3. SimPO loses win rate within its recommended LRs while ranking accuracy rises. Six epochs at 1e-6 form a well. Even at 1 epoch, adding the first 1e-6 run's offset to Zephyr SFT cuts win rate to 14%, vs. 48% for a same-size random vector.
LRwin rate vs. Zephyr SFT (95% CI)ranking acc.
3e-7, official68.0% (62.7-73.0)0.667
5e-760.3% (54.8-65.7)0.694
8e-753.0% (47.3-58.7)0.710
1e-6, order 148.2% (42.5-53.8)0.706
1e-6, order 237.8% (32.3-43.2)0.708

The offset also marks a released model. princeton-nlp/Mistral-7B-Base-SFT-SimPO writes "Confidence: N%" in 14% of answers (0% in Zephyr SFT). Removing the offset at decoding cuts this to 1% (15% for a same-size random vector). Failures sit where objectives push harder (smallest τ, smaller β, larger LR, longer run), and the abstract, §1 and §6 ("Where the failure occurs") state this boundary.

W2 & Q2. Can a fixed alarm separate damaged from healthy runs? Yes. Two alarms fixed before 44 new runs caught 15/17 damaged runs (88%) with 4/24 false alarms (17%), and acting on the alarm rescued training. Switching on an NLL term (weight 0.1) when it fired lifted the two 1e-6 SimPO runs from 48% and 38% to 67% and 65%.

Neither alarm needs to know the eventual well token. The offset alarm fires when the submission's dose passes 1 on whichever token the offset raises most. The entropy alarm fires when output entropy on fixed prompts hits 1.5x its initial value (set on 9 older checkpoints outside the 44).

final win raterunsentropyoffseteither
below 40%1712415
40% to 50%3202
50% or above24404

The offset alarm hit all 3 wells among the 44 plus 3-epoch SimPO at 1e-6 (/******/ in 78% of answers), and 0/24 healthy runs (large offsets alone never triggered it). Lead times are short, so the abstract and §8 call the alarms detectors. They fire early in 6/15 caught runs (50-70 steps in four 1-epoch runs, 160 and 240 in two 3-epoch runs), at the first below-40% check in 8, and 30 steps late in 1. Stopping at the alarm would have beaten the final checkpoint in 12/15 (median +11 pts).

They caught all 15 damaged runs trained without centering. Both misses used centering, and their answers end early (Q4) while entropy falls. Neither alarm fired in W1's Dutch run (outside the 44, since our reward models cannot score Dutch), where dose peaked at 0.90 and entropy fell (App. C.3).

Q1. Does SFT delay or prevent wells? It does not prevent them. From Zephyr SFT, IPO forms a well in 1 epoch and SimPO in 6. From SimPO's Llama-3-8B SFT (Llama-3 SFT), none formed up to 12x normal length, so SFT only delaying wells remains a hypothesis and was removed from §6 (App. C.8).

Q3. Demand or reference model? Demand is a useful predictor of wells, not the full explanation of DPO's resistance (§6 revised). Adding a reference model at fixed demand (42 nat/token) still forms a well (the well token takes 86% of the mass shifted by training vs. 97% without it, Table 2). Unmatched for demand, adding one to SimPO at 1e-6 (same LR, β) lifts win rate from 48% to 58% (App. C.10).

Q4. Does offset control improve quality without a well? An NLL term is what helps there, not offset control. Offset control stops wells where NLL alone fails (Table 3), and §7 and §9 now claim it only for that. Without a well, 0.1 NLL lifts SimPO (1e-6) from 48% to 65% and our Llama-3 SFT run from 34% to 61%, while offset control drops that run to 26% and centering drops official SimPO from 68% to 51%. Without centering, the offset pushes the end-of-sequence token down hardest. Centering removes this push, and a quarter of answers then end within two words.

That 34% used an earlier text format and a 15.7k subset. On full data in the SimPO authors' text format, 3-epoch SimPO holds 69% and centering stays level at 73%. No W1 run uses the old format or subset, and W2 counts match across formats (App. C.3, C.4, C.9).

Thank you for the additional experiments and clarifications.

The new results on SFT-start IPO, DPO, and SimPO address much of my concern in W1. They provide stronger evidence that the identified mechanism is relevant beyond the original experimental setting. The clarifications regarding SFT, demand, and offset control also adequately address Q1, Q3, and Q4.

My main remaining concern is W2. The reported 15/17 detection rate relies on combining the offset-based alarm with an entropy-based alarm. The offset alarm alone detects only 4/17 damaged runs, although it captures all three vocabulary wells. In addition, only 6 of the 15 detected runs were flagged before the first evaluation showing damage. I therefore find the evidence for detecting degradation more convincing than the evidence for predicting it early enough to guide training.

Overall, the rebuttal has strengthened the paper and addressed several of my concerns. I still consider the predictive reliability of the proposed monitoring approach insufficiently established, and will keep my original rating.

AuthorsOpenReview ↗

Clarifying detector scope and mid-training rescue

Thank you for following up, and we are glad that W1, Q1, Q3, and Q4 are resolved.

On W2, we agree with your reading: our results establish online detection rather than advance prediction, and the revised Abstract and §8 call both signals detectors for that reason. Two brief points:

1. Why the offset alarm fires on 4 of 17 damaged runs Out of the 17 damaged runs, only 3 actually formed a vocabulary well (plus 3-epoch SimPO at 1e-6, which repeated /******/ in 78% of answers). Because the offset alarm specifically checks for vocabulary wells, it caught all 4 of those cases with 0 false alarms across all 24 healthy runs. The other 13 runs lost generation quality without forming a well (the second failure mode in §6), which is why we pair it with a simple entropy check.

2. Even as a detector, it still guides training (fires in steps 50–240, no RM needed)

  • Gradual SimPO runs can be rescued: On the original data, 6 of the 7 damaged SimPO runs were flagged before win rate fell below 40% (e.g., at steps 160 and 250 in the 3-epoch runs, whereas untreated win rate first drops below 40% at step 400). Turning on 0.1 NLL when the entropy alarm fires brings all 6 flagged 1- and 3-epoch SimPO runs (SimPO text format) to 65%–67%, including both false alarms (61%–66% vs. 20%–57% on ArmoRM):
SimPO run (original data)Alarm stepNo action0.1 NLL at alarm
5e-7, 1 epoch (false alarm)250 / 472 (300 w/o action)60%66%
8e-7, 1 epoch (false alarm)120 / 47253%65%
1e-6, 1 epoch120 / 47248%67%
1e-6, 1 epoch, 2nd order130 / 47238%65%
1e-6, 3 epochs160 / 1,41624%65%
8e-7, 3 epochs250 / 1,416 (240 w/o action)33%65%
  • Rapid collapses can be stopped early: The other 9 caught runs are mostly harsh settings (IPO at τ ≤ 0.1 or self-sampled negatives) that collapse almost immediately and cannot be rescued by NLL (e.g., IPO at τ = 0.1 ends at 26% vs. 15%). Since ranking accuracy keeps rising, firing within 240 steps still lets you abort a broken setting early without a reward model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.