- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GAW-PO, a token-level reweighting of the rejected-response term in DPO. For each rejected token, the gradient of its log-likelihood is compared, by dot product, with two “preferred” directions: the gradient of the paired chosen response and the chosen-response gradient averaged over the whole dataset. The larger score is mapped through a decreasing sigmoid to a weight that can only reduce, never increase, the token’s standard DPO penalty. The gradients are computed once before training in a frozen low-rank PiSSA adapter that serves only as a probe; DPO training itself updates the full model. On OLMo 3 7B with one preference dataset and 11 benchmarks (mathematics, reasoning, code, question answering), the paper reports the best average score among DPO, IPO, KTO, SimPO, TIS-DPO, TI-DPO and AAO (Table 1), and much better stability than DPO when the regularization coefficient β is made small (Figure 2).
Critical strengths
Strength 1: Instead of estimating how important a rejected token is (TI-DPO uses gradient norms with respect to input embeddings; AAO uses embedding similarity between tokens of the two responses), GAW-PO asks what penalizing the token would do to the chosen response, measured in parameter space. Because the score is signed and depends on context, the same token can be spared where it is correct and penalized where it is wrong (Appendix D.1, Table 3), and the weights are computed once before training without any auxiliary model (Sections 2 and 3.3).
Strength 2: All methods start from the same open SFT checkpoint and preference data, the policy is trained with full-parameter updates, evaluation covers 11 verifiable benchmarks in four capability groups over three seeds, and the code, weight-computation pipeline and filtered data are released (Sections 4.1–4.3). The paper also includes matched-attenuation controls with uniform and random weights (Appendix D.2).
Critical weaknesses
Weakness 1: The baselines in Table 1 are all implemented by the authors, and several of them are modified, so a fair comparison cannot be guaranteed. The headline claim, the best average score among all methods, rests entirely on these re-implementations, and the margins over the strongest baselines are modest (Table 1), so changes to the baselines could plausibly alter the ranking. In particular, Appendix C states that AAO’s ambiguity thresholds, which the original method learns with a small network, are fixed to constants; that TI-DPO is run without the triplet loss of the original paper; and that SimPO uses a learning rate different from its official configuration.
Weakness 2: The advantage of not collapsing at small β can largely be explained by an overall reduction of the rejected-response penalty, and does not necessarily come from gradient alignment. Since no weight exceeds the standard DPO weight, GAW-PO necessarily weakens the total penalty on rejected responses, which by itself makes the objective gentler at small β. The robustness result (Section 5.2, Figure 2) is the paper’s main selling point, so its attribution is central. In the control of Appendix D.2, where every rejected token receives the same discount equal to the mean GAW-PO weight (“Uniform”), the authors themselves state that “Uniform weighting recovers much of the stability”.
Questions for authors
Q1: Does the direction computed in the low-rank adapter agree with the direction computed on the full model? All alignment scores are computed in a frozen PiSSA adapter, a low-rank adapter initialized from the principal singular components of each weight matrix and used only as a gradient probe, while training updates the full model (Section 3.3, Appendix A.2). The justification relies on training curves reported in the original PiSSA paper, and Appendix F leaves “how closely low-rank alignment tracks full-parameter alignment” to future work. Could the authors explain why the sign and ranking of the probe scores should match those obtained from full-model gradients, and whether this has been checked on any subset of the data?
Q2: Why were the baselines not run with the settings of their original papers, or compared with the results reported there, and why were some of them modified? This relates to Weakness 1. Could the authors give the reason for each deviation listed in Appendix C (fixed AAO thresholds, TI-DPO without the triplet loss, a different SimPO learning rate)? In addition, GAW-PO is trained with a larger learning rate than every baseline, several times larger than that of DPO (Section 4.1 versus Appendix C). Why do the methods use different learning rates, and how was each one chosen?
Q3: Why should the absence of collapse at small β be attributed to gradient alignment rather than to an overall reduction of the penalty? This relates to Weakness 2. Given the authors’ own statement that uniform weighting recovers much of the stability, what evidence shows that the stability itself, as opposed to the remaining gap in final scores, comes from aligning token weights with the preferred directions? The Uniform and Random controls are trained only at the smallest β of the sweep (Appendix D.2); how do they behave at the other values of β?
4 comments
| Benchmark | SFT (init) | SFT-chosen | Uniform (β=0.002) | GAW-PO (β=0.002) |
|---|---|---|---|---|
| AIME 2024 | 5.73 ± 0.55 | 8.99 ± 0.39 | 17.12 ± 0.79 | 16.77 ± 0.79 |
| AIME 2025 | 6.88 ± 1.04 | 7.74 ± 0.22 | 16.91 ± 0.37 | 17.53 ± 0.68 |
| MATH | 67.70 ± 0.27 | 68.15 ± 1.19 | 76.71 ± 0.35 | 76.63 ± 0.39 |
| Omega 500 | 11.53 ± 0.76 | 10.93 ± 0.92 | 13.20 ± 0.92 | 12.33 ± 0.70 |
| AGI Eval | 58.51 ± 0.58 | 57.81 ± 0.81 | 65.12 ± 0.41 | 65.49 ± 0.24 |
| BBH (CoT) | 44.40 ± 0.77 | 55.55 ± 1.01 | 64.96 ± 0.40 | 67.79 ± 0.49 |
| HumanEval+ | 68.84 ± 0.48 | 66.28 ± 0.40 | 62.40 ± 0.88 | 75.35 ± 0.57 |
| LiveCodeBench | 20.30 ± 0.34 | 23.51 ± 1.24 | 28.01 ± 0.19 | 30.01 ± 0.79 |
| MBPP+ | 56.38 ± 0.37 | 55.48 ± 0.12 | 55.85 ± 0.56 | 59.11 ± 0.29 |
| GPQA | 31.47 ± 1.74 | 30.58 ± 0.77 | 34.75 ± 1.52 | 35.04 ± 0.39 |
| MMLU (CoT) | 66.75 ± 0.32 | 64.73 ± 0.22 | 68.03 ± 0.35 | 69.98 ± 0.57 |
| Average | 39.86 | 40.89 | 45.73 | 47.82 |
The new experiments help. The probe check on 1000 pairs (Spearman 0.815, sign agreement 90.1%) addresses Q1, and rerunning AAO with its learned threshold module (40.75 versus 40.52) plus SimPO at the recommended learning rate shows those deviations did not flatter your method, while the SFT-chosen control usefully bounds what extreme attenuation can do. Two gaps remain. GAW-PO trains at 1e-5, chosen through preliminary training dynamics, while every baseline keeps a default, and what that choice was tuned on is unstated, so tuning asymmetry is unresolved. The Uniform and Random controls also appear only at the smallest beta, so attributing stability to alignment across the sweep rests on one endpoint.
Thank you for the follow-up!
The training dynamics we monitored were the reward margin and the log-probability of the chosen responses. We compared them for GAW-PO at β = 0.1 with learning rates of 1e-6, 2e-6, 5e-6 and 1e-5.
| Learning rate | Reward margin (end of training) | Sum of chosen log-prob (end of training) |
|---|---|---|
| 1e-6 | ≈ 0.56 | ≈ −412 |
| 2e-6 | ≈ 1.22 | ≈ −405 |
| 5e-6 | ≈ 2.37 | ≈ −391 |
| 1e-5 | ≈ 3.33 | ≈ −382 |
* Here we report the average values for a single optimization step. Higher is better.
We required two things. The chosen log-probability and reward margins had to increase. Training also had to avoid collapse, with both curves converging to a stable plateau. We therefore used 1e-5, the point where the improvements begin to saturate and training remains stable.
For each baseline, we used the hyperparameters recommended by its authors in the original paper or released configuration. These settings reflect the authors' own tuning, and using them is common practice when comparing preference-optimization methods. Running a separate hyperparameter search for every baseline with full-parameter training at 7B scale is beyond the compute available to us in an academic setting. The tuning applied to GAW-PO was itself limited and based on training curves.
To check that the stability claim does not rest on a single β, we add two more matched comparisons between GAW-PO and the Uniform control, at β = 0.001 and β = 0.005 (Tables R1–R3). We also measure degeneration directly, as the share of responses that end in a repetition loop, i.e. whose final 200 tokens have seq-rep-4 ≥ 0.5 (Welleck et al., 2020).
- At β = 0.005 the two weightings are indistinguishable: 47.00 vs. 47.08 average accuracy, with 0.7% of responses looping for both.
- At β = 0.002, GAW-PO loops in only 0.7% of responses, the same level as at β = 0.005. Uniform which has a similar average weight but no alignment information, loops 5.7% on average.
- At β = 0.001 both weightings degrade, but Uniform more: 42.74 vs. 40.52 average accuracy, with 6.8% vs. 10.7% of responses looping. The gap is largest on code, where 16.5% of Uniform responses loop against 5.3% for GAW-PO. We believe this follows from how GAW-PO assigns its weights. Chosen and rejected code solutions typically share most of their tokens (signatures, syntax, boilerplate), so a uniform penalty also suppresses structure the correct solution needs; GAW-PO lowers the penalty on exactly these chosen-aligned tokens.
The advantage of GAW-PO is therefore not tied to one endpoint. It appears across the low-β range, where the attenuated controls begin to degenerate.
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2020). Neural text generation with unlikelihood training. International Conference on Learning Representations (ICLR 2020).
β = 0.005 — GAW-PO learned / uniform
| accuracy ↑ | repetition loops (%) ↓ | |
|---|---|---|
| Code | 54.3 / 54.8 | 0.5 / 0.5 |
| Math | 29.5 / 29.3 | 1.0 / 1.2 |
| QA | 52.7 / 52.1 | 0.2 / 0.2 |
| Reasoning | 65.5 / 65.9 | 0.7 / 0.6 |
| Avg (11) | 47.0 / 47.1 | 0.7 / 0.7 |
β = 0.002 — GAW-PO learned / uniform
| accuracy ↑ | repetition loops (%) ↓ | |
|---|---|---|
| Code | 54.8 / 48.8 | 0.7 / 5.0 |
| Math | 30.8 / 31.0 | 1.1 / 5.7 |
| QA | 52.5 / 51.4 | 0.2 / 7.1 |
| Reasoning | 66.6 / 65.0 | 0.6 / 5.4 |
| Avg (11) | 47.8 / 45.7 | 0.7 / 5.7 |
β = 0.001 — GAW-PO learned / uniform
| accuracy ↑ | repetition loops (%) ↓ | |
|---|---|---|
| Code | 41.1 / 35.4 | 5.3 / 16.5 |
| Math | 29.8 / 28.9 | 7.2 / 7.2 |
| QA | 51.3 / 50.0 | 6.5 / 9.7 |
| Reasoning | 62.5 / 62.0 | 8.4 / 10.1 |
| Avg (11) | 42.7 / 40.5 | 6.8 / 10.7 |
Thank you for your thorough review and questions! We respond to them individually, below and in the second comment. We hope that our responses will address your concerns. If so, we would be grateful if you would consider raising your score.
Additional evaluations on judge-based benchmarks (AlpacaEval 2, Arena-Hard v0.1, and MT-Bench) are included in the revised paper (Appendix F) and summarized in our response to Reviewer bueS.
Q1
We will add an analytical explanation and an empirical comparison in the Appendix.
The full model’s score combines contributions from all parameters. The adapter keeps only contributions from the top singular subspaces, so its sign and ranking should match when those retained contributions dominate the score. The probe uses the same selection and weighting for every token, making the comparisons consistent. We also compared adapter and full-model scores on 1000 randomly sampled preference pairs.
The adapter and full model show strong rank agreement, and their signs agree on 90.1% of tokens. The adapter therefore captures most full-model alignment patterns while using a much smaller space.
W1 & Q2
Most baselines are not actually re-implementations per se: DPO, IPO and KTO use TRL's official trainers directly; SimPO uses TRL's SimPO loss with the authors' released configuration. Where we found no usable implementation, we wrote the code ourselves. This was the case for AAO, whose official implementation lacks the adaptive path — trainer_adative.py is a 0-byte file. TIS-DPO and TI-DPO do publish code, but adaptations need to be made in terms of base model, data pipeline and formatting, logging, and configuration handling.
For AAO, we implemented Eq. 17 directly, with alpha set to 200, the value the paper itself specifies. The constants we picked are not arbitrary: a = 0.6 and b = 0.3 are the authors' own defaults. The original source is missing (see above). We have now also trained AAO with the learned threshold module of Eq. 8 in place of fixed constants: the average score reaches 40.75, against 40.52 for the fixed-constant variant reported in Table 1.
For TI-DPO, we omit the triplet loss, a variant the authors evaluate in their own ablation and that the official code supports. The triplet term is memory-intensive: at every training step, it samples an anchor response from the current policy and requires additional policy and reference forward passes on that anchor. The method is designed and released for short sequences: the authors already acknowledge a substantial computational overhead of TI-DPO in their paper, and the official implementation uses a maximum sequence length of 256 tokens by default (512 in its memory-optimized configuration), far below the 2048 tokens of our setting. Running the triplet term at this length was therefore not feasible within the training setup shared by all methods, and reducing the sequence length for TI-DPO alone would have truncated a large share of our preference pairs and made the comparison inconsistent. In the authors’ own ablation, the variant without the triplet loss remains close to the full method.
For SimPO, we also trained it at the recommended learning rate of 1e-6; it performs slightly worse than the configuration we report (40.29 vs 40.63), therefore we decided to use the better-performing model.
Our principle was to use each baseline’s default or officially released configuration wherever one exists. For our method, preliminary training dynamics identified 1e-5 as the most effective learning rate, which we then used throughout the paper.
W 2 & Q3
We agree that weakening the rejected penalty explains part of the effect and the stability. To separate attenuation from alignment, we compare GAW-PO against the Uniform control (App. D.2) and now a new baseline, SFT-chosen. For SFT-chosen, we fine-tuned the same SFT checkpoint with the standard cross-entropy loss on the chosen responses of the same preference data. This removes the rejected signal entirely, so it is the extreme case of attenuation.
[The results table is in the next comment; markdown formatting takes up a large share of the per-comment character limit]
If a gentler objective were enough, either one of these should match GAW-PO, and neither does. Uniform avoids the collapse but ends 2.09 points lower, and on code it falls below the SFT initialization. Removing the penalty altogether is worse: SFT-chosen reaches 40.89, which is 6.93 points below GAW-PO. So the total size of the penalty does not explain the result on its own. Which tokens keep the penalty matters.