acceptodds
Under review as a conference paper at ICLR 2027

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

25 views

Abstract

Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
preference optimization, token-level credit assignment, post-training, gradient alignment, influence estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_51z3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GAW-PO, a token-level reweighting of the rejected-response term in DPO. For each rejected token, the gradient of its log-likelihood is compared, by dot product, with two “preferred” directions: the gradient of the paired chosen response and the chosen-response gradient averaged over the whole dataset. The larger score is mapped through a decreasing sigmoid to a weight that can only reduce, never increase, the token’s standard DPO penalty. The gradients are computed once before training in a frozen low-rank PiSSA adapter that serves only as a probe; DPO training itself updates the full model. On OLMo 3 7B with one preference dataset and 11 benchmarks (mathematics, reasoning, code, question answering), the paper reports the best average score among DPO, IPO, KTO, SimPO, TIS-DPO, TI-DPO and AAO (Table 1), and much better stability than DPO when the regularization coefficient β is made small (Figure 2).

Critical strengths

Strength 1: Instead of estimating how important a rejected token is (TI-DPO uses gradient norms with respect to input embeddings; AAO uses embedding similarity between tokens of the two responses), GAW-PO asks what penalizing the token would do to the chosen response, measured in parameter space. Because the score is signed and depends on context, the same token can be spared where it is correct and penalized where it is wrong (Appendix D.1, Table 3), and the weights are computed once before training without any auxiliary model (Sections 2 and 3.3).

Strength 2: All methods start from the same open SFT checkpoint and preference data, the policy is trained with full-parameter updates, evaluation covers 11 verifiable benchmarks in four capability groups over three seeds, and the code, weight-computation pipeline and filtered data are released (Sections 4.1–4.3). The paper also includes matched-attenuation controls with uniform and random weights (Appendix D.2).

Critical weaknesses

Weakness 1: The baselines in Table 1 are all implemented by the authors, and several of them are modified, so a fair comparison cannot be guaranteed. The headline claim, the best average score among all methods, rests entirely on these re-implementations, and the margins over the strongest baselines are modest (Table 1), so changes to the baselines could plausibly alter the ranking. In particular, Appendix C states that AAO’s ambiguity thresholds, which the original method learns with a small network, are fixed to constants; that TI-DPO is run without the triplet loss of the original paper; and that SimPO uses a learning rate different from its official configuration.

Weakness 2: The advantage of not collapsing at small β can largely be explained by an overall reduction of the rejected-response penalty, and does not necessarily come from gradient alignment. Since no weight exceeds the standard DPO weight, GAW-PO necessarily weakens the total penalty on rejected responses, which by itself makes the objective gentler at small β. The robustness result (Section 5.2, Figure 2) is the paper’s main selling point, so its attribution is central. In the control of Appendix D.2, where every rejected token receives the same discount equal to the mean GAW-PO weight (“Uniform”), the authors themselves state that “Uniform weighting recovers much of the stability”.

Questions for authors

Q1: Does the direction computed in the low-rank adapter agree with the direction computed on the full model? All alignment scores are computed in a frozen PiSSA adapter, a low-rank adapter initialized from the principal singular components of each weight matrix and used only as a gradient probe, while training updates the full model (Section 3.3, Appendix A.2). The justification relies on training curves reported in the original PiSSA paper, and Appendix F leaves “how closely low-rank alignment tracks full-parameter alignment” to future work. Could the authors explain why the sign and ranking of the probe scores should match those obtained from full-model gradients, and whether this has been checked on any subset of the data?

Q2: Why were the baselines not run with the settings of their original papers, or compared with the results reported there, and why were some of them modified? This relates to Weakness 1. Could the authors give the reason for each deviation listed in Appendix C (fixed AAO thresholds, TI-DPO without the triplet loss, a different SimPO learning rate)? In addition, GAW-PO is trained with a larger learning rate than every baseline, several times larger than that of DPO (Section 4.1 versus Appendix C). Why do the methods use different learning rates, and how was each one chosen?

Q3: Why should the absence of collapse at small β be attributed to gradient alignment rather than to an overall reduction of the penalty? This relates to Weakness 2. Given the authors’ own statement that uniform weighting recovers much of the stability, what evidence shows that the stability itself, as opposed to the remaining gap in final scores, comes from aligning token weights with the preferred directions? The Uniform and Random controls are trained only at the smallest β of the sweep (Appendix D.2); how do they behave at the other values of β?

4 comments

Thank you for your thorough review and questions! We respond to them individually, below and in the second comment. We hope that our responses will address your concerns. If so, we would be grateful if you would consider raising your score.

Additional evaluations on judge-based benchmarks (AlpacaEval 2, Arena-Hard v0.1, and MT-Bench) are included in the revised paper (Appendix F) and summarized in our response to Reviewer bueS.

Q1

We will add an analytical explanation and an empirical comparison in the Appendix.

The full model’s score combines contributions from all parameters. The adapter keeps only contributions from the top singular subspaces, so its sign and ranking should match when those retained contributions dominate the score. The probe uses the same selection and weighting for every token, making the comparisons consistent. We also compared adapter and full-model scores on 1000 randomly sampled preference pairs.

MeasureAgreement (95% CI)
Within-response Spearman ρ\rho0.815 [0.806, 0.824]
Within-response Pearson rr0.874 [0.866, 0.883]
Sign agreement90.1% [89.4, 90.3]
Cohen’s κ\kappa for sign0.692 [0.681, 0.703]
Jaccard overlap of the 10% most conflicting tokens0.668 [0.655, 0.680]

The adapter and full model show strong rank agreement, and their signs agree on 90.1% of tokens. The adapter therefore captures most full-model alignment patterns while using a much smaller space.

W1 & Q2

Most baselines are not actually re-implementations per se: DPO, IPO and KTO use TRL's official trainers directly; SimPO uses TRL's SimPO loss with the authors' released configuration. Where we found no usable implementation, we wrote the code ourselves. This was the case for AAO, whose official implementation lacks the adaptive path — trainer_adative.py is a 0-byte file. TIS-DPO and TI-DPO do publish code, but adaptations need to be made in terms of base model, data pipeline and formatting, logging, and configuration handling.

For AAO, we implemented Eq. 17 directly, with alpha set to 200, the value the paper itself specifies. The constants we picked are not arbitrary: a = 0.6 and b = 0.3 are the authors' own defaults. The original source is missing (see above). We have now also trained AAO with the learned threshold module of Eq. 8 in place of fixed constants: the average score reaches 40.75, against 40.52 for the fixed-constant variant reported in Table 1.

For TI-DPO, we omit the triplet loss, a variant the authors evaluate in their own ablation and that the official code supports. The triplet term is memory-intensive: at every training step, it samples an anchor response from the current policy and requires additional policy and reference forward passes on that anchor. The method is designed and released for short sequences: the authors already acknowledge a substantial computational overhead of TI-DPO in their paper, and the official implementation uses a maximum sequence length of 256 tokens by default (512 in its memory-optimized configuration), far below the 2048 tokens of our setting. Running the triplet term at this length was therefore not feasible within the training setup shared by all methods, and reducing the sequence length for TI-DPO alone would have truncated a large share of our preference pairs and made the comparison inconsistent. In the authors’ own ablation, the variant without the triplet loss remains close to the full method.

For SimPO, we also trained it at the recommended learning rate of 1e-6; it performs slightly worse than the configuration we report (40.29 vs 40.63), therefore we decided to use the better-performing model.

Our principle was to use each baseline’s default or officially released configuration wherever one exists. For our method, preliminary training dynamics identified 1e-5 as the most effective learning rate, which we then used throughout the paper.

W 2 & Q3

We agree that weakening the rejected penalty explains part of the effect and the stability. To separate attenuation from alignment, we compare GAW-PO against the Uniform control (App. D.2) and now a new baseline, SFT-chosen. For SFT-chosen, we fine-tuned the same SFT checkpoint with the standard cross-entropy loss on the chosen responses of the same preference data. This removes the rejected signal entirely, so it is the extreme case of attenuation.

[The results table is in the next comment; markdown formatting takes up a large share of the per-comment character limit]

If a gentler objective were enough, either one of these should match GAW-PO, and neither does. Uniform avoids the collapse but ends 2.09 points lower, and on code it falls below the SFT initialization. Removing the penalty altogether is worse: SFT-chosen reaches 40.89, which is 6.93 points below GAW-PO. So the total size of the penalty does not explain the result on its own. Which tokens keep the penalty matters.

BenchmarkSFT (init)SFT-chosenUniform (β=0.002)GAW-PO (β=0.002)
AIME 20245.73 ± 0.558.99 ± 0.3917.12 ± 0.7916.77 ± 0.79
AIME 20256.88 ± 1.047.74 ± 0.2216.91 ± 0.3717.53 ± 0.68
MATH67.70 ± 0.2768.15 ± 1.1976.71 ± 0.3576.63 ± 0.39
Omega 50011.53 ± 0.7610.93 ± 0.9213.20 ± 0.9212.33 ± 0.70
AGI Eval58.51 ± 0.5857.81 ± 0.8165.12 ± 0.4165.49 ± 0.24
BBH (CoT)44.40 ± 0.7755.55 ± 1.0164.96 ± 0.4067.79 ± 0.49
HumanEval+68.84 ± 0.4866.28 ± 0.4062.40 ± 0.8875.35 ± 0.57
LiveCodeBench20.30 ± 0.3423.51 ± 1.2428.01 ± 0.1930.01 ± 0.79
MBPP+56.38 ± 0.3755.48 ± 0.1255.85 ± 0.5659.11 ± 0.29
GPQA31.47 ± 1.7430.58 ± 0.7734.75 ± 1.5235.04 ± 0.39
MMLU (CoT)66.75 ± 0.3264.73 ± 0.2268.03 ± 0.3569.98 ± 0.57
Average39.8640.8945.7347.82

The new experiments help. The probe check on 1000 pairs (Spearman 0.815, sign agreement 90.1%) addresses Q1, and rerunning AAO with its learned threshold module (40.75 versus 40.52) plus SimPO at the recommended learning rate shows those deviations did not flatter your method, while the SFT-chosen control usefully bounds what extreme attenuation can do. Two gaps remain. GAW-PO trains at 1e-5, chosen through preliminary training dynamics, while every baseline keeps a default, and what that choice was tuned on is unstated, so tuning asymmetry is unresolved. The Uniform and Random controls also appear only at the smallest beta, so attributing stability to alignment across the sweep rests on one endpoint.

Thank you for the follow-up!

The training dynamics we monitored were the reward margin and the log-probability of the chosen responses. We compared them for GAW-PO at β = 0.1 with learning rates of 1e-6, 2e-6, 5e-6 and 1e-5.

Learning rateReward margin (end of training)Sum of chosen log-prob (end of training)
1e-6≈ 0.56≈ −412
2e-6≈ 1.22≈ −405
5e-6≈ 2.37≈ −391
1e-5≈ 3.33≈ −382

* Here we report the average values for a single optimization step. Higher is better.

We required two things. The chosen log-probability and reward margins had to increase. Training also had to avoid collapse, with both curves converging to a stable plateau. We therefore used 1e-5, the point where the improvements begin to saturate and training remains stable.

For each baseline, we used the hyperparameters recommended by its authors in the original paper or released configuration. These settings reflect the authors' own tuning, and using them is common practice when comparing preference-optimization methods. Running a separate hyperparameter search for every baseline with full-parameter training at 7B scale is beyond the compute available to us in an academic setting. The tuning applied to GAW-PO was itself limited and based on training curves.

To check that the stability claim does not rest on a single β, we add two more matched comparisons between GAW-PO and the Uniform control, at β = 0.001 and β = 0.005 (Tables R1–R3). We also measure degeneration directly, as the share of responses that end in a repetition loop, i.e. whose final 200 tokens have seq-rep-4 ≥ 0.5 (Welleck et al., 2020).

  • At β = 0.005 the two weightings are indistinguishable: 47.00 vs. 47.08 average accuracy, with 0.7% of responses looping for both.
  • At β = 0.002, GAW-PO loops in only 0.7% of responses, the same level as at β = 0.005. Uniform which has a similar average weight but no alignment information, loops 5.7% on average.
  • At β = 0.001 both weightings degrade, but Uniform more: 42.74 vs. 40.52 average accuracy, with 6.8% vs. 10.7% of responses looping. The gap is largest on code, where 16.5% of Uniform responses loop against 5.3% for GAW-PO. We believe this follows from how GAW-PO assigns its weights. Chosen and rejected code solutions typically share most of their tokens (signatures, syntax, boilerplate), so a uniform penalty also suppresses structure the correct solution needs; GAW-PO lowers the penalty on exactly these chosen-aligned tokens.

The advantage of GAW-PO is therefore not tied to one endpoint. It appears across the low-β range, where the attenuated controls begin to degenerate.

Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2020). Neural text generation with unlikelihood training. International Conference on Learning Representations (ICLR 2020).

β = 0.005 — GAW-PO learned / uniform

accuracy ↑repetition loops (%) ↓
Code54.3 / 54.80.5 / 0.5
Math29.5 / 29.31.0 / 1.2
QA52.7 / 52.10.2 / 0.2
Reasoning65.5 / 65.90.7 / 0.6
Avg (11)47.0 / 47.10.7 / 0.7

β = 0.002 — GAW-PO learned / uniform

accuracy ↑repetition loops (%) ↓
Code54.8 / 48.80.7 / 5.0
Math30.8 / 31.01.1 / 5.7
QA52.5 / 51.40.2 / 7.1
Reasoning66.6 / 65.00.6 / 5.4
Avg (11)47.8 / 45.70.7 / 5.7

β = 0.001 — GAW-PO learned / uniform

accuracy ↑repetition loops (%) ↓
Code41.1 / 35.45.3 / 16.5
Math29.8 / 28.97.2 / 7.2
QA51.3 / 50.06.5 / 9.7
Reasoning62.5 / 62.08.4 / 10.1
Avg (11)42.7 / 40.56.8 / 10.7
Reviewer_bueSOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a token reweighting method for DPO. Weights are gradient-aligned in the sense that the proposed GAW approach estimates, for each token that is part of the rejected response, whether penalizing it would interfere with the preferred update directions. The latter depend on the preferred response. "Rejected" tokens whose gradients are strongly aligned with the preferred behavior/response receive a weaker negative contribution, while tokens conflicting with the preferred behavior/response retain a stronger penalty.

The approach is evaluated on 11 benchmarks spanning mathematics, reasoning, coding, and question answering. The evaluation compares GAW to the original DPO algorithm as well as a number of competing baseline methods. In these comparisons, GAW achieves the highest average performance, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline.

The paper further shows that, in contrast to DPO, GAW remains stable when the DPO regularization parameter β decreases.

Critical strengths

The paper fills a gap in the growing body of work on token-based approaches for preference optimization. It accounts "for the interaction between rejected-token updates and preferred behavior", a direction of study that has not been explored for DPO. In particular, standard DPO doesn't explicitly distinguish locally correct chunks of a response from tokens that actually determine the preference. The motivation behind the approach ("the observation that training samples interact through their parameter updates") is simple and intuitive.

Critical weaknesses

It is hard to compare the results with those in previous related papers, most of which use the Meng et al 2024 evaluation setting --- testing on AlpacaEval, Arena-Hard and MT-Bench. Do you have these comparisons available using at least one of the base models typically employed, e.g. Llama3-8B-Instruct?

Questions for authors
  1. The paper should include a few other references and potentially comparisons to to work on token-level approaches for preference optimization in its discussion of existing work. I think that the first two papers are possibly the most relevant.

D^2PO Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, Xiang Zhou, Jingang Wang, Xunliang Cai, and Peng Li. 2025. Earlier tokens contribute more: Learning direct preference optimization from temporal decay perspective. In Proceedings of International Conference on Learning Representations.

attentionPO Chengyu Huang, Zhuohang Li, Sheng-Yen Chou, and Claire Cardie. 2026. Token-weighted Direct Preference Optimization with Attention. https://arxiv.org/abs/2605.21883 and EMNLP 2026.

SePO Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. 2025. Selective preference optimization via token-level reward function estimation. In Proceedings of Empirical Methods in Natural Language Processing.

cDPO Zicheng Lin, Tian Liang, Jiahao Xu, Qiuzhi Lin, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. 2024. Critical tokens matter: Token-level contrastive estimation enhances llm’s reasoning capability. arXiv preprint arXiv:2411.19943.

1 comment

Thank you for appreciating our paper and for your suggestions! We responded to them below and updated the paper.

W1

We evaluated all nine Table 1 checkpoints, without re-training, on AlpacaEval 2, Arena-Hard v0.1 and MT-Bench. These additional results are on Appendix F now. The judge is gpt-5.6-sol. The columns of the table below are:

  • MT-Bench: absolute score (1–10), with no reference model.
  • AE2 (vs. GPT-4): AlpacaEval 2 win rate of each method against the standard reference gpt4_1106_preview, both length-controlled (LC) and raw.
  • Arena-Hard (vs. GPT-4): win rate of each method against the standard baseline gpt-4-0314.
  • AE2 / Arena-Hard (vs. GAW-PO): win rate of each method against GAW-PO as the reference/baseline, using the same judge, prompts and scoring protocol. A value below 50% means the judge prefers GAW-PO over that method.
MethodMT-Bench (1–10)AE2 LC % (vs. GPT-4)AE2 raw % (vs. GPT-4)Arena-Hard % (vs. GPT-4)AE2 LC % (vs. GAW-PO)AE2 raw % (vs. GAW-PO)Arena-Hard % (vs. GAW-PO)
SFT5.75020.6919.8816.1745.7245.4746.59
DPO5.95019.7919.4415.0145.3146.1544.41
IPO5.79419.3220.2516.0847.2848.3947.93
KTO5.88119.6620.1916.3046.3946.7745.32
SimPO5.75021.2220.8716.8445.5346.7147.62
AAO5.90620.9720.6216.7844.3445.0348.53
TI-DPO5.86920.1019.7515.1044.8445.2844.06
TIS-DPO5.95020.3820.8716.4242.9844.9746.03
GAW-PO6.23722.2321.7416.7050.00 (ref.)50.00 (ref.)50.00 (ref.)

GAW-PO ranks first on MT-Bench (+0.29 over the next methods) and on AlpacaEval 2 against GPT-4 (both LC and raw). On Arena-Hard against GPT-4 it is within 0.14 points of the best, in a field spanning 1.8 points.

The benchmarks differ in how they score models. MT-Bench assigns each response an absolute score, independent of any reference model. AlpacaEval 2 and Arena-Hard instead measure win rates against GPT-4-class references. Against such strong references, all 7B models win a minority of comparisons, which compresses the scores into a narrow range and reduces the separation between methods.

The last three columns remove this effect by comparing each method directly against GAW-PO. Every baseline has a win rate below 50% against GAW-PO on both benchmarks.

We view these benchmarks as a generalization check. They show that GAW-PO preserves and even improves chat ability. We will add these results to the appendix.

Q1

Thank you for highlighting these papers! We will include all four in our related work. D²PO uses position-based weights, while SePO and cDPO use additional models to identify important tokens. In contrast, GAW-PO uses gradient alignment to weight rejected tokens without requiring an auxiliary model. AttentionPO appeared in proceedings after our initial manuscript was prepared. We also note that cDPO was outperformed by TI-DPO, which is included in our baselines.

Reviewer_sH9POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GAW-PO, a token-level reweighting method for Direct Preference Optimization (DPO). The method is motivated by the observation that rejected responses may contain useful tokens whose negative gradient updates could interfere with learning preferred behavior. Instead of penalizing all rejected tokens equally, GAW-PO estimates their gradient alignment with preferred-response gradients and assigns smaller weights to tokens whose penalization may interfere with preferred behavior.

The method considers both pair-specific and dataset-level preferred gradient directions. To make gradient computation tractable, it uses a low-rank PiSSA adapter as a gradient probe and precomputes token weights before full-parameter DPO training. Experiments on 11 benchmarks covering mathematics, reasoning, coding, and question answering show improvements over several preference optimization baselines. The authors further demonstrate improved stability under aggressive preference optimization, where standard DPO suffers substantial performance degradation as the regularization parameter β decreases.

Critical strengths
  1. The paper addresses an interesting and meaningful problem in preference optimization. The observation that rejected responses may contain useful information, and that suppressing such information could interfere with learning preferred behavior, is well motivated. I particularly appreciate the attempt to understand this issue through token-level gradient interactions. Rather than relying solely on token importance or surface-level similarity, the proposed approach considers how penalizing individual tokens may affect preferred update directions. This provides an interesting perspective on token-level credit assignment and offers useful insights into the optimization dynamics of preference learning.

  2. The experimental results are encouraging, particularly regarding optimization stability. The paper evaluates GAW-PO across multiple capability domains and compares it with both sequence-level and token-level preference optimization methods. The improvements on BBH and LiveCodeBench are noticeable. More importantly, the results under different values of β reveal an interesting phenomenon: while standard DPO suffers substantial degradation under aggressive preference optimization, GAW-PO remains relatively stable and even improves as β decreases. The uniform and random weighting controls also suggest that the benefits cannot be fully explained by simply reducing the overall negative gradient contribution. In addition, the use of low-rank gradient probing and efficient gradient projections makes the approach computationally practical.

Critical weaknesses
  1. While the motivation is interesting, the relationship between the proposed approach and the original preference optimization objective remains somewhat unclear. DPO primarily aims to improve the relative preference of chosen responses over rejected ones. Although this process may decrease the absolute likelihood of chosen responses, such a decrease does not necessarily imply that the relative preference objective is being optimized incorrectly. If the concern is preserving chosen behavior during preference optimization, introducing an explicit chosen-likelihood regularization term would be a natural alternative. The paper discusses related approaches, including DPO-Positive (Pal et al., 2024), but does not compare against them experimentally. More fundamentally, it remains unclear what specific advantage token-level gradient reweighting provides over these more direct approaches to preserving chosen behavior. The reported results mainly focus on downstream benchmark performance, leaving the actual changes in chosen likelihood and relative preference margins unexplored. Consequently, although the performance improvements are encouraging, the connection between the proposed gradient-interference mechanism and the observed benefits is not yet fully established.

  2. Some aspects of the optimization behavior introduced by the proposed reweighting mechanism remain unclear. The central motivation is to reduce negative updates that may interfere with preferred behavior. However, even for tokens identified as strongly aligned with preferred gradients, the method retains a nonzero negative contribution. It is not entirely clear why such updates should be continuously downweighted rather than excluded, or what role the remaining negative contribution plays in achieving the intended objective. Another related issue concerns shared prefixes between chosen and rejected responses. In standard DPO, the gradient contributions from identical prefixes cancel, whereas the asymmetric weighting in GAW-PO breaks this cancellation and introduces additional positive updates on shared content. Such updates may be beneficial, but they are not necessarily attributable to more accurate identification of harmful rejected tokens. Together, these observations make it difficult to distinguish the benefits of gradient-based token-level credit assignment from the effects of changing the balance between positive and negative training signals. The current analysis does not fully clarify this distinction.

Questions for authors
  1. The paper motivates GAW-PO partly through the potential degradation of chosen behavior during DPO. How do the authors distinguish the intended benefits of token-level gradient reweighting from those of existing likelihood-preserving approaches, such as DPO-Positive? In particular, how should we understand the relationship between preserving chosen likelihood and improving relative preference in the proposed framework?

  2. What motivates retaining a nonzero negative contribution for rejected tokens whose gradients are strongly aligned with preferred behavior, rather than excluding their negative contributions? Relatedly, how do the authors interpret the additional updates arising from shared prefixes that would otherwise cancel in standard DPO?

  3. Could the authors clarify the role of the dataset-level preferred gradient? In particular, what additional information does the global direction contribute beyond the pair-specific chosen gradient, and how important is this component to the proposed mechanism?

4 comments

Thank you for your suggestions! We respond to them individually, below. We hope that our responses will addressed your concerns. If so, we would be grateful if you would consider raising your score.

Additional evaluations on judge-based benchmarks (AlpacaEval 2, Arena-Hard v0.1, and MT-Bench) are included in the revised paper (Appendix F) and summarized in our response to Reviewer bueS.

W1 & Q1

DPO-Positive does not have a public code release, so a fair comparison requires us to reimplement it from the paper and train it. This requires additional time. In the meantime, we clarify the conceptual difference between the two approaches.

DPO-Positive and GAW-PO address the decrease in chosen likelihood through different mechanisms. When the chosen and rejected gradients are strongly aligned, decreasing the rejected likelihood can also decrease the chosen likelihood. DPO-Positive adds a term that encourages higher chosen likelihood. However, it does not reduce gradient entanglement directly. A paper by Yuan et al. (2025), shows that strengthening the chosen update can make it harder to decrease the rejected likelihood. GAW-PO instead addresses gradient entanglement at the token level. Our alignment score (s_t) estimates each token's contribution. Tokens with high positive alignment receive lower weights, while tokens with conflicting gradients retain higher weights. This reduces the contribution of strongly aligned gradients without uniformly weakening the rejected-response penalty. We will update our paper to include this explanation and cite the article in our related work.

Empirical evidence: Our Uniform baseline (Appendix D.2) assigns the same weight (c=0.489) to every rejected token, increasing the chosen-to-rejected gradient coefficient ratio to approx. 2. This approximates the relative weighting effect of DPO-Positive, although its activation mechanism differs. GAW-PO uses the same average weight but distributes it according to gradient alignment. It outperforms Uniform by 2.09 points on average and 13 points on HumanEval+ (Table 4), indicating that token-level weight allocation contributes beyond the overall weighting ratio.

Yuan, H., Zeng, Y., Wu, Y., Wang, H., Wang, M., & Leqi, L. (2025, May). A common pitfall of margin-based language model alignment: Gradient entanglement. In International Conference on Learning Representations (Vol. 2025, pp. 5278-5297).

W2 & Q2

GAW-PO uses continuous weights to adjust each rejected token's penalty based on its gradient alignment, rather than selecting or excluding tokens. Strongly aligned tokens receive lower penalties, while conflicting tokens receive higher penalties. This preserves the DPO preference signal across all tokens while adjusting their relative contributions. Per-response normalization ensures comparable weights across responses with different gradient magnitudes. During development we also considered harder variants, including binary weights and mappings without normalization; they did not perform as well, which is consistent with this design.

Q3

The pair direction protects rejected tokens whose behavior is also present in the paired chosen response. However, it may miss useful behavior when the two responses follow different approaches. The global direction, computed as the mean chosen gradient across the dataset, addresses this limitation by capturing patterns consistently supported by preferred responses. Taking the maximum of the two alignment scores allows either direction to protect a rejected token. The pair direction determines the score for 79.2% of rejected tokens, while the global direction determines 20.8% (Section E.2).

—--

For example: Consider a code that computes the square of a number.

Chosen:

print(x**2)

Rejected

# Square of a number

print(sqrt(x))

—--

Shared print syntax: protected by pair alignment.

Comment explaining the code: may have low pair/chosen alignment but can be protected by global alignment because similar comments occur across the dataset.

Error (confusing the square with square root): has low or negative alignment with both directions and retains a higher penalty.

Thank you for the detailed response. The clarifications are helpful and address several of my concerns.

Regarding W1/Q1, I appreciate the rationale for selectively reweighting rejected tokens and the supporting evidence from the Uniform control. I would still find it helpful to see whether GAW-PO better preserves chosen likelihood in practice. Could you compare the log-likelihood of chosen responses on the same evaluation examples using the initial Olmo-3-7B-Instruct-SFT checkpoint and the final models trained with DPO and GAW-PO? Evaluating these existing checkpoints would be sufficient; training curves would also be useful if intermediate checkpoints are already available. This would help connect the proposed mechanism to the actual changes in chosen likelihood.

Regarding W2/Q2, I understand and accept the rationale for continuous weighting and retaining nonzero penalties for strongly aligned rejected tokens. However, I would still appreciate clarification of the shared-prefix effect. For identical shared prefixes, the direct chosen and rejected gradient contributions cancel in standard DPO, whereas the asymmetric weighting in GAW-PO introduces an additional positive contribution on this content. How should this reinforcement be understood in relation to the proposed mechanism of reducing harmful negative updates? A conceptual clarification would be sufficient here.

Regarding Q3, the explanation of how the global direction captures preferred behavior that may be absent from the paired chosen response is clear. Together with the reported statistic that it determines the alignment score for 20.8% of rejected tokens, this clarifies its complementary role. I consider this question addressed.

Thank you for the follow-up!

W1/Q1

Following your suggestion, we tracked the log-likelihood of chosen responses during training. Since training runs for a single epoch, each batch is scored before the model is updated on it, so the values reflect behavior on unseen preference pairs. As a reference, we score the initial Olmo-3-7B-Instruct-SFT checkpoint on the same batches in the same order. We report this for Standard DPO and GAW-PO checkpoints (from Table 1 and Figure 2). As images cannot be included in this comment, the training curves for all runs are provided in the revised paper (Figure 4, Appendix E.3) and in our anonymous repository. The following table shows the summed log-likelihood of chosen responses, averaged over the 1,631 training steps (higher is better).

SFT reference: −417.03DPOGAW-PO
Table 1 checkpoints
−396.17−384.43
β sweep (lr = 1e-5 for both methods)
β = 0.2−393.58−393.05
β = 0.1−391.56−384.43
β = 0.05−391.22−376.57
β = 0.02−408.13−360.77
β = 0.01−509.41−353.35
β = 0.005−675.37−346.64
β = 0.002−1039.38−351.39

As β decreases, GAW-PO increasingly raises the chosen log-likelihood. Standard DPO first stagnates, then falls sharply below the SFT level. These are exactly the settings where its downstream performance collapses (Figure 2). GAW-PO instead remains above SFT in the same regime. The training curves show that this divergence develops early in training.

W2/Q2

This is correct, and the reinforcement is intended. The shared prefix would likely have a small weight in our approach, which would increase the likelihood of those tokens instead of canceling them as in the case of DPO. Since the prefix is part of the chosen response, we can safely assume that this is a desired behaviour, and the mistake happens somewhere later in the rejected response.

This matters when the shared content is not yet well modeled by the policy. Consider a prompt asking for a JSON answer, where the chosen and rejected responses both begin with ```json\n{ and diverge only in the fields (as in Table 4b). If the model rarely produces this format, standard DPO learns nothing about it, because the prefix contributions cancel exactly. GAW-PO instead learns part of the shared prefix, which the preferred response requires. When the policy already predicts the prefix confidently, the token gradients are small, so this residual matters mainly where the shared content has not yet been learned. Thus, GAW-PO extracts useful signal from tokens that would otherwise contribute only a negative update or none at all.

Thank you for the additional analyses and clarifications in response to both my comments and other reviewers’ comments. These address my main concerns, and I have updated my rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.