- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve DPO by masking frequent response-token types. It is motivated by the observation that frequent token types occupy much of the occurrence mass on both sides of preference pairs while showing weaker aggregate chosen–rejected distinction than retained types. The paper tests whether suppressing their direct score contributions can reduce the alignment component of gradient entanglement.
It proposes Anisotropic DPO (ADPO), also called Frequency-hard DPO. Using the existing token-level decomposition of standard DPO’s response score, it applies a fixed binary mask based on corpus frequency to exclude frequent token types from the score. This intervention changes how the response-level score is assembled while preserving all tokens in the conditioning context.
Experiments on Qwen2.5-7B-Instruct and Llama-3-8B-Instruct report improvements on AlpacaEval, MT-Bench, and Arena-Hard, alongside frequency audits, gradient diagnostics, and selection ablations.
Critical strengths
-
The paper’s related-work coverage is relatively comprehensive. It discusses closely related token-selection and reweighting methods, as well as studies of gradient entanglement and likelihood displacement. The discussion distinguishes the proposed frequency mask from existing selectors and clarifies that its use of “anisotropy” refers to unequal token-score coefficients rather than representation geometry, helping readers place the method within the literature.
-
The paper examines the hypothesis that highly frequent tokens carry less preference-relevant information than the remaining tokens. It evaluates frequent-token content using a frozen reward model and existing human preference labels, comparing complete responses, responses containing only frequent tokens, and responses with frequent or randomly selected tokens removed. These comparisons provide useful empirical motivation for the proposed frequency-based selector.
Critical weaknesses
-
Given the well-established use of token masking and reweighting in LLM post-training, particularly in closely related approaches such as SparsePO and SePO, the novelty of the contribution remains a concern. Prior work on gradient entanglement also proposes token masking to reduce coupling. The paper also cites policy-confidence-based approaches that select preference-critical tokens without an auxiliary model or additional inference passes. The paper should directly and clearly demonstrate the additional value of this particular reweighting mechanism and justify the significance of framing it as “anisotropy.”
-
The connection between the frequency observations, gradient alignment, and the proposed weighting scheme is insufficiently established. The paper moves from frequent tokens’ weak aggregate preference-side associations to chosen–rejected gradient cosine similarity, and then uses the observed cosine reduction to support unequal token-score weighting. Although gradient alignment is relevant to entanglement, the analysis does not establish that corpus frequency identifies the contextual contributions responsible for harmful coupling. Moreover, cosine reduction alone does not establish improved likelihood dynamics, which also depend on gradient norms and the gradient contribution of frequent tokens after chosen–rejected subtraction. A clearer connection between these observations would strengthen the motivation for the specific frequency-based mask.
-
The main results show limited improvements over the strongest competing methods, and their reliability is insufficiently established. In Tables 3 and 4, ADPO improves MT-Bench over TDPO by only 0.055 on Qwen and 0.04 on Llama, while its Qwen Arena-Hard improvement is 0.4 percentage points. Although the AlpacaEval gains are larger, the paper does not sufficiently specify the uncertainty estimates or independent training repetitions needed to assess the consistency of these differences. The highest reported point estimates therefore provide limited support for broader claims of robust superiority, particularly given the method’s narrow distinction from existing token-selection approaches.
-
The held-out HH analysis needs a clearer experimental specification and connection to the proposed method. Section 3.4 evaluates edited responses using PairRM and existing human preference labels, but the sample size, selection procedure, and corresponding frequency-mask construction are insufficiently specified. The paper should also explain how this HH diagnostic relates to the UltraFeedback-based training experiments. Although the analysis provides motivation for frequency-based selection, deleting tokens changes the response presented to the reward model, whereas ADPO preserves context and masks score contributions. The implications of the observed ranking differences for preference-training effectiveness therefore require further justification.
-
The evaluation uncertainty and threshold sensitivity need clearer characterization. Tables 3 and 4 do not sufficiently explain whether the reported uncertainty estimates reflect variation across independent training runs. Some gains over the strongest baselines are small. Table 5 also shows substantial threshold sensitivity: on Llama, the AlpacaEval LC win rate is 53.06% when approximately 50 types are masked, versus 48.15% and 42.70% for approximately 25 and 100 types; DPO scores 50.43%. The validation procedure for choosing the threshold and other hyperparameters, the training seeds, and uncertainty in the performance differences between methods should be specified before interpreting the results as robust improvements.
Questions for authors
-
Could the authors clarify the conceptual and methodological contribution beyond existing token-selection and reweighting approaches? In particular, what additional value does the corpus-frequency rule provide, and what insight is gained from framing it as “anisotropy”? Further justification and controlled comparisons would help establish the contribution of this particular mechanism.
-
It would help to track chosen and rejected response log-likelihoods during training and measure the net gradient contribution of frequent tokens after chosen–rejected subtraction. These diagnostics could clarify whether lower gradient cosine similarity corresponds to reduced likelihood displacement or other improvements in training dynamics. An ablation separating token masking from changes in the sigmoid weighting would also strengthen the explanation. Where feasible, results on another model family and additional evaluations, such as Arena-Hard v2.0, verifiable instruction following with IFEval, or DPO applied to math or code preferences, would help establish how broadly the frequency-based method works.
-
Could the authors provide further justification for the connection between token frequency, gradient alignment, and effectiveness? In particular, why should high corpus frequency and weak aggregate preference-side association identify tokens responsible for harmful gradient coupling? Additional analysis or controlled experiments linking frequency-based selection to contextual gradient contributions and improved preference learning would help establish the proposed explanation beyond the observed reduction in gradient cosine similarity.
-
Could the authors provide more details about the HH experiment, including the sample size, selection procedure, tokenizer, and data used to construct the frequency mask? It would also help to explain how this analysis connects to the UltraFeedback-based training experiments and what the ranking differences on edited responses establish about ADPO’s context-preserving score masking.
General Response to All Reviewers
We thank Reviewers hLEq, GuUQ, and VTkt for their detailed and constructive feedback. The reviews raised four main themes: (1) whether frequent tokens are specifically uninformative, beyond the effect of removing score mass, (2) the mechanism linking frequency, gradient coupling, and effectiveness, (3) novelty relative to existing token-selection methods, and (4) threshold sensitivity, hyperparameter selection, and broader evaluation. We added experiments for each, as summarized below. Full results are in the individual responses and will be included in the revision.
1. Why frequency selection helps (hLEq, GuUQ, VTkt).
2. Mechanism (GuUQ, VTkt, hLEq).
3. Novelty (VTkt). Prior token-level methods (SePO, SparsePO, TIS-DPO, ConfPO) rely on an oracle model, learned masks, auxiliary LLMs, or selection recomputed at every step. ADPO shows that a single corpus statistic, computed once before training, suffices. It needs no extra model, parameters, or per-step cost, reduces exactly to DPO when the mask is empty, and is competitive with these more complex selectors. We will also clarify our narrow use of "anisotropy": the per-token coefficient on the implicit reward depends on the token type rather than being uniform.
4. Threshold, uncertainty, and broader evaluation (hLEq, GuUQ, VTkt).
Revisions. We will add all new analyses, revise the abstract and Sec. 3–4 to state the mechanism precisely (shared, low-specificity mass rather than margin dominance), and add Arena-Hard v2.0 and a third model family.
We would be grateful if the reviewers could look over the responses and let us know of any remaining concerns before the discussion period ends.