acceptodds
Under review as a conference paper at ICLR 2027

Masking Frequent Tokens Sharpens Direct Preference Optimization

Abstract

Direct Preference Optimization (DPO) aligns language models by optimizing over sequence-level sums of token-wise implicit reward differences. However, we identify a pervasive pathology in this formulation: a disproportionately small subset of high-frequency token types dominates cumulative sequence scores while appearing symmetrically across both preferred and dispreferred responses. Specifically, under canonical Qwen tokenization on Anthropic HH-RLHF, merely 69 token types account for of all response tokens and of within-pair shared token mass, exhibiting substantially lower preference-side specificity than the remaining vocabulary. This symmetric ubiquity induces gradient entanglement and dilutes the discriminative preference signal propagated through the objective. To resolve this issue, we introduce Anisotropic DPO (ADPO) and its canonical realization, Frequency-Hard DPO. Using a fixed, label-agnostic vocabulary mask, our method zeroes the implicit reward contribution of high-frequency response tokens while assigning unit weight to informative positions, thereby suppressing gradient interference without modifying preference pairs, discarding context, or introducing learned parameters. Here, anisotropy designates non-uniform token-level objective weighting rather than representational geometry. Extensive empirical evaluations on AlpacaEval, MT-Bench, and Arena-Hard demonstrate that Frequency-Hard DPO consistently outperforms standard DPO across Qwen-2.5-7B-Instruct and Llama-3-8B-Instruct, establishing that selectively masking shared high-frequency tokens offers an effective, zero-overhead mechanism for robust preference alignment.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Mechanistic Interpretability, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VTktOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve DPO by masking frequent response-token types. It is motivated by the observation that frequent token types occupy much of the occurrence mass on both sides of preference pairs while showing weaker aggregate chosen–rejected distinction than retained types. The paper tests whether suppressing their direct score contributions can reduce the alignment component of gradient entanglement.

It proposes Anisotropic DPO (ADPO), also called Frequency-hard DPO. Using the existing token-level decomposition of standard DPO’s response score, it applies a fixed binary mask based on corpus frequency to exclude frequent token types from the score. This intervention changes how the response-level score is assembled while preserving all tokens in the conditioning context.

Experiments on Qwen2.5-7B-Instruct and Llama-3-8B-Instruct report improvements on AlpacaEval, MT-Bench, and Arena-Hard, alongside frequency audits, gradient diagnostics, and selection ablations.

Critical strengths
  1. The paper’s related-work coverage is relatively comprehensive. It discusses closely related token-selection and reweighting methods, as well as studies of gradient entanglement and likelihood displacement. The discussion distinguishes the proposed frequency mask from existing selectors and clarifies that its use of “anisotropy” refers to unequal token-score coefficients rather than representation geometry, helping readers place the method within the literature.

  2. The paper examines the hypothesis that highly frequent tokens carry less preference-relevant information than the remaining tokens. It evaluates frequent-token content using a frozen reward model and existing human preference labels, comparing complete responses, responses containing only frequent tokens, and responses with frequent or randomly selected tokens removed. These comparisons provide useful empirical motivation for the proposed frequency-based selector.

Critical weaknesses
  1. Given the well-established use of token masking and reweighting in LLM post-training, particularly in closely related approaches such as SparsePO and SePO, the novelty of the contribution remains a concern. Prior work on gradient entanglement also proposes token masking to reduce coupling. The paper also cites policy-confidence-based approaches that select preference-critical tokens without an auxiliary model or additional inference passes. The paper should directly and clearly demonstrate the additional value of this particular reweighting mechanism and justify the significance of framing it as “anisotropy.”

  2. The connection between the frequency observations, gradient alignment, and the proposed weighting scheme is insufficiently established. The paper moves from frequent tokens’ weak aggregate preference-side associations to chosen–rejected gradient cosine similarity, and then uses the observed cosine reduction to support unequal token-score weighting. Although gradient alignment is relevant to entanglement, the analysis does not establish that corpus frequency identifies the contextual contributions responsible for harmful coupling. Moreover, cosine reduction alone does not establish improved likelihood dynamics, which also depend on gradient norms and the gradient contribution of frequent tokens after chosen–rejected subtraction. A clearer connection between these observations would strengthen the motivation for the specific frequency-based mask.

  3. The main results show limited improvements over the strongest competing methods, and their reliability is insufficiently established. In Tables 3 and 4, ADPO improves MT-Bench over TDPO by only 0.055 on Qwen and 0.04 on Llama, while its Qwen Arena-Hard improvement is 0.4 percentage points. Although the AlpacaEval gains are larger, the paper does not sufficiently specify the uncertainty estimates or independent training repetitions needed to assess the consistency of these differences. The highest reported point estimates therefore provide limited support for broader claims of robust superiority, particularly given the method’s narrow distinction from existing token-selection approaches.

  4. The held-out HH analysis needs a clearer experimental specification and connection to the proposed method. Section 3.4 evaluates edited responses using PairRM and existing human preference labels, but the sample size, selection procedure, and corresponding frequency-mask construction are insufficiently specified. The paper should also explain how this HH diagnostic relates to the UltraFeedback-based training experiments. Although the analysis provides motivation for frequency-based selection, deleting tokens changes the response presented to the reward model, whereas ADPO preserves context and masks score contributions. The implications of the observed ranking differences for preference-training effectiveness therefore require further justification.

  5. The evaluation uncertainty and threshold sensitivity need clearer characterization. Tables 3 and 4 do not sufficiently explain whether the reported uncertainty estimates reflect variation across independent training runs. Some gains over the strongest baselines are small. Table 5 also shows substantial threshold sensitivity: on Llama, the AlpacaEval LC win rate is 53.06% when approximately 50 types are masked, versus 48.15% and 42.70% for approximately 25 and 100 types; DPO scores 50.43%. The validation procedure for choosing the threshold and other hyperparameters, the training seeds, and uncertainty in the performance differences between methods should be specified before interpreting the results as robust improvements.

Questions for authors
  1. Could the authors clarify the conceptual and methodological contribution beyond existing token-selection and reweighting approaches? In particular, what additional value does the corpus-frequency rule provide, and what insight is gained from framing it as “anisotropy”? Further justification and controlled comparisons would help establish the contribution of this particular mechanism.

  2. It would help to track chosen and rejected response log-likelihoods during training and measure the net gradient contribution of frequent tokens after chosen–rejected subtraction. These diagnostics could clarify whether lower gradient cosine similarity corresponds to reduced likelihood displacement or other improvements in training dynamics. An ablation separating token masking from changes in the sigmoid weighting would also strengthen the explanation. Where feasible, results on another model family and additional evaluations, such as Arena-Hard v2.0, verifiable instruction following with IFEval, or DPO applied to math or code preferences, would help establish how broadly the frequency-based method works.

  3. Could the authors provide further justification for the connection between token frequency, gradient alignment, and effectiveness? In particular, why should high corpus frequency and weak aggregate preference-side association identify tokens responsible for harmful gradient coupling? Additional analysis or controlled experiments linking frequency-based selection to contextual gradient contributions and improved preference learning would help establish the proposed explanation beyond the observed reduction in gradient cosine similarity.

  4. Could the authors provide more details about the HH experiment, including the sample size, selection procedure, tokenizer, and data used to construct the frequency mask? It would also help to explain how this analysis connects to the UltraFeedback-based training experiments and what the ranking differences on edited responses establish about ADPO’s context-preserving score masking.

Reviewer_GuUQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a small set of very common token types appears on both the chosen and rejected side of DPO pairs, so their log-ratios add little preference signal while coupling the two gradients. The proposed fix counts token frequencies on part of the training data and gives zero weight in the DPO margin to every type above a frequency threshold, while keeping these tokens in the context. The motivation rests on token-count statistics, a gradient-cosine diagnostic and a reward-model test on edited responses. On Qwen2.5-7B-Instruct and Llama-3-8B-Instruct, the method slightly outperforms DPO and several token-level baselines on AlpacaEval 2.0, MT-Bench and Arena-Hard.

Critical strengths
  1. Almost zero overhead. The mask is a fixed vocabulary-level vector computed once from token counts, so it needs no extra model or learned component and drops into any DPO implementation.
Critical weaknesses
  1. The claim that frequent tokens "dominate cumulative sequence scores" is never measured directly. The only evidence is how often these tokens appear (Fig. 2a, Table 7), not how much of the DPO margin or its gradient they account for. These can differ a lot, since high-confidence function words tend to have near-zero log-ratios and gradients.

  2. The masked set includes tokens that clearly carry preference signal, and several of them are missing from Table 2 but listed in Table 9. Digits are masked for all three tokenizers, so with Qwen2.5's digit-level tokenization a right and a wrong numeric answer look almost the same to the score. EOS, markdown and code symbols such as "=" and "(" are masked too, which affects length, formatting and code, yet neither generation length nor raw win rate is reported. The evaluation also omits math and code benchmarks such as GSM8K, MATH and HumanEval, which would expose these effects.

  3. Fig. 3(a) lacks controls. It compares the masked gradient only with the full gradient, without a random mask of matched occurrence mass or a complement version of Eq. 6 with M replaced by 1−M. Without these, the lower cosine may simply reflect dropping a large share of terms rather than anything specific to frequent tokens.

  4. The PairRM experiment in Sec. 3.4 mostly confirms common sense. A fragment made only of frequent tokens is out-of-distribution text that no reward model can rank, and beating random deletion only shows that content words matter more than function words. Moreover, the experiment scores edited text, while ADPO never edits text and only changes the score.

  5. The analysis data and the training data do not match. The headline statistics in the abstract and Fig. 2(a) come from HH-RLHF, but training uses on-policy UltraFeedback data. On that data the masks cover only 36–41% of tokens and 47–52% of shared mass (Table 7), far below the 55.1% and 85.9% reported on HH.

  6. Results are very sensitive to τ, which appears to be tuned on AlpacaEval. In Table 5, masking about 25, 50 and 100 types gives LC win rates of 48.15%, 53.06% and 42.70% versus 50.43% for DPO, and the RM score swings from 1.65 to 2.20 to 1.44. This points to seed variance or test-set tuning, there is no a reliable rule for choosing τ.

Questions for authors

see the Weaknesses part

Reviewer_hLEqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Frequency-hard DPO, also called Anisotropic DPO (ADPO), which excludes high-frequency vocabulary types from the token log-ratio sums in the DPO preference margin. Frequencies are estimated symmetrically from both responses in training pairs, and the resulting binary mask is fixed. The complete response remains in the causal context, and the full softmax is retained. The motivation is that frequent types account for substantial chosen–rejected overlap but exhibit relatively weak aggregate preference-side associations. The paper presents structural token-frequency audits, a score-gradient cosine diagnostic, a frozen reward-model analysis of edited responses, and alignment benchmarks on Qwen2.5-7B-Instruct and Llama-3-8B-Instruct.

Critical strengths
  1. A simple and precisely specified intervention with encouraging benchmark results. Equations (10)–(13) describe a label-agnostic, fixed selector that requires no learned token scorer. Sections 4.2–4.3 correctly explain that removing direct score terms also changes the pair-level sigmoid gate and does not freeze the corresponding model parameters. Tables 3–4 report improvements over DPO on both model families: AlpacaEval 2.0 LC win rate increases by 3.06 and 2.63 percentage points, respectively. The comparisons include several relevant token-level preference methods.

  2. The frequency prior is supported by more than an in-sample token-count observation. Appendix A.1 separates mask construction, side-association calibration, and evaluation on 1,961 disjoint pairs. Tables 7–8 show the same structural pattern across three tokenizers, with paired confidence intervals for the retained-minus-frequent side-signal difference above zero. Appendix A.2 additionally matches the selected occurrence budget in its randomized structural comparison. The paper also acknowledges the coherence confound in the reward-model response-editing experiment (Section 3.4), which is an appropriate limitation.

Critical weaknesses
  1. The trained ablations do not isolate why frequency selection helps. Table 6 matches the number of masked vocabulary types, whereas random, middle-frequency, and low-frequency sets of 50 types can remove substantially fewer response positions than the most frequent 50 types. Appendix A.2 explicitly recognizes this distinction and provides an occurrence-matched comparison, but only for token-count statistics, not trained models. Thus it does not establish that the benchmark gain comes from removing particularly uninformative types rather than a more general change in score mass or optimization scale. This matters because the scores are not renormalized and both the selected gradient and sigmoid gate change (Section 4.3, lines 343–350). Figure 3(a) measures cosine between masked score gradients at DPO checkpoints; it does not measure the effect of an ADPO update on the original chosen and rejected full-response likelihoods, which also depends on gradient norms and cross-inner products. I recognize the authors' explicit restriction of this diagnostic's interpretation, but the resulting mechanistic evidence is weaker than the abstract and conclusion suggest.

  2. The apparent benefit is highly sensitive to the selected threshold, and its statistical reliability is insufficiently documented. In Table 5, masking approximately 25 or 100 types yields AlpacaEval LC win rates of 48.15% and 42.70%, respectively, both below DPO's 50.43%; masking 50 yields 53.06%. This is a narrow favorable setting rather than evidence of broad robustness. Section 5.1 describes a temperature sweep but does not identify the validation split and selection criterion for temperatures or the frequency threshold. Tables 3–4 do not define the reported uncertainty sufficiently to determine whether it reflects training-seed variation, evaluation-prompt sampling, judge randomness, or another source; the number of independent training runs is also not given. These omissions matter for the modest margins over the strongest competing methods. The unclear description of TGDPO in Section 5.1 further limits assessment of baseline fidelity.

Questions for authors
  1. Can you provide a trained occurrence-matched randomized type-mask control, using multiple random masks, and a control that matches the effective score/update scale? For the mechanism, can you report changes in the original full chosen/rejected likelihoods under actual ADPO updates, rather than only cosine between masked score gradients at DPO checkpoints?

  2. How were the frequency threshold and all methods' hyperparameters selected, and on which disjoint validation data? What exactly do the uncertainty estimates in Tables 3–4 represent, how many independent training runs were used, and do the main gains persist across seeds and a wider threshold neighborhood on both model families?

  3. Which exact TGDPO objective and implementation were used? Section 5.1, lines 421–422, describes a per-token forward-KL regularizer, whereas the cited TGDPO paper distinguishes its token-level reward guidance from TDPO's forward-KL approach (Section 2). This could be a description error, but the distinction is material for the comparison.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewers hLEq, GuUQ, and VTkt for their detailed and constructive feedback. The reviews raised four main themes: (1) whether frequent tokens are specifically uninformative, beyond the effect of removing score mass, (2) the mechanism linking frequency, gradient coupling, and effectiveness, (3) novelty relative to existing token-selection methods, and (4) threshold sensitivity, hyperparameter selection, and broader evaluation. We added experiments for each, as summarized below. Full results are in the individual responses and will be included in the revision.

1. Why frequency selection helps (hLEq, GuUQ, VTkt).

  • Direct margin decomposition. Frequent types carry roughly their occurrence share of the DPO margin. What sets them apart is their distribution across the pair. Compared with random sets of equal occurrence mass, they make up a much larger share of the tokens the two responses have in common and are far less side-specific.
  • Trained occurrence-matched control. We trained with a mask that removes an equal occurrence mass of response-specific content words. It performs clearly worse than both ADPO and DPO, so which tokens are removed matters, not just how much score mass is removed.
  • Score and update scale. The random masks matched by occurrence also match the share of the margin that is removed. In a second experiment, all objectives take updates of equal length in parameter space. Under these controls, ADPO's advantage is not a generic effect of shrinking the margin or rescaling the update.

2. Mechanism (GuUQ, VTkt, hLEq).

  • Decoupling from the informative direction. We added an analysis measured against the informative preference direction, defined by the content-word gradients. Removing the frequent terms raises the update's alignment with this direction. Masking random types of equal mass lowers it. A fixed list of frequent types recovers most of the gain of a POS-tagged oracle, with no tagging or per-response computation.
  • Full-response likelihoods. Under real updates of equal length, ADPO produces the largest increase in the full (unmasked) preference margin on held-out pairs and improves the margin on the most pairs. We state explicitly that this comes from a more preference-efficient update, not from reduced likelihood displacement.
  • Gate vs. selection. Equal-mass controls shift the sigmoid gate by the same amount yet yield smaller gains. The two hybrid objectives that fully separate gate from selection will be added to the revision.

3. Novelty (VTkt). Prior token-level methods (SePO, SparsePO, TIS-DPO, ConfPO) rely on an oracle model, learned masks, auxiliary LLMs, or selection recomputed at every step. ADPO shows that a single corpus statistic, computed once before training, suffices. It needs no extra model, parameters, or per-step cost, reduces exactly to DPO when the mask is empty, and is competitive with these more complex selectors. We will also clarify our narrow use of "anisotropy": the per-token coefficient on the implicit reward depends on the token type rather than being uniform.

4. Threshold, uncertainty, and broader evaluation (hLEq, GuUQ, VTkt).

  • Selection protocol. The threshold and all methods' hyperparameters were selected on a reward-model-scored validation subset of the training data, not on the evaluation benchmarks. Reported uncertainty is across three independent training seeds. We attribute the drop at large thresholds to the mask beginning to remove informative content tokens.
  • Digits, EOS, and code symbols. On GSM8K, MATH-500, HumanEval, and IFEval, masking these tokens causes no systematic loss relative to DPO. ADPO outputs are not longer than DPO's, and termination is unaffected.
  • Consistent statistics. The abstract will report statistics from the UltraFeedback training data rather than HH-RLHF, and the full masked-token list will move to the main text.
  • Clarifications. We corrected the description of TGDPO, which already follows the original implementation. We will add the full HH experiment specification and present that experiment as motivation only.

Revisions. We will add all new analyses, revise the abstract and Sec. 3–4 to state the mechanism precisely (shared, low-specificity mass rather than margin dominance), and add Arena-Hard v2.0 and a third model family.

We would be grateful if the reviewers could look over the responses and let us know of any remaining concerns before the discussion period ends.

AuthorsOpenReview ↗

Polite Nudge

We thank all reviewers again for their thoughtful feedback. We have added additional experiments, and clarified the contribution of the work which have thus made the positioning of the paper much stronger.

As the discussion period ends soon, we would be grateful if the reviewers could read our responses and, if their concerns have been addressed, consider updating their scores. We are happy to answer any remaining questions before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.