acceptodds
Under review as a conference paper at ICLR 2027

Information-Aligned Preference Optimization for Direct Preference Learning

31 views

Abstract

Direct Preference Optimization (DPO) has become a standard approach for aligning large language models with human feedback. Recent work has addressed its sensitivity to response length, yet equalizing token counts alone leaves open which tokens should contribute to preference learning. In this work, we study this question through an information-theoretic lens and introduce Information-Aligned Preference Optimization (IAPO). IAPO uses model entropy as a practical ranking signal, masking low-entropy positions in the longer response to match the retained token count of the shorter response. Our analysis shows that true conditional entropy upper-bounds token-level preference information and that top-K selection minimizes a discarded-entropy surrogate under a fixed token budget. Experiments across multiple model families and scales show that IAPO improves over DPO and achieves strong average performance against recent variants. Matched-budget ablations further demonstrate the benefit of entropy-guided selection beyond length equalization, supporting selective token aggregation as an effective strategy for preference optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_2U5ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Information-Aligned Preference Optimization (IAPO). IAPO selects high-entropy tokens from each response and compares chosen and rejected responses using the same number of selected tokens. The use of entropy for token selection is partially motivated by a theoretical analysis showing that true conditional entropy upper-bounds token-level preference information. Experimental results demonstrate that IAPO outperforms existing DPO variants on conversational, general question-answering, and mathematical reasoning tasks.

Critical strengths

The experimental results demonstrate consistent improvements over existing methods across several tasks and models. The reported performance gain is generally high and would be practically useful.

Critical weaknesses

Insufficient justification for choosing entropy over alternative token-selection metrics:

  • The analysis in Figure 1(a) shows a correlation between token entropy and likelihood. However, since entropy is computed from the predictive probability distribution that also determines token likelihood, this correlation is somewhat expected and provides limited insight into why entropy is an effective token-selection criterion.

  • Figures 1(b) and 1(c) suggest that token entropy may serve as a proxy for the likelihood gap between chosen and rejected responses. However, if the authors consider this gap an important training signal (and I would agree), it would be natural to use it directly for token selection. The advantage of using entropy instead remains unclear.

  • Related to the previous points, the experiments should include comparisons with alternative token-selection criteria, such as the likelihood gap, token likelihood, the first K tokens, and the last K tokens, to substantiate this design choice,

Questions for authors

How much additional computational overhead does IAPO incur by computing entropy for every token in each response, compared with standard DPO and SamPO (the random K-token baseline)?

3 comments

Thank you for recognizing the improvements across tasks and models. To address your main concern about the choice of entropy, we have added comparisons with gap-based, surprisal, and positional selectors.

W. Entropy and alternative selection scores.

Our main contribution is entropy-guided token selection for preference optimization, supported by an information-theoretic motivation and empirical gains beyond length equalization. Entropy measures uncertainty over the full next-token distribution, whereas token likelihood concerns the observed token. Our theoretical motivation is that low true conditional entropy limits token-level preference information; IAPO uses policy entropy as a practical proxy for this selection criterion. Figure 1 provides initial motivation for this choice, while the added comparisons directly test it against the alternatives raised in your review.

We added the requested comparisons on Qwen2.5-Math-7B, keeping the training configuration, retained-token budget, and loss unchanged. Gap-KK selects tokens by the absolute difference between chosen and rejected token log-probabilities; Surprisal-KK ranks positions by observed-token negative log-probability. First-KK and Last-KK retain the first or last KK positions of the longer response, respectively. The eight-benchmark averages are:

SelectorAverage (%)
Random-KK51.5
Gap-KK51.9
Surprisal-KK50.2
First-KK45.2
Last-KK44.0
IAPO54.1

IAPO achieves 54.1%, exceeding Gap-KK by 2.2 percentage points and Surprisal-KK by 3.9 points. Its 2.6-point gain over Random-KK at the same token budget supports the contribution of entropy ranking beyond length equalization. Together with the First-KK and Last-KK comparisons, these results provide direct empirical support for the choice of entropy in this setting. Table 5 additionally reports 54.0±0.454.0\pm0.4 for IAPO versus 51.9±0.351.9\pm0.3 for Random-KK across three independent training runs.

Q1. Computational overhead.

On Qwen2.5-Math-7B, IAPO increases total training time by approximately 5.2% over DPO and 4.3% over Random-KK (SamPO). Each two-epoch run uses 230 optimizer steps. Dividing the recorded total training time by this step count gives the following average time per optimizer step:

MethodAverage time per optimizer step
DPO122.6 s
Random-KK (SamPO)123.7 s
IAPO129.0 s

This corresponds to an additional 6.4 s per step over DPO and 5.3 s over Random-KK.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 2U5Z,

Thank you for your suggestions on evaluating alternative token-selection criteria. We have added the requested comparisons and would welcome your feedback on whether these results address your concern about the choice of entropy. Please let us know if further clarification would be helpful.

Thank you for your time.

Best regards,

The Authors

Reviewer_2U5ZOpenReview ↗

Thank you for the rebuttal

Thank you for the additional experiments and the clarification regarding computational overhead. The new comparisons partially address my concern about the choice of entropy. Although the results are limited to aggregate scores on a single mathematical reasoning model, and I still find the motivation from Figure 1(a–c) insufficient, the additional empirical evidence strengthens the paper. I will therefore raise my rating to weak accept.

Reviewer_ifxiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper tackles the length bias problem in DPO. In the standard form, DPO sums over all tokens and hence will have length bias introduced when there are long responses. compared to existing methods of dealing this, this paper introduces an entropy based selection signal into selecting which tokens to include in the loss. It is motivated by the observation that, predictive entropy correlates well with token negative log-probability. Also the chosen/rejected log-probability gap increases with entropy, although not monotonically.

The method IAPO attempts to shorten the longer response to match the shorter response length, by picking only the high entropy tokens from the longer one. The entropy is calculated from the current policy and recalculated at every update. Only the loss terms are masked to introduce the modification. There are no new hyperparameters also. The paper also gives some theoretical grounding for this method.

The experiements uses multiple models from Llama, Gemma and Qwen families, and compares against baselines of DPO, SimPO, R-DPO, SamPO. IAPO seems to do well on open-ended benchmarks. IAPO seems to have the best average among the compared 3 settings, although not the best on every metric, on general benchmarks. It scores relatively well on the Math benchmarks also, although there are some misses. The independent runs also seem show a consistent win over other methods.

The theory investigates into True distribution information bounds, the computable surrogates, per-token bounds, subsequence bounds etc. This provides a decent motivation for the method. Appendix also provides details about high/low entropy tokens types, winrates on response lengths etc.

Critical strengths
  • It is quite simple and straightforward to adopt the change and implement.
  • Only applies the method to the loss terms and hence doesnt change the training setting of full prefix as context.
  • Diagnostics provide decent motivation and intuitive sense of the method.
  • The comparisons are done at an equal token budget, which makes interpretation much easier.
  • Covers 5 models across 3 families in experimentation.
  • Theoretic investigation into why it might work
  • Statistical confidence intervals are reported on some results.
  • Quite open claims on shortcomings and weaknesses.
Critical weaknesses
  • The bounds hold for a fixed subset and true entropy, whereas the method uses sample dependent mask from policy's own entropy, so its not one to one. But its acknowledged.
  • The diagnostics is probably not clear enough to show the relevance of entropy to preference directly.
  • Not quite sure about statistical intervals on General benchmarks and Math.
  • gains fall within noise level in some cases, for eg. on ArenaHard on open ended benchmarks.
Questions for authors
  • Have you tried a fixed fraction of tokens, as opposed to always going by the shorter response length?
  • Entropy today comes from current policy. How does it compare to entropy from the refrence model?
  • AppendixB shows type level word clouds, but what does IAPO retain in practice, say for Math? Could you give examples?
  • Any sense of confidence intervals for Result tables where its not given?
  • What would be additional steptime needed for the additional computations? Any percentage increase due to this?
2 comments

Thank you for the thoughtful review and for noting the value of the equal-budget comparisons. We have added the requested ablations and a token-selection example, and clarified the uncertainty measures and computational cost.

W. Theoretical interpretation and empirical support.

Our theory gives a one-sided bound: low true conditional entropy limits conditional preference information. The subset information bound applies to fixed subsets under the true data distribution; top-KK selection minimizes the discarded-entropy surrogate under a fixed token budget. IAPO uses current-policy entropy as a practical proxy, and matched-budget comparisons evaluate the resulting selector. Table 5 reports 54.0±0.454.0\pm0.4 for IAPO versus 51.9±0.351.9\pm0.3 for Random-KK across three independent Qwen2.5-Math-7B runs. The repeated runs support the improvement in the eight-benchmark average.

Q1. Fixed fraction versus the shorter-response length.

We evaluated retained-token budgets of 0.9∣ys∣0.9\lvert y_s\rvert and 0.7∣ys∣0.7\lvert y_s\rvert on Qwen2.5-Math-7B, where ysy_s is the shorter response. The eight-benchmark mathematical averages are:

Retained-token budgetAverage (%)
0.9∣ys∣0.9\lvert y_s\rvert51.1
0.7∣ys∣0.7\lvert y_s\rvert53.3
∣ys∣\lvert y_s\rvert (IAPO default)54.1

The default budget performs best among these tested settings and avoids an additional fraction hyperparameter.

Q2. Current-policy versus reference-model entropy.

We compared current-policy entropy with frozen-reference entropy on Qwen2.5-Math-7B, keeping the token budget and other training settings unchanged. Frozen-reference entropy achieves an eight-benchmark average of 52.1%, compared with 54.1% for IAPO. The current-policy ranking gives a 2.0-point improvement in this comparison.

Q3. Retained tokens in mathematical solutions.

We provide a mathematical example from a held-out UltraFeedback pair evaluated with the final Llama3-8B IAPO checkpoint. In the example, the chosen and rejected responses have 140 and 39 recorded completion positions, respectively, including template and end tokens. With K=39K=39, all positions in the shorter response are retained, while the longer response contributes its 39 highest-entropy positions. Figure 4 in Appendix F.2 of the revised manuscript (p. 22) highlights the retained positions in the full text; masked positions remain in the causal context. The mask is reconstructed from saved token-level entropies at the final checkpoint.

Q4. Confidence intervals and variability.

Table 1 reports Arena-Hard 95% confidence intervals and AlpacaEval 2 standard errors. All mathematical reasoning evaluations use greedy decoding. Table 5 reports mean ±\pm SD across three independent Qwen2.5-Math-7B training runs per method: DPO 49.6±0.349.6\pm0.3, Random-KK 51.9±0.351.9\pm0.3, and IAPO 54.0±0.454.0\pm0.4. These summarize training variability in the eight-benchmark average and support the same ordering across repeated runs. The remaining benchmark results are reported as point estimates.

Q5. Computational cost.

We estimate the average time per optimizer step by dividing the recorded total training time by the 230 optimizer steps in each two-epoch Qwen2.5-Math-7B run:

MethodAverage time per optimizer step
DPO122.6 s
Random-KK123.7 s
IAPO129.0 s

IAPO adds approximately 6.4 s per step (5.2%) over DPO and 5.3 s (4.3%) over Random-KK.

Thanks for the additional details. Appreciate it.

Reviewer_g1L1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IAPO, a modification of DPO that equalizes the number of log-likelihood-ratio terms contributed by a chosen/rejected response pair. It retains every position in the shorter response and the same number of positions with the highest current-policy predictive entropy in the longer response. The masks are recomputed during training and held fixed during backpropagation; the full original prefixes remain available to predict every retained token. Relative to SamPO, the principal change is entropy ranking instead of random selection at the same token budget.

The analysis provides true-distribution conditional-information bounds for fixed subsets, a top-K optimality result for a discarded-entropy surrogate, and a conditional proxy-error bound. The paper expressly distinguishes these statements from information-preservation or convergence guarantees for adaptive policy-entropy masks. Evaluation covers open-ended alignment with Llama3, general benchmarks with Llama3/Gemma2, and mathematical reasoning with Qwen2.5-Math. Matched-budget and independent-training experiments provide the strongest evidence for the proposed selection rule.

Critical strengths
  1. The main experiment separates token selection from token count. Table 4 and Appendix E compare full DPO, random-K, bottom-K, and IAPO while keeping the original prefixes. On Qwen2.5-Math-7B, the reported eight-benchmark averages are 49.4, 51.5, 41.2, and 54.1, respectively. Thus the improvement over DPO cannot be attributed only to equalizing lengths. More importantly, Table 5 reports three independently trained models per method: 49.6 ± 0.3 for DPO, 51.9 ± 0.3 for random-K, and 54.0 ± 0.4 for IAPO. This is substantially stronger evidence than a single checkpoint or variability computed across benchmarks (pp. 8–9, Sections 6.4–6.5).

  2. A simple modification has useful breadth and is presented with appropriate qualifications. IAPO has the highest displayed general-benchmark average for all three textual model settings and the highest mathematical average at both Qwen scales (Tables 2–3). On Llama3-Base, it improves length-controlled AlpacaEval2 performance over SamPO while generating shorter responses (20.18 versus 18.74 LC; 1,309 versus 1,498 tokens, Table 1). The paper also discloses unfavorable task-level results, restricts its repetition claim to Qwen-7B, and does not claim a computational saving from retaining fewer loss terms. These qualifications make the positive empirical result more credible.

Critical weaknesses
  1. The conceptual advance is narrower than the information-theoretic framing suggests. Equal-budget downsampling already comes from SamPO, and high-entropy loss selection has been established for RLVR by Wang et al., 2025. Moreover, SamPO, Appendix G already evaluates a deterministic top-K selector based on token log-probability ratios; this is different from entropy ranking, but is a particularly relevant alternative selection signal. Propositions 4.1–4.3 establish an entropy upper bound and optimality of sorting for the chosen surrogate; they do not explain the behavior of current-policy adaptive masking with intact prefixes, as the authors explicitly acknowledge (Section 4.5). The empirical advance is valuable, but distinguishing entropy from other hard-token scores, rather than only random and bottom-entropy selection, would make the contribution more informative. Figure 1(a)'s strong entropy/surprisal association reinforces this unresolved distinction.

  2. The reported training configuration does not fully establish comparability of the objective scales. Table 7 gives a row labeled “DPO beta,” and Appendix D.1 explicitly says this row refers to DPO. The text specifies SimPO's beta and margin for Llama3-Instruct, but does not give the IAPO/SamPO beta values, R-DPO alpha values, or the remaining SimPO configurations and their selection procedure (pp. 7, 16–18). These parameters affect the logistic loss and update scale, so equal token counts alone do not establish that selection is the only experimental difference. This is a reporting gap, not evidence that the experiments used unfair settings; explicit matched configurations for Table 4/5 and the tuning protocol are needed to resolve it.

Questions for authors
  1. For the matched-budget experiments in Tables 4–5, what exact beta, optimizer settings, initialization/reference checkpoints, and checkpoint-selection rule were used for random-K, bottom-K, and IAPO? Were they identical apart from the selector? Please also provide each baseline's missing objective hyperparameters and how they were selected.

  2. Do you have a matched-budget comparison with observed-token surprisal and the log-probability-ratio selector considered in SamPO Appendix G? This would clarify whether predictive entropy itself adds value beyond other selective weighting rules. A frozen-reference-entropy comparison would additionally test whether recomputing the ranking under the current policy is consequential, but the key request is the alternative-score comparison.

  3. Which NuminaMath source subsets and data revision were sampled, how many preference pairs remained after filtering, and what overlap checks were applied against the eight mathematical evaluation sets? The dataset's official source breakdown includes math, gsm8k, amc_aime, and olympiad material, whereas Section 6.1 and Appendix D.1 describe sampling and reward filtering without an overlap-removal protocol. I am asking for a train/test separation check, not asserting that the sampled pairs are contaminated. Shared data across methods helps relative comparability but does not by itself establish held-out generalization.

3 comments

Thank you for the careful review and for recognizing the empirical improvements. We have added the requested selector comparisons and clarified the training and data-processing details.

W. Contribution and theoretical scope.

We propose an entropy-based selection rule for offline preference optimization, supported by information-theoretic analysis and matched-budget comparisons. SamPO introduced equal-budget downsampling and tested a deterministic log-ratio selector in Appendix G; the cited RLVR work studies high-entropy selection during on-policy reinforcement learning. IAPO ranks positions using current-policy entropy within each preference pair. The comparisons below separate the effect of this ranking from length equalization.

In Table 4, IAPO improves over Random-KK by 1.9 percentage points on Arena-Hard, 1.44 points on AlpacaEval 2 length-controlled win rate, and 2.6 points on the mathematical average. Table 5 shows the same ordering across three independent training runs: 54.0±0.454.0\pm0.4 for IAPO versus 51.9±0.351.9\pm0.3 for Random-KK (mean ±\pm SD). Q2 reports the additional comparisons with alternative selection scores.

The theory motivates entropy as an exclusion criterion: under the true data distribution, low conditional entropy limits a token's conditional preference information. Top-KK ranking minimizes the discarded-entropy surrogate at a fixed budget, while IAPO uses policy entropy as the practical proxy. The matched-budget experiments evaluate this selector with adaptive policy-entropy masking.

Q1. Objective scales and training comparability.

Within each setting in Tables 4–5, the token-selection variants use the same training configuration, initialization, and frozen reference policy. Table 7 reports the common DPO-family scale for DPO, IAPO, Random-KK/SamPO, Bottom-KK, and R-DPO. The shared settings are:

SettingText modelsQwen2.5-Math
DPO-family β\beta0.012.0
Learning rateLlama3: 10−610^{-6}; Gemma2: 8×10−78\times10^{-7}10−610^{-6}
Optimizer / scheduleAdamW / cosineAdamW / cosine
Global batch size128128
Epochs / warmup ratio1 / 0.12 / 0
Maximum prompt / sequence length1800 / 20481000 / 4096

The initialization checkpoints are princeton-nlp/Llama-3-Base-8B-SFT, meta-llama/Meta-Llama-3-8B-Instruct, google/gemma-2-9b-it, Qwen/Qwen2.5-Math-1.5B, and Qwen/Qwen2.5-Math-7B. Within each model setting, the reference-based methods share the initialization, and the reference policy is its frozen copy. In Tables 4–5, Random-KK, Bottom-KK (where included), and IAPO differ only in the selector, using K=min⁡(∣yw∣,∣yl∣)K=\min(|y^w|,|y^l|); DPO is the unmasked baseline. These comparisons use the final checkpoints after the fixed training schedule.

SimPO follows the official Princeton-NLP configurations: (β,γ/β,lr)=(2.0,0.5,6×10−7)(\beta,\gamma/\beta,\mathrm{lr})=(2.0,0.5,6\times10^{-7}) for Llama3-Base, (2.5,0.55,10−6)(2.5,0.55,10^{-6}) for Llama3-Instruct, and (10,0.5,8×10−7)(10,0.5,8\times10^{-7}) for Gemma2. Qwen2.5-Math uses the generic SimPO defaults β=2.0\beta=2.0 and γ/β=0.25\gamma/\beta=0.25, with the remaining training settings above. For R-DPO, we conducted a hyperparameter search over α∈{0.05,0.1,1.0}\alpha\in\{0.05,0.1,1.0\} and selected, for each model setting, the configuration with the best final evaluation performance. The selected values are 0.050.05, 0.10.1, 0.10.1, and 0.050.05 for Llama3-Base, Llama3-Instruct, Gemma2, and both Qwen2.5-Math sizes, respectively.

Except for the settings specified here, we retain the defaults of the corresponding official implementations (SimPO and Online-DPO-R1).

Q2. Alternative scores and reference entropy.

We evaluated the requested selectors on Qwen2.5-Math-7B with the same token budget and training setup. Surprisal-KK ranks positions by observed-token negative log-probability; LogRatio-KK uses the signed policy/reference log-probability ratio as in SamPO Appendix G. Reference-Entropy-KK uses the frozen reference model's predictive entropy. The averages across the eight mathematical benchmarks are:

SelectorAverage (%)
Random-KK51.5
Surprisal-KK50.2
LogRatio-KK (SamPO-TopK)51.4
Reference-Entropy-KK52.1
IAPO54.1

IAPO has the highest average in this comparison. Using current-policy entropy instead of frozen-reference entropy raises the average from 52.1% to 54.1%.

Q3. Data provenance and overlap checks.

The 20,000-prompt pool follows the Online-DPO-R1 data-construction pipeline and uses the train split of RLHFlow/numia_prompt_dpo1, revision 4dce4fb1d7370f1fc93da9fc6bf324421b9a2e52, with the NuminaMath source subsets numina_cn_k12, numina_synthetic_math, numina_olympiads, numina_synthetic_amc, numina_aops_forum, and numina_amc_aime. Reward filtering produced 13,629 and 14,803 candidate preference pairs for Qwen2.5-Math-1.5B and 7B, respectively.

We checked training prompts against all eight mathematical evaluation sets by exact matching after removing the training chat wrapper and fixed reasoning suffix, applying Unicode NFC normalization, and standardizing formatting (whitespace, mathematical delimiters, LaTeX spacing and sizing commands, equation tags, quotation marks, and trailing periods), while preserving case, numbers, and mathematical operators. This identified two prompts overlapping with OlympiadBench. These two prompts were removed before training, leaving 13,627 and 14,801 pairs for the reported 1.5B and 7B runs, respectively. The procedure and final counts are now documented in Appendix F.1 of the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer g1L1,

Thank you for your constructive review, which has helped improve our paper. We would appreciate your feedback on whether our response and the additional experiments address your concerns. We would be happy to clarify any remaining points.

Thank you for your time.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We thank the reviewers for their feedback. We have posted our detailed responses and uploaded the revised manuscript. We summarize the main additions below.

Our central contribution is entropy-guided token selection for preference optimization, motivated by information-theoretic analysis. To test its benefit beyond length equalization, we added the requested selector comparisons on Qwen2.5-Math-7B under the same token budget and training configuration. IAPO achieves an eight-benchmark average of 54.1%, compared with 51.5% for Random-K, 51.9% for Gap-K, 50.2% for Surprisal-K, and 51.4% for LogRatio-K.

We also report comparisons with positional selectors, frozen-reference entropy, and fixed-fraction budgets. Based on recorded total training times, IAPO adds approximately 5.2% overhead over DPO and 4.3% over Random-K.

The revised manuscript clarifies the training configurations, data sources, and overlap checks, and includes a token-selection example in Appendix F. These additions provide further evidence for the selection rule and make the experimental setup easier to assess.

We welcome any further feedback from the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.