- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Information-Aligned Preference Optimization (IAPO). IAPO selects high-entropy tokens from each response and compares chosen and rejected responses using the same number of selected tokens. The use of entropy for token selection is partially motivated by a theoretical analysis showing that true conditional entropy upper-bounds token-level preference information. Experimental results demonstrate that IAPO outperforms existing DPO variants on conversational, general question-answering, and mathematical reasoning tasks.
Critical strengths
The experimental results demonstrate consistent improvements over existing methods across several tasks and models. The reported performance gain is generally high and would be practically useful.
Critical weaknesses
Insufficient justification for choosing entropy over alternative token-selection metrics:
-
The analysis in Figure 1(a) shows a correlation between token entropy and likelihood. However, since entropy is computed from the predictive probability distribution that also determines token likelihood, this correlation is somewhat expected and provides limited insight into why entropy is an effective token-selection criterion.
-
Figures 1(b) and 1(c) suggest that token entropy may serve as a proxy for the likelihood gap between chosen and rejected responses. However, if the authors consider this gap an important training signal (and I would agree), it would be natural to use it directly for token selection. The advantage of using entropy instead remains unclear.
-
Related to the previous points, the experiments should include comparisons with alternative token-selection criteria, such as the likelihood gap, token likelihood, the first K tokens, and the last K tokens, to substantiate this design choice,
Questions for authors
How much additional computational overhead does IAPO incur by computing entropy for every token in each response, compared with standard DPO and SamPO (the random K-token baseline)?
3 comments
Follow-up on Our Response
Dear Reviewer 2U5Z,
Thank you for your suggestions on evaluating alternative token-selection criteria. We have added the requested comparisons and would welcome your feedback on whether these results address your concern about the choice of entropy. Please let us know if further clarification would be helpful.
Thank you for your time.
Best regards,
The Authors
Thank you for the rebuttal
Thank you for the additional experiments and the clarification regarding computational overhead. The new comparisons partially address my concern about the choice of entropy. Although the results are limited to aggregate scores on a single mathematical reasoning model, and I still find the motivation from Figure 1(a–c) insufficient, the additional empirical evidence strengthens the paper. I will therefore raise my rating to weak accept.
Thank you for recognizing the improvements across tasks and models. To address your main concern about the choice of entropy, we have added comparisons with gap-based, surprisal, and positional selectors.
W. Entropy and alternative selection scores.
Our main contribution is entropy-guided token selection for preference optimization, supported by an information-theoretic motivation and empirical gains beyond length equalization. Entropy measures uncertainty over the full next-token distribution, whereas token likelihood concerns the observed token. Our theoretical motivation is that low true conditional entropy limits token-level preference information; IAPO uses policy entropy as a practical proxy for this selection criterion. Figure 1 provides initial motivation for this choice, while the added comparisons directly test it against the alternatives raised in your review.
We added the requested comparisons on Qwen2.5-Math-7B, keeping the training configuration, retained-token budget, and loss unchanged. Gap-K selects tokens by the absolute difference between chosen and rejected token log-probabilities; Surprisal-K ranks positions by observed-token negative log-probability. First-K and Last-K retain the first or last K positions of the longer response, respectively. The eight-benchmark averages are:
IAPO achieves 54.1%, exceeding Gap-K by 2.2 percentage points and Surprisal-K by 3.9 points. Its 2.6-point gain over Random-K at the same token budget supports the contribution of entropy ranking beyond length equalization. Together with the First-K and Last-K comparisons, these results provide direct empirical support for the choice of entropy in this setting. Table 5 additionally reports 54.0±0.4 for IAPO versus 51.9±0.3 for Random-K across three independent training runs.
Q1. Computational overhead.
On Qwen2.5-Math-7B, IAPO increases total training time by approximately 5.2% over DPO and 4.3% over Random-K (SamPO). Each two-epoch run uses 230 optimizer steps. Dividing the recorded total training time by this step count gives the following average time per optimizer step:
This corresponds to an additional 6.4 s per step over DPO and 5.3 s over Random-K.