- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes UTIL, which decomposes the DPO preference signal into a length-neutral quality term z_qual and a utility-guided length term z_len, so that length-related learning is modulated bidirectionally in a context-dependent manner. Empirically, UTIL achieves the highest Win Rate and LC Win Rate on Arena-Hard and AlpacaEval 2 with both Llama-3-8B and Qwen2.5-7B, and the ablations indicate that the utility-guided length term is the main source of the gains.
Critical strengths
The paper offers a clear reframing of response length from a "global bias" to a "marginal token utility" perspective. By separating the quality signal z_qual from the utility-guided length signal z_len, UTIL disentangles length from quality more effectively, which in turn enables better policy optimization.
Critical weaknesses
-
Marginal token utility is defined through the expected reward at each token length, but estimating the reward for every length is difficult when only a few, possibly incomplete, candidate responses are available per context. This raises concerns about the reliability and generalization of the method.
-
In z_qual, the chosen and rejected responses are restricted to the same number of tokens, min(L_w, L_l), while z_len is estimated over the remaining tokens. Such a hard split on length makes the quality estimation inaccurate.
Questions for authors
N/A
We thank the reviewer for appreciating our proposed perspective and for raising important questions. We have carefully addressed both concerns with further clarifications and experimental evidence to improve the paper.
W1: Reliability and Generalization of Utility Estimation
We agree that a single slope estimated from a few candidate responses cannot fully capture the length–quality relationship. Our goal is not to recover the full function, but to extract additional information about how response quality changes with length, which conventional preference training does not explicitly model. By estimating average marginal token utility from existing training candidate responses, we introduce a context-specific signal for adaptive length control without additional data collection or scoring costs.
To examine the reliability and generalization of utility estimation, we evaluate two different preference training settings: a same-model dataset with five Llama-generated responses per prompt scored by ArmoRM, and a cross-model dataset with four responses per prompt from diverse sources, spanning the GPT families, LLaMA families, and other models such as Falcon, scored by GPT-4. UTIL achieves consistent performance gains across both backbone models on both benchmarks. These consistent improvements support the effectiveness of utility-guided length control across different candidate sources, scoring methods, and training models. To assess the reliability of utility estimation across these two generalization settings, we conduct a leave-one-candidate-out analysis, recomputing utility after removing each candidate response. The resulting estimates remain consistent with the full-set estimates in both settings, with Pearson correlations of 0.897 and 0.590, and utility direction consistency of 87.92% and 83.53% for the same-model and cross-model settings, respectively. These results support the reliability and generalizability of utility estimation across different training data settings.
W2: Quality–Length Signal Separation
We understand the concern and would like to clarify that UTIL separates quality and length as two complementary signals, rather than dividing responses into different token segments. Motivated by prior work showing that equal-token comparisons can mitigate length bias in preference estimation [1], zqual compares equal numbers of uniformly sampled tokens from the preferred and rejected responses. Our main contribution lies in zlen, which models the response-length difference using context-specific marginal token utility estimated from full responses, rather than the remaining tokens. This design preserves a length-balanced quality comparison while adaptively controlling the contribution of response length according to its utility.
To examine the effect of the equal-token quality comparison, we conduct an ablation in Table 2 that removes this equal-token design from UTIL. As shown below, this variant consistently reduces both Win Rate and LC WR on Arena-Hard and AlpacaEval. These results support the benefit of combining length-balanced quality comparison with context-specific utility-guided length control. |Benchmark|Variant|Win Rate ↑|LC WR ↑|Tokens| |---|---|---|---|---| |Arena-Hard|w/o Equal-Token Comparison|48.00|51.10|576| ||UTIL|52.60|54.90|583| |AlpacaEval|w/o Equal-Token Comparison|38.00|38.08|439| ||UTIL|39.41|39.54|435|
[1] Lu et al., Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence, EMNLP 2024.