acceptodds
Under review as a conference paper at ICLR 2027

Length Is Not Just Bias: Marginal Token Utility in LLM Post-Training

Abstract

Response length is often treated as a source of bias in LLM post-training, motivating methods that globally suppress or normalize its influence on learning. We study this issue in preference-based post-training, where our analysis of training data shows that the relationship between response length and quality may not be universal: additional length is associated with better outcomes in some contexts, contributes little in others, and can be detrimental elsewhere. To capture this context-dependent relationship, we introduce marginal token utility, which measures the gain associated with additional response tokens within each context. This perspective suggests that length should not be uniformly penalized or removed from optimization. Instead, its influence should reflect the marginal gains associated with additional tokens. Building on this insight, we propose a marginal-utility-guided objective that modulates length-related learning according to marginal token utility. Experiments across language models and alignment benchmarks show that our approach consistently outperforms existing length-debiasing methods while avoiding unnecessary verbosity. Our results suggest that response length should be viewed not merely as an optimization bias, but as a context-dependent source of marginal utility in LLM post-training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, post-training, preference alignment, length bias, marginal utility

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GnspOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UTIL, which decomposes the DPO preference signal into a length-neutral quality term z_qual and a utility-guided length term z_len, so that length-related learning is modulated bidirectionally in a context-dependent manner. Empirically, UTIL achieves the highest Win Rate and LC Win Rate on Arena-Hard and AlpacaEval 2 with both Llama-3-8B and Qwen2.5-7B, and the ablations indicate that the utility-guided length term is the main source of the gains.

Critical strengths

The paper offers a clear reframing of response length from a "global bias" to a "marginal token utility" perspective. By separating the quality signal z_qual from the utility-guided length signal z_len, UTIL disentangles length from quality more effectively, which in turn enables better policy optimization.

Critical weaknesses
  1. Marginal token utility is defined through the expected reward at each token length, but estimating the reward for every length is difficult when only a few, possibly incomplete, candidate responses are available per context. This raises concerns about the reliability and generalization of the method.

  2. In z_qual, the chosen and rejected responses are restricted to the same number of tokens, min(L_w, L_l), while z_len is estimated over the remaining tokens. Such a hard split on length makes the quality estimation inaccurate.

Questions for authors

N/A

1 comment

We thank the reviewer for appreciating our proposed perspective and for raising important questions. We have carefully addressed both concerns with further clarifications and experimental evidence to improve the paper.

W1: Reliability and Generalization of Utility Estimation

  • We agree that a single slope estimated from a few candidate responses cannot fully capture the length–quality relationship. Our goal is not to recover the full function, but to extract additional information about how response quality changes with length, which conventional preference training does not explicitly model. By estimating average marginal token utility from existing training candidate responses, we introduce a context-specific signal for adaptive length control without additional data collection or scoring costs.

  • To examine the reliability and generalization of utility estimation, we evaluate two different preference training settings: a same-model dataset with five Llama-generated responses per prompt scored by ArmoRM, and a cross-model dataset with four responses per prompt from diverse sources, spanning the GPT families, LLaMA families, and other models such as Falcon, scored by GPT-4. UTIL achieves consistent performance gains across both backbone models on both benchmarks. These consistent improvements support the effectiveness of utility-guided length control across different candidate sources, scoring methods, and training models. To assess the reliability of utility estimation across these two generalization settings, we conduct a leave-one-candidate-out analysis, recomputing utility after removing each candidate response. The resulting estimates remain consistent with the full-set estimates in both settings, with Pearson correlations of 0.897 and 0.590, and utility direction consistency of 87.92% and 83.53% for the same-model and cross-model settings, respectively. These results support the reliability and generalizability of utility estimation across different training data settings.

W2: Quality–Length Signal Separation

  • We understand the concern and would like to clarify that UTIL separates quality and length as two complementary signals, rather than dividing responses into different token segments. Motivated by prior work showing that equal-token comparisons can mitigate length bias in preference estimation [1], zqualz_{\mathrm{qual}} compares equal numbers of uniformly sampled tokens from the preferred and rejected responses. Our main contribution lies in zlenz_{\mathrm{len}}, which models the response-length difference using context-specific marginal token utility estimated from full responses, rather than the remaining tokens. This design preserves a length-balanced quality comparison while adaptively controlling the contribution of response length according to its utility.

  • To examine the effect of the equal-token quality comparison, we conduct an ablation in Table 2 that removes this equal-token design from UTIL. As shown below, this variant consistently reduces both Win Rate and LC WR on Arena-Hard and AlpacaEval. These results support the benefit of combining length-balanced quality comparison with context-specific utility-guided length control. |Benchmark|Variant|Win Rate ↑|LC WR ↑|Tokens| |---|---|---|---|---| |Arena-Hard|w/o Equal-Token Comparison|48.00|51.10|576| ||UTIL|52.60|54.90|583| |AlpacaEval|w/o Equal-Token Comparison|38.00|38.08|439| ||UTIL|39.41|39.54|435|

[1] Lu et al., Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence, EMNLP 2024.

Reviewer_sv3XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces UTIL, a lightweight method for fine-tuning LLMs to generate responses of appropriate length which uses existing labeled data to determine the marginal utility of generating additional tokens in response to a context. The authors show that this utility can be effectively estimated from responses generated by the initial model or a set of other models. Then, using Direct Preference Optimization (DPO) as the fine-tuning framework, they show that a training objective accounting for both length-neutral quality and expected utility of the difference in length provides guidance superior to standard DPO and several existing methods for addressing bias related to response length.

Critical strengths

Despite being a relatively simple extension of SamPO, UTIL shows significant performance gains over SamPO in many cases, indicating that there is real benefit to accounting for the context-dependent marginal token utility. The simplicity of the algorithm makes it easy to apply without the need for additional hyperparameter tuning.

Critical weaknesses

I found it very difficult to understand the intuition behind the objective function when considering zlenz_{len}'s effect on the gradient (see the questions section). I do not doubt the efficacy of the method based on the reported empirical results, but it would be worth including a gradient analysis in the appendix which clarifies this.

The focus on UTIL as a variant of DPO limits the applicability of the results. Showing that UTIL's benefits extend to on-policy RL training would demonstrate that it can be applied to a much larger set of modern fine-tuning pipelines. In particular, it would be important to address how marginal token utility changes as training progresses. I believe that the work can stand on its own without this extension, but the extension would certainly strengthen it.

Questions for authors

From my understanding, if the length difference of the current pair disagrees with the direction of marginal utility, then zlen<0z_{len} < 0 which increases the magnitude of the gradient. Since the direction of the gradient is determined by zqualz_{qual}, this will encourage the model to mimic ywy_w at the uniformly-sampled token positions. Since those tokens are just being sampled uniformly, it is possible for unnecessary wasted tokens to be encouraged as well. Therefore it is not obvious why increasing the gradient magnitude in these cases is better than increasing the gradient magnitude in cases where the length difference of the current pair agrees with the direction of marginal utility. If UTIL is supposed to preserve additional generation when marginal token utility is positive, why should the gradient be diminished when the length difference of the current pair is also positive? In other words, can you give an explanation for why it makes sense to scale up the gradient when zlen<0z_{len} < 0 and scale down the gradient when zlen>0z_{len} > 0?

3 comments

We thank the reviewer for recognizing the value of our approach and for the constructive suggestions. We have carefully addressed the feedback to clarify our method and further strengthen its evaluation.

Q1 & W1: Gradient Mechanism of Bidirectional Length Control

  • Upon carefully checking our implementation, we identified a sign typo in the manuscript. Specifically, the implementation reverses the sign of the relative utility term obtained through the CDF-based transformation, but this sign reversal was omitted from Eq. (6) for zlenz_{\mathrm{len}}. We will add the missing negative sign to Eq. (6) and update the corresponding explanation. All experiments used the correct implemented formulation and are therefore unaffected. With this correction, the length signal strengthens learning from preferences consistent with marginal utility, without assuming that every uniformly sampled token is useful.

  • To clarify how the length signal affects optimization, we derive the gradient of the UTIL objective as ∇θL=−σ(−zqual−zlen)∇θzqual\nabla_\theta \mathcal{L} = -\sigma(-z_{\mathrm{qual}}-z_{\mathrm{len}})\nabla_\theta z_{\mathrm{qual}}. Specifically, zlenz_{\mathrm{len}} adjusts the gradient weight of each preference pair without changing its gradient direction. With the corrected sign, a negative zlenz_{\mathrm{len}} increases the gradient weight, while a positive value decreases it. We will add this gradient analysis to the appendix.

  • Intuitively, our bidirectional design uses marginal token utility to interpret the observed length difference in each preference pair. As shown in the table, the same length difference can receive opposite training adjustments depending on its context-specific utility. When the length difference agrees with the utility direction, UTIL increases the gradient weight to strengthen that preference; otherwise, it reduces the weight to weaken that preference. |Length Difference|Marginal Utility|Signs Agree?|zlenz_{\mathrm{len}}|Gradient Weight|Training Effect| |---|---|---|---|---|---| |Longer preferred (+)|Positive (+)|Yes|Negative (−)|Increased|Longer preference strengthened| ||Negative (−)|No|Positive (+)|Decreased|Longer preference weakened| |Shorter preferred (−)|Negative (−)|Yes|Negative (−)|Increased|Shorter preference strengthened| ||Positive (+)|No|Positive (+)|Decreased|Shorter preference weakened|

W2: Iterative Preference Optimization

  • We agree that extending UTIL to online RL would be a valuable direction for future work. While extending to a different optimization framework such as PPO may be beyond the scope of this paper, we further evaluate UTIL against DPO and SamPO through two-round iterative preference optimization in the same-model Llama setting, where each model generates its own candidate responses. After the first round, we generate and score new candidate responses from the updated models for the same prompts. For UTIL, we recompute marginal token utility using accumulated responses from both rounds. Utility remains consistent for 76.64% of prompts, showing substantial stability as new training observations are incorporated. We then continue all three methods for a second training round using the new preference data. As shown below, UTIL continues to achieve the highest Win Rate and LC WR on both benchmarks, outperforming SamPO in Win Rate by 16.9% on Arena-Hard and 8.2% on AlpacaEval. These results support the effectiveness of utility-guided length control as training progresses. |Benchmark|Method|Win Rate ↑|LC WR ↑|Tokens| |---|---|---|---|---| |Arena-Hard|DPO|47.50|51.20|577| ||SamPO|46.20|49.90|569| ||UTIL|54.00|54.10|597| |AlpacaEval|DPO|37.02|37.86|427| ||SamPO|36.92|38.34|415| ||UTIL|39.94|40.30|435|
Reviewer_sv3XOpenReview ↗

Response to Author Rebuttal

I thank the authors for their effort in responding to my question. I have no further concerns, and I will maintain my original score.

Thank you for your positive feedback and for taking the time to consider our responses. We are glad that our additional explanations have addressed your concerns. We will carefully incorporate all the changes into the revised paper to strengthen it.

Reviewer_vqCTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

To address the universal treatment of response length as a source of bias in preference optimization, this paper proposes UTIL. It estimates marginal token utility, the gain associated with additional response tokens within each context, from the relationship between token length and quality score among candidate responses, and uses it to adaptively modulate the role of response length in direct preference optimization. Experiments across language models and alignment benchmarks show that UTIL outperforms existing length bias mitigation methods while avoiding unnecessary verbosity.

Critical strengths
  1. The paper offers a novel perspective supported by training data analysis. Instead of treating length as a universal bias, it argues that the value of additional generation depends on the context. Figure 1 shows that within each training setting, marginal token utility is positive for a considerable fraction of contexts and negative for another, indicating that a universal treatment of response length has limitations.
  2. The method is simple and computationally efficient. It only uses the quality scores of candidate responses already available in the training data to estimate marginal token utility for each context, which then determines a token utility guided length signal added to the preference signal in DPO. It requires no additional model training and introduces no additional hyperparameter.
Critical weaknesses
  1. The core mechanism is not sufficiently validated. Marginal token utility is estimated from the relationship between token length and quality score within each context, but a positive value may reflect length bias in the reward model or differences in response sources rather than useful additional generation, and the paper provides no human verification or independent scoring. A single slope fitted from four or five candidate responses cannot capture nonlinear relationships between quality and length, the paper gives no error analysis for this approximation, and the leave one out analysis only shows stability rather than accuracy. The token utility guided length signal is computed entirely from data, so it only rescales the gradient weight of each preference pair, and the paper does not justify how this yields the claimed bidirectional length control. Sampling an equal number of tokens from both responses also does not guarantee that quality is fully separated from length. Finally, the paper lacks analysis of model outputs grouped by marginal token utility and human evaluation, and two qualitative examples are insufficient to support general claims.
  2. The experimental evidence is insufficient to support the conclusions. The gains are modest, and against the strongest baseline SamPO only one of four win rate improvements has a confidence interval excluding zero. Results come from a single random seed, and the gap between SamPO and the ablation without the utility guided length signal suggests nonnegligible variance across runs. UTIL uses the quality scores of all candidate responses while the baselines use only preference pairs, yet the paper lacks a baseline with the same scoring information and control experiments that shuffle or flip marginal token utility, so the source of the gains remains unclear. In addition, the same model and cross model settings differ in several factors at once, and closely related adaptive length methods are not compared, which limits the generality of the conclusions.
Questions for authors
  1. How can the authors distinguish whether positive marginal token utility reflects genuinely useful additional generation, length bias in the reward model, or stronger source models producing longer responses? Could the authors recompute marginal token utility with human annotation or an independent scorer and report its consistency with the original estimate?
  2. Marginal token utility is defined as a function of token length, but it is estimated as a single slope from four or five candidate responses. In cases where moderate additional generation helps but excessive length hurts, how large is the approximation error, and under what conditions does the linear approximation hold?
  3. The token utility guided length signal is independent of model parameters and only rescales the gradient weight of each preference pair, so pairs whose length difference agrees with the utility direction may receive smaller weights. Could the authors explain how this achieves bidirectional length control, favoring shorter responses for negative utility and longer responses for positive utility, and add an experiment that flips the sign of marginal token utility?
  4. Could the authors group evaluation prompts by marginal token utility estimated from sampled candidate responses, and report how UTIL changes response length and quality relative to the baselines in each group? Are there results from human blind evaluation or tasks with verifiable answers?
  5. The ablation without the utility guided length signal is nearly identical to SamPO, so why is it 2.8 points lower in Win Rate and 3.9 points lower in LC Win Rate on Arena Hard for Llama? Since UTIL uses the quality scores of all candidate responses, could the authors add a baseline with the same scoring information, such as DPO weighted by score differences, and a control with shuffled marginal token utility to rule out gains from additional information or general reweighting?
  6. Only one of four Win Rate improvements over SamPO is statistically significant. Could the authors report the mean and variance over multiple random seeds? Since the same model and cross model settings differ in the model, the scoring source, the number of candidates, and the pair construction, could the authors control these factors and test each one separately? Could the authors also compare with closely related adaptive length methods, such as the response conditioned method of Cai et al. 2026?
5 comments

We thank the reviewer for the detailed and constructive feedback. We have carefully addressed the questions through additional analyses and experiments to strengthen the paper's method justification and empirical evidence.

Q1 & W1: Reliability of Marginal Token Utility Across Scoring Sources:

  • Our method uses the existing quality annotations in commonly used preference training datasets, aiming to remain efficient and generalizable without additional annotations. Specifically, the same-model training data is scored by the ArmoRM reward model, while the cross-model data uses GPT-4 annotations. The consistent performance gains across these two settings suggest that our method is effective with both reward model and LLM scoring. To further verify the reliability in the reward model-annotated same-model setting, we additionally use GPT-4 as the independent scorer to evaluate the same Llama candidate responses and recompute their marginal token utility. The resulting estimates show 80.0% consistency in utility direction and a Pearson correlation of 0.762 with the original ArmoRM estimates. These results further support the robustness of marginal token utility estimation across different scoring sources, beyond potential reward model bias.

Q2 & W1: Linear Approximation Accuracy:

  • We agree that a single slope estimated from a few candidate responses cannot fully capture the length–quality relationship. Our goal is not to recover the full function, but to extract additional information about how response quality changes with length, which conventional preference training does not explicitly model. By estimating average marginal token utility from existing training candidate responses, we introduce a context-specific signal for adaptive length control without additional data collection or scoring costs. To quantify the approximation error, we compute the mean absolute error (MAE) between the fitted and observed candidate rewards, yielding an MAE of 0.394. For comparison and interpretation, a length-independent estimate using the mean candidate reward for each prompt yields an MAE of 0.552. The linear fit thus reduces base MAE by 28.7%, indicating that it captures variation in response quality associated with length that the traditional length-independent estimate misses. UTIL's performance improvements over the compared methods further support the value of incorporating this information into preference training.

Q3 & W1: Mechanism of Bidirectional Length Control:

  • Upon carefully checking our implementation, we identified a sign typo in the manuscript. Specifically, the implementation reverses the sign of the relative utility term obtained through the CDF-based transformation, but this sign reversal was omitted from Eq. (6) zlenz_{\mathrm{len}} in the original manuscript. We will add the missing negative sign to Eq. (6) and update the corresponding explanation. All experiments used the correct implemented formulation and are therefore unaffected.

  • Our bidirectional design uses marginal token utility to interpret the observed length difference in each preference pair. As shown in the table, the same length difference can receive opposite training adjustments depending on its context-specific utility. With the corrected sign in Eq. (6), when the length difference agrees with the utility direction, UTIL increases the gradient weight to strengthen that preference; otherwise, it reduces the weight to weaken that preference. |Length Difference|Marginal Utility|Signs Agree?|zlenz_{\mathrm{len}}|Gradient Weight|Training Effect| |---|---|---|---|---|---| |Longer preferred (+)|Positive (+)|Yes|Negative (−)|Increased|Longer preference strengthened| ||Negative (−)|No|Positive (+)|Decreased|Longer preference weakened| |Shorter preferred (−)|Negative (−)|Yes|Negative (−)|Increased|Shorter preference strengthened| ||Positive (+)|No|Positive (+)|Decreased|Shorter preference weakened|

  • To further verify the effect of utility design and direction, we conduct a sign-flip ablation and the shuffled utility ablation suggested in Q5 on Llama, following the same training settings as our existing ablation study. As shown below, both reversing the utility direction and shuffling utility across training samples consistently degrade performance on both benchmarks. These results demonstrate the importance of both utility direction and its context-specific association with each training sample for effective adaptive length control.

  • Table: Additional Ablation Studies. |Benchmark|Variant|Win Rate ↑|LC WR ↑|Tokens| |---|---|---|---|---| |Arena-Hard|Sign-flipped|40.10|47.00|505| ||Shuffled Utility|46.80|50.20|564| ||UTIL|52.60|54.90|583| |AlpacaEval|Sign-flipped|34.17|36.98|403| ||Shuffled Utility|35.10|35.28|434| ||UTIL|39.41|39.54|435|

Q4 & W1: Utility Grouped Evaluation and Human Blind Evaluation:

  • To analyze how UTIL adapts generation according to marginal token utility, we group evaluation prompts into positive and negative utility groups for Llama and compare UTIL with DPO and SamPO. As shown below, UTIL achieves the highest Win Rate across all four groups while maintaining comparable response lengths. Some baselines achieve slightly higher LC WR on Arena-Hard, potentially related to their shorter response lengths. Overall, these results support UTIL's effectiveness across different utility contexts.

  • Table: Utility Grouped Evaluation. |Benchmark|Utility Group|Method|Win Rate ↑|LC WR ↑|Tokens| |---|---|---|---|---|---| |Arena-Hard|Positive|DPO|62.12|64.70|561| |||SamPO|57.82|62.19|530| |||UTIL|63.76|62.82|571| ||Negative|DPO|33.20|32.12|574| |||SamPO|39.31|39.44|562| |||UTIL|40.39|38.24|595| |AlpacaEval|Positive|DPO|38.43|37.49|467| |||SamPO|39.57|40.12|450| |||UTIL|41.46|40.93|470| ||Negative|DPO|31.53|33.79|334| |||SamPO|27.87|30.17|327| |||UTIL|33.78|35.41|339|

  • Our paper focuses on the trade-off between response length and overall quality in LLM preference alignment, including aspects such as clarity and usefulness. We thus evaluate on the representative alignment benchmarks Arena-Hard and AlpacaEval, following their standard LLM evaluation protocols. Tasks with verifiable answers primarily assess final-answer correctness, which does not directly evaluate this trade-off or these aspects of response quality. Therefore, we further conduct a blind human evaluation on 100 AlpacaEval prompts following common practice in prior studies [1, 2]. Three independent English-speaking annotators with graduate-level education compare UTIL against DPO and SamPO on Llama, with randomized response order and ties allowed. All human preference results are averaged across the three annotators. As shown below, human judgments favor UTIL, with clearly more wins than losses against both DPO and SamPO. These results provide additional human evidence supporting UTIL's improvements in response quality.

  • Table: Human Blind Evaluation. |Comparison|Win / Tie / Loss (%)|Win Rate ↑| |---|---|---| |UTIL vs. DPO|54.3 / 5.3 / 40.3|57.0| |UTIL vs. SamPO|60.7 / 6.0 / 33.3|63.7|

  • We will also expand our qualitative analysis in appendix with model outputs from evaluation benchmarks across positive and negative utility contexts, complementing the existing qualitative examples of candidate responses from training datasets.

[1] Mu et al., Learning to Compress Prompts with Gist Tokens, NeurIPS 2023.

[2] Sorensen et al., Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability, ICLR 2026.

Q5 & W2: Training Consistency and Utility Control Experiments

  • Our zqualz_{\mathrm{qual}} design adopts SamPO's core idea of equal-token sampling, without fully following its implementation details. Some differences exist in how tokens are sampled from the two responses and between the policy and reference models, which may affect training trajectories and contribute to the observed performance gap. To further assess training consistency, we evaluate the main methods across multiple random seeds, with results reported in Q6. For example, on Llama Arena-Hard, UTIL achieves a mean Win Rate of 53.00 ± 0.87 versus 48.67 ± 0.58 for SamPO, and LC WR of 54.87 ± 0.75 versus 53.00 ± 1.77. UTIL achieves higher mean Win Rate and LC WR across both models and benchmarks, supporting that its improvements are consistent across training runs rather than dependent on a single random seed.

  • To distinguish whether UTIL's improvements come from additional scoring information or context-specific utility, we introduce two additional experiments. For presentation clarity, we report added baselines Weighted DPO together with Rc-DPO (requested in Q6) in the table below, and the Shuffled Utility ablation in the Additional Ablation Studies table in Q3. Weighted DPO uses the same candidate quality scores as UTIL to weight preference pairs by score differences, yet UTIL consistently achieves higher Win Rate and LC WR across both training settings. The Shuffled Utility ablation randomly shuffles marginal token utility across training samples while preserving its distribution, testing whether the improvements come from context-specific utility rather than general reweighting. The ablation results in Q3 show that shuffled utility consistently reduces performance on both benchmarks, with Win Rate dropping from 52.60 to 46.80 on Arena-Hard and from 39.41 to 35.10 on AlpacaEval.

  • Table: Additional Baseline Comparisons. |Model|Benchmark|Method|Win Rate ↑|LC WR ↑|Tokens| |---|---|---|---|---|---| |Llama|Arena-Hard|Weighted DPO|47.00|49.20|571| |||Rc-DPO|45.00|47.80|555| |||UTIL|52.60|54.90|583| ||AlpacaEval|Weighted DPO|35.89|35.29|439| |||Rc-DPO|33.86|35.02|426| |||UTIL|39.41|39.54|435| |Qwen|Arena-Hard|Weighted DPO|69.60|66.90|662| |||Rc-DPO|73.90|69.70|706| |||UTIL|75.80|73.60|675| ||AlpacaEval|Weighted DPO|36.09|40.60|383| |||Rc-DPO|39.36|41.12|413| |||UTIL|39.37|42.73|397|

Q6 & W2: Statistical Robustness, Training Settings, and Additional Baselines:

  • We agree that the statistical significance deserves further examination. Two of the three nonsignificant comparisons come from the Qwen cross-model setting, where responses are generated by multiple source models. In this setting, performance differences among the compared methods are generally smaller, so the limited significance may partly reflect the heterogeneous training data setting. In contrast, UTIL achieves larger improvements over baselines in the Llama same-model setting.

  • To further assess training stability, we evaluate SamPO and UTIL across three random seeds for both Llama and Qwen on Arena-Hard and AlpacaEval. As shown below, UTIL consistently achieves higher mean Win Rate and LC WR than SamPO across both models and benchmarks. These results provide further evidence that UTIL's improvements are consistent across different training runs.

  • Table: Multi-Seed Training Robustness (Mean ± SD). |Model|Benchmark|Method|Win Rate ↑|LC WR ↑| |---|---|---|---|---| |Llama|Arena-Hard|SamPO|48.67 ± 0.58|53.00 ± 1.77| |||UTIL|53.00 ± 0.87|54.87 ± 0.75| ||AlpacaEval|SamPO|36.43 ± 1.33|37.12 ± 1.61| |||UTIL|39.45 ± 0.47|39.77 ± 0.46| |Qwen|Arena-Hard|SamPO|72.17 ± 0.99|69.20 ± 2.00| |||UTIL|75.43 ± 0.40|71.90 ± 1.75| ||AlpacaEval|SamPO|38.39 ± 0.72|41.01 ± 0.81| |||UTIL|39.68 ± 0.51|42.83 ± 0.51|

  • Regarding the differences between same-model and cross-model settings, we evaluate the methods using two representative preference training datasets with their respective configurations. These two datasets naturally differ in response sources, scoring methods, numbers of candidates, and pair construction. All compared methods within each setting use the same training data and configurations, ensuring fair comparisons.

  • We also extend our baseline comparison to include Rc-DPO (Cai et al., 2026), which addresses length bias through response-conditioned preference modeling. For presentation clarity, we include its results in the Additional Baseline Comparisons table in Q5, covering both models and benchmarks. UTIL consistently achieves better performance than Rc-DPO. For example, on Llama Arena-Hard, UTIL improves Win Rate by 16.9% and LC WR by 14.9% over Rc-DPO, further supporting the effectiveness of our adaptive length control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.