acceptodds
Under review as a conference paper at ICLR 2027

Preference Optimization for Distribution-Level Objectives

Abstract

Post-training typically targets individual-level objectives, which evaluate a policy through the expected value of a per-response reward. Broader goals like diversity, calibration, and fairness, however, demand distribution-level objectives that depend on the collective outputs and probabilities of a policy. In this work, we consider jointly optimizing individual- and distribution-level objectives, and make the following contributions. (i) We formulate the joint optimization problem and derive a per-response linearization reward for the distribution-level objective. This motivates the development of an iterative surrogate optimization method with dynamic linearization, which we show provably converges to a solution of the joint objective. (ii) We adapt existing post-training methods to the surrogate problems, introducing a general framework for incorporating distribution-level objectives into standard post-training pipelines. Experiments across three testbeds demonstrate improved distribution-level outcomes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Preference optimization, language model post-training, distribution-level objectives, state rewards, conditional gradient methods, multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fJFfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers post-training with a KL-regularized individual-level objective together with a distribution-level objective of the form Ω(π)=Ω2(ExΩ1(C(x,π)))Ω(π)=Ω₂(E_x Ω₁(C(x,π))), where C(x,π) is the expected feature vector of responses at prompt xx. Linearizing ΩΩ at an anchor policy yields a per-response reward. The authors use this reward within a generalized conditional-gradient scheme that periodically refreshes the anchor, and Prop. 1 establishes convergence up to an error floor due to reward estimation and inexact inner solves. They then describe three ways to inject the reward into pairwise preference losses (DPO/IPO): through the comparisons, the pair weights, or the reference policy. The main new construction, reference tilting, shifts the reference-relative margin by ΔrΩ/βΔr_Ω/β. Experiments train LoRA adapters on Llama-3.1-8B-Instruct for embedding diversity (WritingPrompts), prompt-level calibration (TriviaQA), and equalized odds (Bias in Bios), and report paired changes relative to the base policy.

Critical strengths

S1. A general formulation and a clean result for reference tilting (Sec. 2–4, App. E.3). The state-based objective (Eq. 2) covers diversity, binned calibration, and group fairness through a single derivation of r_Ω (Eq. 3). The identity LD,w,θ̃_ref=LD,w,θ_ref shows that, for any link and weighting, tilting maps the minimizers of the baseline loss to their Gibbs tilt, and the injection requires no additional evaluations of the background reward. This is what makes the framework applicable beyond the three tasks studied.

S2. Consistent improvements over ordinary preference training on three different objectives, with careful reporting (Tables 5, 7, 8, 11–13). Relative to ordinary IPO, reference-tilting IPO increases Distinct_k by 3.87 and reduces ECE by about 15 pp; it also reduces noise-corrected EOdds relative to the base policy. The paired design, per-arm coverage, gate-success counts, and explicit statements about non-significant contrasts make the results easy to verify.

Critical weaknesses

W1. No demonstrated advantage over existing or simpler alternatives (Tables 5, 7). Given the moderate novelty (an ODPO-style margin offset; DivPO/DDPO), the case for the method rests on its empirical trade-offs, which the current results do not establish.

  • Diversity. DivPO-style task-gated IPO (LitBench +2.63, GRM +2.71, Distinct_k +4.27, Vendi +0.75) falls between reference tilting at λ=4 and λ=8, and the percentile variant (GRM +0.73, Distinct_k +7.38, Vendi +1.40) is higher than tilting at λ=16 (+0.44 / +6.15 / +1.05) on all three metrics. These are single operating points, but they give no indication that tilting improves on the quality–diversity frontier of existing methods.
  • Calibration. The only controls are ordinary IPO and DPO, which receive no calibration signal. The natural individual-level alternative is a per-response proper scoring rule, e.g., a Brier reward as in RLCR (Damani et al., 2026, cited in A.3), which fits the same combined-score interface. Since E[(c−a)²] = (E[c−a])² + Var(c−a) at each prompt, the Brier score upper-bounds the paper's prompt-level objective. Without this baseline, the calibration results do not show that a distribution-level formulation is needed.

A Brier baseline, together with a joint plot of the DivPO gate sweep and the λ sweep, would address this concern.

W2. The iterative scheme, presented as contribution (i), is neither covered by the theory as implemented nor supported by the experiments (Prop. 1; Sec. 5.2; App. F.1). Prop. 1 assumes a convex policy class closed under mixtures and ε-accurate inner solves. The implementation instead uses LoRA, three or four rounds, and validation selection over round components and mixtures; all ten reported nested calibration policies are single-round checkpoints rather than outputs of Algorithm 1. At matched gradient steps, the differences between nested and static training are not significant (ECE −1.70 pp, AUROC +0.41 pp, EOdds −0.0052). The gain in selection reliability (10/10 vs. 4/10 runs passing the gates in calibration; 30/30 vs. 26/30 eligible runs in fairness) is confounded by the larger candidate set available to nested selection, and in fairness the round-level fallback was triggered in 11 of 90 rounds. As it stands, the evidence mainly supports the static (T=1) variant.

Questions for authors

Q1. How does a per-response Brier (or log-score) reward perform on the calibration task when used through the same combined-score or weighting interface? (W1)

Q2. When the DivPO gate-parameter sweep and the λ sweep for reference tilting are plotted on the same axes (GRM/LitBench vs. Distinct_k/Vendi), is there any λ at which tilting dominates? (W1)

Q3. The IPO–DPO comparison appears confounded. On diversity, ordinary DPO passes the selection gate in 4/30 runs and tilted DPO in 0/30 (Table 11), so the DPO results come from fallback checkpoints, and DPO is evaluated at only one λ. On fairness, tilted DPO actually reaches a better operating point than tilted IPO (EOdds −0.0215 at +6.13 pp accuracy vs. −0.0228 at +3.32 pp; Table 8). Does tilted DPO respond more strongly at a larger λ, or at a λ chosen to match IPO's effective margin shift?

Q4. Do the static-versus-nested conclusions still hold when both schedules select from the same number of candidate policies? (W2)

Q5. Could you report the absolute base-policy values of ECE, EOdds, and Distinct_k? In addition, the fairness estimates exclude runs without an eligible checkpoint (tilting 26/30, combined-score 8/10, CDA with tilting 7/10). How do the conclusions change if these runs are included, for example at base-policy values?

8 comments

We thank the reviewer for their thoughtful comments and evaluation of our work. We begin with clarification regarding the described weaknesses (W) and then answer the questions (Q).


W1 (No demonstrated advantage over existing or simpler alternatives). We thank the reviewer for encouraging us to clarify the empirical advantages and methodological contribution. Our results demonstrate useful trade-offs, alongside a substantial advantage in background task-reward information requirements.

  1. Reference tilting achieves stronger performance than DDPO. All three reported reference-tilting strengths exceed the DDPO-style weighting baseline on mean LitBench, Distinct$_k$, and Vendi (Appendix F.2.1, Table 5).

  2. Our margin offset is derived from the distribution-level objective and has an optimizer-correspondence guarantee. ODPO uses an offset reflecting preference strength. Our offset instead comes from the objective’s linearization reward at the current anchor. Proposition 4 establishes that, when the baseline loss identifies the background optimizer, the tilted loss identifies the surrogate optimizer in the full-support tabular class (Appendix E.3.1). Thus, the offset implements a faithful objective-derived optimization target.

  3. The contribution extends beyond a particular offset or diversity heuristic. Conceptually, whereas DivPO and DDPO are designed specifically for diversity, our framework applies to any distribution-level objective admitting the stated feature representation, including discrete mode coverage, equalized odds, and teacher-target matching beyond the experimental settings (Appendix C). Algorithmically, it unifies three injection sites, strengthens comparison-site training, and introduces reference tilting. Theoretically, it provides outer convergence bounds and characterizes optimizer correspondence, injection-site distinctions, and interactions with preference losses, guiding empirical choices (Appendices D and E).

The DivPO comparison and Brier-baseline concern are addressed separately in W1(a) and W1(b) below.


W1(a) (Existing diversity results do not establish a better quality–diversity frontier than DivPO) 1. Reference tilting has a substantial information advantage. DivPO requires the background task reward for each response to reconstruct preference pairs, whereas reference tilting directly reuses existing comparisons without requiring background task rewards. This preserves a central benefit of direct preference optimization: learning from comparisons without separately training a background task-reward model. Moreover, even with the background task reward available, preparing the range-gated DivPO comparisons took 42m 08s, or 115.3% of training time (see shared response #1), while reference tilting doesn't need this step.

  1. The reported DivPO variants are strengthened by our framework. They use IPO in place of DPO, our SBERT-based linearization reward in place of probability-based novelty, and percentile gating alongside the original reward-range gate. Shared response #1 presents the four-dataset exploration of loss, signal, gate, and threshold choices. Strengthening comparison-site training with our objective-derived signal is itself a contribution.

  2. Reference tilting achieves competitive trade-offs, and composition improves the range-gated counterpart. All three reported reference-tilting strengths improve both mean LitBench and Distinct$_k$ over our DPO/probability/range baseline (Appendix F.2.1, Table 5). The strengthened percentile variant achieves larger diversity gains, while reference tilting provides competitive operating points under weaker information requirements. Adding reference tilting to the strengthened range-gated variant further improves both mean LitBench and Distinct$_k$.

W1(b). Calibration lacks a proper-scoring-rule baseline, so the need for a distribution-level objective is unproven.

We thank the reviewer for identifying this missing baseline. We have added pointwise Brier controls at both training interfaces. For combined-score IPO, we replace our score a+gx(a−c)a+g_x(a-c) with correctness plus negative Brier score, a−(c−a)2a-(c-a)^2, using the same frozen response bank and pair-construction procedure. For reference tilting, we keep the preference pairs fixed and replace only gx(a−c)g_x(a-c) with −(c−a)2-(c-a)^2, using λ=4\lambda=4 for both.

The comparison uses the same nine dataset pools, s0–s8, with one initialization per pool, across all methods. All use learning rate 5×10−65\times10^{-6} and 1,200 training updates. Within each interface, the checkpoint grid, selection rule, and held-out evaluation are matched. We retain all selected policies, including fallbacks.

The table reports mean paired changes from each pool’s base model, multiplied by 100; parentheses give standard errors across the nine paired differences.

MethodRunsAccuracy ↑ECE ↓AUROC ↑
Ordinary IPO9/9+0.50 (0.14)−0.27 (0.22)+0.12 (0.13)
Combined-score linearization9/9+1.01 (0.26)−9.61 (2.31)+2.21 (0.70)
Combined-score Brier9/9+0.29 (0.15)+0.10 (0.17)−0.17 (0.19)
Reference-tilting linearization9/9−0.57 (0.37)−16.31 (1.87)+2.52 (0.72)
Reference-tilting Brier9/9−0.33 (0.41)−14.04 (2.07)+1.17 (1.23)

Brier substantially reduces ECE through reference tilting, demonstrating that it is a meaningful baseline. Linearization achieves larger mean ECE reductions and AUROC gains at both interfaces. At the combined-score interface specifically requested by the reviewer, Brier yields little calibration improvement, whereas linearization improves both ECE and AUROC. Under reference tilting, linearization’s larger calibration gains accompany a slightly larger mean accuracy decrease.

Our signal responds to the prompt’s aggregate calibration error under the anchor policy. The state is

gx=Ey∼πanchor(⋅∣x)[c(x,y)−a(x,y)].g_x=\mathbb{E}_{y\sim\pi_{\mathrm{anchor}}(\cdot\mid x)}[c(x,y)-a(x,y)].

Positive values indicate overconfidence and negative values indicate underconfidence. For a correct response (a=1a=1), the combined scores become:

  • Brier: 2c−c22c-c^2, which increases with confidence on [0,1][0,1].
  • Our linearization: 1+gx(1−c)1+g_x(1-c), which decreases with confidence when gx>0g_x > 0.

Thus, for correct responses to the same overconfident prompt, the scores rank confidence in opposite directions: Brier favors confidence approaching one, while our linearization favors lower confidence to correct the prompt’s aggregate overconfidence. This materially changes pair construction: in pool s0, mean chosen-response confidence is 81.1% under linearization versus 97.3% under Brier, and only 7.4% of ordered pairs overlap. This illustrates a mechanism consistent with the observed calibration differences.

The reviewer’s decomposition is correct: Brier penalizes both the squared prompt-level mean residual and residual variance. Consequently, minimizing this upper bound need not optimize the prompt-level objective or produce the same response rankings. The matched experiments demonstrate the practical value of our objective-derived signal, particularly at the combined-score interface requested by the reviewer.

W2(a) (The implemented nested procedure is not covered by Proposition 1 as reported) We thank the reviewer for requesting a clearer distinction between the theoretical policy updates and the practical training and selection procedure.

  1. The outer anchors belong to a convex policy class even though individual LoRA policies do not. Let ΠLoRA\Pi_{\mathrm{LoRA}} denote the class of individual LoRA policies. We train inner policies in ΠLoRA\Pi_{\mathrm{LoRA}} and maintain outer anchors in conv(ΠLoRA)\mathrm{conv}(\Pi_{\mathrm{LoRA}}) through explicit mixture sampling. For example, sampling from 0.7π1+0.3π20.7\pi_1+0.3\pi_2 means selecting one policy with the corresponding probability and generating the entire response from it. This implements the distributional mixture required by the damped update, without requiring convexity of the LoRA parameterization (Appendix F.1, Table 4). The prescribed inner accuracy over the theoretical comparison class remains an oracle assumption, with approximation error explicitly accounted for in Proposition 1.

  2. The appropriate number of rounds depends on the problem and the available inner-training budget. Our experiments primarily study a fixed optimizer-update budget, where additional rounds leave fewer optimization steps per surrogate. They examine this allocation trade-off; pursuing the best achievable nested performance through extensive schedule tuning is beyond the paper’s scope. The worst-case bound does not prescribe a minimum number of rounds. As formalized in shared response #2, refresh frequency depends on curvature-weighted state movement, inner-optimization accuracy, and reward-estimation error. Figures 4–6 provide complementary evidence of objective progress, late stabilization, and diminishing preference-signal adjustments within the tested budgets.

  3. Checkpoint selection preserves the state-refresh mechanism motivating our theory. Algorithm 1 requires an approximately optimized inner policy, not necessarily the final optimizer iterate. We use a validation-selected checkpoint as the inner solver’s output and incorporate it into the prescribed outer mixture; inner accuracy remains the oracle assumption discussed above.

    Proposition 1 provides a worst-case guarantee for πT\pi_T, while the central algorithmic principle is to re-estimate the state and adjust the surrogate as the policy distribution changes. Our nested implementation follows this principle. For practical reporting, we additionally select among round components and mixtures, allowing an earlier component to be retained when it performs better on validation. This selection rule is distinct from the theoretical output and does not change the preceding state-refresh updates.

    Figure 5D separately reports actual policy mixtures and selected components, showing objective progress along the mixture sequence as well as the performance of the selected policies. This provides evidence for the iterative mechanism beyond the final checkpoint selected for reporting.


W2(b) (Few rounds and nonsignificant nested–static gains mainly support the static method) 1. Both static and nested training are contributions of our framework. Static training is the T=1T=1 instance and provides a practical option that avoids repeated state estimation. We acknowledge that the reported calibration and fairness comparisons do not establish significant improvements from nesting. The clearest observed benefit is a higher combined diagnostic JnoKL=VD+ΩJ_{\mathrm{noKL}}=V_{\mathcal D}+\Omega in the diversity experiments, where nested training favors quality at some cost to diversity (Figures 4–5). This is consistent with the rationale of optimizing the combined objective. These results characterize the tested settings, rather than establish a universal ranking of static and nested schedules.

  1. Our experiments study allocation under a fixed optimizer-update budget. More outer rounds leave fewer steps to optimize each inner surrogate, creating a trade-off between inner accuracy and refreshed linearization. Pursuing the best achievable nested performance through extensive schedule tuning is beyond the scope of this paper. Limited gains under this allocation therefore do not establish the limits of nested training.

  2. The appropriate number of rounds depends on the problem instance. As formalized in shared response #2, the relevant quantities are curvature-weighted state movement, inner-optimization accuracy, and reward-estimation error. When training changes the objective-relevant state little, a static surrogate can remain accurate; larger changes make refreshing more valuable. Our framework thus provides a principled basis for choosing the schedule according to the objective, testbed, and available budget.

Reviewer_vT4COpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Standard post-training methods such as DPO and IPO optimize individual-level objectives, the expected value of a per-response reward. This paper extends them to distribution-level objectives, such as diversity, calibration, and fairness, which depend on the distribution of responses generated by the policy. To this end, the paper linearizes the distribution-level objective at the current policy, which yields a per-response linearization reward, and incorporates this reward into existing preference-based training. Across three testbeds, the paper shows that this approach improves distribution-level outcomes.

Critical strengths

The paper addresses distribution-level objectives, which standard post-training objectives do not capture. Existing methods for such goals, such as DivPO and DDPO, are designed for specific objectives, whereas this paper provides a general framework. Since these objectives cannot be represented by any fixed per-response reward, a framework of this kind is necessary. The optimization approach itself, linearizing a nonlinear utility and refreshing the linearization, has precedents in RL with nonlinear utilities, including convex MDPs and multi-objective RL; its adaptation to LLM post-training, including the injection into preference losses, is interesting. The theoretical analysis and the experiments provide adequate support for the approach.

Critical weaknesses

Convergence of the iterative scheme in practice: Although the experiments cover a broad range of settings, it is unclear whether each experiment is run long enough to support the claims. Because the linearization reward depends on the anchor policy, it has to be re-estimated as the policy changes, and Proposition 1 guarantees convergence only as the number of rounds T grows. However, most headline results use the static schedule (T = 1), to which the guarantee does not apply, and the nested schedule uses only T = 3 or 4, at which the bound has barely decreased. It therefore remains unclear whether the joint objective is sufficiently optimized.

Computational cost: Although the nested schedule matches the number of gradient steps, each round requires regenerating and rescoring candidates. According to Tables 14–16, this increases the cost by roughly 2.5 to 4 times relative to the static schedule, while the improvements of the nested schedule over the static one are not statistically significant. A plot of the performance gain against the additional computational cost, for example across the number of rounds T, would help assess this trade-off. I consider this more informative than the current sweep over λ\lambda.

Algorithmic novelty: The optimization scheme, linearizing a nonlinear (e.g., concave) utility at the current policy and refreshing the linearization, follows existing conditional-gradient approaches for convex MDPs [1], and the comparison- and weight-site injections generalize DivPO and DDPO, as the authors note. Apart from reference tilting, the main contribution therefore lies in unifying these techniques under a general framework rather than in a new algorithmic idea, so the algorithmic novelty is somewhat limited.

[1] Zahavy, Tom, et al. "Reward is enough for convex mdps." Advances in Neural Information Processing Systems 34 (2021): 25746-25759.

Questions for authors

Q1. Does the scheme converge in practice? I understand that even a single linearization step (the static schedule) moves the policy in an ascent direction of the joint objective and can therefore improve distribution-level outcomes. However, it remains unclear whether the objective is sufficiently maximized. Did the anchor policy and its linearization reward oscillate across rounds as the policy changed?

Q2. I consider the increase in computational cost unavoidable. Could the authors show how the distribution-level objective, together with the joint objective (or the diagnostic V_D + Ω), improves as T increases at a fixed training budget?

Q3. I value the generality of the framework. However, on diversity, an objective-specific baseline performs better on the distribution-level metrics: in Table 5, the percentile-band DivPO variant achieves larger gains in Distinct_k (+7.38) and Vendi (+1.40) than reference tilting even at its strongest setting (λ = 16; +6.15 and +1.05). Could the authors clarify why the general framework underperforms an objective-specific method on a similar objective?

6 comments

We thank the reviewer for their thoughtful evaluation and appreciation of our paper. We begin with clarifications regarding the described weaknesses (W) and answer the questions (Q).


W1. (Convergence of the iterative scheme in practice) We thank the reviewer for requesting a clearer account of optimization progress. We provide evidence from both the original experiments and a new controlled round-count study, with the corresponding trajectories presented in Figures 4–6 of the updated manuscript.

  1. The original three-round experiment shows substantial progress followed by diminishing gains. Figure 4 in the updated manuscript plots JnoKL=VD+ΩJ_{\mathrm{noKL}}=V_{\mathcal D}+\Omega, excluding the KL penalty, alongside its quality and distribution-level components. Across ten pools, the nested round-endpoint means are approximately −1.65-1.65, 0.360.36, and 0.560.56: the final-round gain is about one-tenth of the preceding gain, and the trajectory levels off within the final round. At λ=8\lambda=8, nested training favors higher quality at some cost to diversity, yielding a higher combined diagnostic than static training. This reflects the intended balance between quality and diversity; diversity alone does not measure progress on the combined target.

  2. The controlled round-count study shows similar objective progress and diminishing reward adjustments. Figure 5 compares T=1,2,4T=1,2,4 and a four-round frozen-state control across three seeds using the same fixed offline pairs, 960 total optimizer updates, and matched checkpoint-evaluation counts. At λ=16\lambda=16, four-round nested training again achieves a higher late mean JnoKLJ_{\mathrm{noKL}} through better quality despite a lower Ω\Omega than static training. Its final two checkpoint means are −1.32-1.32 and −1.41-1.41, also exceeding the corresponding frozen-state control means. Meanwhile, Figure 6B shows that successive RMS changes in fixed-pair linearization-reward gaps decrease from 0.5860.586 to 0.5060.506 to 0.3880.388, with the decrease occurring in all three seeds. Figures 5A–C and 6B thus provide empirical evidence of convergence within the tested four-round budget.

  3. The worst-case bound does not prescribe a minimum number of rounds. Proposition 1 applies for every T≥1T\geq1 under its assumptions; its O(1/T)O(1/T) upper bound does not imply that the actual optimality gap must remain large at small TT. The appropriate inner budget and refresh schedule depend on curvature-weighted state movement, inner-optimization accuracy, and reward-estimation precision. Small state movement can keep a frozen surrogate accurate, while larger movement motivates refreshing its reward. Our additional error decomposition formalizes this trade-off and provides a principled basis for scheduling; see shared response #2, “Static versus nested training: balancing inner accuracy and state refresh,” for details.


W2 (Computational cost) We acknowledge that the reported nested implementation costs approximately 2.5–4 times as much as static training. This includes state re-estimation and regenerating on-policy comparisons, although the latter is not required. Our new controlled study reuses fixed offline preference pairs. Training takes approximately 72 minutes across schedules, with additional state-refresh costs of 51 minutes for T=2T=2 and 162 minutes for T=4T=4, mainly from response generation. Figure 5 compares their performance.

  1. The value of refreshing depends on the problem and objective. Small curvature-weighted state movement leaves the frozen surrogate accurate; larger movement makes refreshing more useful. Shared response #2 formalizes this trade-off and provides a principled basis for choosing refresh frequency.

  2. Both static and nested training are contributions of our framework. Static training offers an economical T=1T=1 option. In both the original and controlled studies (Figures 4–5), nested training achieves higher quality at some cost to diversity, improving the combined diagnostic VD+ΩV_{\mathcal D}+\Omega. The schedules therefore offer different performance–cost trade-offs.

  3. Our experiments study allocation under a fixed update budget, not the best achievable nested performance. With 960 total updates, T=1,2,4T=1,2,4 leave 960, 480, and 240 updates per inner round. More frequent refreshes therefore reduce the optimization effort per surrogate. Pursuing the best achievable nested performance through extensive schedule tuning is beyond the paper’s scope.

W3 (Algorithmic novelty). We thank the reviewer for recognizing reference tilting as an algorithmic contribution. We would like to clarify the contributions beyond the established conditional-gradient machinery.

  1. A general, objective-first formulation. To our knowledge, we are the first to provide a general framework for distribution-level preference optimization in LLM post-training, connecting these objectives to per-response linearization rewards through conditional gradients. The approach shares the same spirit as the convex-MDP literature, as acknowledged in our introduction. Its applicability extends beyond our testbeds: any distribution-level objective that admits our feature-based representation and satisfies the stated assumptions can be optimized through the framework.

  2. Reference tilting is our main algorithmic contribution. Unlike task-gated comparison-site methods such as DivPO, reference tilting directly reuses existing preference datasets without requiring background task rewards, avoiding additional task-reward scoring and comparison reconstruction. This provides broader applicability and saves the computation needed to obtain these scores and rebuild comparisons. Preparing new combined-score comparisons (excluding state estimation) took 136.3%, 50.5–51.2%, and 8.5% of the corresponding training times on diversity, calibration, and fairness, respectively (see shared response #1 for details).

  3. The framework also strengthens comparison-site training and supports composition. Our objective-derived linearization signal strengthens DivPO variants, and reference tilting can be composed with these methods. The resulting empirical trade-offs are detailed in our response to shared response #1.

  4. Additional theoretical contributions guide algorithm design. Beyond the outer convergence analysis, Lemmas 4–8 characterize conditional optimizer correspondence, differences between injection sites, and their interaction with preference losses (Appendices E.3–E.4). These results provide concrete guidance on how to implement distribution-level objectives through preference optimization.


Q1 (Does the scheme converge in practice, and do anchors or linearization rewards oscillate?) We thank the reviewer for suggesting these diagnostics. We have added objective trajectories (Figures 4–5) and state/reward diagnostics (Figure 6) to the updated manuscript.

  1. The combined objective shows substantial improvement and late stabilization. Figure 4 plots the original three-round experiment, which shows a much smaller final-round gain. Figure 5A–C plots validation JnoKL=VD+ΩJ_{\mathrm{noKL}}=V_D+\Omega and its components against optimizer updates in the controlled study. Four-round nested training reaches similar mean objective values at the final two checkpoints (−1.32 and −1.41), exceeding the corresponding means for static training and the frozen-state control. The decomposition shows that its stronger combined objective primarily comes from higher task quality.

  2. The preference signal adapts through progressively smaller adjustments. Figure 6 in the updated manuscript tracks the same prompts and fixed offline preference pairs across all four rounds. Solid curves show changes from the preceding round; dashed curves show distance from initialization.

    • Figure 6, Panel A: state movement. Mean per-prompt state distances are 0.197 → 0.232 → 0.192, while distance from initialization increases 0.197 → 0.291 → 0.330. Thus, the states continue adapting, although successive state movements do not shrink monotonically.
    • Figure 6, Panel B: linearization-reward movement. We measure the RMS change in raw reward gaps on the fixed pairs, weighting prompts equally. Successive changes decrease 0.586 → 0.506 → 0.388, while distance from initialization increases 0.586 → 0.767 → 0.820. The decrease occurs in all three seeds and is measured before target clipping.

    Individual reward adjustments can reverse direction, consistent with the intended feedback mechanism: as a response becomes less novel under the updated distribution, its novelty reward should decrease. The relevant observation is that these successive adjustments become smaller. All shaded bands show ±1 standard error across the three seeds.

Together, these results provide evidence of empirical convergence within both the original three-round budget and the new controlled four-round budget, with diminishing preference-signal changes providing additional support in the latter study. Rigorous certification of global optimality under practical LLM training remains an important direction for future work.

Q2 (How do the distribution-level and joint objectives change with at a fixed training budget?) Our new controlled study compares T=1,2,4T=1,2,4 with 960 total optimizer updates, using the same fixed offline preference pairs. Figure 5A–C in the updated manuscript reports JnoKL=VD+ΩJ_{\mathrm{noKL}}=V_D+\Omega and its quality and distribution-level components throughout training. Four-round nested training achieves a higher late mean combined objective, primarily through improved task quality despite some sacrifice in diversity. Its objective shows late stabilization, while successive fixed-pair reward-gap changes diminish across rounds (Figure 6B; see W1 and Q1 for details). Training time remains approximately equal across schedules; the additional nested cost comes mainly from generating responses for state refreshes (W2).


Q3. (Why does percentile-band DivPO beat reference tilting on Distinct_k and Vendi?) We thank the reviewer for requesting a clearer comparison with DivPO. We want to clarify that the reported percentile-band DivPO is a strengthened variant that already incorporates our objective-derived linearization signal. Its larger diversity gains therefore also demonstrate the value of our framework at the comparison site.

  1. Reference tilting requires substantially less background task-reward information. DivPO requires the background task reward for each response to reconstruct comparisons. Reference tilting reuses existing preference pairs without requiring background task rewards, preserving applicability when only comparisons are available. Performance should therefore be considered alongside these different information requirements. In our timing study, preparing comparisons for the range-gated DivPO variant took 42m 08s, or 115.3% of its training time (see shared response #1).

  2. Our comparison uses substantially strengthened DivPO variants. The original DivPO uses DPO, model-probability novelty, and a reward-range gate. We replaced DPO with IPO, replaced probability-based novelty with our SBERT-based linearization signal, and introduced percentile gating alongside the original range gate. These changes substantially improve the LitBench–diversity trade-off; see shared response #1 for the exploration and final choices. All three reported reference-tilting strengths outperform our implementation of the original DivPO recipe on both mean LitBench and Distinct$_k$ (Appendix F.2.1, Table 5). The stronger percentile variant highlighted by the reviewer therefore incorporates both our objective-derived signal and our new gating rule, which we regards as part of our contribution.

  3. The results show competitive trade-offs and benefits from composition. All three reported reference-tilting strengths exceed the DPO/probability/range baseline on mean LitBench and Distinct$_k$ (Appendix F.2.1, Table 5). The strengthened percentile variant achieves larger diversity gains, while reference tilting offers competitive trade-offs under weaker information requirements. Adding reference tilting to the strengthened range-gated variant further improves both mean LitBench and Distinct$_k$.

Our contribution combines an objective-derived training signal, reference tilting that reuses existing preferences, and composition across training interfaces; it does not depend on outperforming every task-gated operating point.

Reviewer_9K9eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This submission studies distribution-level objectives, such as diversity, calibration, and group fairness. It linearizes the chosen objective at the current policy to obtain a temporary per-response reward, approximately solves the resulting KL-regularized problem, and repeats the process. Equation 7 incorporates this reward by shifting the DPO or IPO margin. The framework is evaluated on story generation, confidence calibration, and occupation prediction.

Critical strengths
  • The connection between distribution-level objectives and per-response rewards is clearly formulated. Example 3 gives an intuitive illustration: responses less similar to the current policy’s responses receive higher rewards.

  • Equation 7 reuses existing comparisons without requiring additional background-reward evaluations. Proposition 1 provides convergence guarantees under explicit assumptions, accounting for reward-estimation and inner-optimization errors.

  • The experiments cover three testbeds and evaluate both distribution-level outcomes and background performance. The IPO results show promising improvements, particularly in diversity and calibration ECE.

Critical weaknesses
  • The theoretical contribution appears incremental. Once the temporary reward is fixed, Equation 5 becomes a standard KL-regularized reward problem, and the outer scheme builds on established linearization and conditional-gradient methods. The additional contribution beyond adapting these ideas to preference training should be explained more clearly.

  • Proposition 1 requires sufficiently accurate inner optimization over a convex policy class. The paper does not establish how the implemented training procedure satisfies these requirements, limiting the guarantee’s applicability to the experiments.

  • The experiments omit a direct reward-based optimization baseline, although all three testbeds have computable background rewards. Such a comparison would help establish the benefit of the preference-based approach. The gains also depend on the preference loss: DPO shows substantially smaller improvements than IPO in diversity and calibration ECE.

Questions for authors
  • How does the proposed method compare with directly optimizing Equation 5 using the rewards available in these experiments?

  • What evidence connects the implemented training procedure to the inner-optimization accuracy required by Proposition 1?

7 comments

We thank the reviewer for their thoughtful evaluation and suggestions. We begin with clarifications regarding the described weaknesses (W) and then answer the questions (Q).


W1 (The theoretical contribution seems incremental beyond standard KL-regularized optimization and established linearization/conditional-gradient methods.) We thank the reviewer for encouraging us to clarify our contribution. We agree that KL-regularized reward optimization and conditional-gradient methods are established tools, and we could have made clearer how our work builds on these foundations.

  1. Eq. (5) is our intended reduction. That “Equation 5 becomes a standard KL-regularized reward problem” is precisely our goal rather than a limitation: our contribution is to reduce distribution-level optimization to subproblems that existing post-training pipelines can solve, allowing us to draw on established techniques to improve the inner training, with Eq. (5) specifying the objective of each subproblem.

  2. The conceptual contribution is a general, objective-first approach. The step preceding “Once the temporary reward is fixed” is central to our contribution: we derive that reward directly from the specified distribution-level objective. This provides a common approach across objectives such as diversity, calibration, and fairness, connecting their mathematical specification to a per-response training signal and an iterative optimization procedure (§§2–4).

  3. Our theoretical and algorithmic contributions include:

    • Formalization and optimization. We formalize a class of distribution-level objectives, derive their linearization rewards, and establish convergence bounds accounting for reward-estimation and inner-optimization errors under the stated assumptions (Propositions 1–2, Appendices D.6 and D.2; Lemmas 1–3, Appendices D.3–D.5).
    • A framework with three injection sites. We generalize comparison- and weight-site approaches such as DivPO and DDPO, strengthen comparison-site training with our objective-derived signal, and introduce reference tilting as a new training mechanism. Reference tilting reuses existing preference pairs without additional background-reward evaluations and empirically improves both background task performance and distribution-level metrics over the base policy (§4.2, Eq. (7); §5.2).
    • Additional theoretical analysis that guides empirical choices. (1) Conditional optimizer correspondence: reference tilting maps baseline minimizers to their Gibbs tilts in the full-support tabular class; when the baseline loss identifies the background optimizer, the tilted loss identifies the surrogate optimizer (Lemma 4 and Proposition 4, Appendix E.3.1). (2) Injection-site distinctions: positive weighting preserves minimizers when pairwise minima are simultaneously attainable; order-only comparison rules cannot uniformly recover the target across score scales, whereas appropriately calibrated labels can under the stated conditions (Lemma 5, Appendix E.3.2; Lemmas 6–7, Appendix E.3.3). (3) Preference-loss choices: the same reference tilt can reverse a pair's scalar gradient coefficient under IPO, but preserves its sign under DPO, whose response to a fixed reward shift vanishes at extreme margins. This provides theoretical guidance for choosing IPO when incorporating distribution-level signals (Lemma 8, Appendix E.4).

W2 (Proposition 1's convex policy class and accurate inner solves are not established for the implementation) We thank the reviewer for raising the relationship between the theoretical assumptions and our implementation. We clarify each assumption separately below.

  1. Convex policy class.

    • Convexity ensures that the damped policy update in Algorithm 1 remains feasible and supports the policy-space stationarity characterization (Appendices D.4 and D.6).
    • Let ΠLoRA\Pi_{\mathrm{LoRA}} denote the class of individual LoRA policies. Our implementation trains inner policies in ΠLoRA\Pi_{\mathrm{LoRA}} and maintains outer anchors in conv(ΠLoRA)\mathrm{conv}(\Pi_{\mathrm{LoRA}}) through explicit mixture sampling. For example, to sample from 0.7π1+0.3π20.7\pi_1+0.3\pi_2, we select π1\pi_1 with probability 0.70.7 or π2\pi_2 with probability 0.30.3, then generate the entire response from that policy. These samples estimate the anchor's state and linearization reward (Appendix F.1, Table 4).
    • Although ΠLoRA\Pi_{\mathrm{LoRA}} is generally nonconvex, conv(ΠLoRA)\mathrm{conv}(\Pi_{\mathrm{LoRA}}) is convex by construction, ensuring the convexity needed for the outer damped update.
  2. Approximate inner optimization.

    • This assumption ensures sufficient improvement in each surrogate problem to establish outer convergence (Proposition 1, Appendix D.6).
    • We do not establish a prescribed global optimality tolerance for our finite-budget LoRA training, so this remains an oracle assumption. Certifying such a tolerance for practical LLM post-training is an unresolved challenge more broadly. Our bound explicitly accounts for inner approximation error through ϵ\epsilon, alongside reward-estimation error through 2ζ2\zeta.
    • Each inner objective is deliberately formulated as the standard KL-regularized reward problem in Eq. (5). Consequently, practical techniques that improve approximation to its optimal policy can be applied directly within our framework, including optimizer and learning-rate tuning, additional training steps, restarts, and improved sampling. Although these techniques do not certify global optimality, our reduction allows improvements in standard post-training optimization to carry over to the inner solves.
  3. Compactness of the policy class.

    • Compactness supports the existence of maximizers of the objective and surrogate under the stated upper-semicontinuity conditions (Appendix D.4).
    • The full stochastic policy class is compact when both the prompt and response spaces are finite. An unrestricted LoRA policy class is not automatically compact. A sufficient condition is to restrict the LoRA parameters to a compact set and require continuity of the policy map under dρd_\rho; the resulting policy class and its closed convex hull are then compact.
  4. Concavity of the objective.

    • Concavity is needed for the global objective-suboptimality guarantee in Proposition 1; the surrogate-gap guarantee does not require it.
    • Our experimental objectives satisfy this assumption. Appendix C also includes concave objectives for discrete mode coverage, per-prompt reliability, equalized odds, and teacher-target matching (Examples 4–7). Mode coverage uses a sum of logarithms of affine policy statistics; the other three use negative quadratic functions of affine policy statistics.
    • Thus, concavity holds for the specific objective constructions considered here. For nonconcave objectives satisfying the remaining assumptions, the surrogate-gap guarantee still applies.

W3(a) (No direct reward-based optimization baseline despite available background rewards) We thank the reviewer for this suggestion. Our contribution is to enable distribution-level optimization through existing preference-training pipelines, including settings where background task rewards are unavailable.

A central motivation for DPO/IPO as an alternative to RLHF is to learn from pairwise comparisons without fitting a separate reward model. Reference tilting preserves this advantage: it reuses the original comparison dataset without requiring background task-reward values or a learned background reward model (§4.2, Eq. (7)). This also distinguishes it from the reward-dependent comparison-site injections in our experiments. In sum, our method works when only pairwise comparisons are available, and our testbeds provide reward access for convenience, not as a requirement.

If time permits, we will add a direct reward-optimization baseline to assess the performance trade-off when background rewards are available.

W4 (Improvements depend on the preference loss; DPO gains are much smaller than IPO in diversity and calibration ECE) We thank the reviewer for highlighting the dependence on the preference loss, but we politely disagree it as a weakness. Explaining this dependence is one of our technical contributions: our analysis shows why the same reference tilt affects IPO and DPO differently, providing guidance on which loss to use. We will make this connection clearer.

Our theory distinguishes how IPO and DPO incorporate the same reward shift. Although DPO is a widely used preference-optimization method, reference tilting preserves the sign of each pair’s scalar gradient coefficient under DPO, whereas it can reverse that sign under IPO. Moreover, for a fixed reward gap, the change in DPO’s coefficient vanishes at extreme positive and negative margins, while IPO’s coefficient change remains proportional to the reward gap, independently of the margin (Lemma 8, Appendix E.4).

The empirical observations are consistent with these theoretical distinctions. The smaller DPO gains in diversity and calibration ECE are consistent with the limitations identified by our analysis (§5.2, Tables 5 and 7). This analysis provides a principled reason to consider IPO when incorporating distribution-level rewards. Our framework therefore supports theory-guided choices of the preference loss.


Q1 (How does the method compare with directly optimizing Eq. (5) using the available rewards?)
Our headline reference tilting methods enables distribution-level optimization directly from pairwise comparisons without requiring background task rewards, as clarified in our response to W3. If time permits, we will add a direct reward-optimization baseline to assess the performance trade-off when those rewards are available.


Q2 (What evidence connects implemented training to Proposition 1's required inner accuracy?) There are two main evidences:

  1. Connection to standard post-training optimization. As explained in W2, point 2, each inner problem is deliberately formulated as standard KL-regularized reward optimization (Eq. (5)). This allows improvements in established post-training methods to carry over directly to our inner solves.
  2. Empirical evidence from a single inner solve. Our static schedule performs one frozen-anchor inner training stage. The resulting policies improve both quality and diversity relative to the base policy (§5.2, Table 5), demonstrating that practical inner training produces useful improvements without requiring multiple outer rounds.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared Response #1: Clarification of the DivPO comparisons

We thank the reviewers for highlighting the comparison with DivPO. We want to clarify that the two task-gated DivPO variants reported in our main comparison are strengthened variants that already incorporate our objective-derived linearization reward as their novelty signal. We clarify their construction and the resulting empirical trade-offs below.

  1. Reference tilting reuses existing preference pairs without requiring background task rewards. DivPO requires the background task reward for each response to construct new comparisons. Even with those rewards available, preparing the strengthened DivPO comparisons took 42m 08s, or 115.3% of its training time.

    Each row reports one run. Preparation includes generation, scoring, and pair construction. We exclude separately timed state estimation because reference tilting also requires it. For calibration and fairness, the ranges reflect a shared timer for scoring and state estimation.

TestbedMethodPreparationTrainingPreparation / training
DiversityCombined-score IPO42m 08s30m 54s136.3%
DiversityDivPO (SBERT, range gate)42m 08s36m 32s115.3%
CalibrationCombined-score IPO18m 40s–18m 56s36m 58s50.5–51.2%
FairnessCombined-score IPO4m 49s–4m 50s57m 03s8.5%
  1. Four implementation choices define the DivPO family considered here: preference loss, novelty signal, gating rule, and threshold. The original DivPO paper uses DPO, model-probability novelty, and a task-reward range gate (i.e., with ρ=0.3\rho=0.3, this gate admits responses within 30%30\% of the observed reward range below the maximum instead of selecting the top 30%30\% of responses by rank.)

    Before our full experiment, we investigated loss and signal choices, alternative gates, and a partial threshold sweep. The table uses the same four datasets with one initialization per dataset. Each entry is a mean change relative to the stored base-policy evaluations; validation fallbacks are retained.

    LossNovelty signalGateρ\rhoΔ\Delta LitBenchΔ\Delta GRMΔ\Delta Distinct$_k$Δ\Delta Vendi
    DPOProbabilityRange0.31.5104.8361.893.235
    DPOSBERTRange0.31.5535.1731.532.446
    IPOProbabilityRange0.32.3303.5933.583.369
    IPOSBERTRange0.12.2692.2712.886.654
    IPOSBERTRange0.32.6873.4244.706.567
    IPOSBERTRange0.51.802−1.3148.6631.941
    IPOSBERTPercentile0.12.1941.4451.879−.029
    IPOSBERTPercentile0.32.933.8898.4991.378

    These comparisons support the four design choices used in our final evaluation:

    • Loss: IPO. Replacing DPO with IPO substantially improves LitBench and both diversity metrics in the matched configurations, while DPO retains higher GRM scores.
    • Signal: our SBERT-based linearization reward. With IPO and range gating at ρ=0.3\rho=0.3, this signal improves LitBench and both diversity metrics over probability-based novelty.
    • Gate: retain both range and percentile variants. Percentile gating provides larger diversity gains and a complementary quality–diversity trade-off.
    • Threshold: ρ=0.3\rho=0.3. This provides a strong balance among the tested settings. Increasing the range threshold to 0.50.5 further improves diversity but substantially reduces quality.

    Thus, the two final variants are IPO + our linearization signal + range/0.3 and IPO + our linearization signal + percentile/0.3. Strengthening comparison-site injection with our objective-derived signal is itself a contribution of our framework.

  2. Reference tilting achieves competitive trade-offs against these strengthened variants, and composition can further improve performance. Despite its weaker task-reward requirements, all three reported reference-tilting strengths improve both mean LitBench and Distinct$_k$ over our DPO/probability/range baseline in the final ten-dataset, three-initialization evaluation (Appendix F.2.1, Table 5). Against the strengthened variants, reference tilting provides competitive quality–diversity trade-offs, while the percentile variant achieves larger diversity gains. Adding reference tilting to the strengthened SBERT/range/0.3 variant further improves both mean LitBench and Distinct$_k$ over its task-gated counterpart (the same table).

Our contribution is therefore not contingent on outperforming every task-gated operating point: it provides an objective-derived training signal (which already improve the DivPO baseline) and a reference-tilting mechanism that reuses existing preferences, achieves competitive empirical trade-offs.

AuthorsOpenReview ↗

Shared Response #2: Static versus nested training: balancing inner accuracy and state refresh

We thank the reviewers for raising questions about the benefits, convergence, and cost of nested training. Both static and nested schedules are contributions of our framework. Static training is the T=1T=1 instance of the proposed scheme and provides a practical option that avoids repeated state estimation.

First, our experiments primarily study a fixed optimizer-update budget, where more outer rounds leave fewer inner steps per round. Pursuing the best achievable nested performance through extensive schedule tuning is beyond the scope of this paper. Additional rounds therefore need not improve performance at every tested configuration: they update the linearization but reduce the optimization effort available per inner problem.

Moreover, the appropriate refresh frequency depends on how much training changes the objective-relevant state and how sensitive Ω\Omega is to that change. Substantial state movement can make the frozen reward inaccurate; re-estimating the state realigns the linearization reward with the current response distribution. When curvature-weighted state movement is small, retaining the existing estimate introduces little linearization error. Our framework therefore suggests a principled basis for choosing inner-training budgets, when to refresh the state, and when further rounds are needed.

To formalize this intuition, we derive the following consequence of our linearization and reward-estimation analysis. We defer its proof, together with the extension to damped outer updates, to the updated manuscript.

Corollary (Inner accuracy–anchor staleness trade-off). Under the regularity and convex-class assumptions of Proposition 1, including concavity of Ω\Omega, suppose JJ attains its maximum at a finite-valued π⋆\pi^\star. Fix an anchor π†\pi^\dagger. Let q∈Πq\in\Pi have finite relative entropy and be an ε\varepsilon-optimal solution of the estimated frozen surrogate J^π†\widehat J_{\pi^\dagger}, with uniform error

∣J^π†(π)−Jπ†(π)∣≤ζ\left|\widehat J_{\pi^\dagger}(\pi)-J_{\pi^\dagger}(\pi)\right|\le\zeta

over feasible policies with finite relative entropy. Then

\le \varepsilon+2\zeta+ \frac{L_\Omega}{2}\mathbb E_x \left\\|C(x,q)-C(x,\pi^\dagger)\right\\|^2.$$ The three terms represent **inner-optimization error**, **reward-estimation error**, and a **bound on anchor-staleness error**, respectively. Here, $L_\Omega$ is the Lipschitz constant of the gradient of $\Omega$. For continuous diversity, $L_\Omega=\lambda$, and the last term equals the frozen linearization error exactly. **Implication for scheduling.** Allocate a desired tolerance as $\tau=\tau_{\mathrm{opt}}+\tau_{\mathrm{est}}+\tau_{\mathrm{lin}}$, with nonnegative budgets. If $$\varepsilon\le\tau_{\mathrm{opt}}, \qquad 2\zeta\le\tau_{\mathrm{est}}, \qquad \frac{L_\Omega}{2}\mathbb E_x \left\\|C(x,q)-C(x,\pi^\dagger)\right\\|^2 \le\tau_{\mathrm{lin}},$$ then $J(\pi^\star)-J(q)\le\tau$. This motivates continuing inner optimization while the linearization allowance is satisfied, refreshing when it is exceeded, and stopping when all three error contributions meet the target. With sampled states, the displacement estimate must account for sampling uncertainty; the inner-accuracy requirement remains that of Proposition 1. Thus, **static training can be sufficient when an accurately optimized surrogate induces little curvature-weighted state movement; more frequent refreshes become relevant when the objective-derived signal changes substantially.** The scheduling choice depends on the testbed, $\Omega$, and inner-training quality, rather than a universal number of rounds. **Additional empirical evidence.** Figures 4–6 are included in the updated manuscript. Figure 4 shows substantial progress followed by diminishing gains in the original three-round experiment. Figure 5 compares $T=1,2,4$ and a four-round frozen-state control at 960 optimizer updates: four-round nested training achieves the highest late checkpoint mean $J_{\mathrm{noKL}}$, primarily through higher task quality. Figure 6 separates state movement (Panel A) from fixed-pair reward-gap movement (Panel B); successive reward-gap changes decrease across all three seeds. These diagnostics support practical stabilization and clarify the quality–diversity trade-off underlying schedule selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.