- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers post-training with a KL-regularized individual-level objective together with a distribution-level objective of the form , where C(x,π) is the expected feature vector of responses at prompt . Linearizing at an anchor policy yields a per-response reward. The authors use this reward within a generalized conditional-gradient scheme that periodically refreshes the anchor, and Prop. 1 establishes convergence up to an error floor due to reward estimation and inexact inner solves. They then describe three ways to inject the reward into pairwise preference losses (DPO/IPO): through the comparisons, the pair weights, or the reference policy. The main new construction, reference tilting, shifts the reference-relative margin by . Experiments train LoRA adapters on Llama-3.1-8B-Instruct for embedding diversity (WritingPrompts), prompt-level calibration (TriviaQA), and equalized odds (Bias in Bios), and report paired changes relative to the base policy.
Critical strengths
S1. A general formulation and a clean result for reference tilting (Sec. 2–4, App. E.3). The state-based objective (Eq. 2) covers diversity, binned calibration, and group fairness through a single derivation of r_Ω (Eq. 3). The identity LD,w,θ̃_ref=LD,w,θ_ref shows that, for any link and weighting, tilting maps the minimizers of the baseline loss to their Gibbs tilt, and the injection requires no additional evaluations of the background reward. This is what makes the framework applicable beyond the three tasks studied.
S2. Consistent improvements over ordinary preference training on three different objectives, with careful reporting (Tables 5, 7, 8, 11–13). Relative to ordinary IPO, reference-tilting IPO increases Distinct_k by 3.87 and reduces ECE by about 15 pp; it also reduces noise-corrected EOdds relative to the base policy. The paired design, per-arm coverage, gate-success counts, and explicit statements about non-significant contrasts make the results easy to verify.
Critical weaknesses
W1. No demonstrated advantage over existing or simpler alternatives (Tables 5, 7). Given the moderate novelty (an ODPO-style margin offset; DivPO/DDPO), the case for the method rests on its empirical trade-offs, which the current results do not establish.
- Diversity. DivPO-style task-gated IPO (LitBench +2.63, GRM +2.71, Distinct_k +4.27, Vendi +0.75) falls between reference tilting at λ=4 and λ=8, and the percentile variant (GRM +0.73, Distinct_k +7.38, Vendi +1.40) is higher than tilting at λ=16 (+0.44 / +6.15 / +1.05) on all three metrics. These are single operating points, but they give no indication that tilting improves on the quality–diversity frontier of existing methods.
- Calibration. The only controls are ordinary IPO and DPO, which receive no calibration signal. The natural individual-level alternative is a per-response proper scoring rule, e.g., a Brier reward as in RLCR (Damani et al., 2026, cited in A.3), which fits the same combined-score interface. Since E[(c−a)²] = (E[c−a])² + Var(c−a) at each prompt, the Brier score upper-bounds the paper's prompt-level objective. Without this baseline, the calibration results do not show that a distribution-level formulation is needed.
A Brier baseline, together with a joint plot of the DivPO gate sweep and the λ sweep, would address this concern.
W2. The iterative scheme, presented as contribution (i), is neither covered by the theory as implemented nor supported by the experiments (Prop. 1; Sec. 5.2; App. F.1). Prop. 1 assumes a convex policy class closed under mixtures and ε-accurate inner solves. The implementation instead uses LoRA, three or four rounds, and validation selection over round components and mixtures; all ten reported nested calibration policies are single-round checkpoints rather than outputs of Algorithm 1. At matched gradient steps, the differences between nested and static training are not significant (ECE −1.70 pp, AUROC +0.41 pp, EOdds −0.0052). The gain in selection reliability (10/10 vs. 4/10 runs passing the gates in calibration; 30/30 vs. 26/30 eligible runs in fairness) is confounded by the larger candidate set available to nested selection, and in fairness the round-level fallback was triggered in 11 of 90 rounds. As it stands, the evidence mainly supports the static (T=1) variant.
Questions for authors
Q1. How does a per-response Brier (or log-score) reward perform on the calibration task when used through the same combined-score or weighting interface? (W1)
Q2. When the DivPO gate-parameter sweep and the λ sweep for reference tilting are plotted on the same axes (GRM/LitBench vs. Distinct_k/Vendi), is there any λ at which tilting dominates? (W1)
Q3. The IPO–DPO comparison appears confounded. On diversity, ordinary DPO passes the selection gate in 4/30 runs and tilted DPO in 0/30 (Table 11), so the DPO results come from fallback checkpoints, and DPO is evaluated at only one λ. On fairness, tilted DPO actually reaches a better operating point than tilted IPO (EOdds −0.0215 at +6.13 pp accuracy vs. −0.0228 at +3.32 pp; Table 8). Does tilted DPO respond more strongly at a larger λ, or at a λ chosen to match IPO's effective margin shift?
Q4. Do the static-versus-nested conclusions still hold when both schedules select from the same number of candidate policies? (W2)
Q5. Could you report the absolute base-policy values of ECE, EOdds, and Distinct_k? In addition, the fairness estimates exclude runs without an eligible checkpoint (tilting 26/30, combined-score 8/10, CDA with tilting 7/10). How do the conclusions change if these runs are included, for example at base-policy values?
8 comments
W1(b). Calibration lacks a proper-scoring-rule baseline, so the need for a distribution-level objective is unproven.
We thank the reviewer for identifying this missing baseline. We have added pointwise Brier controls at both training interfaces. For combined-score IPO, we replace our score with correctness plus negative Brier score, , using the same frozen response bank and pair-construction procedure. For reference tilting, we keep the preference pairs fixed and replace only with , using for both.
The comparison uses the same nine dataset pools, s0–s8, with one initialization per pool, across all methods. All use learning rate and 1,200 training updates. Within each interface, the checkpoint grid, selection rule, and held-out evaluation are matched. We retain all selected policies, including fallbacks.
The table reports mean paired changes from each pool’s base model, multiplied by 100; parentheses give standard errors across the nine paired differences.
| Method | Runs | Accuracy ↑ | ECE ↓ | AUROC ↑ |
|---|---|---|---|---|
| Ordinary IPO | 9/9 | +0.50 (0.14) | −0.27 (0.22) | +0.12 (0.13) |
| Combined-score linearization | 9/9 | +1.01 (0.26) | −9.61 (2.31) | +2.21 (0.70) |
| Combined-score Brier | 9/9 | +0.29 (0.15) | +0.10 (0.17) | −0.17 (0.19) |
| Reference-tilting linearization | 9/9 | −0.57 (0.37) | −16.31 (1.87) | +2.52 (0.72) |
| Reference-tilting Brier | 9/9 | −0.33 (0.41) | −14.04 (2.07) | +1.17 (1.23) |
Brier substantially reduces ECE through reference tilting, demonstrating that it is a meaningful baseline. Linearization achieves larger mean ECE reductions and AUROC gains at both interfaces. At the combined-score interface specifically requested by the reviewer, Brier yields little calibration improvement, whereas linearization improves both ECE and AUROC. Under reference tilting, linearization’s larger calibration gains accompany a slightly larger mean accuracy decrease.
Our signal responds to the prompt’s aggregate calibration error under the anchor policy. The state is
Positive values indicate overconfidence and negative values indicate underconfidence. For a correct response (), the combined scores become:
- Brier: , which increases with confidence on .
- Our linearization: , which decreases with confidence when .
Thus, for correct responses to the same overconfident prompt, the scores rank confidence in opposite directions: Brier favors confidence approaching one, while our linearization favors lower confidence to correct the prompt’s aggregate overconfidence. This materially changes pair construction: in pool s0, mean chosen-response confidence is 81.1% under linearization versus 97.3% under Brier, and only 7.4% of ordered pairs overlap. This illustrates a mechanism consistent with the observed calibration differences.
The reviewer’s decomposition is correct: Brier penalizes both the squared prompt-level mean residual and residual variance. Consequently, minimizing this upper bound need not optimize the prompt-level objective or produce the same response rankings. The matched experiments demonstrate the practical value of our objective-derived signal, particularly at the combined-score interface requested by the reviewer.
W2(a) (The implemented nested procedure is not covered by Proposition 1 as reported) We thank the reviewer for requesting a clearer distinction between the theoretical policy updates and the practical training and selection procedure.
-
The outer anchors belong to a convex policy class even though individual LoRA policies do not. Let denote the class of individual LoRA policies. We train inner policies in and maintain outer anchors in through explicit mixture sampling. For example, sampling from means selecting one policy with the corresponding probability and generating the entire response from it. This implements the distributional mixture required by the damped update, without requiring convexity of the LoRA parameterization (Appendix F.1, Table 4). The prescribed inner accuracy over the theoretical comparison class remains an oracle assumption, with approximation error explicitly accounted for in Proposition 1.
-
The appropriate number of rounds depends on the problem and the available inner-training budget. Our experiments primarily study a fixed optimizer-update budget, where additional rounds leave fewer optimization steps per surrogate. They examine this allocation trade-off; pursuing the best achievable nested performance through extensive schedule tuning is beyond the paper’s scope. The worst-case bound does not prescribe a minimum number of rounds. As formalized in shared response #2, refresh frequency depends on curvature-weighted state movement, inner-optimization accuracy, and reward-estimation error. Figures 4–6 provide complementary evidence of objective progress, late stabilization, and diminishing preference-signal adjustments within the tested budgets.
-
Checkpoint selection preserves the state-refresh mechanism motivating our theory. Algorithm 1 requires an approximately optimized inner policy, not necessarily the final optimizer iterate. We use a validation-selected checkpoint as the inner solver’s output and incorporate it into the prescribed outer mixture; inner accuracy remains the oracle assumption discussed above.
Proposition 1 provides a worst-case guarantee for , while the central algorithmic principle is to re-estimate the state and adjust the surrogate as the policy distribution changes. Our nested implementation follows this principle. For practical reporting, we additionally select among round components and mixtures, allowing an earlier component to be retained when it performs better on validation. This selection rule is distinct from the theoretical output and does not change the preceding state-refresh updates.
Figure 5D separately reports actual policy mixtures and selected components, showing objective progress along the mixture sequence as well as the performance of the selected policies. This provides evidence for the iterative mechanism beyond the final checkpoint selected for reporting.
W2(b) (Few rounds and nonsignificant nested–static gains mainly support the static method) 1. Both static and nested training are contributions of our framework. Static training is the instance and provides a practical option that avoids repeated state estimation. We acknowledge that the reported calibration and fairness comparisons do not establish significant improvements from nesting. The clearest observed benefit is a higher combined diagnostic in the diversity experiments, where nested training favors quality at some cost to diversity (Figures 4–5). This is consistent with the rationale of optimizing the combined objective. These results characterize the tested settings, rather than establish a universal ranking of static and nested schedules.
-
Our experiments study allocation under a fixed optimizer-update budget. More outer rounds leave fewer steps to optimize each inner surrogate, creating a trade-off between inner accuracy and refreshed linearization. Pursuing the best achievable nested performance through extensive schedule tuning is beyond the scope of this paper. Limited gains under this allocation therefore do not establish the limits of nested training.
-
The appropriate number of rounds depends on the problem instance. As formalized in shared response #2, the relevant quantities are curvature-weighted state movement, inner-optimization accuracy, and reward-estimation error. When training changes the objective-relevant state little, a static surrogate can remain accurate; larger changes make refreshing more valuable. Our framework thus provides a principled basis for choosing the schedule according to the objective, testbed, and available budget.
We thank the reviewer for their thoughtful comments and evaluation of our work. We begin with clarification regarding the described weaknesses (W) and then answer the questions (Q).
W1 (No demonstrated advantage over existing or simpler alternatives). We thank the reviewer for encouraging us to clarify the empirical advantages and methodological contribution. Our results demonstrate useful trade-offs, alongside a substantial advantage in background task-reward information requirements.
Reference tilting achieves stronger performance than DDPO. All three reported reference-tilting strengths exceed the DDPO-style weighting baseline on mean LitBench, Distinct$_k$, and Vendi (Appendix F.2.1, Table 5).
Our margin offset is derived from the distribution-level objective and has an optimizer-correspondence guarantee. ODPO uses an offset reflecting preference strength. Our offset instead comes from the objective’s linearization reward at the current anchor. Proposition 4 establishes that, when the baseline loss identifies the background optimizer, the tilted loss identifies the surrogate optimizer in the full-support tabular class (Appendix E.3.1). Thus, the offset implements a faithful objective-derived optimization target.
The contribution extends beyond a particular offset or diversity heuristic. Conceptually, whereas DivPO and DDPO are designed specifically for diversity, our framework applies to any distribution-level objective admitting the stated feature representation, including discrete mode coverage, equalized odds, and teacher-target matching beyond the experimental settings (Appendix C). Algorithmically, it unifies three injection sites, strengthens comparison-site training, and introduces reference tilting. Theoretically, it provides outer convergence bounds and characterizes optimizer correspondence, injection-site distinctions, and interactions with preference losses, guiding empirical choices (Appendices D and E).
The DivPO comparison and Brier-baseline concern are addressed separately in W1(a) and W1(b) below.
W1(a) (Existing diversity results do not establish a better quality–diversity frontier than DivPO) 1. Reference tilting has a substantial information advantage. DivPO requires the background task reward for each response to reconstruct preference pairs, whereas reference tilting directly reuses existing comparisons without requiring background task rewards. This preserves a central benefit of direct preference optimization: learning from comparisons without separately training a background task-reward model. Moreover, even with the background task reward available, preparing the range-gated DivPO comparisons took 42m 08s, or 115.3% of training time (see shared response #1), while reference tilting doesn't need this step.
The reported DivPO variants are strengthened by our framework. They use IPO in place of DPO, our SBERT-based linearization reward in place of probability-based novelty, and percentile gating alongside the original reward-range gate. Shared response #1 presents the four-dataset exploration of loss, signal, gate, and threshold choices. Strengthening comparison-site training with our objective-derived signal is itself a contribution.
Reference tilting achieves competitive trade-offs, and composition improves the range-gated counterpart. All three reported reference-tilting strengths improve both mean LitBench and Distinct$_k$ over our DPO/probability/range baseline (Appendix F.2.1, Table 5). The strengthened percentile variant achieves larger diversity gains, while reference tilting provides competitive operating points under weaker information requirements. Adding reference tilting to the strengthened range-gated variant further improves both mean LitBench and Distinct$_k$.