acceptodds
Under review as a conference paper at ICLR 2027

Stop Training on Pairs You've Already Learned: Policy-Aware Pair Selection for Preference Optimization

Abstract

Reinforcement learning from human feedback (RLHF) is the standard approach to aligning language models with human preferences, and in practice it is most often performed offline with direct methods such as DPO that learn from preference pairs alone. What such methods learn depends heavily on which pairs they see. Following the Delta Learning Hypothesis, which pairs responses from a large and a small model from the same model class, a strong recent rule (MaxMin) selects the best and worst responses from a candidate pool for each prompt. We show that DPO with MaxMin data selection saturates the training signal quickly. In particular, the policy quickly separates the best from the worst responses, its implicit reward margins grow, and the objective's sigmoid flattens, resulting in a small gradient norm and weight updates. We propose ActiveDelta, an adaptive selection rule that follows the policy being trained: for each prompt, it picks the largest-quality-gap pair according to an external oracle feedback which the current policy has not yet confidently separated, keeping gaps large while avoiding saturation. ActiveDelta consistently outperforms tuned MaxMin selection on downstream benchmarks and reward-model scores on a held-out set of prompts. Crucially, ActiveDelta keeps improving where MaxMin plateaus; on the benchmark suite it matches MaxMin with one-sixth of the prompts. A practical variant that ranks candidates with a learned reward model instead of an oracle and queries the reward oracle only for the selected pair retains these gains, which makes ActiveDelta applicable when scoring every response is too costly.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
preference optimization, DPO, RLHF, alignment, data selection, active learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vJFBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper proposes ActiveDelta, a policy-aware preference-pair selection method for Direct Preference Optimization (DPO). The authors observe that selecting the highest- and lowest-reward responses for each prompt (MaxMin) can lead to early gradient saturation, as the policy quickly learns to distinguish these extreme pairs. ActiveDelta addresses this problem by selecting the pair with the largest oracle reward gap subject to an upper bound on the current policy's implicit reward margin. The selected pairs are updated dynamically during training, allowing the policy to focus on informative comparisons that have not yet been confidently separated.

The method is evaluated using a pool of 28 candidate responses per prompt, primarily with Olmo-3-7B-Instruct-SFT, and additionally with Apertus-1.5-8B-SFT. Experiments cover seven downstream benchmarks, different prompt budgets, repeated training seeds, and an active preference collection setting with a learned reward model. The results show consistent improvements over tuned MaxMin selection, with better sample efficiency and stronger alignment to the reward oracle. Additional training-dynamics analyses support the observation that ActiveDelta mitigates gradient saturation.

Critical strengths
  1. Simple, intuitive, and practically effective selection mechanism. ActiveDelta directly addresses a concrete limitation of MaxMin pair selection by combining oracle reward gaps with the current policy's implicit reward margins. The method is easy to implement and requires no modification to the DPO objective. The training-dynamics analysis provides useful evidence that the proposed selection rule can prevent the rapid decline in learning signal observed with fixed extreme pairs.

  2. Strong empirical validation against the principal baseline. The paper evaluates multiple prompt budgets, reports five-seed results for the main comparisons, and tests generalization to a second base model. The improvements over tuned MaxMin are consistent, including a 1.9-point gain in the seven-benchmark average on Olmo-3-7B. The learned-reward-model variant is also valuable, demonstrating that the method can operate with only two oracle calls per prompt while retaining most of its performance advantage.

Critical weaknesses
  1. Limited novelty and insufficient comparisons with closely related policy-aware preference optimization methods.

    The central idea of considering both preference-pair quality and the current policy's learning progress is closely related to several existing methods. In particular, Uni-DPO (Peng et al., 2026) already combines preference quality with model-dependent training dynamics to reduce emphasis on well-fitted pairs. SAGE (Wu et al., 2026) similarly uses evolving gradient utility to select informative training examples, while Curry-DPO (Pattnaik et al., 2024) explores curriculum-based selection among multiple preference pairs.

    ActiveDelta differs from these approaches by selecting the largest oracle-gap pair satisfying a policy-margin constraint rather than reweighting fixed pairs or following a predefined curriculum. This is a useful distinction, but the underlying principle is largely established. The main experiments compare against MaxMin, random selection, and an APL-style baseline, without directly evaluating these more closely related dynamic alternatives.

    Consequently, the results establish the advantage of ActiveDelta over static extreme-pair selection, but do not sufficiently demonstrate its independent advantage over existing policy-aware training strategies. This limits the significance of the proposed contribution.

  2. Evaluation-set reuse and incomplete computational comparisons weaken the empirical conclusions.

    Section 6 explicitly states that the unweighted average of seven downstream benchmarks is used both for hyperparameter selection and as the primary evaluation metric. The methods are tuned over multiple configurations, including extensive searches at the 60k-prompt budget. Selecting configurations using the same benchmarks on which final performance is reported introduces selection bias and makes the reported gains potentially optimistic. Although the five-seed comparisons demonstrate training stability for the selected configurations, they do not resolve this evaluation issue.

    Moreover, ActiveDelta requires evaluating the current policy on all candidate responses, introducing substantial additional computation. Appendix A reports approximately 61 GPU-hours for ActiveDelta versus 25 GPU-hours for MaxMin at 60k prompts. Therefore, the reported sixfold improvement in prompt efficiency should not be interpreted as an equivalent improvement in computational efficiency. A comparison under matched total computation would provide a more complete assessment of the practical benefit.

    Finally, the saturation analysis reports the gradient weight evaluated at the batch-mean implicit margin rather than the mean of individual pair gradient weights. Since the sigmoid is nonlinear, these quantities can differ substantially. Although the measured gradient norms support the qualitative saturation argument, reporting the actual distribution of per-pair gradient weights would make the mechanistic evidence more rigorous.

References

[1] Peng, S., Wang, W., Tian, Z., et al. (2026). Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs. ICLR 2026. https://arxiv.org/abs/2506.10054

[2] Wu, H., Cai, H., Zhao, J., et al. (2026). Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models. arXiv:2602.01207. https://arxiv.org/abs/2602.01207

[3] Pattnaik, P., Maheshwary, R., Ogueji, K., Yadav, V., & Madhusudhan, S. T. (2024). Curry-DPO: Enhancing Alignment Using Curriculum Learning & Ranked Preferences. arXiv:2403.07230. https://arxiv.org/abs/2403.07230

Questions for authors
  1. Evaluation protocol: Section 6 states that the seven-benchmark average is used for both hyperparameter selection and final evaluation. Was there a separate validation split or any independent evaluation of the selected configurations? How should the reported improvements be interpreted given this reuse?

  2. Relationship to existing methods: What is the principal advantage of ActiveDelta over approaches such as Uni-DPO and SAGE, which also account for preference quality and the model's evolving learning state? Are any controlled comparisons or related ablations available?

  3. Compute efficiency: Since ActiveDelta requires approximately 2.4 times the GPU-hours of MaxMin at 60k prompts, can the authors clarify whether the method retains its advantage at matched computational budgets? How much overhead could be removed by caching reference log-probabilities or using a smaller candidate pool?

  4. Gradient saturation measurement: Figures 1 and 4 report the gradient weight computed from the batch-mean margin. Can the authors provide the actual average per-pair gradient weight, or the fraction of pairs with sufficiently small gradient weights, from the existing training logs? This would help distinguish widespread saturation from a highly skewed margin distribution.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer vJFB

Thank you for the careful review and the precise questions, which we answer in order below.

W1, Q2 (Uni-DPO, SAGE, Curry-DPO). The three methods decide how much to learn from a fixed pair (Uni-DPO), whether to keep it (SAGE) or in which order to use it (Curry-DPO); ActiveDelta decides which pair to train on. We trained each with its paper's default settings on the 10k-prompt protocol of Appendix A (Uni-DPO and Curry-DPO at their papers' learning rates, SAGE at the protocol's), and added a control that drops MaxMin's pair once its margin reaches 4 instead of replacing it.

Seed 42StepsMean of 7ActiveDelta (73.4) minus
Uni-DPO (reweighting)62571.1+2.2
SAGE (filtering)44270.0+3.4
Curry-DPO (3 passes)1,87570.8+2.5
ActiveDelta without substitution17371.4+2.0

All four lie within or below MaxMin's five-seed range (70.6 to 71.8); a one-seed difference has a 95% interval of ±1.3. Reweighting, dropping or reordering saturated pairs does not lift the result above MaxMin. The gain requires a substitute pair that is unsaturated and has a large oracle gap: a random unsaturated substitute scores 70.7 against ActiveDelta's 74.4 at 60k prompts (reply to rPwb). Per-benchmark scores and training dynamics: App. M.

W2 (evaluation reuse), Q1. There was no validation split; both methods had the same 38-run budget. To measure the selection effect, we re-estimated both selected configurations on new runs and on three benchmarks specified before evaluation.

EvaluationMaxMinActiveDelta
Grid prompts, seeds 43–46 (C=4)72.874.7 (+1.9; 4/4 seeds, p = 0.006)
Second prompt draw, 5 seeds (C=3)72.774.4 (+1.7; 5/5 seeds, p = 0.003)
Held-out MMLU, ARC-C, MATH-500 (1 seed)65.265.1 (C=3), 65.3 (C=4)

MaxMin's grid run exceeds its new seeds (73.2 against 72.8), which is the size of the selection effect; the gain persists on runs and prompts not used for selection. Held out, ActiveDelta is 0.2 to 0.8 lower on MMLU and MATH-500 and 0.6 to 1.2 higher on ARC-C, each within one SE. These are capability tests, on which Table 1 also shows no difference against tuned MaxMin (C=4: GSM8K, HumanEval and BBH within 0.6). The reported improvement should therefore be read as a difference between two equally tuned methods that holds at +1.7 to +1.9 outside the selection and does not extend to the held-out capability benchmarks (App. O).

W2 (compute), Q3. Matched GPU-hours: see the global response, part 1 (+1.4 to +1.5 with the reference cached, +0.6 to +0.8 as run). Overhead reduction: caching lowers the overhead from 2.6x to 1.9x. Lazy evaluation saves about a third of the candidate cost: finding the largest-gap eligible pair still needs a median 22 of 26 usable candidates (65% of their tokens). A smaller pool saves more: at 10k prompts, keeping the 4 highest- and 4 lowest-scored candidates lowers the overhead from 2.9x to 1.5x and the score from 73.1 to 72.4 (MaxMin 71.0; 3 seeds, differences not significant, p ≥ 0.25) (App. K).

W2 (gradient weights), Q4. See the global response, part 2. The actual mean per-pair weight is 6.5e-3 for MaxMin against 0.160 for ActiveDelta, and 95.3% of MaxMin's pairs have weight below 0.01, against none of ActiveDelta's; the skew you suspected is real, and saturation is widespread all the same.

Reviewer_3kZZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why DPO training on MaxMin preference pairs can saturate early. Once the policy confidently separates the highest- and lowest-scored responses, the DPO gradient weight becomes small and those pairs provide little further training signal. The authors propose ActiveDelta, which selects the largest oracle reward-gap pair among those whose current implicit reward margin is below a threshold. Experiments compare the method with tuned MaxMin across prompt budgets, selection baselines, and two starting policies. The method improves the aggregate benchmark score and matches the 60k-prompt MaxMin result with 10k prompts. A learned-reward variant retains much of the gain while reducing oracle calls.

Critical strengths

The paper identifies a useful limitation of static preference-pair selection. The distinction between the oracle reward gap and the current policy's implicit reward margin is clear and practically relevant. A pair can have a large quality gap but offer little additional learning signal once the policy has separated it. The gradient analysis and training trajectories provide a concrete explanation for why repeatedly favouring MaxMin pairs may stop helping early in training.

The proposed rule is simple and directly addresses the observed problem. ActiveDelta keeps the preference for large oracle gaps while excluding pairs that have become saturated for the current policy. It changes which pair is selected rather than introducing a new policy-training loss. The rule also admits pairs that the policy currently ranks in the wrong direction. This makes it an appealing modification to existing DPO pipelines with multiple candidates per prompt.

The empirical evidence includes meaningful controls and a practical low-label-cost variant. The paper compares against tuned MaxMin at several prompt budgets, reports matched-hyperparameter trajectories, and uses multiple seeds for the main comparisons. At 60k prompts, the seven-benchmark mean increases from 72.7 to 74.6 on Olmo, and 10k prompts already reach 73.2. The learned-reward variant uses two oracle calls per prompt instead of 28 and outperforms DeltaUCB under a paired comparison. The second-policy experiment and the detailed hyperparameter results further support the usefulness of the selection rule.

Critical weaknesses

W1. Prompt efficiency does not yet establish an advantage at equal compute.

The paper acknowledges that comparisons match prompt counts and gradient steps. Appendix A reports 25 GPU-hours for MaxMin versus 61 for ActiveDelta on 60k prompts, with candidate scoring adding 4.8 hours. Although Table 11 provides a useful control testing a second MaxMin epoch in seven-benchmark mean moving from 73.0 to 73.3, it doesn't fully show what a tuned baseline achieves given equal compute or GPU hours.

The sixfold reduction in prompts is valuable, but distinct from training-cost efficiency. Furthermore, since the implementation repeatedly evaluates the frozen reference, caching log-probabilities would improve efficiency. Quality-versus-GPU-hour curves separating candidate scoring and policy updates would strengthen this claim.

W2. The aggregate improvement comes with capability trade-offs that vary across policies and thresholds.

On Olmo, ActiveDelta (C=4C=4) improves the mean by 1.9 points, but IFEval drops from 83.5 to 81.6. On Apertus, the mean gain is 0.9 points, while IFEval drops from 82.8 to 81.7 and AlpacaEval LC from 63.0 to 60.1 (Tables 1 and 6). Appendix B further shows that increasing CC from 3 to 6 lowers BBH from 70.5 to 64.0 while raising AlpacaEval from 57.3 to 60.8. As the paper notes, there is no set rule for choosing the optimal threshold, showing that a higher aggregate score does not guarantee a better policy for instruction-following or reasoning.

Per-task paired differences and confidence intervals would make these trade-offs clearer. Evaluating thresholds under pre-specified task weights or minimum capability thresholds would also help.

W3. Evidence for open-ended generation quality and transfer to stronger starting policies remains limited.

Much of the seven-benchmark suite measures exact answers, code execution, constraint compliance, or choice likelihoods. While AlpacaEval tests open-ended generation, it is only one benchmark, and RewardBench 2 relies on the same oracle used for training annotations. The general alignment claim would be more convincing with evaluations like Arena-Hard or SedarEval's rubric-based scoring to complement the benchmark average and single pairwise judge.

Although testing in two starting policies, the mixed AlpacaEval results leave open how the method scales to a stronger base model such as Qwen3-8B. Evaluating a stronger base would clarify whether useful unsaturated pairs remain and if the same threshold range applies, which would be a helpful extension to reinforce the core contribution.

Questions for authors

Q1. How does pair selection affect output length? Provide a uniform length analysis for SFT, tuned MaxMin, and ActiveDelta (C=3–6C=3\text{--}6) across Olmo and Apertus, including word count distributions and truncation rates. Connecting these to raw versus length-controlled AlpacaEval win rates would clarify if ActiveDelta’s gains stem from improved content or length bias, explaining the divergent performance between Olmo and Apertus.

Q2. Could you show representative selected pairs and generated responses? Examples of a MaxMin pair becoming saturated and the alternative pair selected by ActiveDelta would make the mechanism easier to interpret. Matched generated responses could illustrate where the extra training signal improves content and where instruction following or reasoning regresses. These would complement the aggregate curves without needing a large new evaluation.

Q3. Does the gain persist on a more strongly post-trained starting policy? Testing on a stronger post-trained checkpoint like Qwen2.5 or Qwen3 would address transferability. Explicitly reporting its zero-shot baseline performance and training setup would help verify whether active selection of ActiveDelta still identifies useful unsaturated pairs in more advanced models.

I would be happy to revise my assessment if the authors can clarify these points and provide the further details.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 3kZZ

Thank you for the detailed review and the concrete suggestions. We ran the analyses you asked for and report them below in the order of your points.

W1 (equal compute). See the global response, part 1: at equal GPU-hours ActiveDelta leads tuned MaxMin by 1.4 to 1.5 points with the reference cached and by 0.6 to 0.8 as run. App. K gives score against GPU-hours with scoring and updates separated.

W2 (per-task trade-offs, choice of C). ActiveDelta and tuned MaxMin also differ in β (0.05 against 0.1), so we split each difference into the two effects (five seeds; * = 95% CI excludes zero; total at C=4, effects at C=3).

Total: ActiveDelta − tuned MaxMinEffect of lower β (MaxMin, 0.1 → 0.05)Effect of selection rule (MaxMin → ActiveDelta, β 0.05)
GSM8K+0.5−4.5*+4.7*
HumanEval+0.5+1.1*−0.9
BBH+0.6−7.6*+9.5*
IFEval83.5 → 81.6*−1.6+1.0
AlpacaEval+6.8*+4.4*+1.0
TruthfulQA+2.6*+0.4+1.4*
BBQ+4.2*+1.0*+2.0*
Mean+1.9*−1.0*+2.7*

Lower β raises MaxMin's AlpacaEval and costs it GSM8K and BBH; the selection rule recovers both. The IFEval drop follows β, though neither part is significant alone. On Apertus, both quoted drops have CIs that include zero (AlpacaEval is lower in all five seeds). On the four capability benchmarks the total is −0.1; on the other three, +4.6. A limit of 1.5 or 2 points of loss on any benchmark selects C=3 or C=4 (1 point: none); the oracle-reward plateau selects C=5 (App. N).

W3 (open-ended evaluation). On Arena-Hard-Auto v0.1 (our AlpacaEval judge, one seed), ActiveDelta C=4 wins 77.6% and tuned MaxMin 75.4% (+2.2, 95% CI −1.0 to 5.5); MaxMin at β = 0.05 wins 81.4%, 2.8 above ActiveDelta C=3 (CI −0.4 to 5.9), the β effect of the W2 table. Neither difference is significant; we report Arena-Hard as inconclusive (App. O). RewardBench 2 does measure agreement with the training oracle; rubric-based scoring (SedarEval) is future work.

W3, Q3 (stronger starting policy). On Qwen3-8B (non-thinking mode; itself one of the 28 pool generators) at 10k prompts, both methods tuned, five seeds, the mean of 7 is 72.5 for the starting policy, 71.3 for MaxMin at the same hyperparameters, 73.0 for tuned MaxMin and 73.8 for ActiveDelta (C=4). ActiveDelta leads by 2.5 at the same hyperparameters (p = 0.047) and by 0.7 over tuned MaxMin, which is not significant (p = 0.45). MaxMin's margin ends at 16.1 (Olmo: 7.2), unsaturated pairs remain throughout, and C = 4 is again best (App. Q).

Q1 (output length). AlpacaEval, five seeds:

WordsRawLC
Olmo MaxMin (tuned)35255.152.4
Olmo ActiveDelta (C=4)38564.059.2
Apertus MaxMin39968.963.0
Apertus ActiveDelta (C=4)50369.460.1

On Olmo the gain holds under length control (+6.8 LC, p = 0.003). On Apertus, responses are 26% longer at equal raw win rate, so the LC deficit is a length effect. ActiveDelta's pairs prefer the longer response in 58 to 60% of cases, against 63% for pool pairs with the same oracle gap, so the rule adds no length preference. AlpacaEval truncation is 2.6% for both; SFT rows and distributions are in App. P.

Q2 (examples). App. P shows randomly drawn examples where MaxMin's pair is saturated and ActiveDelta trains on another pair (median gradient weight 1.2e-5 against 0.050), plus an AlpacaEval win and an IFEval regression. One drawn pair is mislabeled by the oracle.

Thank you for the deeper and wider experiments in Appendix K (equal GPU-hours) and Appendix Q (Qwen3-8B), which largely address my questions regarding compute efficiency and stronger starting policies. One question remains on how to interpret these results:

In both regimes, the average improvement over tuned MaxMin is modest and not consistently statistically significant:

  • Equal GPU-hours (Tables 20 & 21): ActiveDelta (60k prompts) outperforms MaxMin (150k prompts) by +0.8+0.8 (p=0.04p = 0.04) at C=4C = 4 and +0.6+0.6 (p=0.09p = 0.09) at C=3C = 3. Moreover, the margin narrows as MaxMin scales (+1.9+1.9 at 60k →\rightarrow +1.5+1.5 at 112k →\rightarrow +0.8+0.8 at 150k), and at 150k the remaining gain is largely driven by AlpacaEval 2 (LC) (+6.3+6.3 at C=4C = 4), partially offset by regressions on IFEval (−2.7-2.7) and BBH (−1.1-1.1).

  • Stronger starting policy (Table 41): On Qwen3-8B, the gain over tuned MaxMin is +0.7±0.9+0.7 \pm 0.9 (p=0.45p = 0.45). Given the standard error, the 95%95\% confidence interval spans roughly [−1.7,+3.2][-1.7, +3.2], making it indistinguishable from either zero or the +2.15+2.15 paired gain observed on Olmo at 10k prompts.

How do you view the practical contribution of ActiveDelta in these two regimes, and do these findings temper its overall practical significance?

AuthorsOpenReview ↗

Reply: practical significance

Thank you for the careful follow-up. Your reading of both tables is correct, and yes, these findings temper the practical claim in those two regimes. Our view:

Equal GPU-hours. Table 20 fixes ActiveDelta at 60k prompts, its most expensive point, and gives MaxMin 2.5 times the data, so the margin narrows as you describe. The same runs read the other way, as the cost of reaching a given score (Table 23, Figure 5):

Mean of 7ActiveDeltaTuned MaxMin
73.7 ± 0.2 vs 73.8 ± 0.620k prompts, 18 GPU-h150k prompts, 59 GPU-h
73.2 ± 0.4 vs 73.0 ± 0.310k prompts, 9 GPU-h112k prompts, 44 GPU-h

ActiveDelta reaches the score of MaxMin at 150k prompts with 7.5 times fewer prompts and less than half the training GPU-hours. Each further prompt for MaxMin also needs 28 generated and oracle-scored responses, which these GPU-hours do not count. The per-benchmark profile at 150k follows the lower β of the C=4 configuration (App. N); at C=3 the differences to MaxMin at 150k are −1.3 on IFEval, +0.3 on BBH and +4.8 on AlpacaEval 2 (LC).

Qwen3-8B. We agree that five seeds cannot separate +0.7 from zero, and we claim no gain over tuned MaxMin there. The experiment supports two narrower points. The saturation mechanism carries over. And the result depends less on tuning: at the same hyperparameters the difference is +2.5 (p = 0.047), and in the grid of Table 40 (one seed per cell) MaxMin's six cells average 70.5, with one above the starting policy (72.5), against 73.0 for ActiveDelta's nine cells, with four above.

Overall. With ample scored data and compute for MaxMin, or a strongly post-trained policy, the advantage over a well-tuned MaxMin is small; the Limitations of the revision state both. The practical contribution is in the regime where scored preference data is the constraint: ActiveDelta reaches a given quality with several times fewer prompts, and its learned-reward variant does so with 2 oracle calls per prompt instead of 28.

Thank you for the thorough follow-up. Your clarification on the comparison of fairness and performance trade-offs against the baseline clearly situates this work within sample-efficient post-training under constrained preference data. Including a more explicit discussion of these scope boundaries in the Limitations section also makes the presentation much more complete and transparent. Most of my concerns have been addressed.

Reviewer_rPwbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of selecting response pairs for DPO training when multiple responses are available for a given prompt. The conventional MaxMin approach selects the two responses with the largest oracle reward gap. However, the authors argue that this approach causes gradient saturation, leading to early stagnation in performance improvement. The authors propose a new method called ActiveDelta, which selects the pair with the largest oracle reward gap among those whose implicit reward margin is below a certain threshold. The authors conduct experiments across various benchmarks and models, and ActiveDelta consistently demonstrates strong performance.

Critical strengths

The performance improvements reported in the paper are interesting. In particular, ActiveDelta achieves better performance with only 10k training examples than MaxMin with 60k examples. This is a practically meaningful result.

The proposed theoretical analysis of gradient saturation appears sound.

Critical weaknesses

Although gradient saturation may occur, the claim that it causes performance degradation in DPO is not sufficiently supported. Whether gradient saturation simply reflects that the model has already learned from the selected pairs, or whether it actually causes premature convergence and performance stagnation, represents two different interpretations of the same phenomenon. Furthermore, it is unclear whether the reported performance improvements result from mitigating gradient saturation or from selecting more informative response pairs.

While I acknowledge the practical value of the experimental results, I believe that a more rigorous causal analysis would improve the quality of the paper. For example, there may be pair-selection strategies that exhibit gradient saturation while still achieving strong performance. Conversely, one could prevent gradient saturation by deliberately providing incorrect preference pairs, in which case gradient saturation might be avoided but performance would deteriorate. Clarifying the relationship between gradient saturation and model performance would make the paper more convincing.

Figure 1 illustrates (\sigma(-m_\theta)), rather than the actual magnitude of the parameter gradient (|\nabla_\theta \mathcal{L}|). The actual parameter gradient is given by

[ \nabla_\theta \mathcal{L} = -\sigma(-m_\theta)\nabla_\theta m_\theta. ]

However, the magnitude of (\nabla_\theta m_\theta) is not considered in the figure. Even when (\sigma(-m_\theta)) is large, we cannot determine how large the actual parameter gradient will be.

Furthermore, how much useful information the model learns depends not simply on the gradient magnitude, but also on the curvature of the loss landscape, as characterized by the Hessian. Approaches based on the Fisher Information Matrix, such as MASS-DPO discussed in the related work, may provide a more informative perspective. The authors should theoretically reconsider whether saturation of the gradient weighting factor can be directly interpreted as saturation of learning. I suspect that the performance improvements of ActiveDelta may instead arise from selecting semantically more informative response pairs.

Questions for authors

My main question is whether the performance improvement of ActiveDelta is primarily due to adaptive pair selection at each training step, rather than the mitigation of gradient saturation itself.

According to the ActiveDelta algorithm, once the model has sufficiently learned from a particular pair, it has the opportunity to learn from other pairs. Consequently, ActiveDelta is expected to expose the model to a larger number of useful training pairs than MaxMin, potentially improving data efficiency.

However, this advantage comes at an additional computational cost compared with MaxMin, since the current policy must perform additional forward passes to evaluate candidate pairs.

Could the authors provide further analysis or experiments to investigate this possibility?

More broadly, I suggest expanding the paper's analysis beyond gradient saturation toward a more comprehensive framework that considers the informativeness of selected training pairs. I believe the proposed method itself is effective, but I find it somewhat disappointing that the analysis is largely restricted to gradient saturation.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer rPwb

Thank you for the review. Compute cost and the gradient magnitude in Figure 1 are answered in the global response (parts 1 and 2): at matched GPU-hours ActiveDelta leads tuned MaxMin by 0.6 to 1.5 points, and the full gradient norm is 0.79 for MaxMin against 11.5 for ActiveDelta, which the margin gradient does not offset.

Stagnation, not degradation. Our claim is that training on saturated pairs stops improving the policy, not that it harms it: with the same hyperparameters, MaxMin stays between 71.5 and 72.0 from 8k to 60k prompts while ActiveDelta rises from 71.7 to 74.4 (Table 12). We agree with your first reading, that saturation reflects what the policy has already learned from those pairs. ActiveDelta acts on it: once a pair is learned, it trains on the largest-gap pair that is not.

Saturation versus adaptivity (main question). We trained two policy-aware rules that keep the gradient weight high without taking the widest oracle gap: smallest ∣mθ∣|m_\theta|, and a uniform draw from ActiveDelta's eligible set (mθ<Cm_\theta < C), which differs from ActiveDelta only in this draw. All rules use ActiveDelta's selected hyperparameters (60k prompts, lr 2e-6, β 0.05, C=3C = 3).

RuleOracle gap (rel. to MaxMin)Weight σ(−mθ)\sigma(-m_\theta), second half of epochMean of 7
MaxMin, same hyperparameters (5 seeds)1.000.00571.7 ± 0.5
Smallest ∥mθ∥\|m_\theta\| (1 seed)0.160.5068.4
Random among mθ<Cm_\theta < C (1 seed)0.230.3470.7
ActiveDelta (5 seeds)0.560.1674.4 ± 0.3

Both cases you describe occur. A rule can saturate and still be strong: MaxMin, and the APL-style rule, which is adaptive at every step (71.0 at this configuration, 71.8 at the best of its six, Table 9). A rule can avoid saturation and be weak: both new rules fall below ActiveDelta's 95% single-seed prediction interval (73.6 to 75.2). Neither adaptivity nor a non-vanishing weight suffices; the gain requires the largest oracle gap among pairs the policy has not yet separated. Seeing more distinct pairs does not explain the gain either: both new rules draw from all 378 rank combinations of a prompt. Keeping the weight up without changing the pair does not help either: at 10k prompts, reweighting MaxMin's pairs (Uni-DPO, 71.1) or dropping the saturated ones (71.4) stays at MaxMin's level, where ActiveDelta reaches 73.4 (reply to vJFB, W1).

Informativeness beyond the gradient weight. We agree that the weight alone does not measure it. Smallest ∣mθ∣|m_\theta| matches ActiveDelta's first-order signal σ(−mθ)⋅Δs\sigma(-m_\theta)\cdot \Delta_s in the second half of the epoch (2.6 against 2.7) and still scores 6.0 points lower; what separates the rules is the oracle gap of the pairs that still carry weight (0.16 against 0.56 of MaxMin's). The revision states the claim in these two terms and reports both quantities over training for every rule (App. M).

Hessian / Fisher view. Both terms of the per-pair Hessian, σ(mθ)σ(−mθ)∇mθ∇mθ⊤\sigma(m_\theta)\sigma(-m_\theta)\nabla m_\theta\nabla m_\theta^\top (the Fisher information) and −σ(−mθ)∇2mθ-\sigma(-m_\theta)\nabla^2 m_\theta, carry the factor σ(−mθ)\sigma(-m_\theta), so curvature vanishes with the weight. Measured per pair under the same parameters, MaxMin pairs carry 0.009 to 0.11 times the Fisher trace of ActiveDelta pairs at 0.9 to 1.1 times their ∥∇mθ∥\|\nabla m_\theta\| (one seed; smaller on 92 to 100% of the 64 prompts in each of 8 comparisons; App. L). The revision limits the claim to the training signal of the selected pairs. MASS-DPO computes its Fisher criterion once, before training; a curvature criterion evaluated at the current policy could rank the eligible pairs and is complementary.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the three reviewers for their careful reading. Two concerns were raised by more than one reviewer and are answered here; the others are answered in the individual replies. All new results are in the revised PDF; the changes are listed at the end.

1. Equal compute (rPwb; 3kZZ W1; vJFB W2, Q3). We agree that prompt efficiency is not compute efficiency. We trained tuned MaxMin on more prompts until its GPU-hours match a 60k ActiveDelta epoch, re-tuning the learning rate at each budget (2e-6 stayed best). p-values are from two-sided Welch t-tests over training seeds: the probability of a gap this large if the two methods were equal.

MaxMin promptsGPU-hMean of 7ActiveDelta 60k, C=4 (74.6), minus MaxMinC=3 (74.4) minus MaxMin
60k (Table 1)2372.7+1.9+1.7
112k, matched with reference log-probs cached (3 seeds)3973.0+1.5 (p = 0.006)+1.4 (p = 0.007)
150k, matched as run (5 seeds)5973.8+0.8 (p = 0.04)+0.6 (p = 0.09)

At matched GPU-hours ActiveDelta stays ahead: by 1.4 to 1.5 points when the reference is cached for both methods, and by 0.6 to 0.8 as run, where only the C=4 difference is significant. In the other direction, ActiveDelta at 20k prompts (18 GPU-h) scores 73.7, above MaxMin at 60k (23 GPU-h). Caching the frozen reference lowers ActiveDelta's overhead from 2.6x to 1.9x; a step is about 31% policy scoring, 32% reference scoring and 36% update. The compute-matched MaxMin also needs 1.9 to 2.5 times the prompts, each with 28 generated and oracle-scored candidates (App. K).

2. Per-pair gradient weights and the full gradient (rPwb; vJFB W2, Q4). We logged every training pair's margin and measured per-pair gradients (end of the 60k epoch, seed 42; MaxMin with ActiveDelta's hyperparameters, as in Figure 1).

MaxMinActiveDelta C=3
Weight at batch-mean margin (Figs. 1, 4)2.0×10−72.0\times10^{-7}0.124
Mean per-pair weight6.5×10−36.5\times10^{-3}0.160
Pairs with weight < 0.0195.3%0%
Effective pairs (of 512)10257

The reviewers are right: the weight at the batch-mean margin understates MaxMin's mean per-pair weight, by a factor of 3×1043\times10^{4}. Saturation is nonetheless widespread: 95% of MaxMin's pairs have weight below 0.01, and the remaining weight sits on the equivalent of 10 of 512 pairs. The margin gradient does not compensate: under the same checkpoint, ∥∇θmθ∥\|\nabla_\theta m_\theta\| of MaxMin pairs is 0.92 to 1.14 times that of ActiveDelta pairs, and the full gradient norm is 0.79 ± 0.09 against 11.5 ± 0.3 (five seeds, Table 18). Figures 1 and 4 now state what they plot (App. L).

Changes in the revision

  • App. K: equal-compute comparison and cost breakdown (caching, candidate pool). [rPwb; 3kZZ W1; vJFB W2, Q3]
  • App. L: per-pair gradient weights, gradient norms and Fisher trace; Figures 1 and 4 relabelled. [rPwb; vJFB W2, Q4]
  • App. M: Table 3 extended with two new selection rules and with Uni-DPO, SAGE and Curry-DPO. [rPwb; vJFB W1, Q2]
  • App. N: per-benchmark differences with CIs, the split into β and selection effects, rules for choosing C. [3kZZ W2]
  • App. O: re-estimation on new seeds, held-out benchmarks, Arena-Hard. [vJFB W2, Q1; 3kZZ W3]
  • App. P: output length and qualitative examples. [3kZZ Q1, Q2]
  • App. Q: Qwen3-8B. [3kZZ W3, Q3]
  • Limitations and discussion: compute, scope of the saturation claim, curvature-based selection. [all; rPwb]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.