- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
This paper proposes ActiveDelta, a policy-aware preference-pair selection method for Direct Preference Optimization (DPO). The authors observe that selecting the highest- and lowest-reward responses for each prompt (MaxMin) can lead to early gradient saturation, as the policy quickly learns to distinguish these extreme pairs. ActiveDelta addresses this problem by selecting the pair with the largest oracle reward gap subject to an upper bound on the current policy's implicit reward margin. The selected pairs are updated dynamically during training, allowing the policy to focus on informative comparisons that have not yet been confidently separated.
The method is evaluated using a pool of 28 candidate responses per prompt, primarily with Olmo-3-7B-Instruct-SFT, and additionally with Apertus-1.5-8B-SFT. Experiments cover seven downstream benchmarks, different prompt budgets, repeated training seeds, and an active preference collection setting with a learned reward model. The results show consistent improvements over tuned MaxMin selection, with better sample efficiency and stronger alignment to the reward oracle. Additional training-dynamics analyses support the observation that ActiveDelta mitigates gradient saturation.
Critical strengths
-
Simple, intuitive, and practically effective selection mechanism. ActiveDelta directly addresses a concrete limitation of MaxMin pair selection by combining oracle reward gaps with the current policy's implicit reward margins. The method is easy to implement and requires no modification to the DPO objective. The training-dynamics analysis provides useful evidence that the proposed selection rule can prevent the rapid decline in learning signal observed with fixed extreme pairs.
-
Strong empirical validation against the principal baseline. The paper evaluates multiple prompt budgets, reports five-seed results for the main comparisons, and tests generalization to a second base model. The improvements over tuned MaxMin are consistent, including a 1.9-point gain in the seven-benchmark average on Olmo-3-7B. The learned-reward-model variant is also valuable, demonstrating that the method can operate with only two oracle calls per prompt while retaining most of its performance advantage.
Critical weaknesses
-
Limited novelty and insufficient comparisons with closely related policy-aware preference optimization methods.
The central idea of considering both preference-pair quality and the current policy's learning progress is closely related to several existing methods. In particular, Uni-DPO (Peng et al., 2026) already combines preference quality with model-dependent training dynamics to reduce emphasis on well-fitted pairs. SAGE (Wu et al., 2026) similarly uses evolving gradient utility to select informative training examples, while Curry-DPO (Pattnaik et al., 2024) explores curriculum-based selection among multiple preference pairs.
ActiveDelta differs from these approaches by selecting the largest oracle-gap pair satisfying a policy-margin constraint rather than reweighting fixed pairs or following a predefined curriculum. This is a useful distinction, but the underlying principle is largely established. The main experiments compare against MaxMin, random selection, and an APL-style baseline, without directly evaluating these more closely related dynamic alternatives.
Consequently, the results establish the advantage of ActiveDelta over static extreme-pair selection, but do not sufficiently demonstrate its independent advantage over existing policy-aware training strategies. This limits the significance of the proposed contribution.
-
Evaluation-set reuse and incomplete computational comparisons weaken the empirical conclusions.
Section 6 explicitly states that the unweighted average of seven downstream benchmarks is used both for hyperparameter selection and as the primary evaluation metric. The methods are tuned over multiple configurations, including extensive searches at the 60k-prompt budget. Selecting configurations using the same benchmarks on which final performance is reported introduces selection bias and makes the reported gains potentially optimistic. Although the five-seed comparisons demonstrate training stability for the selected configurations, they do not resolve this evaluation issue.
Moreover, ActiveDelta requires evaluating the current policy on all candidate responses, introducing substantial additional computation. Appendix A reports approximately 61 GPU-hours for ActiveDelta versus 25 GPU-hours for MaxMin at 60k prompts. Therefore, the reported sixfold improvement in prompt efficiency should not be interpreted as an equivalent improvement in computational efficiency. A comparison under matched total computation would provide a more complete assessment of the practical benefit.
Finally, the saturation analysis reports the gradient weight evaluated at the batch-mean implicit margin rather than the mean of individual pair gradient weights. Since the sigmoid is nonlinear, these quantities can differ substantially. Although the measured gradient norms support the qualitative saturation argument, reporting the actual distribution of per-pair gradient weights would make the mechanistic evidence more rigorous.
References
[1] Peng, S., Wang, W., Tian, Z., et al. (2026). Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs. ICLR 2026. https://arxiv.org/abs/2506.10054
[2] Wu, H., Cai, H., Zhao, J., et al. (2026). Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models. arXiv:2602.01207. https://arxiv.org/abs/2602.01207
[3] Pattnaik, P., Maheshwary, R., Ogueji, K., Yadav, V., & Madhusudhan, S. T. (2024). Curry-DPO: Enhancing Alignment Using Curriculum Learning & Ranked Preferences. arXiv:2403.07230. https://arxiv.org/abs/2403.07230
Questions for authors
-
Evaluation protocol: Section 6 states that the seven-benchmark average is used for both hyperparameter selection and final evaluation. Was there a separate validation split or any independent evaluation of the selected configurations? How should the reported improvements be interpreted given this reuse?
-
Relationship to existing methods: What is the principal advantage of ActiveDelta over approaches such as Uni-DPO and SAGE, which also account for preference quality and the model's evolving learning state? Are any controlled comparisons or related ablations available?
-
Compute efficiency: Since ActiveDelta requires approximately 2.4 times the GPU-hours of MaxMin at 60k prompts, can the authors clarify whether the method retains its advantage at matched computational budgets? How much overhead could be removed by caching reference log-probabilities or using a smaller candidate pool?
-
Gradient saturation measurement: Figures 1 and 4 report the gradient weight computed from the batch-mean margin. Can the authors provide the actual average per-pair gradient weight, or the fraction of pairs with sufficiently small gradient weights, from the existing training logs? This would help distinguish widespread saturation from a highly skewed margin distribution.
Reply to Reviewer vJFB
Thank you for the careful review and the precise questions, which we answer in order below.
W1, Q2 (Uni-DPO, SAGE, Curry-DPO). The three methods decide how much to learn from a fixed pair (Uni-DPO), whether to keep it (SAGE) or in which order to use it (Curry-DPO); ActiveDelta decides which pair to train on. We trained each with its paper's default settings on the 10k-prompt protocol of Appendix A (Uni-DPO and Curry-DPO at their papers' learning rates, SAGE at the protocol's), and added a control that drops MaxMin's pair once its margin reaches 4 instead of replacing it.
All four lie within or below MaxMin's five-seed range (70.6 to 71.8); a one-seed difference has a 95% interval of ±1.3. Reweighting, dropping or reordering saturated pairs does not lift the result above MaxMin. The gain requires a substitute pair that is unsaturated and has a large oracle gap: a random unsaturated substitute scores 70.7 against ActiveDelta's 74.4 at 60k prompts (reply to rPwb). Per-benchmark scores and training dynamics: App. M.
W2 (evaluation reuse), Q1. There was no validation split; both methods had the same 38-run budget. To measure the selection effect, we re-estimated both selected configurations on new runs and on three benchmarks specified before evaluation.
MaxMin's grid run exceeds its new seeds (73.2 against 72.8), which is the size of the selection effect; the gain persists on runs and prompts not used for selection. Held out, ActiveDelta is 0.2 to 0.8 lower on MMLU and MATH-500 and 0.6 to 1.2 higher on ARC-C, each within one SE. These are capability tests, on which Table 1 also shows no difference against tuned MaxMin (C=4: GSM8K, HumanEval and BBH within 0.6). The reported improvement should therefore be read as a difference between two equally tuned methods that holds at +1.7 to +1.9 outside the selection and does not extend to the held-out capability benchmarks (App. O).
W2 (compute), Q3. Matched GPU-hours: see the global response, part 1 (+1.4 to +1.5 with the reference cached, +0.6 to +0.8 as run). Overhead reduction: caching lowers the overhead from 2.6x to 1.9x. Lazy evaluation saves about a third of the candidate cost: finding the largest-gap eligible pair still needs a median 22 of 26 usable candidates (65% of their tokens). A smaller pool saves more: at 10k prompts, keeping the 4 highest- and 4 lowest-scored candidates lowers the overhead from 2.9x to 1.5x and the score from 73.1 to 72.4 (MaxMin 71.0; 3 seeds, differences not significant, p ≥ 0.25) (App. K).
W2 (gradient weights), Q4. See the global response, part 2. The actual mean per-pair weight is 6.5e-3 for MaxMin against 0.160 for ActiveDelta, and 95.3% of MaxMin's pairs have weight below 0.01, against none of ActiveDelta's; the skew you suspected is real, and saturation is widespread all the same.