- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on likelihood displacement in Direct Preference Optimization, where the likelihood of preferred responses can decrease despite an increasing preference margin. It proposes Adaptive Reward Bounds (ARB), which uses sliding-window statistics to set a lower bound for chosen responses and an upper bound for rejected responses, with penalties applied when these bounds are violated. Experiments on Llama3.1-8B and Qwen3-8B models report improved reward trajectories and mostly better downstream benchmark results than standard DPO. The paper also provides a first-order analysis of the possible update directions.
Critical strengths
-
The paper addresses a useful problem by examining why increasing the preference margin does not necessarily improve the likelihood of preferred responses.
-
The method is straightforward to implement, and the paper includes training-trajectory plots and supplementary code.
Critical weaknesses
-
The analysis in Appendix A shows that suitable gradient weights can produce the desired update directions, but does not show that ARB selects such weights. The actual algorithm uses fixed penalty coefficients, with the bounds controlling whether each penalty is active. Figure 3 compares feasibility thresholds without checking whether the actual updates satisfy the derived condition.
-
The first-order analysis is closely related to Chen et al., “Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner” (ICML 2026), which also derives conditions on the relative gradient weights for the desired likelihood updates. This work is not discussed. The authors should clarify the additional theoretical contribution and compare with its proposed method.
-
Equations 7–8 define the penalty using the mean over the batch consumed by an optimizer step. However, the supplementary code computes the penalty separately for each local microbatch before gradient accumulation. These operations are not equivalent and can activate a penalty even when the overall batch mean satisfies the bound.
-
The downstream evaluation mainly uses multiple-choice benchmarks such as HellaSwag, PIQA, and TruthfulQA. These results measure model capabilities but do not directly establish better preference alignment.
Questions for authors
Please see the weaknesses above.
W1. ARB's fixed weights versus the existence of suitable weights
Please see C3.
C3 shows that the fixed weights and gated hinges of ARB realize the intended dynamics whenever their ratio lies in the band. Fig. 10 (App. E) checks this on the actual training updates. At every step it records the effective incentives dw and dl that the realized update places on gw and gl, computed from the fixed λw,λl and the gates of that step. Under DPO the two coincide, so the ratio t=dw/dl stays at one. Under ARB, once the bounds activate, dw rises above the DPO level by more than dl for most of the run, so the realized ratio exceeds one. This is the direction the derived condition requires in the displacement regime, where the binding inequality is t>ρ/Gw>1 (App. A, after Eq. 21). The extra degree of freedom in App. A is therefore not only available in principle. The training updates use it with fixed weights, and they move the ratio in the direction the condition requires.
W2. Relation to Chen et al.— details in App. E
We thank the reviewer for the pointer. App. E (Reward-level or Gradient-level?) now compares ARB with the reward calibration (RC) of Chen et al. The two methods differ in where the weight ratio between gw and gl is decided: RC decides it from the gradient, ARB from the reward.
In short, RC steers the gradient ratio toward a computed target, whereas ARB steers the reward trajectory itself and lets the ratio follow.
W3. Per-microbatch evaluation of the penalty
The reviewer is correct that the code evaluates the hinges on each local microbatch, whereas Eqs. 7–8 define them on the batch mean. This follows from training under gradient accumulation and does not affect the method.
To fit an policy in GPU memory, we train with gradient accumulation. Each microbatch is backpropagated as soon as it is forwarded, so the mean offset over the full batch is not yet available when the hinge gradient of a microbatch is computed. Evaluating the batch gate exactly would require one of two things.
We therefore evaluate each hinge on its microbatch against the bounds of the current step and accumulate its gradient with the others across microbatches and devices. The window statistics carry no gradient. They are aggregated over the full batch and all devices after each optimizer step, exactly as in Eqs. 7–8, so the floor and the ceiling follow the definition in the paper.
Relation to the batch penalty. Let the K microbatches of a step have offsets δˉw(j) with mean δˉw.
W4. Evaluation limited to multiple-choice benchmarks
Please see C2.