acceptodds
Under review as a conference paper at ICLR 2027

Raising the Floor, Lowering the Ceiling: Adaptive Reward Bounds for Direct Preference Optimization

Abstract

An increase in the preference margin optimized by Direct Preference Optimization (DPO) does not imply an increase in the probability assigned to preferred behavior. The margin can also increase if the chosen and rejected rewards both decrease, provided that the rejected decreases more rapidly. This behavior reflects an under-specification in the DPO objective. It constrains only the difference between the policy's and reference model's response scores, rather than their individual evolution. Moreover, the shared prompt, token representations, and parameters induce coupled gradients for the two responses, allowing a simultaneous decrease to constitute a favorable descent direction for the pairwise objective. We propose Adaptive Reward Bounds (ARB), a trajectory-aware regularization framework that augments DPO with explicit directional control. At each step, ARB computes length-normalized changes in the reward, aggregates chosen and rejected values over the batch, and tracks them with sliding-windows. It imposes a lower bound on the chosen statistic and an upper bound on the rejected statistic using detached soft penalties only upon violations. Different from existing methods, ARB adapts its constraints to the training trajectory, tolerates local reversals, and preserves directional pressure through optional biases. Experiments confirm that ARB effectively mitigates the undesired joint decline of chosen and rejected rewards while preserving the preference learning objective.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Large Language Model,Preference Learning,Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DHYaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on likelihood displacement in Direct Preference Optimization, where the likelihood of preferred responses can decrease despite an increasing preference margin. It proposes Adaptive Reward Bounds (ARB), which uses sliding-window statistics to set a lower bound for chosen responses and an upper bound for rejected responses, with penalties applied when these bounds are violated. Experiments on Llama3.1-8B and Qwen3-8B models report improved reward trajectories and mostly better downstream benchmark results than standard DPO. The paper also provides a first-order analysis of the possible update directions.

Critical strengths
  • The paper addresses a useful problem by examining why increasing the preference margin does not necessarily improve the likelihood of preferred responses.

  • The method is straightforward to implement, and the paper includes training-trajectory plots and supplementary code.

Critical weaknesses
  • The analysis in Appendix A shows that suitable gradient weights can produce the desired update directions, but does not show that ARB selects such weights. The actual algorithm uses fixed penalty coefficients, with the bounds controlling whether each penalty is active. Figure 3 compares feasibility thresholds without checking whether the actual updates satisfy the derived condition.

  • The first-order analysis is closely related to Chen et al., “Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner” (ICML 2026), which also derives conditions on the relative gradient weights for the desired likelihood updates. This work is not discussed. The authors should clarify the additional theoretical contribution and compare with its proposed method.

  • Equations 7–8 define the penalty using the mean over the batch consumed by an optimizer step. However, the supplementary code computes the penalty separately for each local microbatch before gradient accumulation. These operations are not equivalent and can activate a penalty even when the overall batch mean satisfies the bound.

  • The downstream evaluation mainly uses multiple-choice benchmarks such as HellaSwag, PIQA, and TruthfulQA. These results measure model capabilities but do not directly establish better preference alignment.

Questions for authors

Please see the weaknesses above.

1 comment

W1. ARB's fixed weights versus the existence of suitable weights

Please see C3.

C3 shows that the fixed weights and gated hinges of ARB realize the intended dynamics whenever their ratio lies in the band. Fig. 10 (App. E) checks this on the actual training updates. At every step it records the effective incentives dwd_w and dld_l that the realized update places on gwg_w and glg_l, computed from the fixed λw,λl\lambda_w,\lambda_l and the gates of that step. Under DPO the two coincide, so the ratio t=dw/dlt=d_w/d_l stays at one. Under ARB, once the bounds activate, dwd_w rises above the DPO level by more than dld_l for most of the run, so the realized ratio exceeds one. This is the direction the derived condition requires in the displacement regime, where the binding inequality is t>ρ/Gw>1t>\rho/G_w>1 (App. A, after Eq. 21). The extra degree of freedom in App. A is therefore not only available in principle. The training updates use it with fixed weights, and they move the ratio in the direction the condition requires.

W2. Relation to Chen et al.— details in App. E

We thank the reviewer for the pointer. App. E (Reward-level or Gradient-level?) now compares ARB with the reward calibration (RC) of Chen et al. The two methods differ in where the weight ratio between gwg_w and glg_l is decided: RC decides it from the gradient, ARB from the reward.

  • No gradient probing. RC computes two additional output-layer gradients at every step, smooths the score norms with a moving average, and rescales the backward pass toward the band center r⋆=∥gl∥/∥gw∥r^\star=\|g_l\|/\|g_w\|. ARB reads only the reference-relative offsets that the loss already computes and adds no forward or backward pass. An active floor raises the ratio (t>1t>1), an active ceiling lowers it (t<1t<1), and the DPO update is left unchanged otherwise.
  • Trend instead of step. RC re-targets a per-step condition at every step. ARB judges each step against the windowed median of its own branch, so its correction follows the sustained trend of each reward rather than single-step fluctuations.
  • Additive instead of multiplicative. Under DPO, RC rescales the common weight aa to a(r⋆)±1/2a(r^\star)^{\pm1/2}, so that dwdl=a2d_wd_l=a^2. It can only shift weight between the two branches and reverts to DPO as r⋆→1r^\star\to1. ARB adds bwb_w and blb_l on top of aa, so a violated branch gains incentive without the other losing any. Fig. 10 shows this: RC's incentives return to those of DPO once the score norms balance, while ARB's stay above DPO after its bounds activate.

In short, RC steers the gradient ratio toward a computed target, whereas ARB steers the reward trajectory itself and lets the ratio follow.

W3. Per-microbatch evaluation of the penalty

The reviewer is correct that the code evaluates the hinges on each local microbatch, whereas Eqs. 7–8 define them on the batch mean. This follows from training under gradient accumulation and does not affect the method.

To fit an policy in GPU memory, we train with gradient accumulation. Each microbatch is backpropagated as soon as it is forwarded, so the mean offset over the full batch is not yet available when the hinge gradient of a microbatch is computed. Evaluating the batch gate exactly would require one of two things.

  • Retaining the computation graphs of all microbatches until the end of the step. For Qwen3-8B at our maximum sequence length, the layer checkpoints alone take about 77 GB per GPU, compared with 4.8 GB per microbatch, which removes the memory benefit of accumulation.
  • An additional gradient-free forward pass over the full batch and a cross-device synchronization before every backward pass. ARB is designed to avoid this: it uses only the offsets that the loss already computes and adds no forward or backward computation.

We therefore evaluate each hinge on its microbatch against the bounds of the current step and accumulate its gradient with the others across microbatches and devices. The window statistics carry no gradient. They are aggregated over the full batch and all devices after each optimizer step, exactly as in Eqs. 7–8, so the floor and the ceiling follow the definition in the paper.

Relation to the batch penalty. Let the KK microbatches of a step have offsets δˉw(j)\bar\delta_w^{(j)} with mean δˉw\bar\delta_w.

  • By convexity, 1K∑jReLU(Fw−δˉw(j))≥ReLU(Fw−δˉw)\frac{1}{K}\sum_j \mathrm{ReLU}(F_w-\bar\delta_w^{(j)})\ge \mathrm{ReLU}(F_w-\bar\delta_w). A microbatch can therefore activate the penalty while the batch mean satisfies the bound, as the reviewer notes.
  • If the batch mean violates the bound, at least one microbatch does too, so every batch-level violation is corrected.
  • When all microbatch gates agree, the two implementations produce identical gradients.

W4. Evaluation limited to multiple-choice benchmarks

Please see C2.

Reviewer_gwEVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets a known failure mode of DPO: the margin between chosen and rejected implicit rewards can widen while both rewards decrease jointly, because the objective constrains only their difference and the two responses share prompt, tokens, and parameters, producing coupled gradients. It proposes Adaptive Reward Bounds (ARB), a regularizer that applies a lower bound (floor) to the batch-mean chosen offset and an upper bound (ceiling) to the batch-mean rejected offset. Each offset is a length-normalized, reference-relative log-probability difference; the bounds are derived from sliding-window medians, displaced by directional biases, and enforced softly by one-sided hinge penalties only upon violation, after warmup and window initialization. The bounds are recomputed periodically according to the slide period SS. A first-order gradient analysis for a single preference triple shows that the intended "chosen up, rejected down" update under DPO requires ρ<min⁡(Gw,Gl)\rho < \min(G_w,G_l), where GwG_w and GlG_l are squared gradient norms. When both hinges are active and the nonnegative penalty weights can be selected appropriately, the corresponding ARB update is feasible under the broader condition ρ<GwGl\rho < \sqrt{G_wG_l}. Validation on Llama3.1-8B and Qwen3-8B, in both Base and Instruct variants, shows that ARB restores healthy reward separation in the reported training runs, that removing either bound breaks the branch it does not constrain in the reported ablations, that the measured ARB feasibility threshold sits above the DPO threshold at essentially every probed step on the two Instruct models, and that six downstream capability benchmarks show mostly gains with limited regressions over DPO.

Critical strengths
  • S1. The two-sided, trajectory-derived design directly targets the joint dynamics of the failure: bounding the batch-mean chosen offset from below and the batch-mean rejected offset from above, both estimated from the model's own recent level rather than a fixed anchor, addresses the two stated limitations of one-sided static guards, and the violation-triggered hinge leaves the DPO gradient untouched when neither bound is violated. The construction is clean and drop-in.

  • S2. The feasibility comparison is backed by direct measurement of the relevant thresholds: Fig. 3 probes GwGl\sqrt{G_wG_l} and min⁡(Gw,Gl)\min(G_w,G_l) along actual DPO training runs and finds the former strictly above the latter at essentially every probed step on both Instruct models. This establishes a gap between the two thresholds, rather than directly verifying that the actual inner product ρ\rho lies in the additional feasible region. The floor-only/ceiling-only ablation gives a coherent account of the benefit of using both bounds in the reported runs. The theory is anchored to observable quantities.

Critical weaknesses
  • W1. The "adaptive" selling point is undermined by a selectively built comparison. Table 1 includes fixed-reference guards, asymmetric reward weighting, and reference-free reward reshaping, but does not contextualize ARB against prior adaptive preference objectives. In particular, β\beta-DPO [2], which dynamically calibrates β\beta at the batch level using preference-pair informativeness statistics, and AlphaDPO [1], which enables instance-adaptive reward margins through an implicit reference distribution, are neither cited nor empirically compared. The paper instead discusses AlphaPO, a distinct method that reshapes the reference-free reward, but does not evaluate it. More importantly, no method-level baseline for this failure mode (e.g. DPO-Positive or Cal-DPO) is evaluated: the reported empirical comparisons use bare DPO and ARB variants, so the paper never tests its implicit claim of improving over existing fixes.

  • W2. The mechanisms named as contributions are asserted rather than demonstrated. (i) Sec. 5.3 ablates only two-sidedness (floor-only vs ceiling-only); the sliding-window statistic and directional biases are not ablated. A variant with both δwbias=δlbias=0\delta_w^{\mathrm{bias}}=\delta_l^{\mathrm{bias}}=0 and a sweep over WW would test whether the adaptive construction and biases contribute to the reported benefit, and whether the benefit depends on the central hyperparameters. (ii) The theory-method gap remains: App. A establishes first-order feasibility for a single preference triple with both hinges active and appropriately selectable penalty weights, where the effective weights bw=λw/∣yw∣b_w=\lambda_w/|y_w| and bl=λl/∣yl∣b_l=\lambda_l/|y_l| involve no window. It does not establish that the fixed weights and window-determined hinge activation used in training realize a feasible update; the lag and noise of the batch-level windowed median are not analyzed. (iii) The two contour plots in Fig. 5 use colorbars of different ranges (approximately 0.55-1.04 for DPO vs 0.55-3.66 for ARB), so the panels are not comparable by color intensity. A common numerical color scale and contour levels, together with an explicit definition of the low-loss region, would be needed to substantiate the claimed enlargement; separately normalizing each panel or marking minimizers alone would not establish that claim.

[1] AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization. arXiv:2410.10148. DOI: 10.48550/arXiv.2410.10148.

[2] beta-DPO: Direct Preference Optimization with Dynamic beta. arXiv:2407.08639. DOI: 10.48550/arXiv.2407.08639.

Questions for authors
  • Q1. Do you have results for a variant with both δwbias=δlbias=0\delta_w^{\mathrm{bias}}=\delta_l^{\mathrm{bias}}=0 and at least two other window sizes WW? If the zero-bias variant performs comparably, the reported outcomes would not establish that the bias is necessary for the observed benefit, although this would not imply that a nonzero bias supplies no corrective gradient pressure.

  • Q2. How do the trajectories and downstream comparison change when DPO-Positive and Cal-DPO are run under your exact setup (same data, models, budget)? These are closely related published fixes for the same failure mode. Could you also report one alignment-quality metric (MT-Bench, AlpacaEval, or reward-model win-rate)? The six current benchmarks assess general capabilities and truthfulness rather than directly evaluating instruction-following preference alignment.

  • Q3. How does ARB relate to β\beta-DPO [2], which dynamically calibrates β\beta at the batch level, and to AlphaDPO [1], which enables instance-adaptive reward margins through an implicit reference distribution? Both are prior adaptive preference objectives, although neither uses ARB's two-sided trajectory-derived bounds. Neither is cited or empirically compared in the supplied manuscript. AlphaDPO should be distinguished from AlphaPO, which is discussed in Sec. 2.2 and Table 1.

  • Q4. Are the reported results from a single run per configuration? If so, do the ARB-DPO gaps hold across seeds?

[1] AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization. arXiv:2410.10148. DOI: 10.48550/arXiv.2410.10148.

[2] beta-DPO: Direct Preference Optimization with Dynamic beta. arXiv:2407.08639. DOI: 10.48550/arXiv.2407.08639.

1 comment

W1 & Q3. Relation to the adaptive objectives β-DPO and AlphaDPO— details in App. H

We thank the reviewer for these references. App. H (Adaptive Reward Margin, Tab. 4) now discusses β-DPO and AlphaDPO.

  • Different scope. Both methods address the heterogeneity of preference data by adapting the margin. β-DPO rescales it with a batch-level temperature βB=β0[1+α(MˉB−M0)]\beta_{\mathcal{B}}=\beta_0[1+\alpha(\bar M_{\mathcal{B}}-M_0)] set by the quality of the pairs. AlphaDPO turns the constant SimPO margin into a per-pair target γ+αM∗\gamma+\alpha M^\ast through an implicit reference. ARB addresses likelihood displacement and adapts bounds on the levels of the two rewards.
  • Margin adaptation leaves the reward levels free. Both objectives act on the margin alone, so the gradients weight gwg_w and glg_l in a ratio fixed by the objective (Tab. 4), and nothing constrains the direction in which each reward moves.
  • Empirical dynamics (Fig. 14, App. H). Under both objectives and on both models, the chosen and rejected rewards still move in lockstep. Adapting the margin changes how far the two rewards separate, but not the direction in which they move.

W1 & Q2. DPO-Positive and Cal-DPO under the same setup, and an alignment-quality metric

Please see C1 and C2.

W2(i) & Q1. Zero-bias variant and additional window sizes

Please see C4.

W2(ii). Fixed weights, window-determined hinge activation, and lag and noise of the windowed median

Please see C3 and C4.

W2(iii). Color scales and low-loss region of the loss contours (see C1)

Please see Fig.5.

Q4. Multiple seeds— details in App. D

Tab. 3 (App. D) shows models with sampling-based decoding over multiple seeds and reports mean ±\pm std.

Reviewer_z4JaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Direct Preference Optimization (DPO) optimizes only the relative margin between paired responses, often resulting in an unintended failure mode where implicit rewards for both chosen and rejected outputs simultaneously decline. This pathology arises from coupled gradient dynamics under shared parameters and prompt representations. To resolve this under-specification, the paper proposes Adaptive Reward Bounds (ARB), a trajectory-aware regularization framework that imposes dynamic, two-sided constraints on absolute reward trajectories. ARB tracks length-normalized reward offsets across batches using a sliding-window median displaced by directional biases, enforcing a lower floor on preferred responses and an upper ceiling on rejected ones via one-sided hinge penalties only upon violations. Theoretical gradient analysis shows that ARB enlarges the feasible optimization set where the intended "chosen up, rejected down" dynamics occur. Empirical evaluations across Llama-3.1-8B and Qwen3-8B confirm that ARB prevents reward collapse and achieves solid improvements across multiple downstream benchmarks.

Critical strengths
  1. The paper targets the critical "both-down" likelihood displacement issue in DPO. By identifying how coupled gradient dynamics degrade preferred responses, the authors formulate a well-motivated, trajectory-aware directional regularization objective.
  2. ARB introduces minimal overhead by using sliding-window median tracking and one-sided hinge penalties. It requires no auxiliary reward models or online rollouts, making it easy to integrate into existing DPO pipelines.
  3. The theoretical analysis formalizes the gradient entanglement barrier in DPO and proves that ARB’s two-sided bounds strictly enlarge the feasible parameter space where healthy "chosen up, rejected down" dynamics occur.
  4. Experiments across four 8B models confirm that ARB prevents joint reward decline and restores healthy separation. Furthermore, ablations demonstrate the complementary necessity of both the lower floor and upper ceiling.
Critical weaknesses
  1. Although the paper reviews methods bounding absolute rewards (e.g., Cal-DPO, DPO-Positive), empirical comparisons only contrast against vanilla DPO. Omitting two-sided regularizers like Cal-DPO makes it impossible to verify the true benefit of adaptive bounds.
  2. Evaluation relies entirely on multi-choice commonsense reasoning benchmarks rather than standard open-ended alignment evaluations like AlpacaEval or Arena-Hard. Consequently, it remains unproven whether corrected reward dynamics translate to better real-world conversational alignment.
  3. The feasibility theorem assumes arbitrary gradient weights on individual pairs. In practice, hinge activations depend on batch-level statistics with fixed hyperparameter coefficients, meaning theoretical feasibility does not guarantee that optimizer steps select the intended direction.
  4. ARB introduces multiple hyperparameters, including window size, slide period, directional biases, and penalty weights. The lack of systematic sensitivity sweeps leaves it unclear how fragile the method is across diverse batch sizes and datasets.
Questions for authors
  1. How does ARB compare empirically against existing bounded DPO variants, particularly Cal-DPO and DPO-Positive, under identical model and data settings?
  2. Can the authors report length-controlled win rates on AlpacaEval 2.0 or Arena-Hard to verify whether healthier reward trajectories yield superior instruction-following and preference alignment?
  3. Given that penalty weights ‭λw,λl\lambda_w, \lambda_l‬‭‬ ‭‬ are fixed and hinges trigger at the batch level, how does the single-pair theoretical feasibility guarantee translate to stochastic mini-batch training?
  4. To isolate the specific contribution of the sliding-window median, how does ARB compare against an otherwise identical baseline using fixed, non-adaptive two-sided bounds?
1 comment

W1 & Q1. Comparison with Cal-DPO and DPO-Positive under identical settings

Pleas see C1.

W2 & Q2. Length-controlled win rates on AlpacaEval 2.0 and Arena-Hard

Pleas see C2.

W3 & Q3. From the single-pair guarantee to fixed weights in minibatch training

Pleas see C3.

W4 & Q4. Hyperparameter sensitivity and comparison with fixed two-sided bounds

Pleas see C4.

Comments on the paper

1 comment

Common Concerns

We thank all reviewers for their valuable feedback and constructive suggestions. Here we address the common concerns raised by multiple reviewers.

C1. Comparison with similar DPO variants (R1-W1/Q1, R2-W1/Q2)— details in Sec. 5.1, Sec. 5.2 and App. F

We now compare ARB with the four objectives of Tab. 1 that regularize the reward levels or shape, DPO-Positive, Cal-DPO, DPO-Shift and AlphaPO, under the same data, models and training budget.

  • Loss landscape (Fig. 5, Sec. 5.1). The loss contours of all objectives over πθ(yw∣x)\pi_\theta(y_w\mid x) and πθ(yl∣x)\pi_\theta(y_l\mid x) show that ARB concentrates the lowest loss where the chosen probability is large and the rejected one small, so descent is drawn toward configurations with a rising chosen probability.
  • Reward margin (Fig. 6, Sec. 5.1). Among the objectives that keep a reference model, ARB widens the margin the most. AlphaPO reaches a larger margin, but it is reference-free with a reshaped reward, and its margin fluctuates by more than an order of magnitude during training.
  • Reward trajectories (Fig. 12, App. F). The baselines fall into two patterns. Either both rewards rise together, which reinforces the rejected response along with the chosen one, or the two rewards move in the intended directions but the margin widens only slowly after an early phase. Their constraints are fixed before training, one-sided or anchored to constant targets, and cannot follow how far each reward has already moved. ARB derives its bounds from the recent trajectory of each branch, so the two rewards keep separating throughout training (Fig. 4).
  • Alignment benchmarks (Tab. 2, Sec. 5.2). All baselines are evaluated on AlpacaEval 2 and Arena-Hard v2 for the instruction-tuned models, where ARB achieves competitive overall results (see C2).

C2. Open-ended alignment evaluation benchmark (R1-W2/Q2, R2-Q2, R3-W4)— details in Sec. 5.2

Tab. 2 (Sec. 5.2) now reports AlpacaEval 2 (length-controlled and raw win rate with standard error) and Arena-Hard v2 (win rate with 95% confidence interval) for all four models, with GPT-4.1 as the judge.

C3. Feasibility under fixed penalty weights in training (R1-W3/Q3, R2-W2(ii), R3-W1)— details in App. A (Minibatch)

App. A (Paragraph Minibatch, Eqs. 22–24) extends the single-pair analysis to what ARB executes in training: fixed weights λw,λl\lambda_w,\lambda_l, applied to the offsets δˉw,δˉl\bar\delta_w,\bar\delta_l of the minibatch of one optimizer step, with each hinge switched on or off by its bound.

  • The bounds only gate, and the realized step is explicit. The bounds are constants within a step and enter only through the gates γw=I[δˉw<Fw]\gamma_w=\mathbb{I}[\bar\delta_w<F_w] and γl=I[δˉl>Cl]\gamma_l=\mathbb{I}[\bar\delta_l>C_l], which do not depend on λw,λl\lambda_w,\lambda_l. The step is therefore Δθ=η(u+λwγwhw−λlγlhl)\Delta\theta=\eta(u+\lambda_w\gamma_w h_w-\lambda_l\gamma_l h_l) (Eq. 22), where uu is the DPO direction of the whole minibatch and ha=∇θδˉah_a=\nabla_\theta\bar\delta_a. Its first-order effect on the two offsets is given in closed form by Eq. 23.
  • Feasibility with fixed weights. With both hinges active, the fixed weights realize the intended dynamics for every scale above max⁡{0,−pw/cw, pl/cl}\max\{0,-p_w/c_w,\,p_l/c_l\} iff their ratio lies in ρˉ/Gˉw<λw/λl<Gˉl/ρˉ\bar\rho/\bar G_w<\lambda_w/\lambda_l<\bar G_l/\bar\rho. Such a ratio exists iff ρˉ<GˉwGˉl\bar\rho<\sqrt{\bar G_w\bar G_l} (Eq. 24). This holds for any DPO direction uu, which may lie outside the span of hw,hlh_w,h_l and only sets how large the weights must be.

C4. Hyperparameter sensitivity (R1-W4/Q4, R2-W2(i)/Q1)details in App. G

We add a one-at-a-time sweep of all six hyperparameters of ARB in App. G (Fig. 11).

  • Sliding window. W=0W=0 removes the window: the bound follows single batches and its hinge reacts to batch noise. W=200W=200 makes the bound smooth but lags it by about W/2W/2 steps behind the current level. Both ends end with the two rewards rising together, while W=100W=100 separates them. The refresh period behaves the same way: S=1S=1 and S=50S=50 both lose the separation that S=10S=10 achieves. The adaptive window therefore matters, and it works by balancing lag against noise.
  • Directional bias. With δwbias=0\delta_w^{\mathrm{bias}}=0 or δlbias=0\delta_l^{\mathrm{bias}}=0, the bound sits at the recent median and only pulls its branch back to where it was. Both rewards then drift up together, so the bias is what turns the bound into a demand for progress. A large bias (0.50.5) also fails, because a bound far ahead of its branch stays violated and the hinge stops responding to the trajectory.
  • Penalty weight. App. G derives an admissible range for each weight from the minibatch dynamics (Eq. 26). A weight below the range cannot correct its own branch, and a weight above it drags the other branch along. The sweep reproduces all four cases: λw=0.1\lambda_w=0.1 lets both rewards fall, λw=0.8\lambda_w=0.8 and λl=0.1\lambda_l=0.1 let both rise, and λl=0.5\lambda_l=0.5 holds the chosen reward at its initial level.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.