- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper works in the setting of off-policy reinforcement learning, where a distribution , different from the policy distribution, is used to compute the expected reward and score function. It notes that as the policy changes, the same negative sample from can be revisited under a different current policy. For a score function used has the form , it shows that revisiting an action with a negative advantage-signal () makes it increasingly improbable under the current policy (they introduce the remoteness to quantify this), while the size of updates from the negative action does not decrease. There is also a contrast with positive actions, for which the size of the updates decreases monotonically under reuse. Thus repetition sustains or amplifies repulsion from negative actions, causing instability in training. The paper proves that: For Gaussian policy, negative feedback dominating positive feedback can make the policy move indefinitely farther away from the stationary point, with unbounded score response from every fixed action. For Categorical policy, the score response remains bounded, but the probabilities can still be pushed toward zero. The learning process need not settle at a finite policy parameter. To mitigate these issues, the authors introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which multiplies negative actions by a thresholded exponential taper to attenuate their effect. The weight is a function of the remoteness of the action, and decays exponentially for large values of . The theoretical results prove that such a tapering scheme can mitigate the instability for both Gaussian and Categorical policies.
The experimental section shows the instability in training in practice. Figure 1 and 2 show that large updates from negative samples occur in practice, and establish that they arise from the gradients rather than from larger advantages. Figure 3 shows the effect of negative dominance on the held-out reward. The authors also perform comparisons with other algorithms, notably the closely related -TOPR on various tasks. DRPO shows competitive overall benchmark results.
Critical strengths
- The theoretical results in section 4 present a complete picture of the issue of self amplification under reuse of negative samples, the effect of the dominance of negative samples on the learning dynamics, and the control implication (form of the taper required to mitigate the issues). The issue of self amplification of negative samples has been addressed in literature before, but this is a novel contribution which analyzes the issue theoretically and proves why an exponential taper is chosen for mitigating the issue.
- The experiments in Section 6.1 demonstrate the issue in practice. The controlled experiments are especially instructive to understand the source of the large updates from negative samples. Also, the controlled experiments in the bottom panel of Figure 2 show that the far-field negative updates, not negative feedback generally, causes the instability in C-U1 experiments.
Critical weaknesses
- The issue that concerns me the most is the comparison between DRPO and -TOPR. Looking at appendix E.8, in particular, the formula for , it seems that the only difference between DRPO and -TOPR is the extra threshold parameter (note that and can be absorbed into the parameter ). Looking at Table 2 (b), we see that DRPO only provides a minor improvement over -TOPR. As such, I see this paper as being weak methodologically, even though the theory portion is strong.
Questions for authors
- Clarification: Could the authors explain intuitively why DRPO shows a slight improvement over TOPR even though they have very similar functional forms of the tapering?
4 comments
TOPR uses importance reweighting to simulate the effect of encountering negative samples less often. Under continued sampling from the current policy, a negative sample whose probability decreases would naturally be encountered less frequently. Fixed historical data can still repeatedly present it. TOPR reduces its negative-update weight using the current-to-historical probability ratio. This adjustment requires the behavior-policy probability.
Theoretically, TOPR identifies the possible unboundedness of the negative branch of signed log-likelihood and provides a theoretical basis for asymmetric weighting. We further study where the policy is ultimately pushed when positive attraction and negative repulsion act together, when it can remain stable, and whether stability necessarily implies good performance.
Section 4.3 quantifies this evolution. Let denote the total strengths of positive and negative feedback, including sample weights and feedback magnitudes, and let . In the exponential-family analysis, and are the attraction and repulsion centers. When and the target is attainable, the equilibrium is
This equilibrium is locally stable under continuous dynamics and discrete updates with sufficiently small step sizes. With both centers fixed, stronger negative feedback shifts the equilibrium farther in the repulsion direction. We also analyze persistent drift at critical balance and the conditions for instability of finite equilibria when negative feedback dominates.
Thus, repulsion from negative feedback does not imply that overall training must be unstable: positive and negative feedback can form a new stable equilibrium. Conversely, stability does not guarantee good task performance. In the controlled task in Section 6.2, moderate repulsion moves the policy beyond the positive-sample center toward a better location and improves returns on unseen contexts. Stronger repulsion can overshoot the beneficial region and reduce returns even while the policy remains stable.
TOPR adjusts historical negative-sample influence using behavior- or reference-relative probabilities, whereas we directly analyze and control the repulsion and extrapolation caused by negative feedback. This analysis provides the control basis for DRPO: per-sample retention of negative updates adjusts the combined attraction and repulsion. The threshold determines where attenuation begins, and its strength determines how quickly far-field influence decreases. This preserves potentially beneficial contributions from nearby negative feedback while reducing excessive repulsion from remote negative feedback. The analysis of evolution, equilibria, stability, and excessive extrapolation is our theoretical contribution beyond identifying an unbounded negative branch or proposing a reweighting rule.
Thank you for explaining the difference between DRPO and -TOPR intuitively. I understand the importance of the threshold parameter better now. I am increasing my score to 4.
Thank you. We’re glad our responses and clarifications addressed your concerns. We greatly appreciate your time and constructive feedback, which helped us improve our work.
Method differences, theoretical contribution, and performance
Thank you for your positive assessment of the theoretical analysis and controlled experiments. We further clarify the difference between DRPO and β-TOPR, and why similar attenuation forms can produce different outcomes.
Intuitively, DRPO asks “how unlikely is the current policy to produce this negative sample?”, whereas TOPR asks “how much has its probability decreased relative to the behavior policy?” Both adjust negative updates through per-sample weights, but different quantities determine those weights.
To compare their structures using the same probability definition, let
Dθ=−logπθ,Dμ=−logμ,
and express DRPO's exponential attenuation in these coordinates:
\boxed{w_{\mathrm{TOPR}}^{-}=\exp\\!\left\\{-[D_{\theta}-D_{\mu}]_{+}\right\\},\qquad w_{\mathrm{DRPO}}^{-}=\exp\\!\left\\{-k[D_{\theta}-\tau]_{+}\right\\}}
Here k=λ/c>0. TOPR's expression is an exact rewriting of its canonical negative weight min{πθ/μ,1}. TOPR's attenuation reference is determined by the behavior probability; DRPO's threshold τ sets the near/far boundary in current remoteness, independently of the behavior probability. See Le Roux et al. (2025), TOPR, Eq. (1) and Algorithm 1. In Appendix E.8, which you refer to, β-TOPR's denominator comes from a fitted reference policy. Absorbing attenuation-strength parameters does not remove this reference probability, so the difference is not simply an additional threshold.
This distinction corresponds to two control routes. DRPO starts from negative-update gradient dynamics and directly controls far-field repulsion. With repeated reuse, a historical negative sample can become increasingly unlikely under the current policy while its repulsive influence persists or even strengthens. We therefore reduce its update weight according to current remoteness. Determining how remote a sample currently is requires only the current model, not its historical generating probability.