acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Curse of Repulsion in Off-Policy Reinforcement Learning

Abstract

Off-policy policy optimization reuses historical behavior, including negative- advantage samples that provide useful corrective feedback. We identify learner- relative remoteness as a key state variable of reuse: repeated updates from a fixed negative sample can make it increasingly remote without weakening its score re- sponse. This creates a self-reinforcing remoteness–reuse loop. Aggregate analysis shows how this feedback competes with positive attraction, ranging from stable displacement beyond the positive-only solution to persistent drift and loss of finite stable equilibria. The framework covers both continuous Gaussian and discrete categorical policies, with distinct far-field manifestations: Gaussian score response grows with standardized distance, whereas categorical score response remains bounded while suppression persists. Dynamic Remoteness-Aware Policy Optimiza- tion (DRPO) exponentially attenuates remote negative updates while preserving near-field feedback, with Gaussian ultimate-boundedness guarantees. Controlled experiments separately isolate policy geometry as the source of the far-field effect and establish its causal transmission to instability. External diagnostics in D4RL locomotion and structured generation recover the predicted near-to-far pattern, while task-level evaluations show that DRPO achieves higher aggregate scores than reported matched controls.

Primary area:
reinforcement learning
Keywords:
Off-Policy Reinforcement Learning, Policy Optimization, Negative-Advantage Policy Updates, Remoteness-Aware Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aYhAOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper works in the setting of off-policy reinforcement learning, where a distribution ν\nu, different from the policy distribution, is used to compute the expected reward and score function. It notes that as the policy changes, the same negative sample from ν\nu can be revisited under a different current policy. For a score function used has the form E(s,a)∼ν[A^(s,a)∇θlog⁡πθ(a∣s)]E_{(s,a) \sim \nu}[\hat{A}(s,a)\nabla_{\theta}\log \pi_{\theta}(a|s)], it shows that revisiting an action with a negative advantage-signal (A^<0\hat{A}<0) makes it increasingly improbable under the current policy (they introduce the remoteness Du(a)D_u(a) to quantify this), while the size of updates from the negative action does not decrease. There is also a contrast with positive actions, for which the size of the updates decreases monotonically under reuse. Thus repetition sustains or amplifies repulsion from negative actions, causing instability in training. The paper proves that: For Gaussian policy, negative feedback dominating positive feedback can make the policy move indefinitely farther away from the stationary point, with unbounded score response from every fixed action. For Categorical policy, the score response remains bounded, but the probabilities can still be pushed toward zero. The learning process need not settle at a finite policy parameter. To mitigate these issues, the authors introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which multiplies negative actions by a thresholded exponential taper to attenuate their effect. The weight is a function of the remoteness DD of the action, and decays exponentially for large values of DD. The theoretical results prove that such a tapering scheme can mitigate the instability for both Gaussian and Categorical policies.

The experimental section shows the instability in training in practice. Figure 1 and 2 show that large updates from negative samples occur in practice, and establish that they arise from the gradients ∇θlog⁡πθ(a∣s)\nabla_{\theta}\log \pi_{\theta}(a|s) rather than from larger advantages. Figure 3 shows the effect of negative dominance on the held-out reward. The authors also perform comparisons with other algorithms, notably the closely related β\beta-TOPR on various tasks. DRPO shows competitive overall benchmark results.

Critical strengths
  1. The theoretical results in section 4 present a complete picture of the issue of self amplification under reuse of negative samples, the effect of the dominance of negative samples on the learning dynamics, and the control implication (form of the taper required to mitigate the issues). The issue of self amplification of negative samples has been addressed in literature before, but this is a novel contribution which analyzes the issue theoretically and proves why an exponential taper is chosen for mitigating the issue.
  2. The experiments in Section 6.1 demonstrate the issue in practice. The controlled experiments are especially instructive to understand the source of the large updates from negative samples. Also, the controlled experiments in the bottom panel of Figure 2 show that the far-field negative updates, not negative feedback generally, causes the instability in C-U1 experiments.
Critical weaknesses
  1. The issue that concerns me the most is the comparison between DRPO and β\beta-TOPR. Looking at appendix E.8, in particular, the formula for wβ(x,y)w_{\beta}(x,y), it seems that the only difference between DRPO and β\beta-TOPR is the extra threshold parameter τ\tau (note that λ\lambda and cc can be absorbed into the parameter β\beta). Looking at Table 2 (b), we see that DRPO only provides a minor improvement over β\beta-TOPR. As such, I see this paper as being weak methodologically, even though the theory portion is strong.
Questions for authors
  1. Clarification: Could the authors explain intuitively why DRPO shows a slight improvement over β−\beta- TOPR even though they have very similar functional forms of the tapering?
4 comments

Method differences, theoretical contribution, and performance

Thank you for your positive assessment of the theoretical analysis and controlled experiments. We further clarify the difference between DRPO and β-TOPR, and why similar attenuation forms can produce different outcomes.

Intuitively, DRPO asks “how unlikely is the current policy to produce this negative sample?”, whereas TOPR asks “how much has its probability decreased relative to the behavior policy?” Both adjust negative updates through per-sample weights, but different quantities determine those weights.

To compare their structures using the same probability definition, let

Dθ=−log⁡πθ,Dμ=−log⁡μ,D_\theta=-\log\pi_\theta,\qquad D_\mu=-\log\mu,

and express DRPO's exponential attenuation in these coordinates:

\boxed{w_{\mathrm{TOPR}}^{-}=\exp\\!\left\\{-[D_{\theta}-D_{\mu}]_{+}\right\\},\qquad w_{\mathrm{DRPO}}^{-}=\exp\\!\left\\{-k[D_{\theta}-\tau]_{+}\right\\}}

Here k=λ/c>0k=\lambda/c>0. TOPR's expression is an exact rewriting of its canonical negative weight min⁡{πθ/μ,1}\min\{\pi_\theta/\mu,1\}. TOPR's attenuation reference is determined by the behavior probability; DRPO's threshold τ\tau sets the near/far boundary in current remoteness, independently of the behavior probability. See Le Roux et al. (2025), TOPR, Eq. (1) and Algorithm 1. In Appendix E.8, which you refer to, β-TOPR's denominator comes from a fitted reference policy. Absorbing attenuation-strength parameters does not remove this reference probability, so the difference is not simply an additional threshold.

This distinction corresponds to two control routes. DRPO starts from negative-update gradient dynamics and directly controls far-field repulsion. With repeated reuse, a historical negative sample can become increasingly unlikely under the current policy while its repulsive influence persists or even strengthens. We therefore reduce its update weight according to current remoteness. Determining how remote a sample currently is requires only the current model, not its historical generating probability.

TOPR uses importance reweighting to simulate the effect of encountering negative samples less often. Under continued sampling from the current policy, a negative sample whose probability decreases would naturally be encountered less frequently. Fixed historical data can still repeatedly present it. TOPR reduces its negative-update weight using the current-to-historical probability ratio. This adjustment requires the behavior-policy probability.

Theoretically, TOPR identifies the possible unboundedness of the negative branch of signed log-likelihood and provides a theoretical basis for asymmetric weighting. We further study where the policy is ultimately pushed when positive attraction and negative repulsion act together, when it can remain stable, and whether stability necessarily implies good performance.

Section 4.3 quantifies this evolution. Let p,qp,q denote the total strengths of positive and negative feedback, including sample weights and feedback magnitudes, and let ρ=q/p\rho=q/p. In the exponential-family analysis, m+\mathbf m_+ and m−\mathbf m_- are the attraction and repulsion centers. When 0<ρ<10<\rho<1 and the target is attainable, the equilibrium is

m⋆(ρ)=m++ρ1−ρ(m+−m−)\boxed{\mathbf{m}^{\star}(\rho)=\mathbf{m}_{+}+\frac{\rho}{1-\rho}\bigl(\mathbf{m}_{+}-\mathbf{m}_{-}\bigr)}

This equilibrium is locally stable under continuous dynamics and discrete updates with sufficiently small step sizes. With both centers fixed, stronger negative feedback shifts the equilibrium farther in the repulsion direction. We also analyze persistent drift at critical balance and the conditions for instability of finite equilibria when negative feedback dominates.

Thus, repulsion from negative feedback does not imply that overall training must be unstable: positive and negative feedback can form a new stable equilibrium. Conversely, stability does not guarantee good task performance. In the controlled task in Section 6.2, moderate repulsion moves the policy beyond the positive-sample center toward a better location and improves returns on unseen contexts. Stronger repulsion can overshoot the beneficial region and reduce returns even while the policy remains stable.

TOPR adjusts historical negative-sample influence using behavior- or reference-relative probabilities, whereas we directly analyze and control the repulsion and extrapolation caused by negative feedback. This analysis provides the control basis for DRPO: per-sample retention of negative updates adjusts the combined attraction and repulsion. The threshold determines where attenuation begins, and its strength determines how quickly far-field influence decreases. This preserves potentially beneficial contributions from nearby negative feedback while reducing excessive repulsion from remote negative feedback. The analysis of evolution, equilibria, stability, and excessive extrapolation is our theoretical contribution beyond identifying an unbounded negative branch or proposing a reweighting rule.

Thank you for explaining the difference between DRPO and β\beta-TOPR intuitively. I understand the importance of the threshold parameter better now. I am increasing my score to 4.

Thank you. We’re glad our responses and clarifications addressed your concerns. We greatly appreciate your time and constructive feedback, which helped us improve our work.

Reviewer_iW2zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies an instability issue that arises from offline / off-policy policy gradient update schemes, where negative examples can induce unbounded logits. The paper then proposes a method of tapering the updates for negative examples with an exponential in the policy log probabilities to alleviate this issue. This results in a practical algorithm which is deployed on D4RL and a structured sequence generation task, achieving strong performance.

Critical strengths

The paper studies an important and timely problem, that being practical policy optimization algorithms that consume only offline data, as well as the particularly interesting phenomenon that occurs as a byproduct of policy gradient optimization. The proposed solution is simple to implement, and appears to achieve very strong structured generation performance when applied to a small language model.

Critical weaknesses

The mathematical content of the paper is effectively incomprehensible due to terms that aren't defined properly – see the Questions section. In addition, I also don't understand the setting of the mathematical analysis, particularly where advantages are kept frozen over the course of policy optimization, which in principle impacts the optimization objective.

Moreover, the experimental results leave a bit to be desired; results aren't shown with confidence intervals for instance. Moreover, the D4RL result doesn't look particularly impressive (indeed, I'd be curious to see a proper comparison, like via rliable for example), and the structured generation result is only on a small language model. I get that this was meant to propose some theory / new algorithms and that empirical results are not the main focus, however I have two concerns with this:

  1. This issue of instability in off-policy policy gradient settings was already discussed in the paper introducing TOPR (Le Roux et al., 2025). This paper wasn't really mentioned in any of the background.
  2. While TOPR also proposed a (in fact similar) method of addressing this issue, the paper doesn't compare against TOPR. For the structured generation experiment, the paper compares against a strange variant of TOPR that I haven't seen in the literature, and the D4RL experiment doesn't compare to TOPR at all.

So, overall, the theoretical issue is something that is already understood to some degree, the mathematical contributions are extremely difficult to parse, and the experimental results leave much room for improvement.

Questions for authors

At the beginning of section 4.1, it says "To isolate action-side geometry, fix context ss and write uu for its policy-output coordinates". What does this mean? Particularly, I don't know what action-side geometry means, and I don't know what policy-output coordinates are. As such, I really don't know how to interpret uu. What's the difference between uu and policy parameters? Likewise, towards the end of this paragaph, what is "output-space update magnitude"?

I am confused about what the "signed coefficient" A^\widehat{A} represents. I would have expected it to be the advantage of a state-action pair for the policy being learned. But then in section 4.2, you say that A^\widehat{A} remains fixed over the reuse window. Why? What is the interpretation of equation 4 then?

In Theorem 4.5, the categorical case introduces a new assumption that the logits sum to 1. But the gradient update schemes analyzed previously don't preserve this property. So what should one conclude from this part of Theorem 4.5?

I don't understand at all what m+,m−\mathbf{m}_{+}, \mathbf{m}_{-} are. They are described as the "positive-only mean-parameter target" and the "magnitude-weighted negative sample moment", respectively, but these are not precise terms and I genuinely don't know how to interpret them (or even what space they live in).

On line 236, it refers to "Gaussian policies with finitely many negative atoms". What does this mean? Gaussian policies are atomless.

In Corollary 5.3, what is K\mathcal{K}? Above, this symbol is described as "a bounded region representing historical data coverage or desirable task performance", but these things don't have the same "type" as each other, nor is it even clear to me what either of those are. Moreover, Corollay 5.3 asserts that m+∈int(K)\mathbf{m}_{+}\in\mathrm{int}(\mathcal{K}). While again I don't understand what that term is, I don't see how "positive-only mealn parameter target" is an element of type "historical data" or "desirable task performance".

What is "Joint fitted reference β\beta TOPR"? I see some description of this in the appendix (not mentioned in the main text as far as I can tell). But I don't understand why you compare against this bespoke TOPR variant rather than TOPR itself.

10 comments

Weakness: Uncertainty reporting in Table 2

Thank you for pointing out the lack of uncertainty reporting in Table 2. We additionally report means and standard deviations for D4RL and structured generation, and will include the tables and statistical details in the revision.

Table 2(a): D4RL-9 normalized return

DatasetBCTD3+BCCQLIQLTOPR (β=4)DRPO
halfcheetah-medium43.2 ± 0.648.3 ± 0.347.0 ± 0.547.4 ± 0.243.2 ± 0.747.3 ± 0.3
hopper-medium54.1 ± 3.859.3 ± 4.253.0 ± 28.566.2 ± 5.762.1 ± 4.057.3 ± 10.1
walker2d-medium70.9 ± 11.083.7 ± 2.173.3 ± 17.778.3 ± 8.783.4 ± 3.785.5 ± 2.2
halfcheetah-medium-replay37.6 ± 2.144.6 ± 0.545.5 ± 0.744.2 ± 1.237.5 ± 2.843.3 ± 0.5
hopper-medium-replay16.6 ± 4.860.9 ± 18.888.7 ± 12.994.7 ± 8.662.9 ± 6.287.8 ± 8.8
walker2d-medium-replay20.3 ± 9.881.8 ± 5.581.8 ± 2.773.8 ± 7.165.0 ± 5.578.1 ± 6.1
halfcheetah-medium-expert44.0 ± 1.690.7 ± 4.375.6 ± 25.786.7 ± 5.378.5 ± 5.591.1 ± 2.5
hopper-medium-expert53.9 ± 4.798.0 ± 9.4105.6 ± 12.991.5 ± 14.376.4 ± 11.398.0 ± 10.2
walker2d-medium-expert90.1 ± 13.2110.1 ± 0.5107.9 ± 1.6109.6 ± 1.0106.5 ± 3.9110.4 ± 1.2
D4RL-9 total (sum of task means)430.7677.4678.4692.4615.5698.8
Overall summary (nine tasks)47.86 (Mean task SD: 5.73)75.27 (Mean task SD: 5.07)75.38 (Mean task SD: 11.47)76.93 (Mean task SD: 5.79)68.39 (Mean task SD: 4.84)77.64 (Mean task SD: 4.66)

Table note: Each task reports mean ± SD. DRPO's original results are based on 10 runs. The total is the sum of the nine task means. The overall summary separately reports the macro-average score across nine tasks and the equally weighted average of their task-specific SDs. Aggregates are computed from the displayed values and rounded to two decimal places. TOPR uses β=4 and four training seeds.

Table 2(b): Structured-generation Pass@8

TaskPositive-onlyAsymREβ-TOPRDPODRPO
Countdown13.98 ± 0.5414.38 ± 0.2414.41 ± 0.128.48 ± 0.8215.72 ± 0.48
Word Sorting62.06 ± 3.4361.72 ± 1.9959.61 ± 0.7834.38 ± 12.8563.38 ± 1.39
Spiral Matrix100.00 ± 0.0099.92 ± 0.11100.00 ± 0.00100.00 ± 0.0099.94 ± 0.14
Mini Sudoku95.91 ± 0.5097.58 ± 0.3396.88 ± 0.8891.41 ± 2.2197.06 ± 0.32
Maze76.74 ± 0.8777.81 ± 0.6675.86 ± 0.5579.69 ± 0.8777.66 ± 0.00
Word Ladder10.68 ± 1.2512.50 ± 0.2311.41 ± 0.454.69 ± 2.1213.38 ± 0.82
Knights & Knaves67.53 ± 2.0070.16 ± 0.8868.59 ± 1.7758.20 ± 1.9869.53 ± 0.00
Graph Coloring98.67 ± 0.5898.91 ± 0.2299.37 ± 0.4596.88 ± 0.6299.88 ± 0.13
WikiSQL85.65 ± 0.9685.78 ± 0.4482.89 ± 1.8878.91 ± 1.0286.69 ± 0.95
Overall summary (nine tasks)67.91 (Mean task SD: 1.13)68.75 (Mean task SD: 0.57)67.67 (Mean task SD: 0.76)61.40 (Mean task SD: 2.50)69.25 (Mean task SD: 0.47)

Table note: Scores are Pass@8 percentages, and SDs are measured in percentage points. Each structured-generation method uses four training seeds. For DRPO's eight transfer tasks, we report the sample standard deviation across the selected five adjacent λ configurations; each configuration's score is the mean over five late evaluations. The overall score is the unweighted macro average over nine tasks; the value in parentheses is the equally weighted average of the nine task-specific SDs. The newly supplied DPO and Countdown SDs are included in the corresponding cells.

DRPO's mean configuration SD across the eight transfer tasks is 0.47 percentage points, with seven of the eight task SDs below 1 percentage point, indicating small score variation across the reported adjacent λ configurations. These statistics supplement the uncertainty information in the original main table and will be reported in the revision.

Weakness: D4RL performance and practical value

Thank you for asking about the D4RL performance and practical value of the method. Regarding the magnitude of the gains, our central contribution is to identify the role of remoteness in off-policy training collapse and to propose a control strategy that attenuates negative updates according to remoteness. The dynamical analysis and controlled experiments explain the basis for this control, while D4RL tests its effectiveness on actual continuous-control tasks.

Behavior-policy constraints and conservative value treatment are representative control approaches in offline RL. We provide another route that adjusts remote negative feedback directly within policy updates. As described in Section 6.3, we add only a negative-branch remoteness taper to an existing advantage-weighted policy update, without an additional explicit behavior-policy constraint or conservative value penalty, and obtain competitive D4RL-9 results. Achieving this performance through a simple change supports the practical value of directly controlling far-field negative feedback in policy optimization.

The same control principle covers continuous Gaussian policies, such as those used in continuous control, and discrete categorical policies, such as those used in language-model structured generation. Competitive results across continuous and discrete tasks provide external-validity evidence and a basis for further methodological development.

The control depends only on current-policy remoteness, without requiring access to or estimation of the behavior-policy density or an additional reference policy. For example, the canonical TOPR negative weight min⁡{πθ/μ,1}\min\{\pi_\theta/\mu,1\} requires the behavior policy μ\mu's probability for each sample. With historical data mixing policies without stored generating probabilities, fitting the behavior distribution introduces additional modeling work and estimation error. DRPO computes remoteness directly from the current model, avoiding this step and making it straightforward to incorporate into existing policy updates.

DRPO also has potential to be combined with AsymRE's feedback adjustment, IQL's value-learning module, and trust-region-style constraints on policy movement. Our study of the remoteness mechanism and its control effectiveness provides a basis for such combinations and further development.

Weakness: Model scale in structured-generation experiments

Thank you for asking about model scale and experimental coverage. We study the reuse dynamics of historical negative feedback under the current policy. The theory analyzes feedback coefficients, learner-relative remoteness, and the resulting policy-update responses. DRPO adjusts update strength using positive and negative feedback and current-policy remoteness. Its control rule does not rely on a particular language prompt or semantic design, and the analysis and control formulas do not presuppose a specific network parameter count. Off-policy describes the relationship between the current policy and historical data sources; it is not specific to small models, large models, or language tasks. Identifying the far-field negative-feedback mechanism and testing its control principle therefore does not require a minimum model size.

There are also precedents for studying training mechanisms with small models. For example, the NeurIPS 2025 paper On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization analyzes the effect of negative gradients on correct-response probabilities using models including Qwen2.5-0.5B [1]. The ICLR 2026 paper Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation also uses a 0.5B model in its core training-dynamics analysis [2]. In such studies, a central role of the experiments is to test the mechanism identified by the theory and the effectiveness of control targeting that mechanism.

We implement the control principle from the controlled analysis in actual Transformer training, including far-field gradient diagnostics and task-performance evaluations. In protocol-matched comparisons, DRPO keeps the base-model size unchanged and improves performance by changing the treatment of negative updates, demonstrating the value of remoteness control in neural policies. The same principle is also tested on D4RL tasks with continuous Gaussian policies. Together, continuous-control and discrete-generation results provide external-validity evidence across policy families and tasks.

The theoretical analysis, controlled experiments, and external task results jointly support the identified far-field negative-feedback mechanism and the value of controlling it.

References for this response

[1] Wenlong Deng, et al. [On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization]

[2] Yize Wu, et al. [Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation]

Weakness: TOPR-related work and theoretical contribution

Theoretically, TOPR identifies the possible unboundedness of the negative branch of signed log-likelihood and provides a theoretical basis for asymmetric weighting. We further study where the policy is ultimately pushed when positive attraction and negative repulsion act together, when it can remain stable, and whether stability necessarily implies good performance.

Section 4.3 quantifies this evolution. Let p,qp,q denote the total strengths of positive and negative feedback, including sample weights and feedback magnitudes, and let ρ=q/p\rho=q/p. In the exponential-family analysis, m+\mathbf m_+ and m−\mathbf m_- are the attraction and repulsion centers. When 0<ρ<10<\rho<1 and the target is attainable, the equilibrium is

m⋆(ρ)=m++ρ1−ρ(m+−m−)\boxed{\mathbf{m}^{\star}(\rho)=\mathbf{m}_{+}+\frac{\rho}{1-\rho}\bigl(\mathbf{m}_{+}-\mathbf{m}_{-}\bigr)}

This equilibrium is locally stable under continuous dynamics and discrete updates with sufficiently small step sizes. With both centers fixed, stronger negative feedback shifts the equilibrium farther in the repulsion direction. We also analyze persistent drift at critical balance and the conditions for instability of finite equilibria when negative feedback dominates.

Thus, repulsion from negative feedback does not imply that overall training must be unstable: positive and negative feedback can form a new stable equilibrium. Conversely, stability does not guarantee good task performance. In the controlled task in Section 6.2, moderate repulsion moves the policy beyond the positive-sample center toward a better location and improves returns on unseen contexts. Stronger repulsion can overshoot the beneficial region and reduce returns even while the policy remains stable.

This analysis provides the control basis for DRPO: per-sample retention of negative updates adjusts the combined attraction and repulsion. The threshold determines where attenuation begins, and its strength determines how quickly far-field influence decreases. This preserves potentially beneficial contributions from nearby negative feedback while reducing excessive repulsion from remote negative feedback. The analysis of evolution, equilibria, stability, and excessive extrapolation is our theoretical contribution beyond identifying an unbounded negative branch or proposing a reweighting rule.

Reviewer_Uwf4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This work studies off-policy RL in which the same historical negative-advantage samples are reused repeatedly as the policy changes.

The authors introduce “remoteness” (the negative log-probability of the action under the current parametric policy). They argue (assuming frozen data and fixed advantages) that repeated negative updates on a fixed sample make it increasingly "remote" without weakening its gradient, creating a self perpetuating loop that pushes gradients arbitrarily large for Gaussian policies, drives suppressed action probabilities toward zero in categorical policies.

To address this, the authors propose Dynamic Remoteness Aware Policy Optimization (DRPO). DRPO leaves positive updates and low-remoteness negative updates unchanged, and exponentially decays negative updates whose remoteness under the current policy exceeds a given threshold.

Empirically, synthetic environments separate remoteness from advantage magnitude and show that removing far field (but not near field) negative updates prevents collapse due to remote samples. On task performance, DRPO achieves the highest scores (in aggregate) among protocol-matched negative-weighting variants (e.g Positive only, Global α\alpha) on D4RL and on structured generation with Qwen2.5-0.5B-Instruct, and is competitive with published offline-RL baselines on D4RL.

Critical strengths

None

Critical weaknesses
  1. (Bad presentation) Paper was hard to follow. Too much notation was introduced, even for simple things, and never reused later. Motivation of the method was highly unclear, and the relationship to existing literature under-emphasised.

  2. (Insufficient novelty) Eq. 1 cites the policy-gradient literature (e.g Sutton et al. 2000), but it defined on off policy samples without importance weights, which would omit a crucial correction term. TOPR (Le Roux et al. 2025) is explicitly reintroduces this correction term on negative advantages (which is what DRPO does as well). The TOPR paper also explores the instability of off-policy negative updates (which would be the key theoretical contribution here).

  3. (Insufficient empirical performance) In practice, the methods seems to yield marginal gains compared to positive advantage only (see Tables 1 and 2 and Figure 6). No confidence intervals in the tables preclude statements about the significance of the results.

Questions for authors
  1. Could the authors clarify whether the far-field instability is distinct from the omission of importance correction?

  2. If it is not distinct, is the justification for DRPO that the data-generating policy is not always known? Alternatively, is there a setting where remoteness-based weighting is preferable to ratio-based weighting even when the data-generating policy is known?

  3. If the justification is that the data-generating policy is not known, wouldn't estimating it from the dataset (e.g behaviour cloning in D4RL, and the frozen initial policy in structured generation), and using it to do clipped importance correction (approximately) achieve the desired result?

5 comments

W1(1): Notation in the main text

Too much notation was introduced, even for simple things, and never reused later.

Thank you for pointing out the reading burden caused by the notation. After revisiting the main text, we agree that some auxiliary notation and repeated definitions can be simplified. In the revision, we will consolidate repeated definitions, inline intermediate variables used only once, and unify the step-size notation to make the presentation more compact.

The principal symbols distinguish different factors in the mechanism analysis. For example, DD denotes remoteness under the current policy, RR denotes the corresponding gradient response, and p,qp,q denote the total strengths of positive and negative feedback. These quantities recur in the subsequent theorems and derivations. Distinguishing them lets us separately analyze feedback magnitude, policy geometry, and aggregate dynamics. We will therefore retain this necessary notation while simplifying auxiliary symbols and improving the explanations at their first introduction.

Motivation, theoretical contribution, and relationship to existing methods

Thank you for asking about our motivation and relationship to prior work. The central question is: when does repeatedly reused historical negative feedback help learning, when does it cause excessive repulsion or instability, and how can its beneficial effects be preserved while its harmful effects are controlled? We therefore study both how individual negative samples affect a policy and how positive and negative feedback jointly determine equilibrium location, stability, and task performance. This analysis motivates DRPO. We clarify the connection to the existing content below.

(1) Is the instability merely caused by omitting importance correction?

Equation (1) analyzes signed actor updates on historical data: positive feedback increases the probability of the corresponding behavior, while negative feedback decreases it. With fixed data and feedback coefficients, this corresponds to a well-defined empirical optimization objective. We study the dynamics of this update rule under repeated reuse of historical samples, rather than treating it as an unbiased gradient estimate of the current policy's expected return.

Importance correction can indeed weaken this source of instability. As the current probability of the same negative sample decreases relative to its historical generating probability, the importance ratio also decreases, reducing its subsequent update influence. This is compatible with the identified mechanism: maintaining strong repulsion after a negative sample has become remote from the current policy can lead to excessive extrapolation or instability. Importance sampling is not the only way to address this problem.

We study an offline setting with zero data refresh and persistent reuse of historical feedback. The policy can continue to change while the data cannot be refreshed accordingly. Section 3.5 of Levine et al. (2020), Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, explicitly notes that large discrepancies between the learned and behavior policies can cause importance weights to degenerate, producing excessive variance in return or gradient estimates. Such estimates are better suited to settings with only limited deviation from the behavior policy.

This difficulty is not confined to products of ratios over long trajectories. Even a single-step ratio can be large when the behavior probability is low and the current probability is high, allowing a few samples to dominate gradient estimates and increase training fluctuations. We therefore study another control route: directly attenuating a negative sample's far-field update influence according to its remoteness under the current policy. The end of Section 5 and Appendix C discuss how this differs from behavior-policy-ratio weighting.

(2) How does DRPO differ from TOPR methodologically and theoretically?

Intuitively, DRPO asks “how unlikely is the current policy to produce this negative sample?”, whereas TOPR asks “how much has its probability decreased relative to the behavior policy?” Both adjust negative updates through per-sample weights, but different quantities determine those weights.

To compare their structures using the same probability definition, let

Dθ=−log⁡πθ,Dμ=−log⁡μ,D_\theta=-\log\pi_\theta,\qquad D_\mu=-\log\mu,

and express DRPO's exponential attenuation in these coordinates:

= \exp\left(-[D_{\theta}-D_{\mu}]_{+}\right)$$ $$w_{\mathrm{DRPO}}^{-} = \exp\left(-k[D_{\theta}-\tau]_{+}\right)$$ Here $k=\lambda/c>0$. TOPR's expression is an exact rewriting of its canonical negative weight $\min\{\pi_\theta/\mu,1\}$. **TOPR's attenuation reference is determined by the behavior probability; DRPO's threshold $\tau$ sets the near/far boundary in current remoteness, independently of the behavior probability.** See [Le Roux et al. (2025), TOPR](https://arxiv.org/abs/2503.14286), Eq. (1) and Algorithm 1. This distinction corresponds to two control routes. **DRPO starts from negative-update gradient dynamics and directly controls far-field repulsion.** With repeated reuse, a historical negative sample can become increasingly unlikely under the current policy while its repulsive influence persists or even strengthens. We therefore reduce its update weight according to current remoteness. Determining how remote a sample currently is requires only the current model, not its historical generating probability. **TOPR uses importance reweighting to simulate the effect of encountering negative samples less often.** Under continued sampling from the current policy, a negative sample whose probability decreases would naturally be encountered less frequently. Fixed historical data can still repeatedly present it. TOPR reduces its negative-update weight using the current-to-historical probability ratio. This adjustment requires the behavior-policy probability. Theoretically, TOPR identifies the possible unboundedness of the negative branch of signed log-likelihood and provides a theoretical basis for asymmetric weighting. We further study **where the policy is ultimately pushed when positive attraction and negative repulsion act together, when it can remain stable, and whether stability necessarily implies good performance.** Section 4.3 quantifies this evolution. Let $p,q$ denote the total strengths of positive and negative feedback, including sample weights and feedback magnitudes, and let $\rho=q/p$. In the exponential-family analysis, $\mathbf m_+$ and $\mathbf m_-$ are the attraction and repulsion centers. When $0<\rho<1$ and the target is attainable, the equilibrium is $$\mathbf{m}^{\star}(\rho)=\mathbf{m}_{+}+\frac{\rho}{1-\rho}\left(\mathbf{m}_{+}-\mathbf{m}_{-}\right)$$ This equilibrium is locally stable under continuous dynamics and discrete updates with sufficiently small step sizes. With both centers fixed, stronger negative feedback shifts the equilibrium farther in the repulsion direction. We also analyze persistent drift at critical balance and the conditions for instability of finite equilibria when negative feedback dominates. Thus, **repulsion from negative feedback does not imply that overall training must be unstable: positive and negative feedback can form a new stable equilibrium. Conversely, stability does not guarantee good task performance.** In the controlled task in Section 6.2, moderate repulsion moves the policy beyond the positive-sample center toward a better location and improves returns on unseen contexts. Stronger repulsion can overshoot the beneficial region and reduce returns even while the policy remains stable. This analysis provides the control basis for DRPO: per-sample retention of negative updates adjusts the combined attraction and repulsion. The threshold determines where attenuation begins, and its strength determines how quickly far-field influence decreases. This preserves potentially beneficial contributions from nearby negative feedback while reducing excessive repulsion from remote negative feedback. **The analysis of evolution, equilibria, stability, and excessive extrapolation is our theoretical contribution beyond identifying an unbounded negative branch or proposing a reweighting rule.**

(3) Is DRPO also performing importance sampling?

The two approaches can produce similar effects, such as reducing the update influence of certain historical negative samples, but their weights have different bases and purposes.

Importance sampling uses the target-to-sampling density ratio to convert expectations over historical data into expectations under the target distribution. DRPO directly sets the retention fraction of negative updates using current remoteness, thereby controlling how remote negative feedback affects the actor.

The formulas above also make their connection explicit: choosing k=1k=1 and setting τ=Dμ\tau=D_\mu separately for each sample recovers TOPR's negative weight. However, DRPO uses a near/far boundary that is independent of the behavior probability, so its general control rule is not equivalent to behavior-policy-ratio correction.

(4) Is DRPO motivated only by an unknown behavior policy?

No. Avoiding the need to fit an unknown behavior policy is a practical advantage of DRPO, but its motivation also includes the dynamical question above and the design of control that directly targets current far-field negative influence.

Even when behavior probabilities are exactly known, “how remote is this sample now?” and “how much has its current probability decreased relative to the behavior probability?” remain different questions. Access to behavior probabilities therefore does not eliminate the relevance of DRPO's analysis or its control criterion.

(5) Why not estimate an unknown behavior policy and then use clipped IS?

Estimating the behavior policy and then using clipped IS is a valid option. Even an exact behavior-policy estimate, however, produces a behavior-relative ratio rather than the current remoteness used by DRPO.

For historical data mixing multiple policies without stored generating probabilities, fitting the behavior distribution also introduces additional modeling work and estimation error. DRPO computes remoteness directly from the current model and avoids this step. The rationale for this route therefore includes both its information requirements and its different control target.

Regarding the frozen initial policy in structured generation, our replay bank contains oracle solutions and model-independent constructed negative responses; it was not sampled entirely from the initial learner. A frozen initial model can provide a reference policy, but it does not automatically equal the data's true generating distribution. The paper already reports joint fitted-reference β\beta-TOPR as a ratio-weighting control with a fitted reference policy; its definition is given in the structured-generation methods appendix.

(6) Why use remoteness control when the behavior policy is known?

A behavior-relative ratio does not directly determine how remote a sample currently is. For example, a negative sample may already have very low probability under the current policy but an even lower probability under the behavior policy. Then πθ/μ>1\pi_\theta/\mu>1, so TOPR retains a negative weight of 1, whereas DRPO attenuates the sample once its current remoteness exceeds the threshold.

Known behavior probabilities therefore do not make the two control rules equivalent. When the goal is to directly suppress current far-field negative influence, DRPO's weights explicitly match that goal: they determine how much of each negative update to retain based on its current remoteness, thereby adjusting aggregate extrapolation and stability. The value of this route does not depend on the behavior policy being unknown.

Reviewer_YPdsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how repeatedly reusing negative-advantage samples can push a policy away from historical actions and destabilize learning. It proposes DRPO, which downweights negative updates when the corresponding actions are already unlikely under the current policy.

Critical strengths

S1: The analysis clearly separates advantage magnitude from policy-score magnitude and distinguishes Gaussian gradient amplification from bounded categorical suppression.

S2: The controlled interventions support the proposed mechanism.

Critical weaknesses

W1: The theory assumes fixed data and advantages and analyzes direct policy-coordinate updates without importance correction. Its relevance to practical off-policy learning with changing critics, shared neural parameters, and data refresh remains unclear.

W2: The external experiments do not directly establish that the proposed mechanism causes training instability. Gradient probes show a near-to-far pattern, but the causal interventions are confined to controlled environments. The modest structured-generation gains also lack uncertainty estimates in the main table.

Questions for authors

Q1: Does the predicted instability persist when advantages are refreshed or off-policy corrections are applied? What are the observed (q/p) trajectories in the external experiments?

Q2: Does removing or capping far-field negative updates rescue D4RL or structured-generation training when total negative-gradient magnitude is matched?

9 comments

We thank the reviewer for carefully reading our paper and for the specific comments on the connection between the theoretical analysis and practical training, external evidence for the mechanism, and uncertainty in the experimental results. We address these points below.

W1: Relevance of the theoretical setting to practical off-policy learning

The theory assumes fixed data and advantages and analyzes direct policy-coordinate updates without importance correction. Its relevance to practical off-policy learning with changing critics, shared neural parameters, and data refresh remains unclear.

Thank you for pointing out the need to clarify the connection between our analytical setting and practical training. We study how repeatedly reusing historical negative feedback, while the current policy continues to change, affects updates through remoteness under the current policy and can lead to repulsion, drift, or support contraction. The fixed-feedback analysis, gradient diagnostics on neural policies, controlled interventions, and external task evaluations provide, respectively, an analytically tractable mechanism, evidence from actual gradients, causal evidence, and evidence of task performance. We address the four aspects you raised below.

(1) Fixed data and data refresh

Thank you for asking about the connection between fixed-data training and practical applications. Learning from fixed historical data has clear applications: offline RL in continuous action spaces learns policies from existing datasets, and language-model SFT also typically uses previously collected data. Collecting a batch of feedback online and then performing multiple rounds of offline optimization is another form of historical-data reuse. Fixed-data training is therefore a practical application setting that we deliberately study.

For example, conventional SFT primarily imitates selected target responses and does not explicitly use the negative feedback provided by incorrect responses. Our Positive-only control in structured generation learns only from correct completions, corresponding to an SFT-style supervised likelihood objective. In terms of training form, the updates we study introduce positive and negative feedback supplied by rewards or advantages into supervised likelihood learning, and then control the influence of negative feedback using current remoteness. Retaining only positive examples with a uniform positive weight recovers supervised likelihood training on those examples. Retaining controlled negative weights additionally uses the repulsive information in incorrect responses. Our comparisons with Positive-only support this route to improving learning with negative feedback. The conclusion also identifies extending language-model SFT with remoteness-controlled, verified negative completions as a promising application for further study.

Fixed data also define a meaningful limiting setting. On-policy learning requires the sampling policy to match the current policy, and timely data refresh maintains this match. As historical samples are retained and reused for longer, they may continue to contribute updates after the policy has changed substantially. Along the data-refresh-frequency axis, completely eliminating refresh gives the zero-refresh limit, an extreme case of off-policy reuse. We choose this setting to expose the cumulative effects of persistently reused historical negative feedback and to test whether this instability pathway can be controlled without assistance from data refresh. The original introduction and conclusion explicitly state this zero-refresh-limit positioning.

The policy continues to change even when the data are fixed. The probability, remoteness, and gradient response of the same historical action under the current policy all change during optimization. Repelling a negative example can make it more remote without necessarily weakening its update influence, so repeated reuse can sustain repulsion. Fixed data allow us to identify this mechanism and its cumulative consequences while holding sample quality and feedback composition unchanged.

DRPO's control principle does not require the data to remain fixed: at each update, it adjusts the influence of negative feedback using the sample's remoteness under the current policy. The same rule applies to samples from a fixed dataset, a continuously refreshed replay buffer, or an earlier policy. Thus, the zero-refresh setting fully exposes the mechanism and supports the theoretical analysis, while controlling remote negative feedback using current remoteness also applies to more general off-policy training with finite reuse, replay, or stale policies. The conclusion explicitly states that remoteness-aware control is not limited to static offline data.

(2) Fixed advantages and changing critics

Thank you for asking about the relationship between fixed-advantage analysis and changing critics. The remoteness used by DRPO is decoupled from advantage estimation: for a given sample and current policy, remoteness is computed directly from the current policy, without depending on the critic's output or the numerical advantage. A critic or another feedback source supplies the learning signal, whereas DRPO uses current remoteness to adjust the update influence of negative feedback. DRPO therefore requires no particular critic and can also be used without a learned critic.

The far-field mechanism also does not presuppose critic estimation error. Repulsion from remote negative examples can persist even when the advantages are exact. Fixed-advantage analysis holds the feedback coefficients unchanged to isolate the geometric effect of policy evolution: repeatedly reusing the same historical negative example can push it farther away without necessarily weakening its gradient response. This separates changes in advantage estimates from the effect of remoteness.

The existing experiments provide concrete evidence. The D4RL experiments use critic-derived advantages; the external gradient diagnostic first fits the critic and then computes and freezes the advantages. Structured generation instead uses positive and negative feedback directly from a verifier, without training a learned value critic. These settings show that DRPO can be combined with critic-derived advantages and used in critic-free feedback training. The feedback sources and training settings are described in the experimental section and the corresponding appendices.

If advantages are recomputed after critic updates, the feedback coefficients received by the actor, including their signs, may change. DRPO can directly use the refreshed feedback and continue to adjust the negative branch using current remoteness. Fixed advantages isolate the mechanism for analysis; the algorithm itself can be combined with advantage refresh.

(3) Direct policy coordinates and shared neural parameters

Direct policy coordinates make the geometric and dynamical relationships analytically tractable. Actual neural-policy training uses full parameter gradients. Let uθ(s)u_\theta(s) denote the network's output means or logits. The chain rule gives

∇θlog⁡πθ(a∣s)=Jθ(s)⊤∇ulog⁡πu(a)\nabla_\theta \log \pi_\theta(a\mid s) = J_\theta(s)^\top \nabla_u \log \pi_u(a)

The full-parameter gradient diagnostic in the paper measures precisely this gradient, including the network Jacobian. The D4RL diagnostic matches the absolute advantages of negative samples, while the structured-generation diagnostic fixes the negative-feedback coefficient. These diagnostics test whether gradient responses in actual neural policies still exhibit a near-to-far pattern with remoteness. The controlled interventions further test whether far-field updates transmit into policy drift and training instability.

DRPO's effect also extends directly to full parameter gradients. Under the stop-gradient definition in Section 5, the following exact relationship holds for the same negative sample:

gθ,DRPO=ω(Dθ),gθ,raw,gθ,raw=−A^τ∇θlog⁡πθ(a∣s).g^{\theta,\mathrm{DRPO}}=\omega(D_{\theta})\\,g^{\theta,\mathrm{raw}},\qquad g^{\theta,\mathrm{raw}}=-\widehat{A}^{\tau}\nabla_{\theta}\log\pi_{\theta}(a\mid s).

DRPO therefore attenuates the full per-sample gradient, including the shared network Jacobian. The output-coordinate analysis identifies the mechanism, while the neural-gradient diagnostics and interventions examine its practical manifestation. Together, these components connect the theory to neural-policy training.

(4) Absence of explicit importance correction

Thank you for asking about importance correction. Optimizing a policy using off-policy data does not require every actor update to explicitly include an importance ratio. Fitting a policy to historical data and obtaining an unbiased estimate of the current policy gradient from historical data are different optimization tasks. AWR provides a direct precedent: it uses the advantage-weighted policy-fitting objective

\mathbb E_{(s,a)\sim\mathcal D} \left[ \exp\!\left(\frac{\widehat A(s,a)}{\beta}\right) \log\pi_\theta(a\mid s) \right],$$ rather than explicitly multiplying actor updates by $\pi_\theta/\mu$. Section 3.1 explicitly defines Eq. (1) as an empirical signed actor field. With the historical distribution $\nu$ and feedback $\widehat A$ fixed, it is the gradient of the empirical signed log-likelihood objective $$g_{\theta,\mathrm{DRPO}}^{-} = \omega(D_{\theta})\,g_{\theta,\mathrm{raw}}^{-}, \qquad g_{\theta,\mathrm{raw}}^{-} = -\widehat{A}^{-}\nabla_{\theta} \log\pi_{\theta}(a\mid s).$$ Thus, the absence of an explicit ratio is not itself an omission in the definition of this empirical update. This object of analysis lets us directly study the actor dynamics produced by repeatedly reusing historical positive and negative feedback. The relationship between IS and our method is discussed in Section 2, the end of Section 5, and Appendix C. IS corrects the distribution using probabilities relative to the behavior policy; DRPO selectively weakens remote negative feedback using remoteness under the current policy. Both change sample influence, but their control criteria differ. DRPO does not require access to or estimation of the behavior-policy density. Raw IS can suffer from high variance. When an action has a low probability or density under the behavior policy but a high probability or density under the current policy, even a single-step importance ratio can be large. Products of ratios over long trajectories can further amplify variance. This high variance can increase fluctuations in gradient estimates and make training less stable, providing a practical motivation for optimization approaches that do not depend on an explicit ratio. As explained in our response to W1(1), we study the zero-refresh limit, where historical data are continuously reused without any refresh, representing an extreme off-policy setting. Section 3.5 of Levine et al. (2020), [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://arxiv.org/pdf/2005.01643), explicitly discusses the difficulties of importance sampling in fully offline training: when the learned policy differs substantially from the behavior policy, the importance weights degenerate, making the variance of return or gradient estimates too large for effective use. The paper consequently notes that importance-sampling estimates are most suitable when the learned policy deviates only modestly from the behavior policy. This provides a direct literature basis for studying control without explicit importance sampling in our setting. We therefore study another form of control based on current-policy geometry, attenuating remote negative updates directly through remoteness. We address how appropriate importance correction changes the uncorrected instability pathway in Q1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.