acceptodds
Under review as a conference paper at ICLR 2027

Directional-Clamp PPO

Abstract

Proximal Policy Optimization (PPO) is one of the most widely used algorithms in deep reinforcement learning. Its objective encourages the importance ratio between the current and behavior policies to move in the "right direction”, i.e., increasing the ratios for positive advantages and decreasing them for negative ones. To prevent overly aggressive updates in the right direction, PPO and its variants employ clipping mechanisms that restrict the magnitude of such updates. However, due to the stochastic nature of the training process, the importance ratios frequently move in the "wrong direction” during PPO optimization. We identify the strictly wrong direction updates as a key but largely overlooked factor that limits the performance of PPO. To address this, we propose the Directional-Clamp PPO algorithm (DC-PPO), which further penalizes the actions going to the "strict wrong direction” regions, where the advantage is positive (negative) and importance ratio falls below (above) (), for a tunable parameter . The penalty is by enforcing a steeper loss slope, i.e., a clamp, in those regions. We demonstrate across a range of environments that DC-PPO consistently outperforms PPO and its variants that modify the objective's behavior in the right direction. Moreover, we show, both theoretically and empirically, that DC-PPO better avoids strict wrong direction updates while keeping the importance ratio closer to . Finally, we prove global convergence for tabular DC-PPO with direct policy parameterization.

Primary area:
reinforcement learning
Keywords:
actor-critic algorithms, proximal policy optimization, reinforcement learning, policy gradient methods

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UZJ9OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They modify PPO’s surrogate so that ratios moving well past 1 in the “wrong” direction, against the sign of the advantage, get a steeper slope, which effectively increases the gradient for those samples. This is supported with a one step lemma showing the update keeps such ratios closer to 1 than PPO, a tabular setting convergence proof, and comparisons against PPO and variants on mujoco+atari.

Critical strengths
  • The idea is simple, cheap, and can be drop-in. It’s a few lines of code with no extra compute. and one fixed setting (α = 3, β = ε) is used everywhere. if the gains hold up, people would actually use it.
  • The Atari evaluation is reasonably broad, and the authors use appropriate quantificaiton( stratified bootstrap CIs).
Critical weaknesses

No major weaknesses that would prevent adoption, (given that results hold for other researchers).

Questions for authors

Why do Tables 3 and 7 disagree for nominally identical runs? which is correct?

1 comment

We thank the reviewer for his positive feedback. Below we address the reviewer's question.

Questions

Why do Tables 3 and 7 disagree for nominally identical runs? which is correct?

Table 3 shows the performance comparisons on MuJoCo using 10 seeds whereas Table 7 shows the same performance comparison on 5 seeds when we compare all variants considered in the paper to DC-RB and DC-Leaky (each evaluated over the same 5 seeds).

Reviewer_6QXDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This proposed DC-PPO, a modified version of PPO for DRL. The key point is to address the limitation that a significant portion of action probabilities still fail to steer policy updates in the right direction by explicitly penalising “strictly wrong direction” updates via a modified surrogate objective, which introduces directional penalties only when updates deviate significantly from the intended direction. Both theoretical and empirical results suggest that DC-PPO outperformed PPO.

Critical strengths

Hot topic and simple but good ideas.

Good results in both theoretical and practical aspects.

Critical weaknesses

No statistical significance tests found in the paper. DRL is a stochastic paradigm, meaning that the same method with the same architecture and parameter values will generate different results in different independent runs for each experiment. This point is a major concern, although I doubt doing so can significantly change the results or conclusions, but not doing so the results are scientifically incorrect.

Questions for authors

Could you please perform multiple independent runs for each experiment, and report the means and standard deviations with statistical significance tests in the results?

It is very strange to present the related work between the theoretical results and empirical results. Could you please move it to the background as part of the literature survey (existing work)?

4 comments

We thank the reviewer for his efforts. Below we address the reviewer’s main concerns and questions.

Weaknesses

The reviewer claims that no statistical significance tests found in the paper. However, we included a statistical significant test in Appendix B.2 of the paper, showing that the DC-PPO yields a statistically significant improvement over PPO with p < 0.05 under paired statistical test, we refer to these experiments in line 433.

We add here our main findings. Following recent work [1] we measure the probability of DC-PPO’s improvement compared to PPO. To compute the probability of improvement for a task m for algorithms X and Y with N=5 and K=5 runs respectively, we use the Mann-Whitney U-statistic (see Eq. A.2 in Appendix A.7 of [1]). More specifically, P(Xm>Ym)=125∑i=15∑j=15S(xm,i,ym,j)\mathbb P(X_m>Y_m)=\frac{1}{25}\sum_{i=1}^{5}\sum_{j=1}^{5}S(x_{m,i},y_{m,j}) where S(x,y)=1S(x,y)=1 if y<xy<x, S(x,y)=12S(x,y)=\frac{1}{2} if y=xy=x, and 00 otherwise, and xm,i,ym,jx_{m,i},y_{m,j} denote the average reward over the final 100 evaluation episodes of runs i and j on environment m, respectively.

Across 39 Atari environments, the estimated probability that a randomly selected DC-PPO run outperforms a randomly selected PPO run is 60.41%, with a 95% stratified-bootstrap confidence interval of [54.77%,65.95%]. This improvement is statistically significant at the 95% confidence level and is unlikely to be explained solely by random-seed variability. DC-PPO obtains a probability of improvement above 50% on 29 of the 39 environments, see our Table 5 in Appendix B.2.

Questions

  1. Following the reviewer’s request we additionally report here the means with the standard deviations for MuJoCo environments, evaluated over 10 random seeds.

Last-10 average return (mean ± standard deviation)

EnvironmentPPOPPO-RBLeaky PPODC-PPO
Ant-v45713.90 ± 367.825888.53 ± 412.975735.24 ± 400.735521.60 ± 525.93
Hopper-v4852.98 ± 251.191015.99 ± 22.60705.10 ± 380.351230.66 ± 541.28
HalfCheetah-v45833.58 ± 443.545614.77 ± 462.205802.79 ± 842.796110.72 ± 588.53
Reacher-v4-4.76 ± 0.30-4.68 ± 0.35-4.73 ± 0.32-4.64 ± 0.25
Walker2d-v43432.08 ± 910.513362.96 ± 816.013809.17 ± 523.903831.43 ± 720.09
Swimmer-v4219.23 ± 101.42265.16 ± 87.64241.02 ± 96.73317.90 ± 22.45
Humanoid-v4932.01 ± 71.35914.02 ± 89.02874.24 ± 91.231154.80 ± 270.64

Top-10 average return (mean ± standard deviation)

EnvironmentPPOPPO-RBLeaky PPODC-PPO
Ant-v46439.55 ± 294.546563.56 ± 307.346469.42 ± 279.126488.28 ± 207.75
Hopper-v4998.72 ± 43.821020.64 ± 25.971009.99 ± 14.561299.70 ± 630.78
HalfCheetah-v46085.68 ± 531.445768.28 ± 489.256019.54 ± 851.896232.81 ± 687.20
Reacher-v4-3.69 ± 0.82-4.00 ± 0.97-3.89 ± 0.83-3.85 ± 0.85
Walker2d-v43563.23 ± 927.573724.54 ± 453.744022.19 ± 438.264209.62 ± 529.29
Swimmer-v4220.62 ± 100.00269.12 ± 86.76243.86 ± 95.02319.29 ± 24.64
Humanoid-v41079.52 ± 251.961091.90 ± 374.951091.53 ± 375.181349.32 ± 423.58
  1. Following the reviewer’s request we will move the related work section to the background as part of the literature survey.

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

We would be happy to address any additional questions or suggestions the reviewer may have.

Thanks the authors for providing responses to my comments and suggestions. Looking at the details on the experiments and results regarding the statistical significance test, it seems that the standard deviations are very big --- this is very likely because you only did 10 independent runs. The responses did not state which statistical significance test was done. So I am not entirely sure how reliable the results are.

I have to say, you should have place the statistical significance test results in the main paper instead of the appendix. I fully understand that there is a page limit issue, but the statistical significance test results must/should be included in the main text --- without these, the results are scientifically not correct.

Before appropriate statistical significance test results over 30 independent runs were carried out, it is very hard to make judgement. So at this stage, I won't change my score.

One point: the PC chair identified the first reviewer might be an AI tool instead of a real reviewer. But that AI tool or false reviewer gave a negative score of weak reject, which seems a bit low to me. So if the authors provide more evidence of my comments above, also considering the PC Chair's request, I am happy to to change my score.

We thank the reviewer for the follow up.

Looking at the details on the experiments and results regarding the statistical significance test, it seems that the standard deviations are very big --- this is very likely because you only did 10 independent runs.

The large stochasticity over continuous control tasks, and in particular in MuJoCo environments, has been reported in prior works [1,2]. Moreover, different MuJoCo environments exhibit varying degrees of instability, for instance, it has been reported in [3] that Half-Cheetah has a much more stable dynamics model, whereas Hopper, on the other hand, is prone to quick failure modes which introduce larger external variance. We are not sure whether additional runs alone would substantially reduce this inherent stochasticity, although they would of course provide a more precise estimate of the performance distribution.

The responses did not state which statistical significance test was done. So I am not entirely sure how reliable the results are.

Following the earlier question, our statistical analysis follows the methodology in [4], which was specifically proposed for reliable evaluation in deep reinforcement learning under a limited number of runs. After estimating the probability of improvement using the equation from earlier response, we then estimate the uncertainty of this aggregate probability using a stratified bootstrap across the 39 Atari environments, from which we obtain the reported 95% confidence interval of [54.77%,65.95%].

Before appropriate statistical significance test results over 30 independent runs were carried out, it is very hard to make judgement.

We would like to highlight that the statistical significance test over 39 Atari environments was already present in the manuscript of the initial submission, and across these environments, the estimated probability that a randomly selected DC-PPO run outperforms a randomly selected PPO run is 60.41%, with a 95% stratified-bootstrap confidence interval of [54.77%,65.95%] which is unlikely to be explained solely by random-seed variability. Since this confidence interval lies entirely above 50%, the observed improvement is statistically significant at the 5% level. In particular, the resulting bootstrap confidence interval tests H0:P(DC−PPO>PPO)≤0.5H_0:\mathbb P(\mathrm{DC-PPO}>\mathrm{PPO})\leq 0.5 against H1:P(DC−PPO>PPO)>0.5.H_1:\mathbb P(\mathrm{DC-PPO}>\text{PPO})>0.5.

For the camera ready version, we will expand the discussion on the statistical significance test results, when another page would be available.

We thank the reviewer for his efforts.

[1] Andrychowicz, Marcin, et al. "What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study." In ICLR 2021.

[2] Henderson, Peter, et al. "Deep reinforcement learning that matters." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

[3] Islam, Riashat, et al. "Reproducibility of benchmarked deep reinforcement learning tasks for continuous control." In ICML 2017.

[4] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

Reviewer_7jTGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Directional-Clamp PPO (DC-PPO), an extension of PPO-Clip that introduces a third surrogate objective term to increase the slope of the loss function when the importance ratio enters a “strict wrong-direction” region. Specifically, this region is defined as ωθ(s,a)<1−β\omega_\theta(s,a) < 1-\beta when the estimated advantage A^(s,a)\hat{A}(s,a) is positive, and ωθ(s,a)>1+β\omega_\theta(s,a) > 1+\beta when the estimated advantage A^(s,a)\hat{A}(s,a) is negative. The authors argue that such ratios frequently occur during PPO optimization and report that DC-PPO reduces the proportion of samples falling within these strict wrong-direction regions while achieving higher returns than PPO, Leaky PPO, and PPO-RB across seven MuJoCo tasks and several Atari tasks. The paper also establishes a local ratio-comparison lemma and claims to prove global convergence in a tabular setting with direct policy parameterization. Although the empirical results are promising, the precise relationship between the proposed objective function and its claimed activation region, as well as the validity of the convergence proof, requires further clarification.

Critical strengths
  1. Simple and targeted modification. DC-PPO modifies only the surrogate objective function without requiring changes to the data collection pipeline or second-order optimization. If the intended activation region is clearly defined, the method offers a practical approach to studying and mitigating substantial drifts of importance ratios in the wrong direction.

  2. Broad empirical evaluation with diagnostic metrics. The paper evaluates DC-PPO across multiple MuJoCo tasks and Atari environments, compares it with PPO and two variants that modify the surrogate objective, and reports the proportion of samples falling within the strict wrong-direction regions. These diagnostic metrics help link the proposed mechanism to actual training behavior, rather than reporting only final returns.

Critical weaknesses
  1. The objective function does not unconditionally implement the claimed penalty that applies “only in the strict wrong-direction region.” In Equation (3), fDCf_{\mathrm{DC}} is defined as a linear function and is included in the minimum together with the PPO terms. The equation does not explicitly gate this term off outside the strict wrong-direction region. For positive advantages, if β>ϵ\beta > \epsilon, then at ωθ(s,a)=1−β\omega_\theta(s,a) = 1-\beta, the additional term evaluates to 1−β1-\beta, which is smaller than the PPO-clipped value 1−ϵ1-\epsilon. Therefore, the additional term can become the active minimum at the boundary of, and potentially outside, the claimed strict wrong-direction region. An analogous issue arises for negative advantages. The authors should either explicitly impose and justify a constraint such as β≤ϵ\beta \leq \epsilon across all experiments, or define the additional term piecewise to ensure that it does not modify the objective outside the intended region.
  2. In Appendix D, Equation (7) expresses the gradient using the sign of the advantage, whereas differentiating the proposed DC-PPO objective in Equation (3) generally yields terms proportional to the actual advantage values. This discrepancy raises concerns about whether the convergence analysis corresponds to the proposed optimization objective. Furthermore, the derivation of Equation (8) appears to assume that the selected positive-advantage action is included in the current optimization batch, while Assumption 1 only guarantees that each state-action pair is visited infinitely often, rather than at every iteration. Consequently, the probability lower bound derived from Equation (8) is not sufficiently justified under the stated assumptions. The authors should clarify the gradient formulation and provide a rigorous derivation of the probability lower bound to establish the validity of the global convergence analysis.
  3. The global convergence proof appears to contain a potentially consequential gap. Around Equations (8)-(9) in Appendix D, the proof states that its unique minimizer is z=ηz=\eta, and subsequently lower-bounds it. However, on the stated domain z=π(a∣s)∈(0,1]z=\pi(a\mid s)\in(0,1], this lower bound does not hold when 0<η<10<\eta<1. In this case, the function attains its minimum at z=ηz=\eta, where its value is
f(η)=ηeηe+1,f(\eta)=\frac{\eta e}{\eta e+1},

which can be smaller than exp⁡(η)exp⁡(η)+1\frac{\exp(\eta)}{\exp(\eta)+1}. Thus, the proof appears to require an additional condition on η\eta or a corrected lower bound together with a revised subsequent argument. Since the convergence theorem relies on a uniform positive lower bound, this issue is more than a minor technical detail.

Questions for authors

See Weaknesses above.

2 comments

We thank the reviewer for the thoughtful comments. Below we address the reviewer's weaknesses.

Weaknesses

  1. We respectfully disagree with the reviewer’s claim that the objective function does not unconditionally implement the claimed penalty. In the counterexample presented by the reviewer, at w=1−βw=1-\beta if β>ϵ\beta>\epsilon then indeed 1−β<1−ϵ1-\beta<1-\epsilon. However, at that point fDC(w)=w=1−βf_{\mathrm{DC}}(w)=w=1-\beta, whereas standard PPO gives min⁡wA,(1−ϵ)A=(1−β)A\min\\{wA,(1-\epsilon)A\\}=(1-\beta)A, because A>0A>0 and 1−β<1−ϵ1-\beta<1-\epsilon. So DClamp has not changed PPO at all there. The reviewer compared the DClamp term only with the clipped term and forgot the first term wAwA in the minimum. More generally, fDC(w)−w=(α−1)(w−(1−β))f_{\mathrm{DC}}(w)-w =(\alpha-1)\bigl(w-(1-\beta)\bigr). Therefore, {fDC(w)<w,w<1−β,fDC(w)=w,w=1−β,fDC(w)>w,w>1−β.\begin{cases} f_{\mathrm{DC}}(w)<w, & w<1-\beta,\\\\ f_{\rm DC}(w)=w, & w=1-\beta,\\\\ f_{\rm DC}(w)>w, & w>1-\beta. \end{cases} Because A>0A>0 multiplying by AA preserves the inequalities. Hence the extra term can beat wAwA in the minimum only when w<1−βw<1-\beta which is exactly the strict wrong-direction region defined in the paper. Outside that region, fDC(w)A≥wAf_{\mathrm{DC}}(w)A\geq wA, so it cannot change the minimum because wAwA is already one of the candidates.

  2. The reviewer is correct that the gradient in Appendix D, Equation (7) uses only the sign of the advantage and differentiating Equation (3) generally yields terms proportional to the actual advantage estimate. However, note that as the positive advantage magnitude weights only rescale the effective EMDA step sizes, we may simply set these weights to one, without loss of generality as in [1], Equation 150 in Appendix E. Moreover, by Assumption 1, the selected positive-advantage state-action pair (s,a’)(s,a’) is contained in the current optimization batch D(t)\mathcal D^{(t)} for infinitely many iterations tt, almost surely, so our lower-bound in Equation 8 applies for infinitely many tt. This gives us a contradiction, from one hand we know that ∑a∈A−∞(s)π(t)(a∣s)≥c>0\sum_{a\in\mathcal A_-^\infty(s)} \pi^{(t)}(a\mid s) \geq c>0 for infinitely many tt by Equation 10. On the other hand, we showed that ∑a∈A−∞(s)π(t)(a∣s)→0\sum_{a\in \mathcal A_-^{\infty}(s)} \pi^{(t)}(a\mid s)\to0 as t→∞t\to\infty. Thus, a sequence converging to zero cannot stay above a fixed c>0c>0 infinitely often. In the revision, we clarified in line 1339 that the positive advantage magnitudes can be absorbed into the effective EMDA step sizes, allowing these weights to be set to one without loss of generality. We also explicitly defined in line 1391 the set Ts,a=t:(s,a)∈D(t)\mathbb T_{s,a}=\\{t:(s,a)\in\mathcal D^{(t)}\\} and restricted the relevant lower bounds to these iterations.

  3. We are not sure where exp⁡(η)exp⁡(η)+1\frac{\exp{(\eta)}}{\exp{(\eta)}+1} appears as a lower-bound in our derivations, we would be happy to correct this typo. Indeed, as noted by the reviewer the function attains its minimum at z=ηz=\eta where its value is f(η)=ηeηe+1f(\eta)=\frac{\eta e}{\eta e+1} as we explicitly write in Equation 9.

[1] Huang, Nai-Chieh, et al. "PPO-clip attains global optimality: Towards deeper understandings of clipping." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 11. 2024.

We would be happy to address any additional questions or suggestions the reviewer may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.