acceptodds
Under review as a conference paper at ICLR 2027

ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields

Abstract

Proximal Policy Optimization (PPO) dominates reinforcement learning and LLM alignment, yet its hard-clipping mechanism and unconstrained alternatives (e.g., SPO) sit at two extremes of a stability, efficiency dilemma. We argue that this dilemma is best understood dynamically: a surrogate objective is a feedback law on the probability ratio, and its clipping/penalty shape defines a gain field that drives the update dynamics. PPO's clip induces a dead zone, zero feedback outside the trust region, leaving the policy to drift open-loop under momentum; SPO's quadratic penalty induces an unbounded, linearly growing gain that stiffens the dynamics and destabilizes under aggressive step sizes. Guided by this view, we derive **Anchored Neighborhood Optimization (ANO)**, which designs the gain field directly: a shaping kernel that anchors the identity map at , peaks exactly at a prescribed trust-region boundary , bounds the push on severely off-policy samples by a tunable , and exerts a bounded, redescending pull of tunable depth on extreme outliers. The three hyperparameters are fully decoupled and all internal constants are solved in closed form. Empirically, ANO ranks first on both Atari (40 games) and MuJoCo in IQM and Median of normalized scores, while the runner-up differs across domains (PAPO on Atari, SPO on MuJoCo), ANO is the only method consistently at the top. Under a learning-rate stress test (), ANO degrades by only whereas PPO collapses by , and the stressed ANO still outperforms PPO and PAPO running at their best-tuned learning rates.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, PPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dm1tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Anchored Neighborhood Optimization (ANO), a drop-in replacement for PPO’s clipped surrogate. The authors interpret the derivative of a ratio-shaping function f′(r) as a “gain field” acting on the probability ratio. Under this view, PPO has a zero-gain dead zone beyond 1+ϵ, and SPO has an unbounded, linearly growing restoring gain. ANO instead constructs a C∞ shaping kernel directly in the derivative domain, with three hyperparameters: ϵ sets the zero-crossing, κ₊ sets the left asymptotic gain, and κ₋ sets the minimum (pull) gain. The gain is bounded and decays to zero for large ratios, in the spirit of redescending M-estimators. All internal constants are obtained in closed form.

On the theory side, the paper gives a two-sided log-ratio lower bound (Thm 3.1) and a monotonic-improvement result for a hard-constrained version of the generalized objective (Thm 3.4). It also proves that the ANO kernel satisfies the required enclosure property (Lemma 4.4). Empirically, ANO ranks first in IQM and Median on Atari (40 games) and MuJoCo (6 tasks) with 5 seeds. It degrades by 0.9% under a 3e-4 → 1e-3 learning-rate increase on MuJoCo, versus 54.5% for PPO. It also matches SPO and outperforms PPO in a single-run experiment judged by an LLM.

Critical strengths
  1. Robustness to aggressive step sizes. The learning-rate stress test (Table 10) is the most informative result in the paper. PPO collapses (−54.5% IQM), while ANO is almost unaffected (−0.9%), and the stressed ANO still exceeds the best-tuned PPO at the nominal learning rate. If this robustness generalizes, it is a practically useful property.

  2. Clean, cheap construction and a useful diagnostic. The kernel has a closed-form parameterization with no iterative solve, and it is easy to drop into existing PPO code. The out-of-boundary diagnostic (Table 9, Fig. 6) makes concrete how many samples receive zero gradient under PPO: about 7.9% on average on Atari, and up to about 20% on Asterix.

Critical weaknesses
  1. Limited novelty, and the specific kernel is not justified against simpler alternatives. Smooth or soft clipping and restoring-force variants of PPO are a crowded design space. For example, the rollback operation in Truly PPO (Wang et al., 2020) already applies a bounded negative gain beyond the clipping boundary; the paper cites this work but does not compare against it. The "gain field" perspective is largely a reframing of the per-sample gradient weight f′(r)f'(r). The authors themselves note that the scalar flow does not imply convergence in parameter space, so the framework provides motivation rather than new guarantees. Likewise, Theorems 3.1 and 3.4 apply to a hard-constrained objective that ANO does not optimize, so they do not certify the proposed algorithm. Most importantly, there is no ablation showing that this particular rational-exponential kernel matters, rather than any bounded, smooth, redescending function, or even a non-redescending bounded one such as rollback. Without such an ablation, the paper's central design claims remain untested.

  2. Empirical gains are small relative to the uncertainty, and the evaluation is thin where it matters.

    • On Atari, ANO and PAPO are statistically indistinguishable (IQM 0.935 vs. 0.929; paired 95% CI [−0.10, 0.11]).
    • On MuJoCo, the IQM confidence intervals of ANO, SPO, and PPO overlap substantially ([0.88, 0.99], [0.85, 0.98], [0.84, 0.98]).
    • ANO's reported configuration was selected from 6 settings (2 (κ+,κ−)(\kappa_+, \kappa_-) pairs × 3 ϵ\epsilon values), whereas PPO and SPO each had 3. With margins this small, best-of-sweep selection on final scores may matter.
    • The claim that the three knobs are "decoupled" is never tested: the two (κ+,κ−)(\kappa_+, \kappa_-) settings vary both parameters together.
    • The robustness result, which is the strongest evidence for the paper's thesis, rests on a single elevated learning rate, a single domain (MuJoCo, 6 tasks), and only three methods.
    • The RLHF experiment uses one run per method, as the authors acknowledge.
Questions for authors
  1. How does ANO compare with (a) PPO with rollback (Truly PPO) and (b) a simpler bounded, smooth kernel, such as a smoothed clip or a tanh-based restoring term? This would show whether the specific functional form or the redescending property is responsible for the gains. (Could change my assessment.)
  2. Can you vary κ+\kappa_+ and κ−\kappa_- independently to support the decoupling claim, and report sensitivity to each?
  3. Can the learning-rate stress test be extended to several learning rates (e.g., a sweep up to the point where ANO itself fails), to Atari, and to PAPO and TRPO? Does the robustness persist when ANO and PPO use the same ϵ\epsilon? Currently ANO uses ϵ=0.3\epsilon = 0.3 and PPO uses ϵ=0.2\epsilon = 0.2 in Table 10.
  4. Given the unequal tuning budgets, how do the results change if all methods are compared at a fixed, shared ϵ\epsilon, or under an equal number of configurations?
4 comments

Thank you for the detailed review. Point-by-point responses follow.

Q1 (novelty: TrulyPPO, simpler kernels, and property attribution). You asked for (a) a direct comparison with TrulyPPO and (b) a simpler bounded smooth kernel to separate form from property. Table R1 contains both, plus one step further. TrulyPPO realizes the bounded pull without redescending relaxation (the rollback penalty never releases). SmoothClip is the smoothed-clip kernel: bounded and smooth, with the pull removed entirely. BumpPull goes one step further than (b): it adds a tanh-style restoring term carrying a redescending pull, but as a crude local patch rather than an integrated design. All three were evaluated on 66 MuJoCo tasks ×5\times 5 seeds under the paper's protocol:

AlternativePull?Redescending?ImplementationIQMDiverged
TrulyPPO (officially recommended config)yes, but persistent: rollback penalty never relaxesnoprior art0.46130/30
SmoothClipno (pure redescending push)yesclean by design0.45390/30
BumpPullyes, a local patchyes (push plus patch)crude: sech2\mathrm{sech}^{2} dip, depth-matched to ANO0.8214 survivors / 0.4811 all9/30
ANOyes, integratedyesclosed-form0.93040/30

Each alternative fails in the direction of its missing ingredient. The persistent pull (TrulyPPO) scores 0.4613, and a KLRANGE sweep (66 configurations ×30\times 30 runs, including the officially recommended setting and a slope variation) peaks at 0.70: tuning does not close the gap. The pure push (SmoothClip) scores 0.4539. The patched pull (BumpPull) diverges on 9/30 runs (policy NaN; all Humanoid, 3/5 HalfCheetah, 1/5 Ant), and with diverged runs scored at 0 its IQM is 0.48, on par with removing the pull entirely. The defect is localized to a single feature of the gain shape: BumpPull's gain rebounds through zero into positive values (r∈(1.75,3.5)r\in(1.75,3.5)) after the pull lobe, pushing overshooting samples out of the recoverable band, whereas ANO's gain redescends to zero monotonically from below and overshooting samples are always pulled back (gain profiles: Figure 2 of the revised PDF).

For attribution, the two kernels share ANO's push side and advantage-branch construction (formulas below), so each failure isolates the missing ingredient rather than any implementation detail. The two kernels are fully specified (same hyperparameters as ANO's MuJoCo configuration, ε=0.3\varepsilon{=}0.3, κ+=10\kappa_{+}{=}10). SmoothClip:

f(r)={1+κ+(r−1)+(1−κ+)κ+εtanh⁡(r−1κ+ε),r≤1, r,1<r<1+ε, 1+ε+εtanh⁡!(r−1−εε),r≥1+ε.f(r)=\begin{cases}1+\kappa_{+}(r-1)+(1-\kappa_{+})\kappa_{+}\varepsilon \tanh \left(\frac{r-1}{\kappa_{+}\varepsilon}\right), & r\le 1,\\\ r, & 1<r<1+\varepsilon,\\\ 1+\varepsilon+\varepsilon \tanh\\!\left(\frac{r-1-\varepsilon}{\varepsilon}\right), & r\ge 1+\varepsilon.\end{cases}

BumpPull:

fBP(r)=fSC(r)−p,sech2!(r−cw)σ!(r−(1+ε)0.1,ε),c=1+3ε, w=2ε,f_{\mathrm{BP}}(r)=f_{\mathrm{SC}}(r)-p\\,\mathrm{sech}^{2}\\!\left(\tfrac{r-c}{w}\right)\sigma\\!\left(\tfrac{r-(1+\varepsilon)}{0.1\\,\varepsilon}\right),\quad c=1+3\varepsilon,\ w=2\varepsilon,

with pp calibrated at startup so that the minimum gain equals ANO's κ−=−1.5\kappa_{-}=-1.5. Both use ANO's advantage-branch loss (A≥0A\ge 0: f(r)f(r); A<0A<0: 2−f(2−r)2-f(2-r)), so Lemma 4.4's enclosure proof applies to them verbatim.

W1 (theory). The gain field (Section 3.2: the per-sample weight f′(r)f'(r) multiplying the policy-gradient direction) is exactly f′(r)f'(r); our claim is the choice of design domain (prescribing the gain profile directly), analogous to designing feedback gains in control rather than trajectories. We state in Section 3.2 that the scalar flow does not imply parameter-space convergence; it provides a design lens and a unified account of failure modes. Theorems 3.1/3.4 apply to the hard-constrained version, exactly as TRPO's own monotonic-improvement guarantee does (Schulman et al., 2015); our bound contributes a new element-wise, two-sided sufficient condition rather than a tighter one.

Q2 (decoupling). The grid is complete (Table R2, 12 configurations): κ+\kappa_{+} has no monotone effect on IQM (non-monotone across κ+\kappa_{+} at every fixed κ−\kappa_{-}); κ−\kappa_{-} deepening improves IQM consistently at κ+≥10\kappa_{+}\ge 10, and the best probed cell (κ+,κ−)=(10,−2.0)(\kappa_{+},\kappa_{-})=(10,-2.0) is nominally above the reported configuration (the full grid is added to the appendix of the revised PDF, red table, and the reported pair stays unchanged for consistency with all main experiments). No destructive interaction appears anywhere on the grid. We also clarify the tuning budget: the main tables use 3ε3\varepsilon values at the fixed (10, -1.5) point (the "6 configurations" count includes RLHF's two grid points, which are independent of the main experiments).

Table R2 (κ\kappa grid, ε=0.3\varepsilon{=}0.3, IQM, 5 seeds):

κ+\kappa_{+} \ κ−\kappa_{-}-0.25-1.0-1.5-2.0
50.84670.87160.86030.8593
100.80460.88650.9304 (paper default)0.9383 (best)
200.86290.88320.91270.9055

Grid complete at MuJoCo scale (5 seeds, 2% window; 3×4=123\times4=12 configurations). Each joint (κ+,κ−)(\kappa_{+},\kappa_{-}) point costs 3ε×53\varepsilon\times 5 seeds ×40\times 40 Atari games =600=600 runs on Atari (Appendix A.4).

Q3 (stress test extension). Completed (Table R3): three LRs, five methods, ANO at its best configuration alongside the paper's original rows. Two findings: (1) PPO's collapse begins already at 2x LR (-42%), and its high-LR IQM is flat across evaluation windows (2%-50%), i.e., an irreversible early collapse, in contrast to the graceful slow degradation of soft-penalty and KL-constrained methods; (2) the five-method picture precisely matches the gain-field prediction: everything with a continuous restoring mechanism (ANO -0.9%, SPO -6.6%, TRPO -9.3%, PAPO +6.1%) survives; the sole fragile design is PPO's hard clip (-54.5%), whose dead zone provides no restoring force. ANO has the highest absolute IQM at 1e-3 (0.9225) among all five methods.

Table R3 (5 seeds, paper protocol; 3e-4 from Table 3 of the submission, 1e-3 from its stress-test table (submission Table 10, revised Table 12); 6e-4 new):

MethodLRIQMMeanMedian
ANO3e-40.93040.96051.0079
ANO6e-40.93350.95640.8969
ANO1e-30.92250.90150.9347
SPO3e-40.92490.91390.9780
SPO6e-40.92410.91620.8721
SPO1e-30.86430.83510.8027
PPO3e-40.91850.86620.9222
PPO6e-40.53380.55740.5789
PPO1e-30.41770.43700.3910
TRPO3e-40.85010.82040.8249
TRPO6e-40.88220.82390.9379
TRPO1e-30.77080.72730.7477
PAPO3e-40.79250.81120.8723
PAPO6e-40.86430.85980.9787
PAPO1e-30.84050.82390.9563

Q4 (equal tuning budget). All methods follow one protocol: each sweeps its own boundary parameter (clip ε\varepsilon / KL / ANO ε\varepsilon) over 3 values, best final IQM selected; full grids are disclosed in Appendix A.4 with per-candidate cost accounting (each Atari grid point costs 3ε×53\varepsilon\times 5 seeds ×40\times 40 games =600=600 runs, which is why the (κ+,κ−)(\kappa_{+},\kappa_{-}) grid is two points in the main paper and now independently grid-searched at MuJoCo scale in R2). As Table R2 shows, tuning (κ+,κ−)(\kappa_{+},\kappa_{-}) would yield further gains; distilling them into independently tunable knobs is part of the design contribution, but for tuning-budget fairness the paper's experiments freeze a single (κ+,κ−)(\kappa_{+},\kappa_{-}) point and leave this margin unexploited.

Matched trust region (the ε\varepsilon sub-question, answered in full). We ran ANO at PPO's clip ε=0.2\varepsilon{=}0.2 (the best configuration for SPO and PPO on MuJoCo) rather than at ANO's own best $\varepsilon=$0.3 for MuJoCo:

MethodLR=3e-4LR=6e-4LR=1e-3degradation
ANO (ε=0.2\varepsilon{=}0.2)0.87640.91570.8417-4.0%
PPO (ε=0.2\varepsilon{=}0.2)0.91850.53380.4177-54.5%

The qualitative conclusion is unchanged: PPO collapses by an order of magnitude more than ANO at the same trust region. ANO's headline -0.9% is measured at its selected ε=0.3\varepsilon{=}0.3 (Table 12 of the submission, Table R3); at the shared ε=0.2\varepsilon{=}0.2 it is -4.0%, still far below PPO. (ANO's ε=0.2\varepsilon{=}0.2 stress runs are new.)

Atari extension (part of Q3). 3 methods ×\times 2 LRs ×\times 40 games ×\times 5 seeds is about 1200 runs, physically infeasible on our single-GPU budget within the rebuttal period; we will add the Atari stress test under the same protocol in the camera-ready.

W2 (statistical separability). ANO vs PAPO on Atari IQM is a statistical tie (paired CI [-0.10, +0.11]); ANO is the only method at the top on both domains across tasks and settings (vs SPO: +0.047, CI [+0.010, +0.103], P=1.000). The ablation and sweep tables above delimit exactly what is claimed.

Thank you for the additional experiments. The matched-threshold comparison addresses one of my main concerns. On MuJoCo, increasing the learning rate from 3×10−43\times10^{-4} to 10−310^{-3} reduces ANO’s IQM by 4.0% and PPO’s by 54.5% at the same clipping threshold. The independent sweep of the two gain parameters also addresses my request for a sensitivity analysis.

My remaining concern is what accounts for ANO’s robustness. In Table R1, BumpPull’s gain becomes positive again beyond the restoring region, so its instability leaves open whether a simpler kernel whose gain approaches zero from below could match ANO. Testing such a kernel in the same learning-rate sweep, with the zero-crossing and peak restoring gain matched, would help isolate the contribution of ANO’s specific functional form.

I would also put limited weight on the MuJoCo results alone when assessing the practical benefit of ANO. SAC is a strong off-policy baseline for continuous control and would provide a reference for sample efficiency on MuJoCo. For a proposed replacement for PPO’s clipped objective, I would find an Atari learning-rate sweep and a multi-seed RLHF comparison against PPO more informative about whether the robustness gains generalize. The current single-run RLHF comparison leaves run-to-run variability unmeasured.

The rebuttal strengthens the evidence for ANO’s robustness on MuJoCo. My original recommendation remains unchanged given the open questions about the specific kernel and the generality of the result.

We are glad the matched-threshold comparison and the gain-parameter sweep address two of the central concerns. On the remaining points:

On the proposed kernel with gain approaching zero from below. This was run, and it is ANO. BumpPull's failure is not located in the depth of its restoring gain; it is located in the sign change. Its gain crosses zero twice, a restoring lobe followed by a positive interval (r∈(1.75,3.5)r\in(1.75,3.5)) that ejects overshooting samples from the recoverable band; the ejected mass accumulates until training collapses on 9/30 runs. A kernel whose gain approaches zero from below is precisely the configuration that avoids this failure mode, and ANO's right tail is already such a kernel: monotone redescending from κ−\kappa_{-} to 0−0^{-}, never crossing zero. The requested kernel is a description of ANO, not an alternative to it. As stated in our previous comment, finding a second family that satisfies all four requirements is the hard part of this work; we found exactly one, which is why the paper constructs its algorithm from ANO alone. We will gladly run any concrete functional form the reviewer proposes.

On SAC. SAC is off-policy with a replay buffer; at a fixed gradient budget its sample-efficiency regime is not comparable to an on-policy protocol. Running it here would confound the comparison, and adapting the protocol would test a different quantity.

On the reviewer's stated condition. We recall the reviewer's question from the first round: a comparison against TrulyPPO and a simpler bounded smooth kernel, noting that it "would show whether the specific functional form or the redescending property is responsible for the gains" and that it "(Could change my assessment.)" Table R1 delivers exactly this comparison, extended with a third matched alternative: TrulyPPO 0.4613, SmoothClip 0.4539, BumpPull 0.4811 with diverged runs scored at 0 (9/30 diverged), ANO 0.9304. The specific form and the redescending property are shown to be jointly necessary, and the failure is localized to the gain shape's sign structure. We ask that this condition, having been met, be weighed accordingly in the assessment.

On Atari and RLHF. Under the paper's experimental protocol (5 seeds, 40 Atari games, full training budgets), the two extensions are ∼800\sim 800 GPU-hours =∼33=\sim 33 GPU-days (Atari sweep) and 540540 GPU-hours =22.5=22.5 GPU-days (multi-seed RLHF at 60 GPU-hours per run, three methods ×\times 3 seeds) respectively on a single GPU; the former is roughly 2.4x the 14-day discussion window (5-18 November), the latter 1.6x. Neither is executable within the current phase. In our rebuttal we listed the Atari sweep and the RLHF extension as camera-ready candidates; given the budget above, we will prioritize the RLHF multi-seed extension and will revisit the Atari sweep if capacity allows. We note that the scope already delivered in this paper comprises substantially more experiments than the closest prior work we are held against (SPO reports no property-attribution ablation of its kernel, no learning-rate stress test, no decoupling grid, no broad hyperparameter sweep across competing methods, and no RLHF experiment).

On RLHF as evidence. The RLHF experiment is supplementary, not load-bearing: the claim it supports is cross-paradigm transfer of the gain-field design, and the paper labels it "suggestive" exactly because it is single-seed. LLM validation is not standard practice in policy-optimization research and was never a requirement for acceptance in this line of work; SPO (the most recent prior work we cite, and a baseline here) reports no LLM experiments, nor do TRPO, PPO, or PAPO. Its presence, even at one seed, is an addition beyond the community norm, not a deficit.

Reviewer_4z7JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper firstly identifies the issue of PPO and SPO that the so-called gain field may lead to a dead zone or destabilised dynamics. The authors believe that the issue comes from the importance ratio used by PPO and SPO. Instead of the clipping rule of PPO, the authors propose a new strategy using a specific function to control the trust region and avoid the existing issues. The authors have analysed the requirements of this new function in theory. The final algorithm ANO is simple and straightforward. The experimental results are promising.

Critical strengths

The identified problem is interesting, fundamental, and significant. The final algorithm is simple and easy to implement. There are a number of theoretical results (I do not understand most of them even though I am doing RL research, but I do not consider them the issue of this work, and it should be my problem). If the problem is resolved, I would see a wide range of applications of such a design.

Critical weaknesses

My main concern is about the experimental results (Tables 2 and 3). Even though the authors used 5 seeds, the standard deviations are not included here. Since the numbers are quite close to each other, it is hard to draw a conclusion based on them.

I think the main advantage of this work is its capability to increase the stability of the training instead of final performance. I am not convinced by the performance improvement as the metric for this design. The authors may need to consider evaluating the stability brought by this new method instead of the final performance.

Questions for authors

In the introduction, the authors claimed the issues of PPO and SPO. Which experiments show these issues, and how does your method resolve them?

Is it possible to give the std for the results? Have you done any two-sample tests on the results to ensure their statistical significance?

1 comment

Thank you for the careful statistical questions; they pushed us to compute three layers of dispersion evidence (Table R4), which we now also display directly in the revised main tables (new "Std" columns).

W1 (no std; numbers close). Three complementary statistics, all computed offline from the same runs behind Tables 2-3:

  1. Seed dispersion: per-task std across the 5 seeds averages 0.100 for ANO on MuJoCo, the tightest among the leading methods (SPO 0.115, PPO 0.141, TRPO 0.164; PAPO's lower 0.062 accompanies a far lower IQM); on Atari ANO (0.907) is again the tightest of the leading methods (PAPO 0.917, SPO 0.978, PPO 1.047).
  2. Two-sample tests: paired bootstrap on per-task means (n=6) gives ANO vs SPO +0.047, CI95 [+0.010, +0.103], P(ANO>SPO)=1.000 (a different statistic from the pooled IQM bootstrap CIs of Table 3). ANO vs PPO is positive but not significant at n=6 (+0.094, CI95 [-0.057, +0.320]). On Atari, where HNS is heavy-tailed, the pooled paired bootstrap applies: ANO and PAPO are statistically tied on Atari IQM (CI [-0.10, +0.11]). ANO is the only method at the top (or tied at the top) on both domains: the advantage holds across tasks and settings (Atari Median; MuJoCo IQM, Median, and robustness), while the runner-up differs by domain (PAPO on Atari, SPO on MuJoCo).
  3. Stability (your suggestion to evaluate stability rather than only final performance): coefficient of variation of the episodic return over the last 20% of training, pooled IQM: MuJoCo ANO 0.175 < SPO 0.181 < PPO 0.188 < TRPO 0.208; Atari shows no significant difference between methods. ANO is simultaneously the highest-performing and the smoothest on MuJoCo.

Q1 (which experiments expose the PPO/SPO pathologies, and how ANO resolves them). Three complementary pieces of evidence: (1) Figure 1/2's gain fields and phase lines (PPO's zero-gain dead zone; SPO's unbounded gain); (2) the out-of-boundary diagnostic: 7.88% of Atari samples receive zero gradient under PPO (up to ~20% on Asterix) versus 1.21% for ANO; (3) the learning-rate sweep (Table R3): PPO collapses by 54.5% at 3.3x LR while ANO degrades 0.9%.

Q2 (std and tests). See W1 above; the Std columns are now also in the revised Tables 2-3.

Reviewer_fPsbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Anchored Neighborhood Optimization (ANO), an on-policy reinforcement learning (RL) method that modifies the clipping function in PPO. The paper interprets existing surrogate objectives used in on-policy RL algorithms as a feedback law on the probability ratio, and constructs a shaping function with a desired gain profile. This results in a new surrogate objective used by ANO, which is compared against PPO, SPO, and other baselines on MuJoCo and Atari tasks. Experiments also report results for an LLM alignment task over a single random seed.

Critical strengths
  • [S1]: The paper introduces a novel and interesting perspective for on-policy policy optimization methods that treats the surrogate objective as a feedback law on the probability ratio.
  • [S2]: Despite the added complexity in the derivation of the ANO objective, updating the policy with this objective does not appear to introduce additional computational overhead during training compared to PPO and SPO.
Critical weaknesses
  • [W1]: It would be more convincing to focus experimental analysis on recent relevant applications of PPO such as massively parallel training in Isaac Lab or LLM finetuning, rather than older benchmark domains such as MuJoCo and Atari. The RLHF experiments in the paper are a promising direction, but they currently feel more like a placeholder given their very limited scope.
  • [W2]: The paper describes SPO’s unbounded gain as SPO’s major limitation, but there is no experimental analysis that demonstrates this unbounded gain leads to performance issues for SPO. The original SPO paper actually shows that SPO remains robust even when combined with large networks. There is also no ablation demonstrating that the redescending restoring force in ANO is critical.
  • [W3]: The method introduces additional hyperparameters κ+,κ−\kappa_{+}, \kappa_{-} compared to PPO and SPO, and there is no ablation study to demonstrate the sensitivity to these additional hyperparameters (1 setting used for MuJoCo / Atari, 2 settings used for RLHF).
  • [W4]: The paper does not contain a Related Work section, which should include other existing work on on-policy policy optimization methods (e.g., mirror learning [1], supervised policy updates [2], and other modifications to the clipping function design in PPO [3]).

References:

[1] Kuba et al. Mirror Learning: A Unifying Framework of Policy Optimisation. In ICML 2022.

[2] Vuong et al. Supervised Policy Update for Deep Reinforcement Learning. In ICLR 2019.

[3] Wang et al. Truly Proximal Policy Optimization. In UAI 2020.

Questions for authors

The paper could be strengthened through experimental analysis aimed at resolving [W1]-[W3].

1 comment

Thank you for the positive assessment of the perspective and the constructive weaknesses. Point-by-point:

W1 (benchmarks / RLHF scope). Large-scale deployments are the natural application target of PPO-style methods; isolating a mechanism, however, requires controlled comparisons, and MuJoCo/Atari are what allow PPO, SPO, PAPO, and TRPO to be compared under identical networks, budgets, and evaluation protocols. The RLHF experiment is labeled "suggestive" in the paper and serves as a cross-paradigm transfer check; a 3-seed extension (about 60 hours per run) cannot fit in the rebuttal window and is planned for the camera-ready. Massively parallel benchmarks (e.g., Isaac Lab) are an orthogonal engineering axis, which we list as future work.

W2 (SPO's unbounded gain; redescending ablation). (1) The robustness established in the original SPO paper concerns network scale (larger, wider networks), which is orthogonal to the axis studied here, optimization aggressiveness (learning rate and hyperparameters): under the 3.3×3.3\times learning-rate stress test SPO degrades by 6.6%, 7x ANO's 0.9% (Table R3). The triggering quantity is the policy's ratio distribution, not network scale: SPO's gain grows linearly and without bound in r, so its effect is governed by the tail of the empirical ratio distribution. When ratios stay concentrated (benign environments, conservative steps, as in the original SPO experiments), extreme-ratio samples are rare, the unbounded gain is never exercised, and the method behaves robustly; when the distribution develops heavy tails (large steps, high-variance tasks), the unbounded gain amplifies exactly those tail samples in proportion to their deviation, and the update dynamics stiffen (SPO's low out-of-boundary fraction in the diagnostic, revised Table 11 / submission Table 9, is achieved precisely through this unbounded force; the diagnostic measures sample location, not force magnitude). ANO's bounded gain removes the dependence on the distribution's shape: no matter how heavy the tail, every sample's force is capped within [κ−,κ+][\kappa_{-},\kappa_{+}]. (2) The redescending ablation is Table R1 (posted in our reply to Reviewer dm1t), which tests each property with a matched alternative.

W3 (κ\kappa sensitivity). The ε\varepsilon sweep at fixed (κ+,κ−)(\kappa_{+},\kappa_{-}) appears in the full-sweep tables (Tables 7-8 of the submission; 9-10 in the revised PDF; 3 values, full IQM/Mean/Median panels). The full 3×43\times 4 (κ+,κ−)(\kappa_{+},\kappa_{-}) grid is new in this rebuttal (Table R2): κ+\kappa_{+} shows no monotone effect on IQM (non-monotone across κ+\kappa_{+} at every fixed κ−\kappa_{-}); κ−\kappa_{-} deepening helps consistently up to κ−=−1.5\kappa_{-}{=}{-}1.5 at κ+≥10\kappa_{+}\ge 10 (+0.05+0.05 to +0.13+0.13), and keeps improving toward κ−=−2.0\kappa_{-}{=}{-}2.0 at κ+=10\kappa_{+}{=}10 (the full grid is added to the appendix of the revised PDF, red table). The reported pair comes from the paper's coarse sweep and is kept unchanged for consistency with all main experiments. No destructive interaction appears anywhere on the grid, consistent with the decoupled-knobs design claim.

W4 (Related Work). Accepted and fixed: the revised PDF adds a Related Work section (Section 6) covering mirror learning (Grudzien et al., 2022), supervised policy updates (Vuong et al., 2018), and Truly PPO (Wang et al., 2020), with explicit positioning of ANO against each.

Table R2 (κ\kappa grid, ε=0.3\varepsilon{=}0.3, IQM):

κ+\kappa_{+} \ κ−\kappa_{-}-0.25-1.0-1.5-2.0
50.84670.87160.86030.8593
100.80460.88650.9304 (paper default)0.9383 (best)
200.86290.88320.91270.9055

Grid complete at MuJoCo scale (5 seeds, 2% window; 3×4=123\times4=12 configurations). Each joint (κ+,κ−)(\kappa_{+},\kappa_{-}) point costs 3ε×53\varepsilon\times 5 seeds ×40\times 40 Atari games =600=600 runs on Atari (Appendix A.4).

Comments on the paper

2 comments

We thank the reviewers. Before the point-by-point replies, we clarify what this work is: a change of design domain. Several questions below are best answered once that domain is made explicit.

ANO is not merely another PPO variant whose value is a leaderboard position; the ranking is a consequence, not the contribution. The contribution is a change of design domain, and it has two parts. First, the principles: we argue, with the detailed analysis of Section 3, that a policy-update gain field should satisfy enclosure, boundedness, smoothness, and redescending relaxation, and we fix these into the function family as non-negotiable structure, not as tunable options. What is not principled, namely how strong the push and pull should be for a given scenario, is exactly what we distill into user-facing knobs (κ+,κ−,ε)(\kappa_{+},\kappa_{-},\varepsilon), so that adaptation lives in degrees of freedom that carry no design risk. Second, the construction: finding a family that satisfies all the principles while exposing those knobs is far harder than stating the principles. Ours is, to our knowledge, the only such family among existing on-policy policy-optimization objectives: closed-form, elementary to compute (no iterative solving; evaluation adds negligible overhead over the network's forward-backward), and yet provably enclosed. Table R1 is direct evidence of how narrow this feasible region is: three alternative families engineered to satisfy as many of the principles as possible all fail, two of them catastrophically. The new experiments below answer the reviewers' questions with controlled evidence: the property-by-property ablation with matched kernels (R1), a full κ\kappa grid (R2), a five-method learning-rate sweep with a matched-trust-region panel (R3), seed-level dispersion and paired tests (R4), and a window sweep (R5). Tables R1 and R2 are included in the revised PDF as Table 5 and Table 4 (both red). Per-reviewer responses follow.

Table R5 (window invariance; IQM at the 2%/10%/30%/50% evaluation windows):

Variant2%10%30%50%
SmoothClip (no pull)0.45390.45230.44820.4383
BumpPull (surviving runs)0.82140.82040.81060.7918
BumpPull (diverged scored 0)0.48110.48200.47620.4676
TrulyPPO0.46130.44660.43190.4286
ANO ε=0.3\varepsilon{=}0.3, LR 3e-40.93040.94130.91550.8991
ANO ε=0.3\varepsilon{=}0.3, LR 6e-40.93350.92690.90620.8804
ANO ε=0.3\varepsilon{=}0.3, LR 1e-30.92250.92590.90100.8354
ANO ε=0.2\varepsilon{=}0.2, LR 3e-40.87640.86620.85190.8297
ANO ε=0.2\varepsilon{=}0.2, LR 6e-40.91570.90860.88580.8415
ANO ε=0.2\varepsilon{=}0.2, LR 1e-30.84170.82620.76110.6510
SPO ε=0.2\varepsilon{=}0.2, LR 3e-40.92490.93120.92530.9090
SPO ε=0.2\varepsilon{=}0.2, LR 6e-40.92410.92710.91240.8650
SPO ε=0.2\varepsilon{=}0.2, LR 1e-30.86430.84290.76620.6793
PPO ε=0.2\varepsilon{=}0.2, LR 3e-40.91850.90820.89630.8716
PPO ε=0.2\varepsilon{=}0.2, LR 6e-40.53380.54220.54880.5404
PPO ε=0.2\varepsilon{=}0.2, LR 1e-30.41770.41470.41060.4077

Every conclusion in R1-R3 holds at each window. The κ\kappa grid is likewise window-stable in ordering: across the 2%-50% windows every cell retains its relative rank and no grid-based conclusion changes. PPO's flat rows at high LR (0.42/0.53 across all windows) reflect its irreversible early collapse.

Dear Reviewers,

The authors have provided rebuttals to your review. Could you please check their responses and provide any further feedback or questions for the authors to address at your convenience?

Thank you for your time and contribution.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.