- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes Anchored Neighborhood Optimization (ANO), a drop-in replacement for PPO’s clipped surrogate. The authors interpret the derivative of a ratio-shaping function f′(r) as a “gain field” acting on the probability ratio. Under this view, PPO has a zero-gain dead zone beyond 1+ϵ, and SPO has an unbounded, linearly growing restoring gain. ANO instead constructs a C∞ shaping kernel directly in the derivative domain, with three hyperparameters: ϵ sets the zero-crossing, κ₊ sets the left asymptotic gain, and κ₋ sets the minimum (pull) gain. The gain is bounded and decays to zero for large ratios, in the spirit of redescending M-estimators. All internal constants are obtained in closed form.
On the theory side, the paper gives a two-sided log-ratio lower bound (Thm 3.1) and a monotonic-improvement result for a hard-constrained version of the generalized objective (Thm 3.4). It also proves that the ANO kernel satisfies the required enclosure property (Lemma 4.4). Empirically, ANO ranks first in IQM and Median on Atari (40 games) and MuJoCo (6 tasks) with 5 seeds. It degrades by 0.9% under a 3e-4 → 1e-3 learning-rate increase on MuJoCo, versus 54.5% for PPO. It also matches SPO and outperforms PPO in a single-run experiment judged by an LLM.
Critical strengths
-
Robustness to aggressive step sizes. The learning-rate stress test (Table 10) is the most informative result in the paper. PPO collapses (−54.5% IQM), while ANO is almost unaffected (−0.9%), and the stressed ANO still exceeds the best-tuned PPO at the nominal learning rate. If this robustness generalizes, it is a practically useful property.
-
Clean, cheap construction and a useful diagnostic. The kernel has a closed-form parameterization with no iterative solve, and it is easy to drop into existing PPO code. The out-of-boundary diagnostic (Table 9, Fig. 6) makes concrete how many samples receive zero gradient under PPO: about 7.9% on average on Atari, and up to about 20% on Asterix.
Critical weaknesses
-
Limited novelty, and the specific kernel is not justified against simpler alternatives. Smooth or soft clipping and restoring-force variants of PPO are a crowded design space. For example, the rollback operation in Truly PPO (Wang et al., 2020) already applies a bounded negative gain beyond the clipping boundary; the paper cites this work but does not compare against it. The "gain field" perspective is largely a reframing of the per-sample gradient weight . The authors themselves note that the scalar flow does not imply convergence in parameter space, so the framework provides motivation rather than new guarantees. Likewise, Theorems 3.1 and 3.4 apply to a hard-constrained objective that ANO does not optimize, so they do not certify the proposed algorithm. Most importantly, there is no ablation showing that this particular rational-exponential kernel matters, rather than any bounded, smooth, redescending function, or even a non-redescending bounded one such as rollback. Without such an ablation, the paper's central design claims remain untested.
-
Empirical gains are small relative to the uncertainty, and the evaluation is thin where it matters.
- On Atari, ANO and PAPO are statistically indistinguishable (IQM 0.935 vs. 0.929; paired 95% CI [−0.10, 0.11]).
- On MuJoCo, the IQM confidence intervals of ANO, SPO, and PPO overlap substantially ([0.88, 0.99], [0.85, 0.98], [0.84, 0.98]).
- ANO's reported configuration was selected from 6 settings (2 pairs × 3 values), whereas PPO and SPO each had 3. With margins this small, best-of-sweep selection on final scores may matter.
- The claim that the three knobs are "decoupled" is never tested: the two settings vary both parameters together.
- The robustness result, which is the strongest evidence for the paper's thesis, rests on a single elevated learning rate, a single domain (MuJoCo, 6 tasks), and only three methods.
- The RLHF experiment uses one run per method, as the authors acknowledge.
Questions for authors
- How does ANO compare with (a) PPO with rollback (Truly PPO) and (b) a simpler bounded, smooth kernel, such as a smoothed clip or a tanh-based restoring term? This would show whether the specific functional form or the redescending property is responsible for the gains. (Could change my assessment.)
- Can you vary and independently to support the decoupling claim, and report sensitivity to each?
- Can the learning-rate stress test be extended to several learning rates (e.g., a sweep up to the point where ANO itself fails), to Atari, and to PAPO and TRPO? Does the robustness persist when ANO and PPO use the same ? Currently ANO uses and PPO uses in Table 10.
- Given the unequal tuning budgets, how do the results change if all methods are compared at a fixed, shared , or under an equal number of configurations?
4 comments
Table R2 ( grid, , IQM, 5 seeds):
| \ | -0.25 | -1.0 | -1.5 | -2.0 |
|---|---|---|---|---|
| 5 | 0.8467 | 0.8716 | 0.8603 | 0.8593 |
| 10 | 0.8046 | 0.8865 | 0.9304 (paper default) | 0.9383 (best) |
| 20 | 0.8629 | 0.8832 | 0.9127 | 0.9055 |
Grid complete at MuJoCo scale (5 seeds, 2% window; configurations). Each joint point costs seeds Atari games runs on Atari (Appendix A.4).
Q3 (stress test extension). Completed (Table R3): three LRs, five methods, ANO at its best configuration alongside the paper's original rows. Two findings: (1) PPO's collapse begins already at 2x LR (-42%), and its high-LR IQM is flat across evaluation windows (2%-50%), i.e., an irreversible early collapse, in contrast to the graceful slow degradation of soft-penalty and KL-constrained methods; (2) the five-method picture precisely matches the gain-field prediction: everything with a continuous restoring mechanism (ANO -0.9%, SPO -6.6%, TRPO -9.3%, PAPO +6.1%) survives; the sole fragile design is PPO's hard clip (-54.5%), whose dead zone provides no restoring force. ANO has the highest absolute IQM at 1e-3 (0.9225) among all five methods.
Table R3 (5 seeds, paper protocol; 3e-4 from Table 3 of the submission, 1e-3 from its stress-test table (submission Table 10, revised Table 12); 6e-4 new):
| Method | LR | IQM | Mean | Median |
|---|---|---|---|---|
| ANO | 3e-4 | 0.9304 | 0.9605 | 1.0079 |
| ANO | 6e-4 | 0.9335 | 0.9564 | 0.8969 |
| ANO | 1e-3 | 0.9225 | 0.9015 | 0.9347 |
| SPO | 3e-4 | 0.9249 | 0.9139 | 0.9780 |
| SPO | 6e-4 | 0.9241 | 0.9162 | 0.8721 |
| SPO | 1e-3 | 0.8643 | 0.8351 | 0.8027 |
| PPO | 3e-4 | 0.9185 | 0.8662 | 0.9222 |
| PPO | 6e-4 | 0.5338 | 0.5574 | 0.5789 |
| PPO | 1e-3 | 0.4177 | 0.4370 | 0.3910 |
| TRPO | 3e-4 | 0.8501 | 0.8204 | 0.8249 |
| TRPO | 6e-4 | 0.8822 | 0.8239 | 0.9379 |
| TRPO | 1e-3 | 0.7708 | 0.7273 | 0.7477 |
| PAPO | 3e-4 | 0.7925 | 0.8112 | 0.8723 |
| PAPO | 6e-4 | 0.8643 | 0.8598 | 0.9787 |
| PAPO | 1e-3 | 0.8405 | 0.8239 | 0.9563 |
Q4 (equal tuning budget). All methods follow one protocol: each sweeps its own boundary parameter (clip / KL / ANO ) over 3 values, best final IQM selected; full grids are disclosed in Appendix A.4 with per-candidate cost accounting (each Atari grid point costs seeds games runs, which is why the grid is two points in the main paper and now independently grid-searched at MuJoCo scale in R2). As Table R2 shows, tuning would yield further gains; distilling them into independently tunable knobs is part of the design contribution, but for tuning-budget fairness the paper's experiments freeze a single point and leave this margin unexploited.
Matched trust region (the sub-question, answered in full). We ran ANO at PPO's clip (the best configuration for SPO and PPO on MuJoCo) rather than at ANO's own best $\varepsilon=$0.3 for MuJoCo:
| Method | LR=3e-4 | LR=6e-4 | LR=1e-3 | degradation |
|---|---|---|---|---|
| ANO () | 0.8764 | 0.9157 | 0.8417 | -4.0% |
| PPO () | 0.9185 | 0.5338 | 0.4177 | -54.5% |
The qualitative conclusion is unchanged: PPO collapses by an order of magnitude more than ANO at the same trust region. ANO's headline -0.9% is measured at its selected (Table 12 of the submission, Table R3); at the shared it is -4.0%, still far below PPO. (ANO's stress runs are new.)
Atari extension (part of Q3). 3 methods 2 LRs 40 games 5 seeds is about 1200 runs, physically infeasible on our single-GPU budget within the rebuttal period; we will add the Atari stress test under the same protocol in the camera-ready.
W2 (statistical separability). ANO vs PAPO on Atari IQM is a statistical tie (paired CI [-0.10, +0.11]); ANO is the only method at the top on both domains across tasks and settings (vs SPO: +0.047, CI [+0.010, +0.103], P=1.000). The ablation and sweep tables above delimit exactly what is claimed.
Thank you for the additional experiments. The matched-threshold comparison addresses one of my main concerns. On MuJoCo, increasing the learning rate from to reduces ANO’s IQM by 4.0% and PPO’s by 54.5% at the same clipping threshold. The independent sweep of the two gain parameters also addresses my request for a sensitivity analysis.
My remaining concern is what accounts for ANO’s robustness. In Table R1, BumpPull’s gain becomes positive again beyond the restoring region, so its instability leaves open whether a simpler kernel whose gain approaches zero from below could match ANO. Testing such a kernel in the same learning-rate sweep, with the zero-crossing and peak restoring gain matched, would help isolate the contribution of ANO’s specific functional form.
I would also put limited weight on the MuJoCo results alone when assessing the practical benefit of ANO. SAC is a strong off-policy baseline for continuous control and would provide a reference for sample efficiency on MuJoCo. For a proposed replacement for PPO’s clipped objective, I would find an Atari learning-rate sweep and a multi-seed RLHF comparison against PPO more informative about whether the robustness gains generalize. The current single-run RLHF comparison leaves run-to-run variability unmeasured.
The rebuttal strengthens the evidence for ANO’s robustness on MuJoCo. My original recommendation remains unchanged given the open questions about the specific kernel and the generality of the result.
We are glad the matched-threshold comparison and the gain-parameter sweep address two of the central concerns. On the remaining points:
On the proposed kernel with gain approaching zero from below. This was run, and it is ANO. BumpPull's failure is not located in the depth of its restoring gain; it is located in the sign change. Its gain crosses zero twice, a restoring lobe followed by a positive interval () that ejects overshooting samples from the recoverable band; the ejected mass accumulates until training collapses on 9/30 runs. A kernel whose gain approaches zero from below is precisely the configuration that avoids this failure mode, and ANO's right tail is already such a kernel: monotone redescending from to , never crossing zero. The requested kernel is a description of ANO, not an alternative to it. As stated in our previous comment, finding a second family that satisfies all four requirements is the hard part of this work; we found exactly one, which is why the paper constructs its algorithm from ANO alone. We will gladly run any concrete functional form the reviewer proposes.
On SAC. SAC is off-policy with a replay buffer; at a fixed gradient budget its sample-efficiency regime is not comparable to an on-policy protocol. Running it here would confound the comparison, and adapting the protocol would test a different quantity.
On the reviewer's stated condition. We recall the reviewer's question from the first round: a comparison against TrulyPPO and a simpler bounded smooth kernel, noting that it "would show whether the specific functional form or the redescending property is responsible for the gains" and that it "(Could change my assessment.)" Table R1 delivers exactly this comparison, extended with a third matched alternative: TrulyPPO 0.4613, SmoothClip 0.4539, BumpPull 0.4811 with diverged runs scored at 0 (9/30 diverged), ANO 0.9304. The specific form and the redescending property are shown to be jointly necessary, and the failure is localized to the gain shape's sign structure. We ask that this condition, having been met, be weighed accordingly in the assessment.
On Atari and RLHF. Under the paper's experimental protocol (5 seeds, 40 Atari games, full training budgets), the two extensions are GPU-hours GPU-days (Atari sweep) and GPU-hours GPU-days (multi-seed RLHF at 60 GPU-hours per run, three methods 3 seeds) respectively on a single GPU; the former is roughly 2.4x the 14-day discussion window (5-18 November), the latter 1.6x. Neither is executable within the current phase. In our rebuttal we listed the Atari sweep and the RLHF extension as camera-ready candidates; given the budget above, we will prioritize the RLHF multi-seed extension and will revisit the Atari sweep if capacity allows. We note that the scope already delivered in this paper comprises substantially more experiments than the closest prior work we are held against (SPO reports no property-attribution ablation of its kernel, no learning-rate stress test, no decoupling grid, no broad hyperparameter sweep across competing methods, and no RLHF experiment).
On RLHF as evidence. The RLHF experiment is supplementary, not load-bearing: the claim it supports is cross-paradigm transfer of the gain-field design, and the paper labels it "suggestive" exactly because it is single-seed. LLM validation is not standard practice in policy-optimization research and was never a requirement for acceptance in this line of work; SPO (the most recent prior work we cite, and a baseline here) reports no LLM experiments, nor do TRPO, PPO, or PAPO. Its presence, even at one seed, is an addition beyond the community norm, not a deficit.
Thank you for the detailed review. Point-by-point responses follow.
Q1 (novelty: TrulyPPO, simpler kernels, and property attribution). You asked for (a) a direct comparison with TrulyPPO and (b) a simpler bounded smooth kernel to separate form from property. Table R1 contains both, plus one step further. TrulyPPO realizes the bounded pull without redescending relaxation (the rollback penalty never releases). SmoothClip is the smoothed-clip kernel: bounded and smooth, with the pull removed entirely. BumpPull goes one step further than (b): it adds a tanh-style restoring term carrying a redescending pull, but as a crude local patch rather than an integrated design. All three were evaluated on 6 MuJoCo tasks ×5 seeds under the paper's protocol:
Each alternative fails in the direction of its missing ingredient. The persistent pull (TrulyPPO) scores 0.4613, and a KLRANGE sweep (6 configurations ×30 runs, including the officially recommended setting and a slope variation) peaks at 0.70: tuning does not close the gap. The pure push (SmoothClip) scores 0.4539. The patched pull (BumpPull) diverges on 9/30 runs (policy NaN; all Humanoid, 3/5 HalfCheetah, 1/5 Ant), and with diverged runs scored at 0 its IQM is 0.48, on par with removing the pull entirely. The defect is localized to a single feature of the gain shape: BumpPull's gain rebounds through zero into positive values (r∈(1.75,3.5)) after the pull lobe, pushing overshooting samples out of the recoverable band, whereas ANO's gain redescends to zero monotonically from below and overshooting samples are always pulled back (gain profiles: Figure 2 of the revised PDF).
For attribution, the two kernels share ANO's push side and advantage-branch construction (formulas below), so each failure isolates the missing ingredient rather than any implementation detail. The two kernels are fully specified (same hyperparameters as ANO's MuJoCo configuration, ε=0.3, κ+=10). SmoothClip:
f(r)=⎩⎨⎧1+κ+(r−1)+(1−κ+)κ+εtanh(κ+εr−1), r, 1+ε+εtanh!(εr−1−ε),r≤1,1<r<1+ε,r≥1+ε.
BumpPull:
fBP(r)=fSC(r)−p,sech2!(wr−c)σ!(0.1,εr−(1+ε)),c=1+3ε, w=2ε,
with p calibrated at startup so that the minimum gain equals ANO's κ−=−1.5. Both use ANO's advantage-branch loss (A≥0: f(r); A<0: 2−f(2−r)), so Lemma 4.4's enclosure proof applies to them verbatim.
W1 (theory). The gain field (Section 3.2: the per-sample weight f′(r) multiplying the policy-gradient direction) is exactly f′(r); our claim is the choice of design domain (prescribing the gain profile directly), analogous to designing feedback gains in control rather than trajectories. We state in Section 3.2 that the scalar flow does not imply parameter-space convergence; it provides a design lens and a unified account of failure modes. Theorems 3.1/3.4 apply to the hard-constrained version, exactly as TRPO's own monotonic-improvement guarantee does (Schulman et al., 2015); our bound contributes a new element-wise, two-sided sufficient condition rather than a tighter one.
Q2 (decoupling). The grid is complete (Table R2, 12 configurations): κ+ has no monotone effect on IQM (non-monotone across κ+ at every fixed κ−); κ− deepening improves IQM consistently at κ+≥10, and the best probed cell (κ+,κ−)=(10,−2.0) is nominally above the reported configuration (the full grid is added to the appendix of the revised PDF, red table, and the reported pair stays unchanged for consistency with all main experiments). No destructive interaction appears anywhere on the grid. We also clarify the tuning budget: the main tables use 3ε values at the fixed (10, -1.5) point (the "6 configurations" count includes RLHF's two grid points, which are independent of the main experiments).