- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CP-SPO, which caps a natural-gradient trust-region step using an objective-gap-based step size. It characterizes this rule through local certificates, provides a separate chance-feasibility cap, and evaluates synthetic problems and a practical RL variant.
Critical strengths
- The practical cap is simple and inexpensive to incorporate into existing trust-region methods.
- The synthetic curvature tests and matched cap-only ablation help isolate individual components. The paper also clearly distinguishes its theoretical guarantees from empirical heuristics.
Critical weaknesses
-
Theorem 1 follows directly from intersecting two scalar admissibility intervals. Keeping a lower Taylor model above a known objective bound does not itself guarantee descent, as the paper’s own counterexample demonstrates. The optimization benefit of this particular certificate over conventional descent-based step selection remains unclear.
-
The RL implementation uses neither a certified objective bound nor the finite-curvature rule or chance-safety cap. Although acknowledged, this leaves the main guarantees unvalidated in policy learning. In Table 4, all three mean costs exceed the budget, while PointPush1 retains only 40.6% of the selected baseline’s return.
-
The cap-only ablation covers just one task and three seeds. More importantly, the generic-shortening comparison remains explicitly preliminary, with missing protocol details, paired uncertainty, and unresolved issues. Its similar returns and reported 5.2% reduction in cost-excess area do not yet establish an advantage attributable to objective information. Completing this controlled comparison is essential.
Questions for authors
Please check the weaknesses section.
2 comments
Thank you for your reply. After considering your response, I would prefer to keep my original score.
Response to Reviewer 5L2k
Thank you for the careful review and for asking us to clarify what the objective-based step adds beyond generic shortening.
The analysis identifies an objective-dependent scale and conditions under which it leads to convergence. This answers the optimization part of your question; attributing the empirical improvement to objective information also requires the generic-shortening control.
Theorem 1 has a short proof because the scalar admissible sets are intervals. For a given objective gap and direction, the candidate becomes independent of the radius once δ≥2dηsmPol2. On the exact quadratic L(x)=2x2 with F=β=1 and Llb=0, the smooth branch yields ηsmPol=2−1, so x+=(2−2)x and the objective contracts by approximately 0.343. A bare fixed-radius step instead has constant displacement 2δ. A conventional descent line search can also control excessive steps; the paper does not establish that this certificate dominates such a line search.
Descent requires more than a valid lower bound. With Llb=L∗, let χ=d2βqΔ and a=1+1+2χ2. On the objective branch the upper Taylor bound gives Δ+≤2(1−a)Δ, which is a strict contraction when χ<4. Together with the stated Fisher-PL condition, the paper establishes convergence across both branches for every fixed positive radius and bounds entry into the objective-binding region. The changing-Fisher convex analysis addresses the corresponding metric drift. The clipped Polyak construction has clear precedent in SPSmax Loizou et al., 2021; our contribution is the particular smooth certificate and its interaction with Fisher movement and the branch analysis.
The exact finite-curvature rule is tested in the submitted E1 sensitivity experiment. Across 525 trajectories and 31,500 updates, valid smoothness overestimates preserve the true-curvature certificate to the declared −10−10 tolerance. At radii 0.1, 0.3 and 1.0, all 15 starts reach gap 10−10 within 60 iterations at 1×, 2× and 4× the true curvature. A 10× bound slows progress, and the deliberately invalid 0.5× estimate fails the certificate on 37% of updates at δ=0.3. The synthetic cost experiment separately tests the feasibility cap. Both concern settings where the required inputs are available; they do not verify the neural-policy assumptions.
Under a fixed augmented controller, the three-seed PointGoal1 ablation reports +8.6% final return, +8.1% return area and -5.7% cost-excess area when the cap is enabled. Accepted controller-active updates differ materially on 7.0% of cases, and the median of the three seed-level median candidate ratios is 0.83. The cap can therefore affect the training trajectory selectively, even when later controller labels obscure its activity. These are descriptive results from one task; the 1.10 return-SD ratio provides no evidence of variance reduction.
Attributing that improvement specifically to objective information requires the generic-shortening control. Table 3 remains incomplete: its 20.5 versus 20.7 returns and 5.2% lower cost-excess area cannot establish dominance, and marginal standard deviations cannot recover paired uncertainty. The draft sentence and broken appendix reference are our reporting errors. The cap-on/off result supports an effect under its specified controller, while the comparison with generic shortening remains unresolved.
The practical implementation also leaves the finite-curvature and chance-feasibility guarantees unused. Its running-best target is not a certified bound, and its mean-cost controller does not meet the budget in the primary experiments. The navigation results retain 92.7% to 96.8% of the radius-selected baseline's return at the shared δ=0.01 setting, while the pushing result retains 40.6% and has lower, still infeasible cost. The analysis and controlled tests establish the objective-scaling mechanism. The remaining practical questions are cost control and attribution of the cap's benefit to objective information.