acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Trust Radius: Objective-Aware Steps for Policy Learning

Abstract

A fixed trust radius controls policy movement without using the gap between the current loss and a certified lower bound. We connect these two scales through Chentsov–Polyak Safe Policy Optimization, which takes the smaller of a quadratic Fisher trust-region step and a smooth Polyak step. Given a certified objective lower bound and smoothness control, this is the largest step along the fixed natural-gradient direction consistent with both the lower-Taylor and movement bounds. Objective information can therefore tighten the permitted step within a fixed trust region. For constrained learning, a separate closed-form cap preserves cumulative-cost chance feasibility from a surrogate-feasible start under sub-Gaussian costs and certified cost and curvature bounds. Synthetic experiments show reduced sensitivity to oversized radii and report no chance-infeasible iterates among 20,000 tested with the safety cap. On three Safety-Gymnasium tasks, we use a running-best reward proxy for the objective bound, a Lagrangian fallback, and one shared radius. At one million environment steps, our method retains 93–97% of radius-selected trust-region policy-optimization-Lagrangian return on two navigation tasks, but 41% on pushing, with lower pushing cost; mean costs exceed the budget on all three tasks. This exposes a distinction for policy learning: controlling policy movement, scaling optimization steps, and certifying cost constraints require different information.

Primary area:
reinforcement learning
Keywords:
trust region, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5L2kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CP-SPO, which caps a natural-gradient trust-region step using an objective-gap-based step size. It characterizes this rule through local certificates, provides a separate chance-feasibility cap, and evaluates synthetic problems and a practical RL variant.

Critical strengths
  1. The practical cap is simple and inexpensive to incorporate into existing trust-region methods.
  2. The synthetic curvature tests and matched cap-only ablation help isolate individual components. The paper also clearly distinguishes its theoretical guarantees from empirical heuristics.
Critical weaknesses
  1. Theorem 1 follows directly from intersecting two scalar admissibility intervals. Keeping a lower Taylor model above a known objective bound does not itself guarantee descent, as the paper’s own counterexample demonstrates. The optimization benefit of this particular certificate over conventional descent-based step selection remains unclear.

  2. The RL implementation uses neither a certified objective bound nor the finite-curvature rule or chance-safety cap. Although acknowledged, this leaves the main guarantees unvalidated in policy learning. In Table 4, all three mean costs exceed the budget, while PointPush1 retains only 40.6% of the selected baseline’s return.

  3. The cap-only ablation covers just one task and three seeds. More importantly, the generic-shortening comparison remains explicitly preliminary, with missing protocol details, paired uncertainty, and unresolved issues. Its similar returns and reported 5.2% reduction in cost-excess area do not yet establish an advantage attributable to objective information. Completing this controlled comparison is essential.

Questions for authors

Please check the weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5L2k

Thank you for the careful review and for asking us to clarify what the objective-based step adds beyond generic shortening.

The analysis identifies an objective-dependent scale and conditions under which it leads to convergence. This answers the optimization part of your question; attributing the empirical improvement to objective information also requires the generic-shortening control.

Theorem 1 has a short proof because the scalar admissible sets are intervals. For a given objective gap and direction, the candidate becomes independent of the radius once δ≥dηsmPol22\delta\ge\frac{d\eta_{\mathrm{smPol}}^2}{2}. On the exact quadratic L(x)=x22L(x)=\frac{x^2}{2} with F=β=1F=\beta=1 and Llb=0L_{\mathrm{lb}}=0, the smooth branch yields ηsmPol=2−1\eta_{\mathrm{smPol}}=\sqrt{2}-1, so x+=(2−2)xx^+=(2-\sqrt{2})x and the objective contracts by approximately 0.343. A bare fixed-radius step instead has constant displacement 2δ\sqrt{2\delta}. A conventional descent line search can also control excessive steps; the paper does not establish that this certificate dominates such a line search.

Descent requires more than a valid lower bound. With Llb=L∗L_{\mathrm{lb}}=L^*, let χ=βqΔd2\chi=\frac{\beta q\Delta}{d^2} and a=21+1+2χa=\frac{2}{1+\sqrt{1+2\chi}}. On the objective branch the upper Taylor bound gives Δ+≤2(1−a)Δ\Delta^+\le2(1-a)\Delta, which is a strict contraction when χ<4\chi<4. Together with the stated Fisher-PL condition, the paper establishes convergence across both branches for every fixed positive radius and bounds entry into the objective-binding region. The changing-Fisher convex analysis addresses the corresponding metric drift. The clipped Polyak construction has clear precedent in SPSmax Loizou et al., 2021; our contribution is the particular smooth certificate and its interaction with Fisher movement and the branch analysis.

The exact finite-curvature rule is tested in the submitted E1 sensitivity experiment. Across 525 trajectories and 31,500 updates, valid smoothness overestimates preserve the true-curvature certificate to the declared −10−10-10^{-10} tolerance. At radii 0.1, 0.3 and 1.0, all 15 starts reach gap 10−1010^{-10} within 60 iterations at 1×1\times, 2×2\times and 4×4\times the true curvature. A 10×10\times bound slows progress, and the deliberately invalid 0.5×0.5\times estimate fails the certificate on 37% of updates at δ=0.3\delta=0.3. The synthetic cost experiment separately tests the feasibility cap. Both concern settings where the required inputs are available; they do not verify the neural-policy assumptions.

Under a fixed augmented controller, the three-seed PointGoal1 ablation reports +8.6% final return, +8.1% return area and -5.7% cost-excess area when the cap is enabled. Accepted controller-active updates differ materially on 7.0% of cases, and the median of the three seed-level median candidate ratios is 0.83. The cap can therefore affect the training trajectory selectively, even when later controller labels obscure its activity. These are descriptive results from one task; the 1.10 return-SD ratio provides no evidence of variance reduction.

Attributing that improvement specifically to objective information requires the generic-shortening control. Table 3 remains incomplete: its 20.5 versus 20.7 returns and 5.2% lower cost-excess area cannot establish dominance, and marginal standard deviations cannot recover paired uncertainty. The draft sentence and broken appendix reference are our reporting errors. The cap-on/off result supports an effect under its specified controller, while the comparison with generic shortening remains unresolved.

The practical implementation also leaves the finite-curvature and chance-feasibility guarantees unused. Its running-best target is not a certified bound, and its mean-cost controller does not meet the budget in the primary experiments. The navigation results retain 92.7% to 96.8% of the radius-selected baseline's return at the shared δ=0.01\delta=0.01 setting, while the pushing result retains 40.6% and has lower, still infeasible cost. The analysis and controlled tests establish the objective-scaling mechanism. The remaining practical questions are cost control and attribution of the cap's benefit to objective information.

Thank you for your reply. After considering your response, I would prefer to keep my original score.

Reviewer_pGrmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes new step-size rules for constrained MDPs in a natural policy-gradient-style algorithm. The first (Theorem 1) takes the minimum of the TRPO step size and a Taylor-based bound that ensures the smoothness-based lower bound on the objective remains above a specified lower bound. The second, employed in Algorithm 1, takes the minimum of the TRPO step size and a smooth Polyak-rule-style bound. The paper provides further theoretical results on convergence under additional assumptions and a separate step-size cap intended to guarantee cost feasibility under a chance constraint. The method is evaluated on three Safety Gymnasium tasks.

Critical strengths

None

Critical weaknesses

1. Unclear motivation and novelty. The paper's narrative is difficult to follow, and it is unclear what the main theorems establish beyond the stated bounds or why these results constitute a substantial contribution. Theorem 1 characterises the largest step size satisfying both a quadratic trust-region bound and a smoothness-based lower-Taylor bound. This characterisation appears relatively straightforward; the paper does not convincingly explain why it is useful in practice or what insight it provides beyond combining these two constraints.

2. Unclear constraint handling and mismatch in constraint formulations. The paper refers to a “Lagrangian backbone” without adequately specifying the constrained optimisation procedure or explaining how the proposed step-size rules interact with it. Moreover, the theoretical formulation uses a chance constraint on cumulative trajectory cost (Eq. 1), whereas the experiments evaluate episodic mean cost (Table 4). The relationship between these formulations, and the extent to which the theoretical guarantees apply to the experimental setting, remain unclear.

3. Unclear practical applicability and mismatch between theory and algorithm. The theoretical results depend on quantities such as smoothness constants, eigenvalue bounds, and a valid objective lower bound, but the paper does not explain how these quantities can be obtained in practice or what guarantees remain when they are unavailable. Furthermore, Algorithm 1 appears to use a step-size rule that differs from those established in Theorems 1 and 2. This leaves the relationship between the theoretical results and the implemented algorithm unclear.

4. Unconvincing empirical evidence and insufficient baseline comparisons. The experiments do not establish a consistent performance advantage over TRPO-Lagrangian, and the method fails to meet the specified mean-cost budget on all three tasks. The main paper reports only one baseline, which is insufficient to assess the method against the range of existing constrained policy-optimisation approaches. Although the appendix includes additional comparisons, their outcomes and implications are not clearly articulated. The use of only three random seeds also makes the empirical conclusions difficult to assess reliably.

The evaluation protocol further needs clarification: how is δ\delta selected for TRPO-Lagrangian and CP-SPO, and are the selection procedures comparable? In particular, the abstract claims reduced sensitivity to δ\delta, but the experiments presented in the main paper do not convincingly substantiate this claim.

5. Incomplete presentation and lack of inspectable formalisation artefacts. Several references appear to be missing, and the paper contains author-facing draft comments, giving the impression of an insufficiently finalised submission. The AI-use statement says that Lean was used to formalise the proofs, but the submission provides no code or sufficient details to inspect or reproduce this formalisation.

Questions for authors
  1. What concrete guarantees do the theoretical results provide for the proposed method, and what do they establish beyond the individual step-size bounds?

  2. Which of the theoretical guarantees carry over to Algorithm 1 as implemented? What is the gap between the assumptions required by the theory and the quantities available in practice?

  3. How does stochastic estimation of the policy gradient and other quantities used by the step-size rules affect the validity of the theoretical guarantees and the method's practical performance?

  4. How is the trust-region parameter δ\delta selected for CP-SPO and the TRPO-Lagrangian baseline? Is the selection procedure comparable across methods, and what evidence supports the claim that CP-SPO is less sensitive to δ\delta?

  5. What are the main limitations of the proposed method?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pGrm

Thank you for the detailed questions. We analyze an objective-dependent scale within a Fisher movement budget. Theorem 1 identifies the largest scalar satisfying both local certificates. With Δ=L−Llb\Delta=L-L_{\mathrm{lb}}, d=g⊤F−1gd=g^\top F^{-1}g and q=∥F−1g∥2q=\lVert F^{-1}g\rVert^2, the root dη+βqη22=Δd\eta+\frac{\beta q\eta^2}{2}=\Delta gives the switching radius δswitch=dη22\delta_{\mathrm{switch}}=\frac{d\eta^2}{2}. Above it, increasing the radius leaves the candidate unchanged. With the exact optimum as target, the later results give changing-metric convex rates and convergence across both branches under the stated Fisher-PL and χ=βqΔd2<4\chi=\frac{\beta q\Delta}{d^2}<4 conditions. On the objective branch, a=21+1+2χa=\frac{2}{1+\sqrt{1+2\chi}} gives Δ+≤2(1−a)Δ\Delta^+\le2(1-a)\Delta. Capping a Polyak step has precedent in SPSmax; our analysis concerns the smooth root, its Fisher geometry and the resulting branch transitions. A loose lower bound alone does not ensure descent.

Algorithm 1 uses the algebraic β=0\beta=0 specialization and an empirical reward target. It bounds the proposal in the estimated metric, then applies an empirical KL check. These checks do not establish population descent or convergence for the full controller. Known reward bounds can supply Llb=−Rmax⁡1−γL_{\mathrm{lb}}=-\frac{R_{\max}}{1-\gamma} for discounted bounded rewards, although it can be too loose to be useful. Damping stabilizes the Fisher solve but does not certify its population assumptions. Curvature and cost-tail certificates require additional problem information.

In loss notation, a Lagrangian direction uses ∇L+λ∇Jc\nabla L+\lambda\nabla J_c, with a nonnegative multiplier. The reported settings are budget 25, controller margin 2.5 and dual stepsize 0.05. The reported primary algorithm applies the mean-cost controller after forming its reward candidate. The cap-only experiment holds an augmented reward-cost controller fixed, so the protocols differ.

Theorem 2 instead bounds P(Z>C)\mathbb{P}(Z>C) for a fresh trajectory at each fixed policy. It requires a surrogate-feasible start and certified sub-Gaussian tail, derivative and curvature information. The RL runs disable this chance cap and remain above the mean-cost budget; they do not test that theorem. The synthetic cost experiment tests it where the inputs are available. CPO also constrains expected costs, making it a mean-cost comparator.

The appendix tracks estimation error. If the natural-direction error is at most ζt\zeta_t in Fisher norm and the scalar error is at most btb_t, the displacement error is bounded by

rt=ηtζt+bt(dt+ζt).r_t=\eta_t\zeta_t+b_t\left(\sqrt{d_t}+\zeta_t\right).

The rate retains an accumulated potential-error term. A statistical guarantee requires valid simultaneous error bounds along the implemented trajectory. The running-best stationarity result assumes exact unsmoothed returns and true-loss Armijo acceptance. Reward confidence bounds concern evaluated policies, not the best unvisited policy; they do not validate the experimental EWMA target.

CP-SPO uses δ=0.01\delta=0.01 throughout. TRPO-Lagrangian is evaluated at 0.001,0.01,0.05,0.1\\{0.001,0.01,0.05,0.1\\} at 500k steps; the δ=0.05\delta=0.05 configuration, selected by mean return across three seeds, is extended to 1M. There is no complete 1M sweep or real-RL CP-SPO radius sweep. Reduced radius sensitivity is supported by the switching analysis and synthetic tests. The finite-curvature study contains 525 trajectories and 31,500 updates: no valid-bound residual falls below −10−10-10^{-10}, while the invalid 0.5×0.5\times estimate fails on 37% of updates at δ=0.3\delta=0.3.

The appendix compares compact CPO-style and PPO-Lagrangian implementations, official OmniSafe CPO, and seven methods on pushing. Their settings and averaging windows differ. Official CPO has lower cost and lower return at the reported markers. These comparisons show different return-cost tradeoffs, not constrained dominance. Three seeds, target sensitivity, above-budget costs and the pushing return deficit limit the practical conclusions.

The unfinished Table 3 text and broken appendix reference are our reporting errors. The generic-shortening comparison remains inconclusive without its paired seed record. We have prepared an anonymous proof package with pinned Lean 4.33.1/mathlib, a claim-to-theorem map and the recorded audit of 351 public theorem/lemma statements. It covers conditional mathematics, not the RL trainer or measurements, and still needs to be made available anonymously for review.

Thank you for the response. It largely confirms my concerns about limited novelty and relevance to the practical algorithm.

Moreover, the Lagrangian constraint-handling mechanism remains insufficiently specified.

Several of my questions also are insufficiently addressed, particularly regarding the practical implications of the theoretical results and the empirical evidence for reduced sensitivity to the trust-region radius. I therefore maintain my original assessment.

Reviewer_Q85NOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CP-SPO, which takes the smaller of a trust-region step and a smooth Polyak step along the natural-gradient direction (Theorem 1). The smooth Polyak step is determined by the lower bound LlbL_{lb} and β-smoothness. The paper also gives convergence rates under Llb=L⋆L_{lb}=L^\star and a separate cap that preserves the cumulative-cost chance constraint (Theorem 2).

Experiments are conducted on synthetic problems (E1–E5) and three Safety-Gymnasium tasks. On Safety-Gymnasium, the paper uses a practical variant consisting of the first-order cap with β=0, a running-best reward proxy, and a Lagrangian fallback; the cap of Theorem 2 is not used. In the cap-on/off comparison on PointGoal1, final return improves by 8.6%. At one million environment steps, CP-SPO retains 93–97% (navigation) and 41% (PointPush1) of the return of radius-selected TRPO-Lagrangian. Mean costs exceed the budget on all three tasks.

Critical strengths

None

Critical weaknesses
  • The theory is disconnected from the experiments, and the theory itself is thin. The theory covers a rule that uses a certified lower bound and β, as well as the cap of Theorem 2. The Safety-Gymnasium experiments use a different rule, and the cap of Theorem 2 is disabled (Section 5). In Proposition 1, the result closest to the practical variant, the gap equals exactly the prescribed margin mtm_t (Remark 3), so the size of the cap is determined by mtm_t rather than by the objective value. The convergence results (Theorems 6 and 9) are, as the authors acknowledge in Remark 7, specialized exact-oracle calculations. Theorem 1 itself only takes the intersection of two intervals. Apart from the trust-region upper bound 2δ/d\sqrt{2\delta/d} and the natural-gradient direction, its min(Polyak step, upper bound) form is shared with SPS$_{\max}$ (Loizou et al., 2021).

  • The main results are incomplete, and the empirical advantage is unclear. Section 6 still contains the draft sentence "Author draft: supplied summaries; measurement and protocol confirmation pending." The comparison with generic shortening (Table 3) consists of preliminary values, with details deferred to "Appendix ??". Even with these preliminary values, the cap does not improve return over generic shortening (20.5 vs. 20.7); the 5.2% lower cost-excess area is reported without uncertainty. The cap-only ablation (Table 2) reports only relative changes over three seeds and uses a controller different from the primary implementation. In Table 8, which compares the methods at the same δ=0.01, CP-SPO's mean return is below that of TRPO-Lagrangian on all three tasks, and its lower cost on CarGoal1 and PointPush1 still exceeds the budget.

Questions for authors
  • On the comparison with generic shortening: Can the authors provide the finalized version of Table 3 (per-seed values and uncertainty)?

  • On the cap-only ablation: How does the cap-off configuration in Table 2 relate to TRPO-Lagrangian (δ=0.01) in Table 8?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Q85N

Thank you for the careful review and for pointing out where the controller comparisons needed clarification.

The cap-only ablation in Table 2 and the TRPO-Lagrangian row in Table 8 use different controllers. Table 2 holds the augmented reward-cost controller, direction, target, radius, acceptance checks, and three paired seeds fixed, and toggles the cap. Table 8 compares the primary implementations, whose update rules differ even at the same radius. The reported +8.6% return and -5.7% cost-excess area in Table 2 concern the cap within that fixed controller; they do not imply that the primary CP-SPO implementation outperforms TRPO-Lagrangian at δ=0.01\delta=0.01.

Table 3 is incomplete; the unfinished text and broken appendix reference are our errors. The two marginal means and standard deviations cannot determine the paired uncertainty: that requires the seed-level differences. The reported 20.5 versus 20.7 return and 5.2% cost-excess reduction suggest a tradeoff, but the comparison with generic shortening remains inconclusive without the paired record. Table 2 supports the cap-on/off effect under its specified controller, with three seeds and without a significance claim. Completing the generic-control record is necessary for the stronger attribution claim.

As you point out, SPSmax already caps a sample-loss Polyak step by a maximum learning rate Loizou et al., 2021. We analyze the finite-curvature objective root along a changing natural-gradient direction, subject to a Fisher movement budget. With Δ=L−Llb\Delta=L-L_{\mathrm{lb}}, d=g⊤F−1gd=g^\top F^{-1}g and q=∥F−1g∥2q=\lVert F^{-1}g\rVert^2, the objective root solves dη+βqη22=Δd\eta+\frac{\beta q\eta^2}{2}=\Delta. The movement bound gives the switching radius, δswitch=dη22\delta_{\mathrm{switch}}=\frac{d\eta^2}{2}. Once the objective branch binds at a given iterate, increasing the radius leaves that candidate unchanged. The convergence analysis then covers changing Fisher metrics and progress on both branches.

For L(x)=x22L(x)=\frac{x^2}{2}, F=β=1F=\beta=1 and Llb=0L_{\mathrm{lb}}=0, the smooth objective step has η=2−1\eta=\sqrt{2}-1. When this branch binds, the update is x+=(2−2)xx^+=(2-\sqrt{2})x and the objective contracts by 6−426-4\sqrt{2}, approximately 0.343. By comparison, the unbacktracked fixed-radius step has displacement 2δ\sqrt{2\delta}, which does not shrink with x. The example explains the scaling; it does not compare performance with a well-tuned descent line search. More generally, the paper derives contraction under the stated exact-optimum, Fisher-PL and χ=βqΔd2<4\chi=\frac{\beta q\Delta}{d^2}<4 conditions, including progress before the objective branch becomes active. These guarantees are conditional on those assumptions, which we have not established for the neural policies.

The submitted E1 curvature study tests the rule on 525 trajectories and 31,500 updates. At valid smoothness overestimates, the minimum true-curvature certificate residual is −1.55×10−15-1.55\times10^{-15}, within the declared −10−10-10^{-10} tolerance. At δ∈0.1,0.3,1.0\delta\in\\{0.1,0.3,1.0\\}, every start reaches objective gap 10−1010^{-10} within 60 iterations at 1×1\times, 2×2\times and 4×4\times the true curvature. The deliberate 0.5×0.5\times underestimate fails the certificate on 37% of updates at δ=0.3\delta=0.3. The practical RL runs test the proxy cap under different conditions.

As you observe, Proposition 1 uses exact monotone returns, under which the running-best gap reduces to the prescribed margin mtm_t. The proposition establishes that a shrinking target margin, normalized by the natural-gradient quantity dtd_t and combined with true-loss Armijo acceptance, can give a stationarity bound without knowing L∗L^*. It does not show that the target estimates the remaining global objective gap. The noisy EWMA target and subsequent cost controller in the experiments are outside that result.

The Safety-Gymnasium results show a return-cost tradeoff and do not establish constrained dominance. At 1M steps the shared-radius method retains 96.8% and 92.7% of the selected baseline's navigation return, while the pushing task shows a substantial return loss. All primary mean costs remain above 25. The objective scale has a precise switching characterization and conditional convergence results, supported by direct finite-curvature tests. The paper also gives a separate cost-feasibility analysis and tests the inexpensive practical cap. Attribution of its empirical benefit still needs the paired generic-control evidence.

Thank you for the response and for clarifying the relation between Table 2 and Table 8. The response confirms the main points of my review: the comparison with generic shortening (Table 3) is still incomplete and no finalized values are provided, the convergence results do not cover the neural policies or the practical variant, and the Safety-Gymnasium results do not show an advantage over TRPO-Lagrangian, with all mean costs above the budget. I therefore maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.