acceptodds
Under review as a conference paper at ICLR 2027

From Tangent Geometry to Finite-Step Curvature in Safe Policy Optimization

Abstract

Constrained trust-region methods choose updates using first-order tangent models, but evaluate finite policy steps with nonlinear reward and cost surrogates. We characterize exactly what the tangent model determines and how curvature first alters that decision at finite step sizes. With linearized constraints, the Fisher trust region's complete reward–constraint score image is preserved by , the unique smallest linear subspace with this property. Reward–Constraint Subspace (RCS) uses this tangent-exact domain to reoptimize projected reward and constraint curvature, then applies empirical-KL correction and nonlinear same-batch checks before committing an update. We also quantify the approximation from restricting updates to : along a regular small-radius branch, the full-space and restricted solutions share the same tangent term and first differ by an Fisher-orthogonal displacement governed by cross-domain active-Lagrangian curvature. The resulting reward-surrogate gap is . Under weaker Fisher-relative remainder bounds, projection provides an finite-surrogate comparator for arbitrary . On Safety-Gymnasium, matched curvature diagnostics show improved finite-step model fidelity and update reorientation, while an independent nonlinear full-space reference recovers the predicted leakage magnitude, scaling, and direction.

Primary area:
reinforcement learning
Keywords:
Constrained Reinforcement Learning, Policy Optimization, Trust-Region Methods, Information Geometry, Constrained Markov Decision Processes

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_nBNLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a clean geometric characterization of constrained trust-region updates and an interesting analysis of curvature-induced departures from the tangent subspace.

Critical strengths

The paper identifies the smallest subspace that preserves all first-order reward and constraint scores, then characterizes the leading curvature-induced departure, including its direction and asymptotic error, under explicit regularity assumptions.

Critical weaknesses
  1. The theory’s relevance to actual training remains unclear. Proposition 2 analyzes a regular constrained reward-optimization branch, whereas approximately 71% of Goal1 updates use restoration. The leakage diagnostic also modifies the observed residuals and certifies only one of five seeds at the training radius. Please quantify the subspace-restriction error using actual training residuals, including restoration updates.

  2. The practical benefit of curvature needs stronger evidence. On Goal1, Full improves return from (19.44$\pm$1.34) to (20.35$\pm$1.49), with essentially unchanged final cost. On Push1, both variants yield a return of 1.05, although Full reduces the cumulative cost excess. The improved surrogate accuracy is convincing, but the robustness of the downstream gains is less clear. Please report confidence intervals for the paired differences across seeds.

  3. The broader baseline gains are difficult to attribute. RCS includes a conservative rolling-window cost estimator and nonlinear acceptance rules. On Goal1, the Linear variant already reduces cumulative mean-cost excess from PPO-Lagrangian’s 3,680 to 864, compared with Full’s 680. Thus, most of this improvement occurs without curvature. A comparison that uses the same cost estimator and acceptance rules for CPO would help establish which components drive the gains.

Questions for authors

Please check the weaknesses section.

Reviewer_9eG2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper characterizes a minimal Fisher subspace preserving first-order reward–constraint information and derives the leading curvature-induced departure from this subspace. It proposes RCS, which optimizes projected quadratic models with nonlinear acceptance checks. Validation includes Safety-Gymnasium benchmarks, matched curvature ablations, and numerical tests of the predicted displacement and surrogate-gap scaling.

Critical strengths

The finite-step analysis provides an explicit characterization of the omitted curvature direction. The independent full-space diagnostics and matched ablations offer useful evidence supporting the proposed mechanism.

Critical weaknesses
  1. The first-order subspace structure is closely related to existing CPO geometry. The main theoretical extension relies on a regular local branch, whose relevance to actual training remains insufficiently demonstrated, especially given the dominant use of restoration updates.
  2. The practical benefit attributable to curvature remains limited in the reported ablations. Cross-method comparisons do not clearly isolate its contribution from conservative cost estimation and recovery rules, while paired uncertainty estimates for the outcome improvements are missing.
Questions for authors
  1. Do the baselines use the same conservative cost estimator and acceptance rules? If not, what evidence separates their effects from the proposed curvature correction?
  2. How frequently do actual training updates satisfy the regular-branch conditions, and does the leakage prediction remain informative with the original training residuals?
Reviewer_6geuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mismatch in CPO-style constrained trust-region methods between choosing an update with a first-order tangent model and evaluating the finite step with nonlinear surrogates. t shows that Sm=range(H−1[g,b1,...,bm])\mathcal S_m = \text{range}(H^{-1}[g, b_1,...,b_m]) is the minimal subspace preserving the first-order reward–constraint score image. Then derives that the leading full-space departure from Sm\mathcal S_m is an O(δ)O(\delta) displacement governed by cross-domain active-Lagrangian curvature, and builds the RCS optimizer on this subspace.

Critical strengths
  1. The paper clearly separates what is already exact at first order from what finite-step curvature newly introduces, which gives a useful lens on CPO-style updates.
Critical weaknesses
  1. The theoretical novelty is limited. Prop. 1 and Cor. 1 appear to be straightforward extensions of the KKT characterization in CPO, and Prop. 2 uses classical parametric sensitivity analysis, which is cited neither in the main text nor in the proof.
  2. The connection between theory and practice is questionable. Fig. 5a shows that the leakage outside the subspace is not small at δ=10−2\delta = 10^{-2}, but he main experiments use exactly this value as the target KL, and the authors provide no clear explanation for this.
Questions for authors
  1. Please distinguish new contributions from extensions of existing results and add relevant citations.
  2. Why was δ=10−2\delta=10^{-2}, a radius at which the leakage is large, chosen as the training target KL, and how much does this leakage affect the practical performance of RCS?
Reviewer_wWEFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the geometry of constrained trust-region policy updates. It shows that the reward gradient and constraint gradients, after applying the inverse Fisher, span a minimal subspace that preserves all first-order reward/constraint scores in the trust region. It then studies how curvature causes a finite-step solution to leave this subspace and proposes RCS, which solves a small curvature-corrected problem in the subspace and uses nonlinear checks before accepting the update. Experiments are on Safety-Gymnasium and include curvature ablations and diagnostics of the predicted leakage direction.

Critical strengths
  • The paper gives a fairly clean geometric interpretation of what information is actually needed for the first-order constrained problem. I found the distinction between score-preserving tangent geometry and curvature-induced finite-step effects useful.
  • I also liked that the experiments try to test fairly specific theoretical predictions rather than only showing benchmark performance. In particular the leakage magnitude/direction tests make the theory more concrete.
Critical weaknesses
  • I am concerned that most of the evidence for improved finite-step modeling is essentially in-sample. The projected Hessians are estimated from a batch, the candidate is chosen using these models, and then the nonlinear reward/cost surrogate used for the final check is evaluated on the same batch. The model-error numbers are also measured at these committed same-batch steps. So it is not clear to me how much of the improved surrogate fidelity reflects genuinely better local prediction versus fitting the particular batch used to construct and select the update. A held-out batch, or at least evaluation on the next independently collected batch, would make this much more convincing.

  • There is also some distance between the exact geometric statement and the numerical implementation that is not quantified. The theory treats H and H^{-1} as exact, whereas the implementation uses a damped empirical Fisher and approximate conjugate-gradient solves. Since the main claim is that this particular subspace exactly preserves the first-order score image, I think it matters whether the actually constructed basis has noticeable score-preservation error. This seems especially relevant because the reduced optimization and the later curvature analysis are built on that basis. I could not find a numerical diagnostic showing how accurately g and the cost gradients are preserved after the approximate Fisher solves.

Questions for authors
  • Can you evaluate the quadratic-model prediction error and/or acceptance quality on data not used to estimate the gradients and projected Hessians? For example, what happens if the committed update is evaluated on a held-out rollout batch or on the next batch collected after the update?

  • In the actual implementation, how large is the numerical error in the claimed score preservation due to Fisher damping and the finite CG solve? For example, can you report the relative errors in g^T d and b_i^T d between a full candidate and its numerically constructed Fisher-subspace projection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.