- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides a clean geometric characterization of constrained trust-region updates and an interesting analysis of curvature-induced departures from the tangent subspace.
Critical strengths
The paper identifies the smallest subspace that preserves all first-order reward and constraint scores, then characterizes the leading curvature-induced departure, including its direction and asymptotic error, under explicit regularity assumptions.
Critical weaknesses
-
The theory’s relevance to actual training remains unclear. Proposition 2 analyzes a regular constrained reward-optimization branch, whereas approximately 71% of Goal1 updates use restoration. The leakage diagnostic also modifies the observed residuals and certifies only one of five seeds at the training radius. Please quantify the subspace-restriction error using actual training residuals, including restoration updates.
-
The practical benefit of curvature needs stronger evidence. On Goal1, Full improves return from (19.44$\pm$1.34) to (20.35$\pm$1.49), with essentially unchanged final cost. On Push1, both variants yield a return of 1.05, although Full reduces the cumulative cost excess. The improved surrogate accuracy is convincing, but the robustness of the downstream gains is less clear. Please report confidence intervals for the paired differences across seeds.
-
The broader baseline gains are difficult to attribute. RCS includes a conservative rolling-window cost estimator and nonlinear acceptance rules. On Goal1, the Linear variant already reduces cumulative mean-cost excess from PPO-Lagrangian’s 3,680 to 864, compared with Full’s 680. Thus, most of this improvement occurs without curvature. A comparison that uses the same cost estimator and acceptance rules for CPO would help establish which components drive the gains.
Questions for authors
Please check the weaknesses section.