acceptodds
Under review as a conference paper at ICLR 2027

SC-PPO: Secant-Corrected Proximal Policy Optimization for Execution-Robust Navigation

Abstract

This paper investigates execution-robust navigation in confined spaces, where a learned policy must remain collision-free even when its actions are executed with small errors. While canonical reinforcement learning (RL) algorithms such as proximal policy optimization (PPO) target the maximal expected return under exact execution, we argue that a deployed robot should trade a small amount of nominal return for insensitivity to execution errors, since limited clearance and closed-loop feedback turn these errors into collisions. However, the mismatch between the parameter space where PPO updates the policy and the action space where execution errors enter makes this goal difficult to pursue: any action sensitivity in the kernel of the policy Jacobian is invisible to the parameter gradient, so a policy can be flat in parameter space yet sharp in action space. To address the challenge, we propose a model-free RL algorithm, namely secant-correction PPO (SC-PPO), which compares two gradient evaluations within PPO's frozen evaluation context and uses their difference to correct the policy update without any disturbance model. Its action-space correction operates directly where execution errors enter and yields an action-risk bound under a measurable critic-fidelity condition, and the same operator supplies a gap-based criterion that replaces KL-divergence early stopping. SC-PPO is compared with PPO and flatness-based baselines on the BARN benchmark under persistent-bias and random-noise execution faults, and achieves the highest nominal success rate and the highest average success rate under faults. We further demonstrate the trained policy on a full-size humanoid robot.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot navigation, execution uncertainty, action robustness, execution errors

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pPBEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SC-PPO, a modification of PPO for navigation under action execution errors. It combines parameter- and action-space secant corrections with a revision-gap monitor for controlling policy updates. Experiments on BARN report modest improvements under execution disturbances while preserving nominal performance, alongside a qualitative demonstration on a humanoid robot.

Critical strengths
  1. The paper addresses the mismatch between commanded and executed actions, a practical source of navigation failures in constrained spaces.
  2. The evaluation covers persistent biases and random action noise, uses five training seeds and matched interaction budgets, and includes component ablations.
Critical weaknesses
  1. The results show gains under the tested disturbances, but do not establish that reduced action sensitivity explains them. Validating the Q-head’s predicted return changes against actual outcomes under small action perturbations, together with matched analyses of trajectories, speeds, and corrective behavior, would help distinguish this mechanism from general improvements in policy learning.
  2. The paper reports deployment without retraining, but does not sufficiently describe the simulated execution dynamics, including control response, delays, and velocity tracking errors, or their correspondence to the humanoid’s locomotion controller. Missing details on calibration and interface adaptation limit reproducibility and make the transfer conditions difficult to assess.
Questions for authors
  1. Does the goal in Fig. 2 represent the final destination or an externally generated waypoint/subgoal? If the system uses a global planner, what information does it provide to the policy, and how are route selection and local obstacle avoidance divided between these modules?
  2. How many episodes were evaluated per test map and disturbance condition? Based on existing evaluations, could the authors provide results for each training seed and uncertainty estimates for SC-PPO’s performance gains over PPO and SAM+PPO?
2 comments
AuthorsOpenReview ↗

Response to Reviewer pPBE

We thank the reviewer for the constructive suggestions. Several of them, such as validating the Q-head's predicted return changes against actual outcomes, are part of our revision plan.

Mechanism. We agree that the paper does not show that reduced action sensitivity explains the results. Given the analysis in point 1 of the general response, we now expect the current method not to reduce curvature-level sensitivity. In the revision we will test this directly: we will compare Q-head-predicted ΔQ\Delta Q with Monte-Carlo Δ\Deltareturn under small action perturbations, and report retention, i.e. success under faults given nominal success, together with matched statistics of trajectories, speeds and minimum clearance.

Execution dynamics and humanoid transfer. We agree that the paper lacks these details. Training and evaluation use the standard BARN Gazebo/Jackal setup. Beyond the injected test-time faults, the simulator models no additional actuation latency or velocity-tracking error. No calibration or interface adaptation to the humanoid was performed: the policy's (v,ω)(v,\omega) command is sent directly as forward-velocity and yaw-rate targets to the onboard locomotion controller. Transfer therefore relies on the policy tolerating the humanoid's tracking error without any adaptation. We will add these details to the appendix. We will also state explicitly that the humanoid run is a qualitative demonstration and not evidence of transfer.

Q1. Goal in Fig. 2. The goal is an intermediate waypoint produced by a global planner, not the final destination. The global planner is responsible for route selection, and the policy receives the waypoint as its goal input and handles local obstacle avoidance. We will state this division explicitly in the revision.

Q2. Episodes and uncertainty. Nominal evaluation uses one deterministic episode per test map per training seed (45 maps × 5 seeds = 225 episodes), so the nominal SC-PPO vs PPO difference is a single episode. Faulted families are evaluated on the same maps at each nonzero fault level. With five seeds, the SC-PPO gains over PPO and SAM+PPO are within seed noise. The revision will use multiple perturbed start poses per map, paired per-map tests, hierarchical bootstrap CIs and per-seed tables.

Reviewer_pPBEOpenReview ↗

Response to Rebuttal of Submission779

Thanks for acknowledging this concern. The proposed checks of Q-head predictions against actual return changes, together with examples comparing paths, speeds, and corrective actions under the same conditions, would help explain whether the robustness results come from reduced action sensitivity. Since these analyses have not been provided yet, I’m keeping my original assessment.

Reviewer_iFfLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the execution-robust navigation in the narrow spaces, which argues that the parameter-space flatness (pursued by SAM+PPO) does not imply an action-space robustness. The proposed SC-PPO then applies a secant-correction operator in both the parameter space and action space, aiming to guide the policy optimization towards a gradient flat direction.

Critical strengths

This paper studies a robot navigation problem with a useful observation, where robots fail the navigation task in tight spaces because commands are executed imperfectly is a true sim-to-real issue. This paper then provides a clear point: a policy can look stable when we perturb its weights and still be fragile when we perturb its actions. This is a reasonable argument for working on the actions directly, instead of relying on the parameter-space flatness.

Critical weaknesses
  1. The authors need to carefully reorganize the paper's writing, as the current logical expression is not friendly for reading. The unnecessary mathematical packaging needs to be avoided. For example, some terms are not defined upon their first appearance, while others do not significantly contribute to the subsequent logic after being mentioned. Otherwise, it is hard to clarify the definitions and relationships of complex terms in this paper.

  2. The proposed method is built upon the secant correction, but in Table 2, turning this correction off inside the action-space does not hurt the performance: scoring 40.7% under errors vs. 39.8% with it. While turning off the parameter-space version also makes no meaningful difference. What actually matters is adding a Q-head and pushing actions along its gradient, which is close to the well-known ideas. As a result, the claimed contribution is not sufficiently supported by the evidence.

  3. The motivation of this paper initially focused on the task scenarios involving narrow passages, while the experimental part did not provide a strong empirical evidence. In Figure 3, both the baseline method PPO and SAM+PPO failed, but SC-PPO did not successfully pass through the same narrow passage. Instead, it chooses to detour on another wide road, which is inconsistent with the motivation this paper aims to address.

Questions for authors

Please interpret the experimental results as reported in the paper, to explain whether the claimed contributions have brought the actual performance improvements.

1 comment
AuthorsOpenReview ↗

Response to Reviewer iFfL

We thank the reviewer for a frank and clear assessment.

Ablation. We agree with the reading of Table 2. Please see points 1 and 2 of the general response, which also give the reason we have found for why the secant correction has no measurable effect.

Q: Do the claimed contributions bring actual improvements? Read strictly, the paper does not establish an improvement over PPO: the nominal and faulted differences are both within seed noise. Within SC-PPO, the effects in Table 2 come from other components. Removing the action term (ηa=0\eta_a=0) lowers nominal SR by 5.8 points. Removing the revision-gap monitor nearly doubles the seed standard deviation of faulted SR (6.47 vs 3.37), so it mainly stabilizes training. Even these effects need more seeds to confirm.

Fig. 3 (detour instead of the narrow passage). Fig. 3 shows a single humanoid run under a persistent angular bias (bω=−0.3b_\omega=-0.3 rad/s). Route selection in our system is handled by a global planner that supplies waypoints to the policy, so the route taken in one run cannot be attributed to the policy alone. The reviewer is right that one qualitative run cannot support the motivation. A proper test needs success and retention rates stratified by passage clearance, and we will add them.

Presentation. We agree. In the revision we will define every term at first use, add a notation table, and move results that the main argument does not use (the conditional bound with the visibility constant, finite-sample conditions, the scalar step-size example) out of the main text. We will also restate the method around the components that the evidence actually supports.

Reviewer_Pz56OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a modification of PPO to make navigation policies more robust in confined environments. It observes that PPO updates parameters whereas execution errors enter in action space, and that components of the action-value gradient in the kernel of the policy Jacobian are invisible to the parameter gradient. It proposes SC-PPO: a secant correction operator applied to the clipped surrogate in parameter space (PSC) and to a learnt Q-head in action space, pulled back through the policy Jacobian as a DPG-style term (ASC), plus a revision-gap monitor (RGM) that replaces KL early stopping and adapts the probe step. An action-risk bound is given under a critic-fidelity assumption. The method is evaluated on BARN against PPO, SAM+PPO and GR+PPO under four test-time fault families, with matched ablations and a qualitative run on a humanoid robot.

Critical strengths

The problem is well formulated and genuinely interesting. The proposed method is grounded and requires neither additional samples neither changes at inference time.

Critical weaknesses
  1. Table 2 reports that the plain action gradient (no secant correction) is within seed noise of the proposed method (same for λθ=0\lambda_\theta = 0), undercutting the whole argument. What is left is PPO plus a Q-head with a DPG-style action-gradient term, a known combination, and a stopping rule whose main effect is variance reduction. The framing and title are therefore not supported by the evidence.

  2. The paper's ablation shows that SC-PPO is not statistically better than PPO (table 2: 39.84±3.3739.84\pm 3.37 vs 37.47±2.3837.47\pm 2.38 within seed noise and nominal 41.78 vs 41.33). They acknowledge this but conclude saying "the evidence is that the gain over PPO has the same sign in every family" which is not valid since each family is evaluated using the same trained policies. Furthermore, table 1 does not report STDs and bold values should be reserved for the best values with statistical significance (41.78 is identical to 41.33, not better, if the STD is 2). Furthermore, these runs are not even compute-matched since "SC-PPO [...] costs about 3$\times$ the actor compute of PPO" (line 325). Some obvious baselines are also missing (e.g. action noise (Tessler, 2019) (cited line 124)). Also, GR+PPO and SAM+PPO losing 4 to 7 nominal points also suggests tuning artefacts.

Questions for authors
  1. Why did you not report STDs in table 1 and for the nominal column in table 2? Do the differences in table 1 survive with them included?
  2. How does SC-PPO compare with PPO trained under Gaussian action noise of the test-fault magnitudes, which needs only a trivial "model"?
  3. Given λa=0\lambda_a = 0 matches or beats full SC-PPO, what evidence supports presenting the secant correction, rather than the Q-head/DPG term, as the contribution?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Pz56

We thank the reviewer for a precise and constructive review. Points 1–3 of the general response cover the main concerns. Below we answer the specific questions.

Q1. Missing STDs. Table 1 reports seed means only. With STDs (Table 2), none of the Table 1 differences between SC-PPO and PPO survives: the nominal difference is one episode out of 225, and the faulted difference is within one seed standard deviation. The bold entries should be removed.

Q2. PPO with Gaussian action noise. We did not run this, and we agree it is the most important missing baseline. As the general response notes, it also controls the policy-std confound that our comparison leaves open.

Q3. Why credit the secant correction rather than the Q-head/DPG term? On the evidence in the paper, nothing supports crediting the secant correction. The general response gives the reason we have since identified: the correction scales with ∥∇aQ∥\|\nabla_a Q\| and vanishes as the DPG term converges. We will reframe the contribution accordingly.

Compute and baseline tuning. SC-PPO matches PPO in environment interactions but not in actor compute, so a compute-matched PPO is required. For SAM+PPO and GR+PPO, the grids in Appendix J are small (ρ∈{0.001,0.005,0.01,0.05}\rho\in\{0.001,0.005,0.01,0.05\}; λGR∈{10−4,…,10−1}\lambda_{GR}\in\{10^{-4},\dots,10^{-1}\}). We agree that their 4–7 point nominal drop may reflect insufficient tuning rather than an inherent trade-off, and we will widen these sweeps.

I would like to thank the authors for this surprisingly candid rebuttal. All of my concerns are conceded. Thus, I do not see any reason to change my rating.

This is reinforced by the fact that the general response of the authors acknowledge that the necessary changes would be too significant to be evaluated fairly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for their careful reading. They agree on several points, and after re-examining our results we agree with them. We address these points here, and below each review we answer the remaining specific questions.

1. The ablation does not isolate the secant correction (Pz56, iFfL). This reading of Table 2 is correct. With the plain action gradient (λa=0\lambda_a=0), faulted SR is 40.68±3.2140.68\pm3.21, versus 39.84±3.3739.84\pm3.37 for full SC-PPO. Removing PSC (λθ=0\lambda_\theta=0) also stays within seed noise. On this evidence the secant correction is not shown to contribute, and the title and framing overstate it.

Prompted by the reviews, we found a structural reason for this result. The action-space probe is placed along the gradient, apr=μ−τaua^{pr}=\mu-\tau_a u. To first order this gives d(a)−u≈λaτaHfud^{(a)}-u\approx\lambda_a\tau_a H_f u, so the correction is proportional to u=−∇aQu=-\nabla_a Q. The DPG term itself drives μ\mu toward a maximizer of the learned QQ, where u→0u\to0, so the correction vanishes exactly where it should matter. This term controls the first-order part ε∥u∥\varepsilon\|u\| of the bound in Theorem 1. It does not control the curvature term βaε2/2\beta_a\varepsilon^2/2, which dominates near convergence and which is what governs sensitivity to execution errors. We think this explains why λa=0\lambda_a=0 matches the full method. It also points to a fix: placing the probe along random directions instead of the gradient makes the two-evaluation estimator target curvature, and the estimator then does not vanish at stationary points.

2. The evidence for improvement over PPO is not statistically supported (Pz56, pPBE). We agree. Nominal evaluation uses one deterministic episode per test map (45 maps × 5 seeds = 225 episodes). The nominal difference, 41.78% vs 41.33%, is therefore 94 vs 93 successful episodes, and it should not have been bolded. The faulted-SR difference (39.84±3.3739.84\pm3.37 vs 37.47±2.3837.47\pm2.38) is also within seed noise. Reviewer Pz56 is right that our "same sign in every family" argument is invalid, because all families are evaluated on the same trained policies. We withdraw the claim that SC-PPO improves on PPO.

3. Missing controls (Pz56). We agree that the following controls are needed and were not run: PPO trained with Gaussian action noise at the test-fault magnitudes; a compute-matched PPO (SC-PPO uses about 3× PPO's actor compute); and PPO with the revision-gap monitor alone. The first control matters especially, because PPO's own Gaussian exploration already optimizes an action-smoothed objective. The final policy standard deviation is therefore a confound that we neither controlled nor reported.

Planned revisions. We plan to (i) replace the gradient-aligned action probe with a curvature-targeting one; (ii) add the three controls above; (iii) evaluate with multiple perturbed start poses per map and use paired per-map tests with hierarchical bootstrap CIs, reporting per-seed results; (iv) report retention, i.e. success under faults given nominal success, which separates robustness from nominal capability; and (v) restructure the presentation as iFfL suggests. These changes alter the method and the main claims, so we do not think they can be completed and fairly re-evaluated within this discussion period. We are grateful for reviews that identified the right problems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.