acceptodds
Under review as a conference paper at ICLR 2027

VPPO: Verified Pathwise Policy Optimization

Abstract

Proximal policy optimization (PPO) reuses on-policy data through a clipped surrogate, but clipping alone does not explicitly bound policy change. Incorporating critic action gradients offers a complementary direction for policy improvement, provided that their reliability can be assessed. We introduce Verified Pathwise Policy Optimization (VPPO), an on-policy algorithm that combines clipped natural-gradient updates with probe-calibrated pathwise refinement. More specifically, VPPO performs multiple re-linearized steps on each rollout batch, subject to explicit KL bounds and nondecrease of the regularized surrogate. To calibrate trust in the critic-guided refinement, we use paired simulator trajectories with common random numbers to evaluate the proposed action direction. A damped-Fisher projection then preserves first-order compatibility between the refinement and surrogate directions. A matched-budget ablation shows that clipping remains necessary for stable batch reuse even under explicit KL control. On GPU-parallelized benchmarks, VPPO improves humanoid locomotion performance over the standard PPO configuration while achieving comparable final returns on additional control tasks. Under matched entropy regularization, VPPO reaches intermediate performance milestones earlier than PPO. Compared with off-policy baselines, VPPO also reaches target returns in less wall-clock time on most tested control tasks, despite equiring more environment steps.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, proximal policy optimization, natural policy gradients, trust-region methods, critic-gradient refinement, humanoid locomotion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uTDeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the VPPO algorithm to improve the reliability of PPO.

Critical strengths

A clear, rigorous paper on a well-motivated problem. The experimental result that VPPO outperforms PPO on some IsaacLab tasks is particularly interesting.

Critical weaknesses

The original TRPO solves a constrained optimization problem with approximated Hessian and backtracking line search. Despite being theoretically sound, its computational overhead is high as it estimates the Hessian of neural network parameters which is very large. PPO makes a trade-off: it is much less rigorous than TRPO, but simplifies the implementation by replacing those optimization-inspired methods with clipping. In the past few years, some algorithms have tried to make PPO as rigorous as TRPO by bringing back components like quadratic constraints and/or line search. This paper looks just like another work of this kind. The proposed algorithm is more complicated, slower, but does not offer better performance in most cases.

By the way, I believe that a detailed discussion on why VPPO outperforms PPO on IsaacLab tasks offers more insights and strengthens the paper, but the current empirical performance is not appealing enough to convince me to use VPPO in my problems.

Questions for authors

Please see Weaknesses.

Reviewer_GhL1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

VPPO combines PPO's clipped objective with repeated natural-gradient updates checked against empirical KL limits and the batch surrogate. It adds critic-gradient refinement, weighted using paired simulator probes and projected to preserve directional compatibility. The experiments support clipping in the tested walker-run configuration and show earlier G1 learning milestones when entropy coefficients are matched. Final returns are comparable to PPO on the six DMC-style tasks. The paper carefully distinguishes its theoretical results and batch-level checks from a guarantee of improved true return.

Critical strengths
  1. The update rule is clear. The Fisher projection has a direct geometric justification, the analysis includes approximate-solve error, and the line search checks the finite update separately (Eqs. 7–8; Appendix B).
  2. The clipping experiment retains explicit KL control and includes a larger-budget single-step comparison. This helps examine why repeated use of a rollout can benefit from clipping (Table 5).
  3. Matching the entropy coefficient and reporting curriculum levels are valuable controls. They show which apparent reward gains survive a closer comparison and why reward alone can misrepresent progress on rough terrain (Tables 6–7).
  4. The paper reports low-trust tasks, failed runs, and additional computation. The five-seed DMC-style results and the explicit limitations make the evidence easier to assess.
Critical weaknesses
  1. The probe signal needs a more independent check. The default probe runs for only two steps before bootstrapping from the same critic ensemble whose direction it is testing. Excluding calibration labels from critic training is useful, but it does not remove shared errors in the terminal estimate. Trust is also transferred from mean-action probe states to rollout actions and changing policy means, while the buffer can contain labels spanning roughly 1,280 iterations. Please compare probe signs with longer independent return estimates and measure calibration on fresh queries where refinement is actually applied. Testing the positive trust floor would also help, since Appendix B.5 correctly notes that its guarantee does not cover the implemented weight.
  2. It is difficult to identify what the probe-calibrated component adds. The no-refinement baseline also removes probes and ensemble training. It therefore does not show whether calibrated weights improve on constant weights, or how much the directional loss and Fisher projection contribute. This is especially relevant given the lack of a walker-run mean improvement and the small or conditional G1 gains in Section 7.2. A few controlled comparisons with constant or shuffled trust, the directional loss removed, and the aggregate projection removed would address this directly.
  3. The G1 results need a stronger comparison at matched difficulty. The existing entropy control removes the final flat-terrain reward gap. Rough-terrain results use only three seeds, and Figure 5 highlights the progressing subset, although Table 6 usefully reports the other runs. The fixed-command evaluation compares against standard-entropy PPO rather than the matched control. Please evaluate every final seed on common terrain and command distributions, including matched-entropy PPO and the updater without refinement. This would clarify whether faster curriculum progress also produces reliably better locomotion.
  4. Earlier progress in iterations does not yet establish a computational benefit over PPO. VPPO takes about 1.6–2.3 times longer per iteration, and its threshold times in Table 8 are slower than PPO's on all six DMC-style tasks. Appendix D also notes that timing was not process-isolated. A controlled timing comparison is needed to establish the practical tradeoff. The probe budget needs reconciliation too: advancing 256 environments for 40 steps per refresh adds substantial interactions before the branches used for the approximately 0.1% figure. Please report complete counters and distinguish sample or iteration gains from wall-clock gains.
  5. The clipping result supports a narrower conclusion than necessity under KL control. The main comparison uses one task and two seeds per variant. Although the nominal cap is shared, the unclipped updater moves about five times farther in realized rollout KL than the clipped updater (Section 7.1). A tighter-cap unclipped control at comparable realized KL would help determine whether clipping provides a distinct benefit or mainly reduces the effective step. Until then, I would state the result as evidence for clipping in this tested configuration.
Questions for authors
  1. How often do the short probe signs agree with longer independent return differences, and how accurate is calibration at current refinement queries? What happens with zero-floor or lower-confidence-bound weights?
  2. Do constant or shuffled trust weights perform worse when other training is unchanged? How much do the directional loss and aggregate projection each contribute?
  3. What are the results for all G1 seeds under fixed-difficulty, matched-entropy evaluation against PPO and the no-refinement updater, including uncertainty and completion rates?
  4. Which milestones are reached sooner in isolated wall-clock comparisons with matched PPO, and do probe-step totals include the full state-generation phase?
  5. Can the unclipped multi-step updater recover performance when its cap is reduced to match the clipped updater's realized KL?
Reviewer_QXZfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VPPO, which combines clipped natural-gradient updates with critic refinement calibrated through paired simulator probes. Experiments show faster learning on some control tasks and generally comparable final returns to PPO, at increased computational cost.

Critical strengths
  • The probe calibration method is interesting and inspiring. It evaluates critic-proposed directions using paired simulator rollouts with common random numbers, then calibrates ensemble disagreement against held-out probe labels excluded from critic training. This provides a more grounded measure of directional reliability than relying on ensemble agreement alone.
  • The direction-fusion mechanism has a clear mathematical justification. The damped-Fisher projection ensures that, under exact Fisher solves, the combined direction remains an ascent direction for the regularized surrogate to first order.
  • The empirical results show that VPPO reaches intermediate performance thresholds in fewer training iterations on some tasks, including under matched entropy coefficients.
Critical weaknesses
  • Paired perturbations measure the return response along the critic-proposed direction. Positive labels do not establish strong alignment with the true action gradient or substantial improvement: a nearly orthogonal direction with a small positive component may still receive a high trust weight.
  • The probes use only two simulator steps before critic bootstrapping, leaving potential terminal-value bias that common random numbers cannot eliminate. Mean-policy continuation also introduces a mismatch with the stochastic policy used for training.
  • Full VPPO does not outperform the clipped natural-gradient backbone without refinement on walker-run, and refinement provides only a small gain on G1 flat terrain. Positive results on rough terrain are limited by the small number of seeds and analyses restricted to successful runs. Further ablations are needed to isolate the contributions of probe supervision, trust weighting, and the two projection stages.
  • After matching entropy coefficients, the final-return advantage on G1 flat terrain largely disappears, while performance on the DMC-style tasks is generally comparable to PPO. The evidence does not support broad improvements in final performance.
Questions for authors

No

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.