- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Proximal Behavior Reshaping (PBR), which combines a PPO-trained Gaussian policy with a Q-guided flow transformation to generate richer behavior. Experiments on navigation and continuous-control benchmarks show improvements over PPO with additional computational cost.
Critical strengths
- The methodology is clear and practical. By retaining PPO to optimize a Gaussian anchor, using a Q-function to guide local action modifications, and learning a flow-based behavior transformation, the method offers a useful tradeoff between optimization simplicity and behavioral expressiveness.
- The KL-regularized policy improvement formulation explains how action values and transport costs jointly determine the target distribution, providing a good mathematical motivation for action target construction.
- The experimental evaluation is conducted across six Isaac Lab tasks and six MuJoCo Playground tasks with 5 random seeds. This broad coverage provides substantial empirical support for the method’s effectiveness.
- The local Q-value alignment analysis and intervention experiments that switch to direct Q-guided optimization of the Gaussian mean at different training stages provide meaningful insights into the method’s underlying mechanisms.
Critical weaknesses
- The failure of a Gaussian policy on Wall2D cannot be directly attributed to the coupling between policy optimization and behavior execution. The success of gSDE suggests that temporally structured exploration offers an alternative explanation. The current evidence therefore does not sufficiently establish the claimed architectural limitation.
- The ablation studies do not sufficiently isolate the contribution of the flow-based transformation. Comparisons with simpler alternatives, such as a residual MLP trained on the same action targets or a single Q-gradient-based action refinement step, are missing. Moreover, the direct Q-sampling ablation is limited to one task. It therefore remains unclear whether the gains specifically depend on flow or could be achieved with a simpler behavior transformation.
- The paper provides no theoretical guarantee for the joint updates of the Gaussian policy, flow transformation, and Q-function. In particular, PPO clipping constrains the Gaussian anchor’s probability ratios but does not directly constrain changes in the executed policy induced by flow updates. Consequently, the stability benefits attributed to this separation remain primarily empirically motivated.
- QFLEX is a particularly relevant baseline because it also combines a Gaussian source policy with a Q-guided flow. However, its Isaac Lab results rely on cross-framework adaptation, which introduces implementation-related uncertainty and limits the strength of conclusions about PBR’s advantage over QFLEX.
- The computational overhead is substantial: on Ant, PBR requires approximately the training time of PPO for the same number of environment interactions. Although the paper reports this cost, comparisons under a fixed wall-clock budget or in terms of time to reach a target return would be needed to establish its practical training efficiency.
Questions for authors
No.
3 comments
Response to Reviewer to8C (2/2)
W3. Theoretical guarantee for the joint updates.
We agree that PPO clipping alone does not constrain the executed policy. The revision now bounds the executed policy's change directly (new Appendix D), in three steps:
- What PPO controls. Within an iteration the flow is fixed, so from the anchor's perspective it is part of the environment, and rollouts store the Gaussian sample itself. PPO's ratio is therefore exact for the anchor. The remaining question is how much the executed policy changes between iterations, when the flow is updated too.
- Decomposition (Proposition D.1). Between iterations, the executed policy moves by at most the anchor's change, which PPO regulates through its KL-adaptive update, plus the flow's displacement at the two iterations. This holds for any flow and any critic.
- Bounded flow displacement (Lemmas D.2 and D.4). The flow's training targets have bounded expected transport cost, in units of the anchor's standard deviation, for any critic, because the policy proposes the candidates and the critic only compares them. An ideal flow moves actions no further than its targets.
Measured drift is consistent with this: across six tasks, the executed policy's drift between checkpoints is 0.95–1.19× the anchor's, and both decrease over training. Only the step from targets to the trained network relies on an idealized flow, which is why we measure the drift directly. The stability argument therefore no longer rests on empirical motivation alone: the per-update change of the executed policy is bounded by quantities PPO and the target construction control, and the measured drift stays at the anchor's scale.
W4. Reliability of the QFLEX comparison.
We agree that a cross-framework port carries implementation uncertainty. We therefore added a controlled comparison inside our own pipeline: variant (3) trains PBR's flow on QFLEX's critic-ascent targets (20 gradient steps with QFLEX's step rule), with everything else unchanged. It falls below PBR on both Ant and Allegro (new Section 8.3). Because this changes a single design choice within the same implementation, it avoids confounding from differences between codebases. We keep the port in Appendix B.2 only as a best-effort transfer.
W5. Computational cost.
PBR reaches PPO’s return at the original training budget after about 0.11 h on Ant, before PPO reaches that budget at 0.18 h. At the same 0.18 h budget, PBR’s mean return is already higher (134.06 vs. 124.64). With PPO trained further to PBR’s full wall-clock budget of 0.61 h, PBR remains higher (145.32 vs. 130.43) and leads in every training run. In addition, PBR's Gaussian alone retains most of its gain at deployment (new Table 2), so the deployed policy can run at PPO's inference cost, and the overhead applies to training only.
Summary of revisions: new Section 8 (Gaussian-only deployment, proximity, design ablations) and new Appendix D (Proposition D.1, Lemma D.2, measured drift).
We would be glad to clarify anything further.
Dear Reviewer to8C,
Thank you again for your review. We have updated the paper with the completed ablation results. In our earlier response, the Allegro results were at 2.5k training iterations. We have now finished the full-length runs and added Digit Loco-Manipulation (revised Section 8.3, Tables 3 and 4).
W2 and W4. Ablations of the transformation and its targets, with PPO still updating the Gaussian:
| Method | Ant | Allegro Repose | Digit Loco-Manipulation |
|---|---|---|---|
| Single Q-gradient step | 126.00 ± 15.46 | 140.35 ± 36.29 | 60.20 ± 3.09 |
| Direct candidate selection | 139.64 ± 3.55 | −3.25 ± 2.40 | −9.85 ± 5.40 |
| QFLEX-style critic-ascent targets | 132.64 ± 8.55 | 150.57 ± 21.25 | 55.86 ± 8.62 |
| PBR | 145.48 ± 13.03 | 159.42 ± 18.77 | 62.26 ± 1.54 |
W2. Residual MLP instead of the flow (Ant and G1 Rough unchanged):
| Task | Return: PBR | Return: MLP | Spread: PBR | Spread: MLP |
|---|---|---|---|---|
| Allegro Repose | 159.42 ± 18.77 | 59.58 ± 17.45 | 101% | 50% |
All three variants fall below PBR on every task, and direct selection collapses on Allegro and Digit. On Allegro, the MLP remains far below PBR and still narrows the executed behavior, while the flow preserves the targets' spread.
As the discussion period nears its end, we would welcome any further questions or comments.
Response to Reviewer to8C (1/2)
We thank the reviewer for the detailed review, and for highlighting the practical tradeoff PBR offers, the KL-regularized motivation for target construction, the breadth of the evaluation, and the value-alignment and intervention analyses. We address each weakness below with new experiments, a new formal analysis, and revisions to the paper.
W1. Attribution of PPO's failure on Wall2D, and gSDE.
We agree that temporally structured exploration is an alternative route to success on Wall2D. The paper does not attribute PPO's failure there to the coupling of policy optimization and behavior execution. In Section 4, we note that gSDE reaches the goal through temporally persistent, state-dependent exploration, and conclude only that neither exploration magnitude nor policy expressiveness alone is sufficient. In Section 6.1, we then present PBR as a different route from gSDE.
The two routes differ in mechanism. gSDE changes PPO's noise process, making it state-dependent and temporally correlated but independent of action value. PBR keeps PPO's Gaussian and its state-independent standard deviation unchanged, and uses the critic to reshape behavior. Notably, PBR's Gaussian evaluated alone, without the flow, reaches 89.3% mean success on Wall2D, nearly the same as full PBR with the flow (90.0%), versus 19.5% for PPO with the identical policy class. The same holds at scale, without the flow, PBR's Gaussian still outperforms PPO on Ant and Allegro (new Table 2).
The limitation PBR addresses is therefore one of learning, not of the policy class: PPO's own Gaussian can represent the bypass, but does not discover it from its own behavior. gSDE resolves this by changing the policy's noise, whereas PBR resolves it while keeping the policy exactly as PPO optimizes it. The two are complementary routes rather than competing explanations.
W2. Isolating the flow: simpler alternatives.
We added all three suggested comparisons (new Section 8.3), each changing one component of PBR within the same implementation.
Without a learned transformation. Both variants execute actions derived from the critic directly, with PPO still updating the Gaussian:
A single Q-gradient step falls below PBR on both tasks. Direct candidate selection, now with exactly PBR's weights, transport penalty and identity option, extends the original Digit comparison to Ant and Allegro. It also falls below PBR on both and collapses on Allegro.
A residual MLP instead of the flow. The MLP is trained on PBR's own targets. Besides return, we report spread: the within-state standard deviation of executed actions as a percentage of the targets' (Appendix E).
The MLP outperforms PBR on Ant but is far worse on Allegro and G1 Rough. As it is trained with squared error, it predicts the mean target for each source action and narrows the executed behavior, and its spread follows the same order as its return. The flow preserves the full spread of the targets on all three tasks.
PBR's gains therefore come mainly from its target construction, and the flow is what makes them robust across tasks.
(continued in the next comment)