acceptodds
Under review as a conference paper at ICLR 2027

Proximal Behavior Reshaping

Abstract

Policy optimization in continuous control commonly uses the same parametric distribution both as the object of optimization and as the behavior distribution for environment interaction. This coupling can be restrictive when effective exploration requires richer behavioral structure, while directly using learned action values for policy improvement can make learning sensitive to critic error. We introduce Proximal Behavior Reshaping (PBR), which separates the distribution directly optimized by PPO from the distribution executed for interaction. PBR retains a Gaussian policy as the optimization anchor and learns a residual flow that reshapes its sampled actions into a richer composed policy. Learned action values guide local, policy supported reshaping without directly determining the Gaussian policy update, and are trained using the current rollout together with a short buffer of recent policy data. On a goal reaching task with wall avoidance, PBR produces coherent obstacle bypassing behavior while making effective use of locally useful but imperfect action value estimates. Across six high dimensional Isaac Lab tasks spanning locomotion and manipulation, PBR exceeds PPO across the benchmark while providing strong and consistent performance relative to more direct guided and flow based alternatives.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_to8COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Proximal Behavior Reshaping (PBR), which combines a PPO-trained Gaussian policy with a Q-guided flow transformation to generate richer behavior. Experiments on navigation and continuous-control benchmarks show improvements over PPO with additional computational cost.

Critical strengths
  • The methodology is clear and practical. By retaining PPO to optimize a Gaussian anchor, using a Q-function to guide local action modifications, and learning a flow-based behavior transformation, the method offers a useful tradeoff between optimization simplicity and behavioral expressiveness.
  • The KL-regularized policy improvement formulation explains how action values and transport costs jointly determine the target distribution, providing a good mathematical motivation for action target construction.
  • The experimental evaluation is conducted across six Isaac Lab tasks and six MuJoCo Playground tasks with 5 random seeds. This broad coverage provides substantial empirical support for the method’s effectiveness.
  • The local Q-value alignment analysis and intervention experiments that switch to direct Q-guided optimization of the Gaussian mean at different training stages provide meaningful insights into the method’s underlying mechanisms.
Critical weaknesses
  • The failure of a Gaussian policy on Wall2D cannot be directly attributed to the coupling between policy optimization and behavior execution. The success of gSDE suggests that temporally structured exploration offers an alternative explanation. The current evidence therefore does not sufficiently establish the claimed architectural limitation.
  • The ablation studies do not sufficiently isolate the contribution of the flow-based transformation. Comparisons with simpler alternatives, such as a residual MLP trained on the same action targets or a single Q-gradient-based action refinement step, are missing. Moreover, the direct Q-sampling ablation is limited to one task. It therefore remains unclear whether the gains specifically depend on flow or could be achieved with a simpler behavior transformation.
  • The paper provides no theoretical guarantee for the joint updates of the Gaussian policy, flow transformation, and Q-function. In particular, PPO clipping constrains the Gaussian anchor’s probability ratios but does not directly constrain changes in the executed policy induced by flow updates. Consequently, the stability benefits attributed to this separation remain primarily empirically motivated.
  • QFLEX is a particularly relevant baseline because it also combines a Gaussian source policy with a Q-guided flow. However, its Isaac Lab results rely on cross-framework adaptation, which introduces implementation-related uncertainty and limits the strength of conclusions about PBR’s advantage over QFLEX.
  • The computational overhead is substantial: on Ant, PBR requires approximately 3.4×3.4\times the training time of PPO for the same number of environment interactions. Although the paper reports this cost, comparisons under a fixed wall-clock budget or in terms of time to reach a target return would be needed to establish its practical training efficiency.
Questions for authors

No.

3 comments
AuthorsOpenReview ↗

Response to Reviewer to8C (1/2)

We thank the reviewer for the detailed review, and for highlighting the practical tradeoff PBR offers, the KL-regularized motivation for target construction, the breadth of the evaluation, and the value-alignment and intervention analyses. We address each weakness below with new experiments, a new formal analysis, and revisions to the paper.

W1. Attribution of PPO's failure on Wall2D, and gSDE.

We agree that temporally structured exploration is an alternative route to success on Wall2D. The paper does not attribute PPO's failure there to the coupling of policy optimization and behavior execution. In Section 4, we note that gSDE reaches the goal through temporally persistent, state-dependent exploration, and conclude only that neither exploration magnitude nor policy expressiveness alone is sufficient. In Section 6.1, we then present PBR as a different route from gSDE.

The two routes differ in mechanism. gSDE changes PPO's noise process, making it state-dependent and temporally correlated but independent of action value. PBR keeps PPO's Gaussian and its state-independent standard deviation unchanged, and uses the critic to reshape behavior. Notably, PBR's Gaussian evaluated alone, without the flow, reaches 89.3% mean success on Wall2D, nearly the same as full PBR with the flow (90.0%), versus 19.5% for PPO with the identical policy class. The same holds at scale, without the flow, PBR's Gaussian still outperforms PPO on Ant and Allegro (new Table 2).

The limitation PBR addresses is therefore one of learning, not of the policy class: PPO's own Gaussian can represent the bypass, but does not discover it from its own behavior. gSDE resolves this by changing the policy's noise, whereas PBR resolves it while keeping the policy exactly as PPO optimizes it. The two are complementary routes rather than competing explanations.

W2. Isolating the flow: simpler alternatives.

We added all three suggested comparisons (new Section 8.3), each changing one component of PBR within the same implementation.

Without a learned transformation. Both variants execute actions derived from the critic directly, with PPO still updating the Gaussian:

MethodAntAllegro Repose
PBR145.48 ± 13.03130.13 ± 14.91
Single Q-gradient step126.00 ± 15.46103.93 ± 32.03
Direct candidate selection139.64 ± 3.55−0.48 ± 0.04

A single Q-gradient step falls below PBR on both tasks. Direct candidate selection, now with exactly PBR's weights, transport penalty and identity option, extends the original Digit comparison to Ant and Allegro. It also falls below PBR on both and collapses on Allegro.

A residual MLP instead of the flow. The MLP is trained on PBR's own targets. Besides return, we report spread: the within-state standard deviation of executed actions as a percentage of the targets' (Appendix E).

TaskReturn: PBRReturn: MLPSpread: PBRSpread: MLP
Ant145.48 ± 13.03165.89 ± 2.13103%79%
Allegro Repose130.13 ± 14.9151.84 ± 14.07101%53%
G1 Rough51.76 ± 0.47−7.59 ± 1.79100%33%

The MLP outperforms PBR on Ant but is far worse on Allegro and G1 Rough. As it is trained with squared error, it predicts the mean target for each source action and narrows the executed behavior, and its spread follows the same order as its return. The flow preserves the full spread of the targets on all three tasks.

PBR's gains therefore come mainly from its target construction, and the flow is what makes them robust across tasks.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer to8C (2/2)

W3. Theoretical guarantee for the joint updates.

We agree that PPO clipping alone does not constrain the executed policy. The revision now bounds the executed policy's change directly (new Appendix D), in three steps:

  • What PPO controls. Within an iteration the flow is fixed, so from the anchor's perspective it is part of the environment, and rollouts store the Gaussian sample itself. PPO's ratio is therefore exact for the anchor. The remaining question is how much the executed policy changes between iterations, when the flow is updated too.
  • Decomposition (Proposition D.1). Between iterations, the executed policy moves by at most the anchor's change, which PPO regulates through its KL-adaptive update, plus the flow's displacement at the two iterations. This holds for any flow and any critic.
  • Bounded flow displacement (Lemmas D.2 and D.4). The flow's training targets have bounded expected transport cost, in units of the anchor's standard deviation, for any critic, because the policy proposes the candidates and the critic only compares them. An ideal flow moves actions no further than its targets.

Measured drift is consistent with this: across six tasks, the executed policy's drift between checkpoints is 0.95–1.19× the anchor's, and both decrease over training. Only the step from targets to the trained network relies on an idealized flow, which is why we measure the drift directly. The stability argument therefore no longer rests on empirical motivation alone: the per-update change of the executed policy is bounded by quantities PPO and the target construction control, and the measured drift stays at the anchor's scale.

W4. Reliability of the QFLEX comparison.

We agree that a cross-framework port carries implementation uncertainty. We therefore added a controlled comparison inside our own pipeline: variant (3) trains PBR's flow on QFLEX's critic-ascent targets (20 gradient steps with QFLEX's step rule), with everything else unchanged. It falls below PBR on both Ant and Allegro (new Section 8.3). Because this changes a single design choice within the same implementation, it avoids confounding from differences between codebases. We keep the port in Appendix B.2 only as a best-effort transfer.

W5. Computational cost.

PBR reaches PPO’s return at the original training budget after about 0.11 h on Ant, before PPO reaches that budget at 0.18 h. At the same 0.18 h budget, PBR’s mean return is already higher (134.06 vs. 124.64). With PPO trained further to PBR’s full wall-clock budget of 0.61 h, PBR remains higher (145.32 vs. 130.43) and leads in every training run. In addition, PBR's Gaussian alone retains most of its gain at deployment (new Table 2), so the deployed policy can run at PPO's inference cost, and the overhead applies to training only.

Summary of revisions: new Section 8 (Gaussian-only deployment, proximity, design ablations) and new Appendix D (Proposition D.1, Lemma D.2, measured drift).

We would be glad to clarify anything further.

Dear Reviewer to8C,

Thank you again for your review. We have updated the paper with the completed ablation results. In our earlier response, the Allegro results were at 2.5k training iterations. We have now finished the full-length runs and added Digit Loco-Manipulation (revised Section 8.3, Tables 3 and 4).

W2 and W4. Ablations of the transformation and its targets, with PPO still updating the Gaussian:

MethodAntAllegro ReposeDigit Loco-Manipulation
Single Q-gradient step126.00 ± 15.46140.35 ± 36.2960.20 ± 3.09
Direct candidate selection139.64 ± 3.55−3.25 ± 2.40−9.85 ± 5.40
QFLEX-style critic-ascent targets132.64 ± 8.55150.57 ± 21.2555.86 ± 8.62
PBR145.48 ± 13.03159.42 ± 18.7762.26 ± 1.54

W2. Residual MLP instead of the flow (Ant and G1 Rough unchanged):

TaskReturn: PBRReturn: MLPSpread: PBRSpread: MLP
Allegro Repose159.42 ± 18.7759.58 ± 17.45101%50%

All three variants fall below PBR on every task, and direct selection collapses on Allegro and Digit. On Allegro, the MLP remains far below PBR and still narrows the executed behavior, while the flow preserves the targets' spread.

As the discussion period nears its end, we would welcome any further questions or comments.

Reviewer_keMAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Proximal Behavior Reshaping (PBR), which combines a PPO-trained Gaussian policy with a learned flow transformation for action execution. The transformation is trained using locally sampled candidates weighted by their estimated values and transport costs. Experiments show promising improvements over PPO across several continuous-control tasks.

Critical strengths
  • The method is clearly motivated and retains a relatively simple PPO optimization component.
  • The evaluation covers multiple locomotion and manipulation tasks, with five seeds on the Isaac Lab benchmarks.
  • The paper includes useful component ablations and reports computational overhead.
Critical weaknesses
  1. The methodological novelty appears limited. QFLEX already combines a learned Gaussian source policy with a Q-guided flow transformation trained through conditional flow matching. PBR mainly changes the Gaussian update to PPO and constructs flow targets through regularized candidate reweighting. These are reasonable modifications, but the paper does not sufficiently establish a substantive advance beyond this combination of existing techniques.
  2. The claimed separation between optimization and behavior needs clarification. Updating the Gaussian policy using source-action likelihoods is reasonable when the transformation is fixed. However, the transformation also changes during training, and PPO clipping on the Gaussian alone does not constrain changes in the complete executed policy. The implications for the claimed stability benefits are insufficiently discussed.
Questions for authors
  • What is the main contribution beyond combining PPO with an existing Gaussian-plus-flow architecture?
  • How is the change in the executed policy controlled when both the Gaussian and transformation are updated?
  • Does direct candidate selection remain inferior when using exactly the same weighting, transport penalty, and identity candidate?
2 comments
AuthorsOpenReview ↗

Response to Reviewer keMA

We thank the reviewer for the clear review and for noting the simple PPO component, the multi-task evaluation and the component ablations. Each of the three questions now has a direct answer in the revision:

  • Q1 (contribution): The policy proposes and the critic only compares, and Q never updates the Gaussian. As a result, target displacement is provably bounded for any critic.
  • Q2 (executed-policy change): The change is now bounded explicitly, by the anchor's change, which PPO regulates, plus the flow's displacement, which is bounded for any critic. Measured drift tracks the anchor's (0.95–1.19×).
  • Q3 (direct selection): With exactly PBR's weighting, transport penalty and identity candidate, direct selection falls below PBR on both tasks tested and collapses on Allegro.

W1 / Q1. Contribution beyond combining PPO with a Gaussian-plus-flow architecture.

PBR's contribution is a specific role for the learned critic: the critic may only adjust the policy's own behavior, locally and at a bounded cost, while PPO alone updates the policy. PBR shares QFLEX's parameterization, as the revision now states explicitly, but assigns the critic this role through two different choices:

  1. How the critic builds targets. QFLEX follows the critic's gradient, so the critic alone determines where each target lands. In PBR, the policy proposes and the critic only compares: targets are chosen among anchor samples, with a transport penalty and the option of not moving. As a result, for any critic, the expected transport cost of a target, in units of the anchor's standard deviation, is bounded by a constant set only by the number of proposals, the action dimension and the score weights (new Lemma D.2). This holds because every target is either the source or a sampled proposal, and the standardized value gains are capped. Critic errors can therefore change which proposal is chosen, but never how far the target can move. Gradient-ascent targets have no such bound relative to the anchor's spread (Remark D.7).
  2. Where the critic acts. QFLEX trains its Gaussian off-policy to maximize Q. In PBR, Q never enters the Gaussian's objective: it only reshapes the actions used to collect experience, and PPO updates the anchor from the resulting rollouts.

Both choices are supported empirically. QFLEX's critic-ascent targets inside our pipeline fall below PBR on Ant and Allegro (new Section 8.3), and without the flow, PBR's Gaussian still outperforms PPO by 18.5 on Ant and 47.9 on Allegro (new Table 2).

W2 / Q2. Controlling the executed policy when both components are updated.

We agree that PPO clipping alone does not constrain the executed policy, and the revision bounds it directly (new Appendix D).

  • Within an iteration, the PPO update is exact for the anchor. The flow is fixed during a rollout, so it acts as part of the environment for the anchor, and rollouts store the Gaussian sample itself. PPO's ratio is therefore the correct one. What remains is how much the executed policy changes between iterations, when the flow is updated too.
  • Between iterations, the change has two controlled terms (Proposition D.1): the executed policy moves by at most the anchor's change plus the flow's displacement at the two iterations. The anchor's change is regulated by PPO's KL-adaptive update.
  • The flow's displacement is bounded for any critic. Its training targets have bounded expected transport cost (Lemma D.2), and an ideal flow moves actions no further than its targets (Lemma D.4).
  • Measured drift is consistent with this. Proposition D.1 holds for any flow and any critic, and only Lemma D.4 idealizes the flow, so we also measure drift directly. Across six tasks, the executed policy's drift between checkpoints is 0.95–1.19× the anchor's, and both decrease over training (Figure 13).

Q3. Direct candidate selection with identical weighting, penalty and identity candidate.

Yes. New variant (1) executes a candidate sampled with PBR's exact weights, transport penalty and identity option, removing only the flow (Section 8.3):

MethodAntAllegro (2.5k iters)
PBR145.48 ± 13.03130.13 ± 14.91
(1) Direct candidate selection139.64 ± 3.55−0.48 ± 0.04

PBR outperforms direct selection on both tasks, and on Allegro direct selection collapses. Executing the selection rule directly is therefore not a substitute for the learned flow.

Summary of revisions: new Section 8 (Gaussian-only deployment, proximity, design ablations) and new Appendix D (formal proximity analysis and measured drift).

We would be glad to clarify anything further.

Dear Reviewer keMA,

Thank you again for your review. We have updated the paper with the completed ablation results. In our earlier response, the Allegro results were at 2.5k training iterations. We have now finished the full-length runs and added Digit Loco-Manipulation (revised Section 8.3, Table 3):

MethodAntAllegro ReposeDigit Loco-Manipulation
(1) Direct candidate selection139.64 ± 3.55−3.25 ± 2.40−9.85 ± 5.40
(3) Critic-ascent targets132.64 ± 8.55150.57 ± 21.2555.86 ± 8.62
PBR145.48 ± 13.03159.42 ± 18.7762.26 ± 1.54

For Q3, direct selection with PBR's exact weights, transport penalty and identity candidate falls below PBR on all three tasks and collapses on both Allegro and Digit. For Q1, critic-ascent targets in place of PBR's candidate-selected targets also fall below PBR on all three tasks.

As the discussion period nears its end, we would welcome any further questions or comments.

Reviewer_2dovOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PBR, which decouples the distribution optimized by PPO from the distribution executed in the environment. PPO continues to optimize a Gaussian “anchor” policy, while sampled Gaussian actions are passed through a learned residual flow before execution. The flow is trained by conditional flow matching toward local targets, which are selected from policy-supported candidates by a Boltzmann rule that trades off critic-estimated value improvement against a σ\sigma-normalized transport cost. The critic is trained on the current rollout plus a short window of recent-policy data, and it guides only the behavior reshaping, not the Gaussian update. The paper analyzes PBR on a 2D wall-avoidance task, and reports strong results on six Isaac Lab tasks against PPO, RePPO, FastTD3 and FPO++, with additional gains over PPO on six MuJoCo Playground tasks, at about 3.4$\times$ PPO’s wall-clock cost.

Critical strengths

Simple, practical idea with strong results on challenging benchmarks. PBR is a lightweight addition to standard PPO that keeps the PPO update unchanged, and it performs competitively or best on demanding high-dimensional Isaac Lab tasks, while being cheaper than the value-guided and flow-based alternatives. The Wall2D analysis gives an intuitive picture of how value-guided reshaping produces structured behavior that a Gaussian alone does not.

Critical weaknesses
  1. The source of the performance gain is not established. The paper attributes the gains to richer, structured exploration that helps learning, but the evaluated policy also executes the reshaped action. Reshaping toward higher-Q actions at evaluation time is effectively a test-time, critic-guided local improvement step on top of the Gaussian. The current experiments therefore cannot distinguish better learning of the Gaussian anchor thanks to reshaped exploration, from a better executed action at evaluation time.

  2. Novelty relative to QFLEX is not clearly articulated or empirically tested. QFLEX also augments a Gaussian source policy with a Q-guided flow used for interaction. The differences (PPO instead of off-policy actor updates, local candidate selection with a transport cost, recent-policy data reuse) are presented mainly through the design-space narrative, and QFLEX is compared only in an appendix on two tasks using the authors’ own adaptation that did not recover reasonable performance.

Questions for authors
  1. Does the trained Gaussian anchor itself outperform PPO?

  2. What precisely is new relative to QFLEX beyond using PPO for the Gaussian update? Could you include a controlled ablation, e.g., PBR with the Gaussian updated off-policy as in QFLEX, or QFLEX-style global Q-guided transport inside the PBR pipeline?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2dov

We thank the reviewer for the careful and constructive review, and for recognizing PBR as a simple, practical addition to PPO with strong results on demanding Isaac Lab tasks. Both weaknesses ask where PBR's gain comes from and how PBR differs from QFLEX. We answer both with new experiments and revisions to the paper.

W1 / Q1. Does the trained Gaussian anchor itself outperform PPO?

Yes. We evaluated every final checkpoint deterministically in two ways: executing the Gaussian mean directly, and passing the same mean through the flow (new Section 8.1, Table 2).

TaskPPOPBR, Gaussian onlyPBR, Gaussian + flow
Ant125.06 ± 14.90143.53 ± 14.12145.32 ± 13.58
Allegro Repose109.19 ± 29.49157.08 ± 16.43158.53 ± 15.52
Digit Loco-Manipulation56.77 ± 6.5065.23 ± 0.9762.03 ± 1.21
Franka Lift158.03 ± 2.96157.60 ± 0.90158.56 ± 0.76
G1 Rough50.54 ± 0.3751.73 ± 0.4451.78 ± 0.39
Go2 Rough39.83 ± 0.1239.69 ± 0.1539.77 ± 0.07

Without the flow, the anchor still outperforms PPO by 18.5 on Ant and 47.9 on Allegro and is best on Digit. Elsewhere the three are close, and the flow adds at most 1.8 return. The gain therefore lies in the Gaussian that PPO learns from the reshaped rollouts, not in a test-time improvement step, and PBR can be deployed as a plain Gaussian policy at PPO's inference cost.

W2 / Q2. What is new relative to QFLEX?

We agree that PBR shares QFLEX's parameterization, a Gaussian source with a flow used for interaction, and the revision now states this explicitly. PBR's contribution is how the critic is used, in two respects.

  1. How the critic builds targets. QFLEX follows the critic's gradient, so the critic alone determines where each target lands. In PBR, the policy proposes and the critic only compares: targets are chosen among anchor samples, with a transport penalty and the option of not moving. As a result, for any critic, the expected transport cost of a target, in units of the anchor's standard deviation, is bounded by a constant set only by the number of proposals, the action dimension and the score weights (new Lemma D.2). This holds because every target is either the source or a sampled proposal, and the standardized value gains are capped. Critic errors can therefore change which proposal is chosen, but never how far the target can move. Gradient-ascent targets have no such bound relative to the anchor's spread (Remark D.7).
  2. Where the critic acts. QFLEX trains its Gaussian off-policy to maximize Q. In PBR, Q never enters the Gaussian's objective, so the PPO update itself is unchanged. Q only reshapes the actions used to collect experience, and PPO updates the anchor from the resulting on-policy rollouts. The answer to W1 shows that this is where the gain ends up.

We address both suggested ablations as follows:

QFLEX-style Q-guided transport inside PBR (how). New variant (3) in Section 8.3 trains the flow on QFLEX's targets (20 critic-gradient ascent steps with QFLEX's step rule):

MethodAntAllegro Repose (2.5k iters)
PBR145.48 ± 13.03130.13 ± 14.91
(3) QFLEX-style critic-ascent targets132.64 ± 8.55122.30 ± 2.94

Critic-ascent targets fall below PBR on both tasks. Using the critic even more directly, without a flow, is less robust still: executing PBR's selected candidate directly collapses on Allegro (Section 8.3).

Gaussian updated by Q instead of PPO (where). In Section 6 (Figure 4, Wall2D), we replace PPO's update of the Gaussian mean with a direct Q-gradient update, holding the critic, reshaping and rollouts fixed. Switched early in training, it slows learning and fails to find the bypass within the training horizon. It preserves performance only once the behavior is already established.

Each ablation changes a single design choice within the same implementation, which avoids confounding from differences between codebases. We therefore base the comparison with QFLEX on these ablations, and keep our cross framework port (Appendix B.2) only as a best effort transfer.

Summary of revisions: new Section 8 (Gaussian-only deployment, proximity, design ablations) and new Appendix D (formal proximity analysis and measured drift).

We would be glad to answer any further questions.

Dear Reviewer 2dov,

Thank you again for your review. We have updated the paper with the completed ablation results. In our earlier response, the Allegro results were at 2.5k training iterations. We have now finished the full-length runs and added Digit Loco-Manipulation (revised Section 8.3, Table 3):

MethodAntAllegro ReposeDigit Loco-Manipulation
PPO120.55 ± 15.35119.12 ± 30.6854.25 ± 7.68
(3) QFLEX-style critic-ascent targets132.64 ± 8.55150.57 ± 21.2555.86 ± 8.62
PBR145.48 ± 13.03159.42 ± 18.7762.26 ± 1.54

With everything else held fixed, QFLEX-style critic-ascent targets fall below PBR on all three tasks.

As the discussion period nears its end, we would welcome any further questions or comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.