acceptodds
Under review as a conference paper at ICLR 2027

RAM-A: Bounded Advantage Regression for Flow Policy Optimization

Abstract

Flow-matching policies provide an expressive class of action distributions for robotic control, but reinforcement learning with such policies remains challenging. Existing approaches such as Flow Policy Optimization (FPO) construct policy updates through exponentiated differences of flow-matching losses and rely on proximal clipping to stabilize optimization. We introduce RAM-A, an advantage-based extension of Reinforce Adjoint Matching (RAM) that casts reinforcement learning as regression toward advantage-shifted flow-velocity targets. We show that RAM-A and FPO share the same first-order policy-improvement direction at the behavior policy, but differ under repeated optimization of a rollout batch, where unbounded advantage regression diverges for negative advantages. RAM-A instead bounds the norm of the regression target, which makes each update self-limiting and robust to corrupted advantage estimates. We evaluate RAM-A across dense-reward continuous control and sparse-reward robotic manipulation, together with targeted experiments on multimodality and multitask retention. While matching existing methods in task performance, RAM-A retains both modes of a multimodal pretrained policy and better preserves skills not targeted by the fine-tuning reward. Pretrained generative policies can thus be improved one skill at a time without erasing their diversity or capabilities.

Primary area:
reinforcement learning
Keywords:
Flow Matching, Reinforcement Learning, Robot Learning, Policy Fine-Tuning, Policy Optimization, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W21MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents RAM-A, which is a diffusion-based model's RL technic which bounds the grad in the RL stage, to keep the model's pre-training abilities and multi-modality, while staying competitive among SOTA RL methods.

Critical strengths
  1. The advantage of RAM-A on multi-modality and pre-training ability keeping is clearly shown and well presented.
  2. Ablation study shows how the RAM-A helps to prevent major shifts, and therefore keeping the multi-modality and pre-training skills.
Critical weaknesses
  1. The motivation is questionable. To my understanding, post-training is designed for turning the pre-trained model into a specific application scope. It is unclear the motivation for keeping pre-training skills on other tasks in this stage, other than doing a weighted pre-training, or a weighted SFT.
  2. This paper is majorly talking about RL technics. In this sense, I think real robot experiments, though already clearly stated in the limitations, might still be important.
  3. The proposed method's novelty is questionable to me. I would be interested in its advantages over grad clipping.
Questions for authors

Please see the weaknesses.

2 comments

I keep my score since I don't see author's rebuttals.

We apologize for posting our response so shortly before the rebuttal deadline. We just realized that our response was not submitted correctly. We regret that this leaves little time for further discussion.

We thank the reviewer for the review and for recognizing the retention and multimodality results.

Regarding the questions:

1. Why preserve pre-trained capabilities during post-training?

Robots operate in dynamic environments, where learning through interaction is valuable. We believe this learning should also help build broader capabilities over time and not just specialize it for one task. A robot learning better insertion may still need to pick and stack. We study whether RL can improve the rewarded skill while preserving existing abilities and alternative successful behaviors. Weighted SFT could also help, but requires revisiting demonstrations. Our setting uses only interactions and rewards from the current task.

2. Real robot experiments

We see our primary contribution as algorithmic. RAM-A introduces a bounded advantage-regression update for on-policy optimization of flow policies. The central properties that we investigate are: optimization stability, task performance, and retention during fine-tuning. Simulation allows us to study these properties through reproducible comparisons and targeted ablations. We do not consider hardware validation necessary to show these algorithmic claims.

3. Advantages over gradient clipping

Gradient clipping limits the size of each update, but repeated updates can still move the policy far from its reference. RAM-A instead bounds the regression target around a reference prediction. This limits the influence of extreme advantage estimates before gradients are combined. Furthermore, these mechanisms are complementary and can be used together. In fact, we do have gradient clipping applied in all our experiments. But gradient clipping alone will not stabilize the algorithm as the new ablation results in the revised experiments section shows.

Reviewer_FHbgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAM-A, an on-policy method for training flow policies through regression toward advantage-dependent velocity targets. It analyzes the instability that sufficiently negative advantages can cause when an unbounded regression update is repeatedly optimized on a fixed batch. The proposed method bounds the target displacement and combines this with reference-policy anchoring and behavior-policy proximity control. Experiments compare RAM-A with FPO++ and ReinFlow on continuous-control and manipulation tasks, with additional evaluations of skill retention, multimodal behavior, and sensitivity to corrupted advantages.

Critical strengths

The paper provides an explicit analysis of the unbounded regression update rather than introducing target clipping without motivation. The distinction between limiting individual target corrections and controlling accumulated policy movement is useful for understanding the complete algorithm.

The evaluation considers what a policy retains during fine-tuning, including skills outside the current training objective and multiple successful behaviors. These are relevant properties of pretrained generative policies that task reward alone does not capture. Using shared architectures and initializations also makes the comparisons more informative.

Critical weaknesses

(a) The practical advantage of RAM-A is not sufficiently established by the current experiments. Figure 1 does not show a consistent improvement over the strongest baseline: Finger Spin is the clearest favorable result, while RAM-A is comparable to or worse than ReinFlow on several other tasks in sample efficiency and/or final performance. This makes the retention experiments central to the contribution. However, in the two-goal experiment, both goals receive the same reward and a policy that uses only one mode still achieves near-perfect success. Figure 3 therefore demonstrates preservation of mode coverage, but does not establish when this preservation improves task performance, robustness, or subsequent adaptation. The evidence is also narrow: mode preservation is tested in one symmetric two-goal setting, and skill retention uses three tasks in a single fine-tuning order. More diverse evaluations are needed to assess the generality and practical value of these effects. In particular, it remains unclear whether RAM-A offers a better adaptation–retention trade-off or simply makes more conservative updates.

(b) The paper does not clearly establish which benefits are specific to its flow-policy formulation. The sample-level stability argument concerns a reference-regularized quadratic regression objective with signed weights; the argument itself does not depend on flow dynamics. Target clipping, reference anchoring, and proximity control are also broadly applicable optimization ideas. Although the velocity targets and path-based penalty provide a concrete implementation for flow policies, the paper needs to distinguish this implementation from its more general methodological claims. It is also difficult to determine whether the observed retention comes from bounded advantage regression or from the additional constraints. Comparisons with FPO++ or ReinFlow equipped with comparable reference anchoring and update control would be particularly useful. An analogous evaluation with a suitable non-flow policy would further help establish whether the benefit depends on the policy representation or reflects a general regularization effect.

Questions for authors

(a) Can the authors compare retention at matched target-task performance, including baselines with stronger or adaptive update constraints? This would help determine whether RAM-A improves the adaptation–retention trade-off rather than obtaining retention primarily by restricting adaptation.

(b) In Figure 3, collapsing to one mode does not reduce task success. What evidence supports the practical value of preserving both modes? A setting in which access to multiple behaviors affects robustness or subsequent task performance would materially strengthen this part of the contribution.

(c) Which parts of the analysis and optimization mechanism require a flow-policy representation? Could the authors clarify the scope of the contribution and whether an analogous bounded-regression update would provide similar benefits for other suitably expressive policy classes?

(d) How sensitive are the retention results to the reference update rate, target bound, proximity budget, and task order? The current experiments do not sufficiently isolate these factors, particularly their contribution to the results in Figures 2 and 3.

4 comments
AuthorsOpenReview ↗

Response to Reviewer FHbg

We thank the reviewer for their thoughtful and constructive feedback, which helped us sharpen both the scope of our contribution and the evidence supporting it. We appreciate their recognition of the importance of analyzing unbounded advantage regression and preserving pretrained capabilities during fine-tuning. Their questions prompted substantial additions to the paper, including broader retention experiments, component ablations, and optimization diagnostics.

The reviewer’s question about applicability beyond flow matching was particularly valuable. It prompted us to distinguish the general target construction from its flow-specific implementation and develop extensions to Gaussian and Gaussian mixture policies. This analysis helped us articulate a broader framework and clarify which conclusions depend on the policy representation.

The reviewer also correctly notes that RAM-A does not consistently outperform ReinFlow on target-task success (Fig. 1). This raises an important distinction: whether RAM-A offers a favorable adaptation–retention tradeoff or simply makes more conservative policy updates. We have expanded the experiments to examine this question through systematic sweeps of adaptation aggressiveness.

We address these points in three sections below, with references to our general comments and the revised manuscript.

AuthorsOpenReview ↗

Practical Value of Mode Retention

We agree that, because both goals receive the same reward, a single-mode policy can achieve near-perfect success in symmetric PickCube. The experiment deliberately separates task performance from behavioral preservation. It tests whether fine-tuning retains multiple pretrained solutions when the reward does not require their preservation.

The practical motivation comes from settings where alternative behaviors become useful under changing conditions. Kumar et al. (2020) demonstrate that learning diverse successful behaviors supports adaptation to environmental perturbations by allowing selection among alternative solutions. Similarly, Wang et al. (2021) show that modeling diverse strategies supports human–robot collaboration when human partners exhibit different behaviors or change strategies during execution.

These findings motivate retaining behavioral alternatives that the fine-tuning reward alone may not distinguish. Our experiment demonstrates preservation of those alternatives, but does not directly demonstrate improved robustness or collaboration performance. We have clarified this distinction and present the symmetric task as a controlled diagnostic of mode retention.

Motivated by the reviewer’s suggestion, we are also developing an additional experiment to test whether retaining alternative behaviors provides a practical advantage when conditions change. We hope to include the results before the rebuttal deadline. This evaluation is ongoing, and our current claims remain limited to the preservation demonstrated in the existing experiments.

AuthorsOpenReview ↗

Extensions Beyond Flow Policies

We thank the reviewer for this question, which prompted us to examine the scope of our approach more closely and develop explicit extensions beyond flow matching. We have added Appendix C to distinguish the general mechanism from its flow-specific implementation, with derivations for Gaussian and Gaussian mixture policies with fixed covariances.

Reference anchoring and bounded advantage-weighted target corrections do not require flow dynamics. The instability threshold at scaled advantage A=−1A=-1 follows from the quadratic fitting loss and its weighting relative to reference regularization. The same analysis applies to fixed-covariance Gaussian means when both terms use the corresponding quadratic metric and matched weighting. For Gaussian mixtures, the likelihood is not quadratic, so this threshold does not carry over. Nevertheless, Appendix C provides a counterexample showing that negative advantages can also make the uncapped mixture objective unbounded below. The target correction can be bounded in both policy classes without relying on a universal curvature threshold.

The flow-specific ingredients are the velocity-target construction, ODE-based action generation, and the stored-path proximity penalty with its action-displacement analysis. Other representations require appropriate fitting losses and proximity measures. Appendix C uses likelihood losses, exact Gaussian KL, and a tractable upper bound on mixture KL. The mixture extension also demonstrates that the construction applies to a non-flow policy class capable of representing multimodal action distributions.

In these constructions, clipping bounds the target correction in the chosen output space. The prediction-dynamics guarantee concerns direct updates toward bounded targets with a fixed reference. It does not establish stability of arbitrary neural-network parameter updates. Whether these extensions yield comparable empirical benefits in task performance and behavioral retention remains open.

AuthorsOpenReview ↗

Regularization Strength and Retention

We agree with the reviewer that preservation must be evaluated alongside adaptation and may depend on the strength of regularization. This concern motivated us to expand both retention experiments with systematic hyperparameter sweeps. We examine whether stronger constraints improve baseline retention and whether RAM-A retains pretrained behavior when its own constraints are relaxed.

For multimodality, we vary RAM-A’s target bound and proximity budget while testing increasingly restrictive clipping for FPO++ and ReinFlow. RAM-A’s average mode coverage across seeds remains nonzero in every tested configuration, although some loose budgets reduce both retention and task performance, and individual seeds can lose a mode. ReinFlow becomes effectively single-mode throughout its clipping sweep. FPO++ retains appreciable mode coverage with tighter clipping. However, separate normal-PickCube experiments show that its most restrictive settings substantially impair adaptation, illustrating the importance of evaluating preservation together with learning progress.

For multitask retention, we sweep actor learning rates, target bounds, and method-specific proximity constraints. Each experiment independently fine-tunes the same pretrained policy on one task and evaluates retention on the two untrained tasks. RAM-A’s retention-selected configurations reach 75.0% PegInsertionSide success with a 0.7-percentage-point average loss on untrained tasks, and 92.3% StackCube success with a 9.3-point improvement on untrained tasks. These configurations demonstrate that substantial adaptation can coexist with retention. The full sweeps show how this tradeoff varies with hyperparameter choice.

All experiments fully refresh the reference from the current actor between rollout batches and keep it fixed during optimization of each batch. Slower reference updates are supported by the framework but remain empirically untested.

These experiments broaden the evidence across optimization settings and fine-tuning tasks, but do not establish robustness to sequential task order. They also evaluate retention under the complete method and do not isolate target bounding as the cause of preservation.

Our general comments “Multitask Retention Ablation” and “Multimodal Behavior Retention Ablation” provide quantitative summaries. Appendix E reports the full sweeps and per-seed results.

Reviewer_xfbdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes RAM-A, an on-policy actor–critic method for flow matching policies. it adapts the reward-dependent velocity regression target in Reinforce Adjoint Matching by replacing rewards with advantages. to stabilize optimization on rollout batches, it proposes 1. bounded regression targets, and 2. behavior-policy proximity control through a path-divergence penalty, and 3. a gating rule. experiments show RAM-A achieves competitive performance against FPO++ and ReinFlow on continuous-control and robotic manipulation tasks. some ablations also shows RAM-A has better retention of previously learned task performance and multimodal behavior than the baselines

Critical strengths

The paper provides a clear analysis of the instability of unbounded advantage regression under repeated optimization and proposes targeted fixes

Critical weaknesses
  1. limited and somewhat unclear contribution relative to prior work: the unbounded advantage-regression update is already present in Advantage Weighted Matching. the paper's methodological novelty therefore seems to be in its adaptation to sequential actor–critic learning and some additional stabilization tricks. namely norm clipping, reference regularization, and trust-region-style optimization. but they are based on already well-established techniques. the main extension is their combination for sequential actor–critic learning, but the paper does not compare against an AWM-style objective under the same training protocol. the writing and presentation also makes the paper's contributions difficult to discern. the paper is overloaded with technical detail but lacks a clear overarching explanation of its main contributions and how the individual components fit together
  2. insufficient ablations to justify the proposed design: the main experiments compare complete algorithms without isolating RAM-A's individual proposed components. fig4 shows policy drift on a fixed rollout batch and fig5 shows gradient sensitivity to artificially corrupted advantages. but neither evaluates the effects of target bounding, the path-divergence penalty, and the gating rule separately on downstream task performance or behavioral preservation.
  3. limited evidence for the preservation claim: fig1 shows that RAM-A does not consistently outperform existing methods in task performance. the claimed preservation property is only shown through one three-task sequential finetuning and a symmetric two-goal PickCube task with fixed initial config. they are insufficient to support broad claims about preserving pretrained capabilities and multimodal behavior.
Questions for authors
  1. can the authors provide downstream performance and preservation ablations comparing unbounded AR, bounded AR, bounded AR with the path-divergence penalty, and the complete method with gating? can the authors provide stronger evidence about the claimed property of multimodality?
  2. the paper kind of proves instability in a simplified optimization model, but does not establish whether that instability is a meaningful problem during its actual RL training. what fraction of rollout transitions have scaled advantages < -1? how does this change during optimization? and how do bounded and unbounded advantage regression compare in training stability and downstream performance under the same optimization settings?
6 comments
AuthorsOpenReview ↗

Response to Reviewer xfbd

We thank the reviewer for their thoughtful feedback and for appreciating our analysis of the instability of unbounded advantage regression. The reviewer raises important concerns regarding the distinction from prior work, the contributions of individual components, and the experimental evidence supporting behavioral preservation. We address these points below and have revised the presentation to make our main contributions and their relationship clearer.

We address the concerns in four sections below.

AuthorsOpenReview ↗

Relationship to AWM and Scope of Our Contribution

We agree with the reviewer that AWM’s core update is closely related to unbounded advantage regression. We now make this relationship explicit in the manuscript: under matched weighting and reference regularization, the two formulations yield equivalent gradients for a single gradient step.

However, AWM was introduced for contextual-bandit reward optimization rather than sequential environment interaction with bootstrapped advantage estimates. We therefore evaluate its adaptation to sequential PickCube across five seeds. All five exhibit substantial performance deterioration, reaching only 1.4 ± 1.3% final success after 200 updates, despite using a single optimizer step per rollout batch. Appendix F provides the comparison, adaptation details, and evaluation limitations. These results demonstrate difficulty transferring the evaluated AWM recipe to this setting, rather than a general failure of AWM in sequential decision-making.

Our contribution lies in the targeted regression formulation for sequential actor–critic learning, its analysis under repeated optimization, and the corresponding stabilization mechanisms. We additionally demonstrate its ability to retain pretrained behavioral modes and performance on untrained tasks while adapting. We have revised the presentation to distinguish these contributions from the shared single-step update underlying AWM and unbounded AR.

AuthorsOpenReview ↗

Ablation of Individual Components

We agree that the original diagnostics did not sufficiently establish how the proposed components affect downstream learning. We have therefore added a component-ablation subsection comparing unbounded AR, bounded AR, unbounded AR with proximity control, and the complete bounded method with proximity control across three manipulation tasks, with five seeds per configuration.

Proximity control is a modular component of the framework whose choice is conceptually independent of the regression objective. We instantiate it using a path-divergence penalty and directional gate because the required quantities are straightforward to compute for our flow policies and this combination performs well on the evaluated tasks. We do not claim this particular proximity mechanism as a methodological novelty.

The ablation separates the contributions of regression-target bounding and proximity control. Target bounding limits individual regression-target displacements, while proximity control constrains policy movement during optimization. Our boundedness result concerns direct updates of an individual prediction with a fixed reference. It does not guarantee stability of a shared neural policy across changing rollout distributions. The new ablations show that target bounding alone is insufficient to maintain downstream performance, motivating separate behavior-policy proximity control.

Under the tight proximity budget used in the main experiments, proximity control is the principal contributor to sustained downstream performance. On PickCube, unbounded and bounded AR without proximity control finish at 2.8% and 0.4% mean success, whereas their proximity-controlled counterparts reach 96.6% and 95.6%. StackCube shows the same pattern: 0% without proximity control, compared with 94.0% and 95.8% with it. Under this tight proximity budget, bounding maintains comparable task performance on PickCube and PegInsertionSide. On StackCube, the bounded variant learns faster in the reported curves and achieves numerically higher mean final success. These observations do not establish that the measured changes in internal optimization dynamics cause the downstream performance differences.

To examine the benefit of target bounding under less restrictive proximity control, we evaluated all three tasks at a path-divergence budget of 0.08, using five seeds per condition. With c=1c=1, mean final success increases from 20.2% to 80.2% on PickCube and from 51.0% to 75.2% on StackCube. On PegInsertionSide, bounding does not consistently prevent performance deterioration. These results indicate that bounding can substantially improve performance under looser proximity control, but its benefit is task-dependent. Appendix E.3 provides the complete comparison, including variability across seeds.

Together, these experiments clarify the complementary roles of proximity control and target bounding. Proximity control is essential for sustained performance in the evaluated component ablations. Target bounding limits individual regression corrections and substantially improves downstream performance on PickCube and StackCube under the looser proximity budget, although this benefit does not extend uniformly across tasks.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Multimodal Behavior Retention Ablation

We agree with xfbd and FHbg that the scope of our original multimodality evaluation was too limited. We have expanded it across a wider range of optimization constraints, testing whether tighter proximity control prevents mode collapse in ReinFlow and FPO++, and whether relaxing proximity control induces it in RAM-A.

We find that RAM-A preserves substantial average coverage of both goal-reaching behaviors in symmetric two-goal PickCube across a broad range of settings. FPO++ requires substantially tighter clipping, while ReinFlow becomes effectively single-mode throughout the tested sweep.

On the symmetric two-goal PickCube task, we reduce FPO++ and ReinFlow clipping thresholds down to 10−510^{-5} and 10−610^{-6}, respectively. For RAM-A, we sweep proximity budgets δ∈{0.002,0.005,0.01,0.05}\delta\in\{0.002,0.005,0.01,0.05\} and target bounds c∈{0.5,1,2}c\in\{0.5,1,2\}, relaxing proximity control up to, and including, a budget at which overall task performance becomes unstable.

The tables report minority-mode share (%) 40 iterations after first reaching 98% success, to distinguish sustained retention from delayed mode collapse. Values are mean ± sample standard deviation across seeds. 50% indicates balanced coverage and 0% complete collapse. RAM-A results use c=1c=1.

Appendix E.1 provides the full sweep, per-seed results, measurements at first reaching 98% success, and target-bound ablations.

FPO++

Clipping thresholdMinority-mode share (%)
10−210^{-2}0.0 ± 0.0
5×10−35 \times 10^{-3}0.0 ± 0.0
10−310^{-3}16.6 ± 22.8
10−410^{-4}23.3 ± 13.4
10−510^{-5}24.8 ± 16.2

ReinFlow

Clipping thresholdMinority-mode share (%)
10−310^{-3}0.0 ± 0.0
5×10−45 \times 10^{-4}0.0 ± 0.0
10−410^{-4}0.0 ± 0.0
10−510^{-5}0.2 ± 0.4
10−610^{-6}0.0 ± 0.0

RAM-A (c=1c=1)

Proximity budget δ\deltaMinority-mode share (%)
0.00236.8 ± 11.5
0.00527.0 ± 11.8
0.0132.7 ± 12.9
0.057.5 ± 11.6

All summaries include five seeds except RAM-A at δ=0.05\delta=0.05, where only four reach the success threshold. At this largest proximity budget, RAM-A’s reduced mode coverage coincides with instability in its overall task performance (see Appendix E.1 for details).

FPO++’s restrictive clipping also substantially slows adaptation: on normal PickCube, thresholds 10−410^{-4} and 10−510^{-5} yield only 38% and 37% success after approximately 20 million steps. In comparison, both FPO++ with baseline hyperparameters and RAM-A with baseline proximity control (δ=0.005\delta=0.005) reach approximately 98% success after around 10 million steps. Thus, RAM-A’s mode retention does not require the substantial adaptation slowdown associated with these restrictive FPO++ settings.

AuthorsOpenReview ↗

Multitask Retention Ablation

We agree with xfbd and FHbg that our original multitask evaluation was too limited. We expanded it by sweeping learning rates and proximity constraints to test whether RAM-A offers a better adaptation–retention tradeoff, rather than merely making more conservative updates. We call the task used for fine-tuning the primary task and measure retention on the other two tasks, which receive no fine-tuning updates.

RAM-A achieves the best observed retention on both fine-tuning tasks while substantially improving primary-task performance. On PegInsertionSide, its retention-selected setting reaches 75.0% success with only a 0.7-point retention loss, compared with losses of 2.3 points for FPO++ and 10.2 points for ReinFlow at their retention-selected settings. On StackCube, RAM-A reaches 92.3% success while improving untrained-task performance by 9.3 points, whereas FPO++ and ReinFlow lose 3.0 and 13.5 points, respectively. RAM-A is the only evaluated method with a configuration that improves average untrained-task performance.

From the same pretrained policy, we independently fine-tune on StackCube or PegInsertionSide and evaluate all three tasks. We omit PickCube fine-tuning here because pretrained success is already approximately 100%.

The study comprises a 144-run grid, with three seeds per configuration and 125 training iterations. We sweep actor learning rates {10−5,10−6}\{10^{-5},10^{-6}\}, FPO++ clipping {0.002,0.02,0.05}\{0.002,0.02,0.05\}, ReinFlow clipping {0.001,0.01,0.05}\{0.001,0.01,0.05\}, and RAM-A proximity budgets {0.002,0.005,0.01}\{0.002,0.005,0.01\} with c∈{1,2}c\in\{1,2\}. Evaluations use 100 episodes per task.

For each method and task, we report the settings with highest retention and highest primary-task success at the final iteration, breaking success ties by retention. Retention is the change in average success on the two untrained tasks; positive values indicate improvement and negative values forgetting. Results are mean ± sample standard deviation across three seeds per configuration.

Appendix E.2 provides the full sweeps, individual-task results, training trajectories, and comparisons at matched primary-task success.

Fine-tuning on PegInsertionSide (initial BC success: 32.7%)

MethodSelected forHyperparametersPrimary success (%)Untrained-task change (pp)
FPO++Retentionclip=0.002, LR=10−510^{-5}80.0 ± 3.0−2.3 ± 0.3
FPO++Primary successclip=0.002, LR=10−510^{-5}80.0 ± 3.0−2.3 ± 0.3
ReinFlowRetentionclip=0.001, LR=10−610^{-6}63.0 ± 11.1−10.2 ± 6.4
ReinFlowPrimary successclip=0.01, LR=10−510^{-5}99.3 ± 0.6−74.3 ± 10.7
RAM-ARetentionδ=0.002\delta=0.002, c=1c=1, LR=10−510^{-5}75.0 ± 5.2−0.7 ± 4.5
RAM-APrimary successδ=0.005\delta=0.005, c=1c=1, LR=10−510^{-5}86.7 ± 5.0−7.7 ± 0.6

Fine-tuning on StackCube (initial BC success: 71.4%)

MethodSelected forHyperparametersPrimary success (%)Untrained-task change (pp)
FPO++Retentionclip=0.002, LR=10−610^{-6}85.3 ± 8.1−3.0 ± 1.3
FPO++Primary successclip=0.02, LR=10−510^{-5}96.7 ± 0.6−22.5 ± 6.1
ReinFlowRetentionclip=0.001, LR=10−610^{-6}96.7 ± 0.6−13.5 ± 3.3
ReinFlowPrimary successclip=0.001, LR=10−510^{-5}98.7 ± 0.6−14.8 ± 3.3
RAM-ARetentionδ=0.005\delta=0.005, c=2c=2, LR=10−610^{-6}92.3 ± 2.1+9.3 ± 4.5
RAM-APrimary successδ=0.002\delta=0.002, c=2c=2, LR=10−510^{-5}96.7 ± 1.5−16.0 ± 2.0

The primary-success selections illustrate the tradeoff. On PegInsertionSide, RAM-A reaches 86.7% success with a 7.7-point retention loss, whereas ReinFlow reaches 99.3% with a 74.3-point loss. FPO++ offers a competitive conservative setting at 80.0% success and a 2.3-point loss. Together with the StackCube results, these findings support RAM-A as a useful option when both adaptation and preservation matter.

AuthorsOpenReview ↗

Summary of Manuscript Revisions

We have substantially revised the manuscript, particularly the Method and Results sections, to address the reviewers’ concerns about clarity and experimental support. The main revisions are:

  • Clearer method presentation: We simplified the formulation and clarified the role of each component and its relationship to prior work.
  • Component ablations: We added comparisons in the main text and appendix that distinguish the effects of target bounding and proximity control.
  • Retention ablations: We expanded the appendix with multimodal and multitask retention experiments examining the trade-off between adaptation and preservation.
  • Broader theoretical framework: We extended bounded advantage regression beyond flow matching in a dedicated appendix.
  • Comparison with AWM: We added a theoretical comparison and a five-seed PickCube evaluation alongside unbounded advantage regression and full RAM-A.
  • Internal optimization analysis: We added evidence that naturally occurring A<−1A<-1 samples contribute disproportionately to regression loss, and that target bounding reduces their absolute gradient energy in matched comparisons.
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their constructive feedback and the discussions throughout the rebuttal. Their questions have lead to additional experiments and analyses, which have strengthened the paper. We would like to summarize here, how these additions address the main concerns raised in the original reviews.

  • Relationship to prior work and novelty. We clarified the relationship between RAM-A and AWM, by adding a five-seed PickCube comparison with an on-policy adaptation of AWM. RAM-A maintains substantially higher success under the tested settings.

  • Role of the individual components. We added five-seed ablations across three manipulation tasks, comparing unbounded and bounded advantage regression, each with and without proximity control. These experiments show complementary roles: proximity control is important for sustained learning, while target bounding stabilizes the internal optimization dynamics. We also evaluated looser proximity budgets to better understand when bounding matters.

  • Practical relevance of the stability analysis. We measured negative advantages in actual training rollouts and examined their behavior during repeated optimization of the same batch. These samples can contribute disproportionately to the regression loss, and bounding reduces amplification of their gradient contributions. This connects the analysis to observed training behavior.

  • Retention and conservative updates. We expanded the two-goal analysis with different target bounds, proximity budgets, baseline clipping thresholds, and learning rates, including comparisons at matched task success. These experiments clarify the adaptation–retention trade-off: conservative baseline settings can also preserve multiple modes, however, their update needs to be so conservative that they become practically unusable.

  • Applicability beyond flow policies. We added Appendix C with explicit extensions to Gaussian and Gaussian mixture policies with fixed covariances. These derivations show that reference anchoring and bounded target corrections also apply beyond flow matching.

Together, these additions provide stronger evidence for RAM-A as a regression-based approach to flow-policy reinforcement learning and a clearer account of when its stabilization mechanisms help. Our contribution is primarily algorithmic: understanding and controlling advantage-regression updates during repeated rollout-batch optimization, with competitive task performance and retention benefits in the evaluated settings.

We hope these additional results and clarifications will be considered in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.