acceptodds
Under review as a conference paper at ICLR 2027

Sample-Efficient RL for Policy Improvement via Flow-Reversed Action Anchors

Abstract

Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our anonymous website https://anonymous.4open.science/r/AnchorQ/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Robot Learning, Vision Language Models, Post-Training, Flow Matching Policy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9udeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

AnchorQ proposes a new online RL method for VLA finetuning.

AnchorQ uses flow reversal to map actions back into the noise space (namely noise anchors), where perturbations can be applied to sample new actions for exploration. A Q function is learned to score and select anchors. The paper provides theoritical studies and conduct empirical studies on libero, robotwin, and 3 real robot experiments.

Critical strengths
  1. The paper is well written and technically sound. The real robot RL results are promising.
  2. The core idea using flow reversal to construct a structured action space for more efficient exploration is intuitive and well motivated.
Critical weaknesses
  • Missing ablations. AnchorQ is compared to only one fixed beta which does not prove the advantage of adaptive beta based on uncertainty estimation. Also it seems the weighted averaging has not been ablated. A possible way is to use particles with higest Q to ablate the effect of weighted averaging.
  • Fig 8 introduces "no demo" but didn't explain it adequately in texts or captions. Could the author provide more clarifications about "no demo" and prefilling setup for the methods involved
Questions for authors

See weaknesses

2 comments

We thank the reviewer for recognizing our contributions and for suggesting these ablations.

Highest-Q particle versus weighted averaging

We have conducted the suggested ablation using the same trained critic ensemble:

SuiteWeighted averagingHighest-Q particleGain (percentage points)
LIBERO-1594.73 ± 0.52%84.60 ± 1.49%+10.13 ± 1.09
LIBERO hard-1060.35 ± 3.20%52.65 ± 2.63%+7.70 ± 0.60
RoboTwin-677.25 ± 2.55%67.17 ± 2.27%+10.08 ± 3.22

Weighted averaging improves success on all three suites. We have included this comparison in Appendix D.1, Table 5.

Fixed versus adaptive β and σ

We conducted sweeps over β ∈ {0.1, 1, 3, 10} and σ ∈ {0.3, 0.5, 1.0}. For each suite, we selected the best-performing fixed combination (β = 0.1 and σ = 0.3) for the reported ablation. We have stated the sweep ranges explicitly in Appendix C.2 and clarified this selection in the ablation table caption. These results show the advantage of adapting β and σ over the best fixed combination from these sweeps.

Demonstration prefill and “no demo” in Fig. 8

Only the Hard-10 suite uses additional demonstrations. We add one successful trajectory per task in replay in the demo-assisted comparisons, following FRS. For DSRL + FRS, we use the same implementation described by Tang et al.: an auxiliary behavior-cloning (BC) actor loss is computed on demonstration transitions sampled from the buffer, using flow-reversed demonstration noises as targets. Please refer to the FRS paper for implementation details. AnchorQ uses the demonstration for replay prefill and candidate-Q supervision. “No demo” removes both replay prefill and the demonstration loss. We have clarified this in Appendix C.2 and the caption of Fig. 8.

We hope these address the reviewer's concerns and welcome any further conversation.


References

  • Tang et al. Improving Robotic Generalist Policies via Flow Reversal Steering. arXiv:2606.13675.
Reviewer_9udeOpenReview ↗

Response

Thank the authors for the adding new experiments and clarifications which are informative and have addressed all my concerns. My recommendation remains acceptance.

Reviewer_2kNFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper adapts a frozen flow-matching VLA with online RL in its initial-noise space. At each decision, a fixed library of reference motions (six Cartesian directions; 36 combinations for the bimanual setting) is mapped to state-conditioned noise anchors by flow reversal (the FRS primitive of Tang et al., 2026), plus a "defer" anchor sampled from the prior. Around every anchor, Gaussian particles are combined into a barycenter weighted by exp⁡(β⋅Q)\exp(\beta·Q) under a 10-head critic ensemble, with β\beta and the particle scale adapted to ensemble disagreement; the refined latent with the highest value is executed. Only the critic is trained (REDQ-style TD learning); there is no latent actor. A ~60-page appendix provides theory on anchor-restricted identification, a bias–complexity trade-off, properties of exponential tilting, and an uncertainty-adaptive tilt. The experiments use two held-out LIBERO-90 splits, six RoboTwin tasks, and three real Franka/DROID tasks with 100 online episodes each. At 500k steps, AnchorQ reaches 95.5% vs 48.3% (DSRL) on LIBERO-15T, 62.1% vs 25.2% (DSRL+FRS) on Hard-10 and 70.5% vs 47.2% on RoboTwin; on the real robot, about 87% vs 63% (DSRL) vs 55% (base policy).

Critical strengths
  1. AnchorQ consistently outperforms DSRL and residual RL by a large margin across simulation and real-robot experiments, with fairly thorough evaluations.
  2. The paper addresses an interesting question about action representations for RL on frozen VLAs, and Fig. 4 clearly shows how AnchorQ maintains exploration where DSRL struggles.
  3. The ablations are comprehensive and show the contribution of each component, particularly the benefit of particles on precise-placement tasks.
  4. The method is practical, keeping the VLA frozen and training only a small critic ensemble without an additional actor.
  5. I appreciate that the appendix openly discusses the limitations of the theoretical analysis and its gap with the actual implementation.
Critical weaknesses
  1. It's unclear how much the particles and tilting actually contribute, since anchor-only already achieves 81.6% on LIBERO-15T vs. 95.5% for the full method, and the same ablations are not clearly reported for the other suites.
  2. It's unclear which experiments use demonstrations, and the baseline comparisons are inconsistent across benchmarks, with some important baselines missing and residual RL performing unusually poorly on RoboTwin.
  3. The claimed 97% throughput improvement seems to mainly reflect higher success rates rather than actual speedups, and there is no wall-clock or compute comparison despite the additional inference overhead.
  4. I'm not sure how well the hand-designed Cartesian anchors would generalize to other action spaces or more complex robots, especially since some implementation details are missing.
  5. Several details are inconsistent across the paper, including sampling vs. barycenter selection, which anchors are refined, hyperparameter settings, and the real-robot implementation.
Questions for authors
  1. What are the wall-clock time per step, peak GPU memory and total training time for AnchorQ, DSRL and Rainbow-FRS? How exactly is "completions/hr" computed?
  2. Why does Residual RL get 0% on RoboTwin, below the base policy?
  3. How does best-of-N from the noise prior, using the same critic ensemble and no anchors, compare?
  4. How would anchors be defined for joint-space or absolute-pose policies? How does the VLA behave on anchor latents with such small norms?
5 comments

We thank the reviewer for the detailed feedback. We have conducted the requested experiments and address each concern below.

Wall-clock time, GPU memory, training time, and completions/hour

We use the same frozen VLA for both methods, which accounts for most of their GPU memory and inference cost. In a representative AnchorQ inference, VLA computation takes about 315 ms, while critic scoring and particle selection add about 25 ms. With synchronous execution in the simulation benchmarks, we estimate 0.6 seconds per decision for all methods, including inference and simulation. In practice, we can overlap inference with execution of the current action chunk to reduce the waiting time between chunks.

We compute completions/hour by dividing the number of successful episodes by total evaluation time in hours, including failed attempts and resets. Higher success rates and fewer failed attempts improve this metric even when decision times are similar. We have clarified this distinction in Appendix D.2 and included the compute comparison in Table 6.

We report GPU peak memory and training time per task/seed run at a 500k nominal-action budget below. AnchorQ achieves higher success rates and throughput at compute costs comparable to DSRL.

SuiteAnchorQ time (h)DSRL time (h)AnchorQ memory (GiB)DSRL memory (GiB)
LIBERO-1538.4532.3310.1910.05
LIBERO hard-1040.0833.4610.0410.02
RoboTwin-617.8015.3939.6839.01

Best-of-N sampling from the noise prior

We compare AnchorQ with best-of-N and best-of-2N sampling from the noise prior using the same trained critic ensemble. N matches the number of particles used by AnchorQ.

SuiteAnchorQBest-of-N from the priorBest-of-2N from the prior
LIBERO-1594.73 ± 0.52%8.63 ± 0.50%8.70 ± 0.44%
LIBERO hard-1060.35 ± 3.20%1.45 ± 0.29%1.45 ± 0.29%
RoboTwin-677.25 ± 2.55%54.67 ± 0.88%54.83 ± 1.06%

AnchorQ outperforms both controls on all three suites, while doubling N has little effect. This shows that the anchors and their refinement are essential for inference. We have included these results in Appendix D.1, Table 5.

Coverage of RL baselines

Thank you for raising this question. Our work considers both direct policy fine-tuning and lightweight adaptation of frozen VLAs, with a focus on sample-efficient online RL under limited interaction budgets. Direct policy fine-tuning methods such as DPPO and FPO++ use on-policy RL, which can require substantial online interaction data as well as updates to the policy weights. DSRL compares against DPPO and reports a 5–10× sample-efficiency improvement over the compared approaches on its Robomimic and Gym benchmarks. FPO++ studies large-scale training in simulation and does not report RL training on a real robot.

Our work focuses on sample-efficient RL that can run on real robots, where we allow only 100 online episodes per task. We therefore compare against DSRL and DSRL + FRS as directly relevant noise-space steering baselines. We also consider lightweight adaptation methods such as RL-Token and PLD. These methods’ implementations were not available to us, so we tried our best to implement a residual RL baseline for our setting. We use an online-only PLD Stage-1 implementation with a frozen VLA; we exclude subsequent policy distillation and other stages. Together, these comparisons cover both noise-space steering and residual-action adaptation under our focus on efficient online RL. We have added brief references to FPO++ and RL-Token in Appendix A.

References

  • Ren et al. Diffusion Policy Policy Optimization. arXiv:2409.00588.
  • Wagenmaker et al. Steering Your Diffusion Policy with Latent Space Reinforcement Learning. arXiv:2506.15799.
  • Yi et al. Flow Policy Gradients for Robot Control. arXiv:2602.02481.
  • Tang et al. Improving Robotic Generalist Policies via Flow Reversal Steering. arXiv:2606.13675.
  • Xu et al. RL Token: Bootstrapping Online RL with Vision-Language-Action Models. arXiv:2604.23073.
  • Xiao et al. Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. arXiv:2511.00091.
AuthorsOpenReview ↗

Rebuttal (continued)

Residual RL performance on RoboTwin

In our experiments, Residual RL tends to dip to very low success rates despite nonzero success from the base VLA, making learning on RoboTwin particularly challenging. We use KL regularization to limit residual actions, but even small corrections from a randomly initialized actor can disrupt useful base-policy behavior. RoboTwin's coordinated bimanual tasks also make exploration harder: without successful rollouts, sparse rewards provide little learning signal. These two factors help explain the poor result. Noise-space steering generates actions through the frozen VLA, with DSRL and AnchorQ starting near the noise prior. As a result, these methods maintain higher success rates achieved by the base policy, with AnchorQ continously improving to higher performance.

Use of demonstrations

We follow the demo-assisted setting used for DSRL + FRS in the FRS paper by Tang et al. on Hard-10, with one successful demonstration per task. We provide the same demonstrations to each RL method and compare AnchorQ against DSRL + FRS in this setting. The other benchmarks use no additional demonstrations. For DSRL + FRS, flow reversal maps the demonstrated actions to noise targets used to prefill replay. During RL, we sample demonstration transitions from the buffer and add an auxiliary behavior-cloning (BC) loss to the actor objective to match these noise targets. We use the same DSRL + FRS implementation described by Tang et al. and refer the reviewer to the FRS paper for implementation details. For AnchorQ, we use the demonstrations to initialize replay and supervise candidate Q scores. “No demo” removes both replay prefill and the demonstration loss. We have clarified this in Appendix C.2 and the caption of Fig. 8.

Generalization of Cartesian anchors

We show that Cartesian anchors extend to coordinated bimanual manipulation in RoboTwin, using 36 combinations of six directions per arm plus a defer anchor. We also ablate different choices of noise anchors in Rainbow-FRS and show that this simple anchor design works well.

We thank the reviewer for raising this direction. We have clarified the anchor construction in Appendix C.2. We leave extensions to dexterous hands and humanoids to future work, including how to learn suitable anchors for their action spaces.

Joint-space and absolute-pose policies; anchor latent norms

We construct anchors directly in Cartesian action space for LIBERO. For DROID and RoboTwin, we use inverse kinematics to convert Cartesian reference motions into joint-position actions before flow reversal, as the base policies for these tasks output joint position targets. Absolute-pose policies can use the same approach with references expressed in their coordinate frame. Small latent norms do not imply small robot motions. In Fig. 4(b), we show that AnchorQ selects effective latents with norms between those of DSRL and the Gaussian prior.

Selection rule, refined anchors, hyperparameters, and real-robot setup

In Algorithms 1–3, all anchors are refined into Q-weighted barycenters. To select the best action, we evaluate the critic at each barycenter and decode the highest-valued one through the frozen policy. We have clarified barycenter execution in the introduction and the refinement of defer in Appendix C.1. Hyperparameters are detailed in Appendix Table 4.

For the real world setup, we use the standard DROID setup with pi05_joint_pos as the base policy; we have corrected the model name in Section 3.1 and included the checkpoint name and action representation in Table 4. We welcome the reviewer to point out any remaining concerns and would be happy to address them.


We have included the additional inference results in Appendix D and the implementation clarifications in Appendix C. We hope these modifications effectively address the reviewer's concerns and would appreciate if the reviewer would consider raising their score. We welcome further questions.

References

  • Ren et al. Diffusion Policy Policy Optimization. arXiv:2409.00588.
  • Wagenmaker et al. Steering Your Diffusion Policy with Latent Space Reinforcement Learning. arXiv:2506.15799.
  • Yi et al. Flow Policy Gradients for Robot Control. arXiv:2602.02481.
  • Tang et al. Improving Robotic Generalist Policies via Flow Reversal Steering. arXiv:2606.13675.
  • Xu et al. RL Token: Bootstrapping Online RL with Vision-Language-Action Models. arXiv:2604.23073.
  • Xiao et al. Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. arXiv:2511.00091.

Thank you for the response and the additional experiments. The clarification on demonstrations, the compute table and the best-of-N control address most of my concerns. However, one issue remains: the new table reports AnchorQ at 77.25 ± 2.55 on RoboTwin-6, while Table 1 reports 70.5 ± 3.0. Could you explain what differs between the two evaluations? If this is clarified, I will raise my rating from 2 to 3.

Reviewer_5NCCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AnchorQ which is a method to improve frozen flow-matching VLAs. The method fixes a small set of reference actions and samples Gaussian particles around them reweighted by exponential tilting with a learned critic. The paper has theoretical contributions and empirical contributions. Theoretically, it shows searching over a few anchors takes fewer samples than searching the whole noise space. Empirically, it is evaluated on LIBERO and RoboTwin. It is also evaluated on three real Franka tasks.

Critical strengths

The critical strength of the paper is large empirical gains across simulation and real world environments. The improvement over DSRL is large and it is often two times the performance or more. In LIBERO, the performance is especially high. In the real world, it also shows consistent gains. This is consequential because performance is often the thing that is cared about the most for RL fine-tuning, and a larger improvement makes it more desirable to apply the method.

Critical weaknesses

The critical weakness is that it is missing key baselines. While it compares to DSRL and residual RL, RL fine-tuning of flow matching models is a large area with a lot of prior work. It is hard to compare the performance and put it in the larger picture.

Questions for authors

In section 3.1 the paper states the DROID π0 was used while in section 3.4 and figure 5 it states π0.5. What was the base policy that was used?

How does the method compare to other RL fine-tuning baselines?

1 comment

We thank the reviewer for recognizing the empirical strengths of our work and for the constructive feedback.

RL baselines

Thank you for raising this question. We consider both direct policy fine-tuning and lightweight adaptation of frozen VLAs, with a focus on sample-efficient online RL under limited interaction budgets. Direct policy fine-tuning methods such as DPPO and FPO++ use on-policy RL, which can require substantial online interaction data as well as updates to the policy weights. DSRL (Wagenmaker et al.) compares against DPPO and reports a 5–10× sample-efficiency improvement over the compared approaches on its Robomimic and Gym benchmarks.

The FPO++ paper studies large-scale training in simulation and does not report RL training on a real robot. Our work focuses on sample-efficient RL that can run on real robots, where we allow only 100 online episodes per task. We therefore compare against DSRL and DSRL + FRS as directly relevant noise-space steering baselines. We also consider lightweight adaptation methods such as RL-Token and PLD. These methods’ implementations were not available to us, so we tried our best to implement a residual RL baseline for our setting. We use an online-only PLD Stage-1 implementation with a frozen VLA; we exclude subsequent policy distillation and other stages for fair comparison. Together, these comparisons cover both noise-space steering and residual-action adaptation under our focus on efficient online RL. We have added brief references to FPO++ and RL-Token in Appendix A.

Base model used for DROID

All methods in our real-world experiments use pi05_joint_pos, a DROID-pretrained pi 0.5 model with joint-position actions. We have corrected the model name from pi0 to pi0.5 in Section 3.1 and included pi05_joint_pos and its action representation in Table 4, “Hyperparameters by benchmark.”

We hope these address the reviewer's concerns and welcome any further conversation.


References

  • Ren et al. Diffusion Policy Policy Optimization. arXiv:2409.00588.
  • Wagenmaker et al. Steering Your Diffusion Policy with Latent Space Reinforcement Learning. arXiv:2506.15799.
  • Yi et al. Flow Policy Gradients for Robot Control. arXiv:2602.02481.
  • Tang et al. Improving Robotic Generalist Policies via Flow Reversal Steering. arXiv:2606.13675.
  • Xu et al. RL Token: Bootstrapping Online RL with Vision-Language-Action Models. arXiv:2604.23073.
  • Xiao et al. Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. arXiv:2511.00091.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised paper with changes in blue

We thank all reviewers for their feedback. We have revised the paper to clarify anchor construction and implementation details. We also added ablations and computational-cost results in the appendix, updated related work, and corrected the real-robot model description. All changes are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.