acceptodds
Under review as a conference paper at ICLR 2027

\(\pi\)-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs

Abstract

Flow-based vision-language-action (VLA) models have emerged as a powerful paradigm for embodied control, but multi-step denoising makes policy-gradient online reinforcement learning (RL) difficult because the final-action likelihood induced by the sampler is not directly tractable. We identify an explorationsupervision mismatch within the denoising trajectory, where stochastic differential equation (SDE) sampling explores a wider local stochastic transition space while supervision derived from the final denoised output does not directly use the adjacent transitions observed during sampling. We propose π-StepNFT (Step-wise Negative-aware Fine-Tuning), a flow-native online RL framework that couples wider stochastic exploration with finer step-wise supervision without computing the final-action likelihood. π-StepNFT treats observed SDE transitions as supervision units, using variance-normalized transition errors to rank mirrored positive and negative velocity branches and map rollout feedback to local velocity updates. On LIBERO, starting from shared few-shot SFT initializations, π-StepNFT raises average success from 57.6% to 92.0% for πо and from 77.1% to 94.0% for πo.5. On ManiSkill, it achieves the highest reported mean success among the evaluated RL baselines on five test-only visual perturbations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Online Reinforcement Learning, Flow-Based Policies

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JX95OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces π-StepNFT, an online RL method for flow-based vision-language-action models. The key idea is to address an exploration-supervision mismatch created by stochastic SDE sampling: rather than supervising intermediate denoising states using only the final generated action, π-StepNFT directly learns from observed adjacent SDE transitions. It uses variance-normalized transition errors and contrastive ranking between positive and negative velocity branches.

Critical strengths
  1. The method is well motivated for flow-based VLA policies. The exploration-supervision mismatch is clearly identified, and using observed SDE transitions for local supervision provides an intuitive solution that avoids computing the final-action likelihood.

  2. The paper provides strong mechanistic analysis. The authors carefully ablate SDE exploration, variance normalization, and etc. These experiments make it relatively clear where the performance improvements come from.

Critical weaknesses

The main LIBERO results do not outperform the strongest PPO-based baseline. π-StepNFT reaches 92.0%/94.0% average success for π0/π0.5, or 93.9%/96.0% with a critic, while Flow-SDE + PPO reaches 96.0%/97.9%. This weakens the argument for π-StepNFT as a generally better online RL method. Similar result can be observed in Maniskill experiment. This is my main concern, although its likelihood-free formulation and efficiency remain attractive.

Questions for authors

What do the authors see as the main advantage or potential advantage of π-StepNFT over Flow-SDE + PPO given that PPO remains stronger on the overall LIBERO averages?

4 comments

We thank the reviewer for recognizing the motivation and mechanistic analysis of π-StepNFT. We clarify its main advantages in flow-native actor optimization, compatibility with critic-based feedback, and denoising-transition utilization. For your convenience, please also refer to our revised manuscript for the corresponding updates.

 

A1 (W1, Q1): Main advantages over Flow-SDE PPO (Part 1)

Flow-native actor optimization. π-StepNFT directly optimizes the flow-matching velocity field through contrastive ranking of mirrored branches on observed xt→xt−x_t \rightarrow x_{t^-} SDE transitions. In contrast, πRL Flow-SDE PPO uses advantage-weighted current-to-old transition-policy probability ratios, while Flow-Noise PPO additionally retains the multi-step denoising solver chain during actor optimization. By mapping rollout feedback directly to local velocity-field updates, π-StepNFT avoids PPO-style importance ratios and backpropagation through the full denoising chain. This provides a direct velocity-prediction optimization interface for flow-based policies.

Standalone actor loss and critic compatibility. π-StepNFT introduces an actor loss that operates independently with binary success/failure feedback and can also incorporate critic-derived temporal credit without changing the ranking objective. This allows environment-time credit assignment to complement denoising-time supervision, as demonstrated by the following results.

Benchmarkπ-StepNFTπ-StepNFT + CriticFlow-SDE PPOFlow-Noise PPO
LIBERO π0\pi_0 Avg.92.093.996.097.6
LIBERO π0.5\pi_{0.5} Avg.94.096.097.998.3
ManiSkill π0.5\pi_{0.5} ID84.391.090.989.7
Visual Avg. (5)75.376.166.269.8
OOD Avg. (9)58.061.254.258.3

Success rates are in %. OOD Avg. averages the nine shared visual, semantic, and execution variants, excluding ID.

Critic-derived feedback improves performance across multiple settings. On LIBERO, GAE improves the overall averages from 92.0% / 94.0% to 93.9% / 96.0% for π0/π0.5\pi_0/\pi_{0.5}, with substantial gains on Long (86.7% → 90.0% and 79.8% → 88.4%, respectively). On ManiSkill, ID success increases from 84.3% to 91.0%, reaching PPO-level performance. Semantic success improves from 30.5% to 34.4%, exceeding both PPO baselines (33.1% / 33.3%), while execution success increases from 42.2% to 50.8%, surpassing Flow-SDE PPO (45.4%). The overall OOD average rises from 58.0% to 61.2%, exceeding both PPO baselines, while preserving the visual-shift advantage (76.1% vs. 66.2% / 69.8%). These results demonstrate that critic-derived temporal credit complements the proposed actor loss, improving long-horizon performance and robustness across the evaluated distribution shifts.

A1 (W1, Q1): Main advantages over Flow-SDE PPO (Part 2)

Potential for improved denoising-transition utilization. We further evaluate π-StepNFT and πRL Flow-SDE PPO under the same Last-2 configuration, retaining two late denoising transitions per sampled path instead of one randomly selected transition in their default Random-1 configurations. Both methods use half as many environment rollouts per iteration while maintaining the same number of retained training transitions and optimizer updates.

TaskMethodRollouts/Iter.Peak SR70%80%90%
Object-0π-StepNFT Last-21696.88%1.60 / 2241.82 / 2562.38 / 336
Object-0πRL Flow-SDE PPO Last-21647.50%NRNRNR
Long-5π-StepNFT Last-23280.00%0.91 / 2563.47 / 1,024NR
Long-5πRL Flow-SDE PPO Last-23274.38%4.74 / 1,344NRNR

Threshold entries report GPU-hours / cumulative environment rollouts required to first reach each smoothed on-policy training-success threshold. NR indicates that the threshold was not reached.

Under Last-2, π-StepNFT reaches 90% on Object-0 with 336 rollouts and 2.38 GPU-hours, whereas PPO does not reach 60%. On Long-5, π-StepNFT reaches 70% with 256 rollouts and 0.91 GPU-hours, compared with 1,344 rollouts and 4.74 GPU-hours for PPO, and reaches 80% while PPO does not. Compared with its default Random-1 configuration, π-StepNFT Last-2 reduces the rollouts required to reach 90% on Object-0 from 1,280 to 336 and 80% on Long-5 from 2,752 to 1,024 (training curves). These results suggest that the proposed actor loss has the potential to utilize multiple observed denoising transitions from each collected rollout more effectively, improving environment-data utilization in the evaluated single-task settings.

Thank you to the authors for their detailed responses and additional experiments. The clarifications and experimental results address part of my concerns and confirm some of my initial assumptions. I will keep my score unchanged, as I believe it already appropriately reflects my overall assessment of the paper.

Thank you again for your time and for considering our clarifications and additional experiments. We respect your assessment and would like to better understand the remaining concerns so that we can address them more precisely.

We acknowledge that PPO retains higher overall LIBERO averages. Our response highlights the observed-transition ranking objective, its compatibility with critic-derived temporal credit, and the accompanying robustness and transition-utilization results.

Without a critic, Table 1 of the original submission reports higher LIBERO average success than Flow-SDE GRPO for both backbones: 92.0% vs. 90.0% for π0\pi_0 and 94.0% vs. 91.5% for π0.5\pi_{0.5}, corresponding to gains of 2.0 and 2.5 percentage points.

With critic-derived feedback, the added ManiSkill experiment shows that π-StepNFT + Critic achieves 91.0% ID success, comparable to 90.9% / 89.7% for Flow-SDE / Flow-Noise PPO, while obtaining higher visual success (76.1% vs. 66.2% / 69.8%) and overall OOD success (61.2% vs. 54.2% / 58.3%). These results show that the visual advantage can coexist with PPO-comparable ID performance.

The additional Last-2 diagnostic also provides evidence of potential improvements in denoising-transition utilization. On Long-5, π-StepNFT reaches the 70% smoothed training-success threshold after 256 rollouts, compared with 1,344 for Flow-SDE PPO under its Last-2 configuration. The corresponding updates are included in our revised manuscript.

If time permits, could you briefly clarify whether the remaining concern primarily relates to the LIBERO performance gap or to the proposed objective and interpretation of the additional results? It would also be helpful to understand which initial assumptions you consider confirmed. We would appreciate your guidance in addressing these points and improving the work.

Reviewer_zquFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces π-StepNFT, an online RL method for flow-based robot policies, using successful and failed rollouts. Usual Negative-aware Fine-Tuning (NFT) baselines take the final generated action, add noise, and train the flow model to reconstruct it, with the loss weighted by the success. The proposed method saves a denoising transition from the rollout, then compares the current velocity prediction and its reflection around the old policy’s prediction. They then rank how well each predicts the observed next state, after normalizing errors by noise variance.

Critical strengths
  1. Clear training objective: It links observed denoising transitions to success feedback.
  2. Better visual robustness: On ManiSkill with π0.5, it improves average success by 5.5 percentage points over the strongest baseline under visual changes.
Critical weaknesses
  1. Limited performance gains: It trails PPO on LIBERO and standard ManiSkill tasks. Its clearest advantage concerns visual changes.
  2. Narrow validation: The authors test only two related models in simulation.
Questions for authors
  • Why does visual robustness improve while task success degrades compared to PPO?
  • The method changes both the training target and the loss (observed denoising transitions instead of final-action, and ranking instead of weighted regression). Can you ablate the two? Maybe compare ranking versus weighted regression on the same observed denoising transitions?
5 comments

We thank the reviewer for the thoughtful review and helpful suggestions. Below, we clarify the distinction between supervision targets and learning objectives, and provide additional evidence on critic compatibility, cross-model generalization, and physical-robot performance. For your convenience, please also refer to our revised manuscript for the corresponding updates.

 

A1 (W1, Q1): Visual robustness and in-distribution performance

Temporal credit and the ID performance gap. The default π-StepNFT uses trajectory-level binary feedback, whereas PPO employs critic-derived GAE for environment-time temporal credit assignment. To investigate whether this difference contributes to the performance gap, we evaluate π-StepNFT + Critic on ManiSkill with π0.5\pi_{0.5}, incorporating GAE feedback while retaining the original observed-transition ranking objective.

ManiSkill π0.5\pi_{0.5}Flow-SDE PPOFlow-Noise PPOπ-StepNFTπ-StepNFT + Critic
ID90.989.784.391.0
Visual Avg. (5)66.269.875.376.1
Semantic Avg. (2)33.133.330.534.4
Execution Avg. (2)45.454.942.250.8

Adding critic-derived feedback improves ID success from 84.3% to 91.0%, reaching PPO-level performance. Semantic and execution success also increase from 30.5% to 34.4% and 42.2% to 50.8%, respectively. These results suggest that the absence of fine-grained temporal credit assignment contributes to the default method's performance gap with PPO.

Visual robustness as an empirical finding. The visual-shift advantage persists after introducing critic feedback, with success improving from 75.3% to 76.1%, compared with 66.2% / 69.8% for the two PPO baselines. The original π-StepNFT also outperforms both baselines with π0\pi_0 (66.5% versus 60.4% / 64.4%). This advantage is an empirical generalization finding. Our primary contribution concerns aligning rollout feedback with observed stochastic denoising transitions rather than explaining the underlying mechanism of visual generalization.

Together, these results show that critic-derived temporal credit complements observed-transition ranking, narrowing the ID performance gap while preserving the empirical visual robustness advantage.

A2 (Q2): Independent ablations of supervision target and ranking objective

Controlled comparisons in the original submission. Our original submission already includes two independent ablations on LIBERO-Object Task 0. Figure 2b compares supervision targets with the ranking objective fixed, while Figure 2d compares contrastive ranking against weighted MSE (wMSE) on the same observed xt→xt−x_t \rightarrow x_{t^-} SDE transitions. We additionally conduct a wMSE-based target ablation on Object-0, reproducing the stability pattern in Figure 2b. The observed xt−x_{t^-} target remains effective at α0=0.1\alpha_0=0.1, whereas x0x_0-derived supervision collapses.

Cross-task extension. We further extend both ablations to LIBERO-Long Task 6 and LIBERO-Spatial Task 6, each trained for 200 iterations, varying only the corresponding component while keeping all other settings fixed.

The terminal success rates (mean ± SD) are:

ObjectiveTargetα0\alpha_0Long-6Spatial-6
wMSExt−x_{t^-}0.172.14 ± 0.39%93.42 ± 0.12%
Rankingxt−x_{t^-}0.175.23 ± 0.68%97.09 ± 0.12%
Rankingx0x_0-derived0.10.00 ± 0.00%0.00 ± 0.00%
Rankingxt−x_{t^-}0.959.98 ± 0.32%81.30 ± 0.36%
Rankingx0x_0-derived0.964.24 ± 1.04%85.10 ± 0.64%

Learning objective. With identical observed-transition supervision, ranking outperforms wMSE by 3.09 pp on Long-6 and 3.67 pp on Spatial-6, extending Figure 2d (Long-6 curve, Spatial-6 curve).

Supervision target. At α0=0.1\alpha_0=0.1, observed-transition supervision achieves 75.23% / 97.09%, while the x0x_0-derived target yields 0% on both tasks. At α0=0.9\alpha_0=0.9, the x0x_0-derived target performs better on both tasks. These results further confirm that observed-transition supervision improves training stability under aggressive EMA initialization, rather than uniformly outperforming the x0x_0-derived target across EMA settings (Long-6 curve, Spatial-6 curve).

A3 (W2): Cross-model generalization and physical-robot validation

Cross-model generalization. π-StepNFT operates on the shared flow-based action-generation mechanism rather than model-specific architectures. Beyond π0\pi_0 and π0.5\pi_{0.5}, we additionally evaluate NVIDIA GR00T N1.5, a distinct VLA model family, starting from a few-shot SFT checkpoint trained by the RLinf team. We replace dropout layers in the action expert with identity operations to prevent stochastic dropout from interfering with policy updates. We then apply π-StepNFT to all four LIBERO suites, obtaining the following success rates (%):

MethodSpatialObjectGoalLongAvg.Δ Avg.
GR00T N1.5 SFT41.458.648.261.952.5—
GR00T N1.5 + π-StepNFT92.097.886.381.489.4+36.9

π-StepNFT improves success across all four suites, raising the average from 52.5% to 89.4% (+36.9 pp). Together with the submitted π0\pi_0 and π0.5\pi_{0.5} results, this provides cross-model evidence that the observed-transition ranking objective generalizes beyond the π-series architectures.

Physical-robot validation. We additionally conduct a Real2Sim2Real pick-and-place experiment on a Franka robotic arm. Starting from SFT on 20 real-robot demonstrations, we perform π-StepNFT RL fine-tuning entirely in a reconstructed simulator and directly deploy the resulting policy on the physical robot without additional real-world RL training. Physical-task success improves from 1/10 (10%) to 6/10 (60%), with corresponding simulation and real-robot execution videos.

Reviewer_EAy6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

π-StepNFT improves flow-based VLAs using feedback on sampled denoising transitions rather than supervision reconstructed from the final action. It combines stochastic exploration with a ranking loss and evaluates the method on LIBERO and ManiSkill.

Critical strengths

s1. The paper gives a clear motivation for matching the training target to the transitions actually sampled during exploration.

s2. I appreciate the detailed ablations, including multiple seeds and comparisons that separate exploration, supervision targets, and the ranking loss.

s3. The visual-perturbation results are promising, and the method can use either binary success or critic-based feedback.

Critical weaknesses

w1. The benefits depend on the setting. PPO remains stronger overall on LIBERO, while the ManiSkill advantage is mainly on visual rather than semantic or execution changes.

w2. Most component ablations use one LIBERO task, and the CALVIN analysis uses one fixed sequence. It is unclear how broadly these findings hold.

w3. There is no real-robot evaluation yet. The reported compute savings are useful, but interaction cost remains a practical concern.

Questions for authors

q1. Why does the method help more with visual changes than with semantic or execution changes?

q2. Could you repeat the main target and loss ablations on a few more tasks, including longer ones?

q3. How many environment interactions and GPU-hours does each method need to reach the same success rate?

q4. Could you include a small real-robot evaluation, even if training remains in simulation?

6 comments

We thank the reviewer for the careful reading and constructive feedback. We address the concerns below with additional analyses and experiments on critic feedback, cross-task ablations, training efficiency, and physical-robot validation. For your convenience, please also refer to our revised manuscript for the corresponding updates.

 

A1 (W1, Q1): Visual-shift robustness and critic feedback

Temporal credit and the performance gap. The default π-StepNFT uses trajectory-level binary feedback, whereas PPO employs critic-derived GAE for environment-time temporal credit assignment. To investigate whether this difference contributes to the performance gap, we evaluate π-StepNFT + Critic on ManiSkill with π0.5\pi_{0.5}, incorporating GAE feedback while retaining the original observed-transition ranking objective.

ManiSkill π0.5\pi_{0.5}Flow-SDE PPOFlow-Noise PPOπ-StepNFTπ-StepNFT + Critic
ID90.989.784.391.0
Visual Avg. (5)66.269.875.376.1
Semantic Avg. (2)33.133.330.534.4
Execution Avg. (2)45.454.942.250.8

Adding critic-derived feedback improves ID success from 84.3% to 91.0%, semantic success from 30.5% to 34.4%, and execution success from 42.2% to 50.8%. These results suggest that the absence of fine-grained temporal credit assignment contributes to the default method's performance gap on ID and non-visual shifts.

Visual robustness as an empirical finding. Importantly, the visual-shift advantage persists after introducing critic feedback, with mean success improving from 75.3% to 76.1%, compared with 66.2% / 69.8% for the two PPO baselines. The original π-StepNFT also outperforms both baselines with π0\pi_0 (66.5% versus 60.4% / 64.4%). This robustness advantage is an empirical generalization finding. Our primary contribution concerns aligning rollout feedback with observed stochastic denoising transitions rather than identifying the underlying causes of visual generalization.

Together, these results show that critic-derived temporal credit complements observed-transition ranking, substantially narrowing the ID and non-visual performance gaps while preserving the empirical visual robustness advantage.

A2 (W2, Q2): Cross-task validation of target and loss ablations

We extend the main ablations from LIBERO-Object Task 0 to LIBERO-Long Task 6 and LIBERO-Spatial Task 6, each trained for 200 iterations. The new experiments independently examine two components:

  • Supervision target. Holding contrastive ranking, SDE sampling, variance normalization, outcome feedback, and training settings fixed, we compare the observed xt−x_{t^-} and x0x_0-derived targets at each EMA initialization.
  • Learning objective. Holding the observed xt→xt−x_t \rightarrow x_{t^-} transitions and training settings fixed, we compare contrastive ranking against step-wise weighted MSE (wMSE).

The terminal success rates (mean ± SD) are:

ObjectiveTargetα0\alpha_0Long-6Spatial-6
wMSExt−x_{t^-}0.172.14 ± 0.39%93.42 ± 0.12%
Rankingxt−x_{t^-}0.175.23 ± 0.68%97.09 ± 0.12%
Rankingx0x_0-derived0.10.00 ± 0.00%0.00 ± 0.00%
Rankingxt−x_{t^-}0.959.98 ± 0.32%81.30 ± 0.36%
Rankingx0x_0-derived0.964.24 ± 1.04%85.10 ± 0.64%

Learning objective. With identical observed-transition supervision, ranking outperforms wMSE by 3.09 pp on Long-6 and 3.67 pp on Spatial-6, extending the finding in Figure 2d to additional task categories (Long-6 curve, Spatial-6 curve).

Supervision target. At α0=0.1\alpha_0=0.1, observed-transition supervision achieves 75.23% / 97.09%, while the x0x_0-derived target yields 0% on both tasks. At α0=0.9\alpha_0=0.9, the x0x_0-derived target performs better on both tasks. Together with the supervision-target comparison in Figure 2b, these results demonstrate that the observed-transition target provides improved training stability under aggressive EMA initialization, rather than uniformly higher performance across EMA settings (Long-6 curve, Spatial-6 curve).

Additional CALVIN validation. We extend the four subgoal-feedback constructions in Figure 2f to CALVIN Trial 1. The average numbers of completed subtasks (mean ± SD) are 4.84 ± 0.02 for Traj-Relative, 4.67 ± 0.01 for Seg-Relative, 4.54 ± 0.03 for Traj-Success, and 4.42 ± 0.01 for Seg-Success. Traj-Relative again achieves the highest performance, consistent with the original Trial-0 results (training curves).

A3 (W3, Q3): Environment rollout and computational efficiency

Default-configuration comparison. We first compare π-StepNFT and πRL Flow-SDE PPO using the default Random-1 transition-selection configurations of their respective papers, each retaining one randomly selected denoising transition per sampled path. We evaluate both methods on LIBERO-Object Task 0 and LIBERO-Long Task 5 under matched rollout budgets. The table reports GPU-hours and cumulative environment rollouts required to first reach each smoothed on-policy training-success threshold. NR indicates that the threshold was not reached.

TaskMethodRollouts/Iter.Peak SR60%70%80%90%
Object-0π-StepNFT Random-13291.25%0.69 / 2560.77 / 2881.25 / 4803.22 / 1,280
Object-0πRL Flow-SDE PPO Random-13295.00%0.33 / 960.43 / 1280.62 / 1923.77 / 1,248
Long-5π-StepNFT Random-16480.94%0.66 / 2561.86 / 7686.43 / 2,752NR
Long-5πRL Flow-SDE PPO Random-16483.75%0.66 / 2560.97 / 3844.59 / 1,856NR

Threshold entries report GPU-hours / cumulative environment rollouts.

On Object-0, PPO reaches lower success thresholds faster, whereas π-StepNFT reaches 90% in 3.22 GPU-hours, compared with 3.77 GPU-hours for PPO. On Long-5, both methods reach 60% in 0.66 GPU-hours, while PPO reaches 70% and 80% faster.

Additional Last-2 comparison. We further evaluate both methods under a Last-2 configuration that retains two late denoising transitions from each sampled path instead of one randomly selected transition. This halves the number of collected environment rollouts per iteration while maintaining the same number of retained training transitions.

TaskMethodRollouts/Iter.Peak SR60%70%80%90%
Object-0π-StepNFT Last-21696.88%1.28 / 1761.60 / 2241.82 / 2562.38 / 336
Object-0πRL Flow-SDE PPO Last-21647.50%NRNRNRNR
Long-5π-StepNFT Last-23280.00%0.49 / 1280.91 / 2563.47 / 1,024NR
Long-5πRL Flow-SDE PPO Last-23274.38%1.17 / 3204.74 / 1,344NRNR

Under Last-2, π-StepNFT reaches 90% on Object-0 with 336 rollouts and 2.38 GPU-hours, whereas PPO does not reach 60%. On Long-5, π-StepNFT reaches 70% with 256 rollouts and 0.91 GPU-hours, compared with 1,344 rollouts and 4.74 GPU-hours for PPO, and reaches 80% while PPO does not. Compared with π-StepNFT Random-1, Last-2 also reduces the rollouts required to reach 90% on Object-0 from 1,280 to 336 and 80% on Long-5 from 2,752 to 1,024 (training curves).

These single-task results suggest that π-StepNFT has the potential to exploit multiple observed denoising transitions from each collected rollout more effectively, improving transition-level supervision utilization without increasing environment data collection.

Full-suite training-system efficiency. The submitted system profiling on 8× A100 80GB GPUs reports 286 / 310 / 354 seconds per global iteration for π-StepNFT / Flow-SDE PPO / Flow-Noise PPO, respectively (Appendix D.2), corresponding to 7.7% / 19.2% shorter iteration times for π-StepNFT relative to the two PPO baselines.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their constructive feedback. We have incorporated the clarifications and additional experiments into our revised manuscript. The main updates and supporting evidence are summarized below.

Our work identifies an exploration–supervision mismatch in online RL for flow-based VLAs and addresses it through step-wise ranking on observed denoising transitions. The resulting flow-native actor objective supports both binary outcomes and critic-derived feedback. Without a critic, the original LIBERO results exceed Flow-SDE GRPO by 2.0 / 2.5 percentage points for π0\pi_0 / π0.5\pi_{0.5} (Table 1). With critic-derived feedback, the new ManiSkill results achieve PPO-comparable ID success and higher mean visual and overall OOD success (Table 2). The Last-2 diagnostics further suggest potential for improved environment-data utilization.

Original denotes evidence in the submitted manuscript. New denotes experiments added during this rebuttal. Clarifications of existing evidence are labeled separately from new experiments.

ConcernRaised byEvidence sourceResponse and key evidenceLocation
Methodological contribution and feedback compatibilityJX95Original (clarified)Clarified observed-transition ranking as a flow-native actor objective, distinct from PPO's transition-policy-ratio objective. The original formulation accepts binary or critic-derived feedback, with LIBERO + Critic results supporting this compatibility.Original submission, Secs. 3, 4.2 and Table 1
ID performance and visual robustnessEAy6, zquF, JX95NewAdded ManiSkill π0.5\pi_{0.5} + Critic results, compared with the original PPO baselines: ID success rises from 84.3% to 91.0%, comparable to Flow-SDE / Flow-Noise PPO's 90.9% / 89.7%. Visual success reaches 76.1% versus 66.2% / 69.8%, and overall OOD success reaches 61.2% versus 54.2% / 58.3%.Sec. 4.2, Table 2
Independent ablations and cross-task supportEAy6, zquFOriginal + NewOriginal: Figure 2b / 2d already isolate target and objective effects on Object-0. New: Added an Object-0 target comparison under wMSE and extended comparisons to Long-6 and Spatial-6: ranking exceeds wMSE by 3.09 / 3.67 pp. With ranking fixed, observed-transition supervision remains effective at α0=0.1\alpha_0=0.1, where x0x_0-derived supervision collapses. Added CALVIN Trial-1 feedback validation.Sec. 4.3, Figure 2; Apps. D.4–D.5
Validation beyond the π-serieszquFNewAdded GR00T N1.5 experiments on all four LIBERO suites: average success improves from 52.5% to 89.4% (+36.9 pp).Sec. 4.2, Table 1
Physical-robot validationEAy6, zquFNewAdded a Franka Real2Sim2Real pick-and-place experiment. From SFT on 20 real demonstrations, simulator-only RL improves physical-task success from 1/10 to 6/10 trials, with direct deployment and no additional real-world RL.Sec. 4.5, Figure 4
Environment-data utilizationEAy6, JX95NewAdded single-task Random-1 / Last-2 diagnostics supporting potential for improved rollout utilization. Under Last-2, Long-5 reaches 70% smoothed training success with 256 rollouts versus 1,344 for PPO. On Object-0, π-StepNFT reaches 90% with 336 rollouts, while PPO Last-2 does not reach that threshold.App. D.2
Computational costEAy6, JX95Original + NewOriginal: The 8×A100 system profile reports 286 seconds per iteration, versus 310 / 354 seconds for PPO (7.7% / 19.2% shorter iterations). New: Added GPU-hours-to-threshold comparisons on Object-0 and Long-5.App. D.2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.