- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces π-StepNFT, an online RL method for flow-based vision-language-action models. The key idea is to address an exploration-supervision mismatch created by stochastic SDE sampling: rather than supervising intermediate denoising states using only the final generated action, π-StepNFT directly learns from observed adjacent SDE transitions. It uses variance-normalized transition errors and contrastive ranking between positive and negative velocity branches.
Critical strengths
-
The method is well motivated for flow-based VLA policies. The exploration-supervision mismatch is clearly identified, and using observed SDE transitions for local supervision provides an intuitive solution that avoids computing the final-action likelihood.
-
The paper provides strong mechanistic analysis. The authors carefully ablate SDE exploration, variance normalization, and etc. These experiments make it relatively clear where the performance improvements come from.
Critical weaknesses
The main LIBERO results do not outperform the strongest PPO-based baseline. π-StepNFT reaches 92.0%/94.0% average success for π0/π0.5, or 93.9%/96.0% with a critic, while Flow-SDE + PPO reaches 96.0%/97.9%. This weakens the argument for π-StepNFT as a generally better online RL method. Similar result can be observed in Maniskill experiment. This is my main concern, although its likelihood-free formulation and efficiency remain attractive.
Questions for authors
What do the authors see as the main advantage or potential advantage of π-StepNFT over Flow-SDE + PPO given that PPO remains stronger on the overall LIBERO averages?
4 comments
A1 (W1, Q1): Main advantages over Flow-SDE PPO (Part 2)
Potential for improved denoising-transition utilization. We further evaluate π-StepNFT and πRL Flow-SDE PPO under the same Last-2 configuration, retaining two late denoising transitions per sampled path instead of one randomly selected transition in their default Random-1 configurations. Both methods use half as many environment rollouts per iteration while maintaining the same number of retained training transitions and optimizer updates.
| Task | Method | Rollouts/Iter. | Peak SR | 70% | 80% | 90% |
|---|---|---|---|---|---|---|
| Object-0 | π-StepNFT Last-2 | 16 | 96.88% | 1.60 / 224 | 1.82 / 256 | 2.38 / 336 |
| Object-0 | πRL Flow-SDE PPO Last-2 | 16 | 47.50% | NR | NR | NR |
| Long-5 | π-StepNFT Last-2 | 32 | 80.00% | 0.91 / 256 | 3.47 / 1,024 | NR |
| Long-5 | πRL Flow-SDE PPO Last-2 | 32 | 74.38% | 4.74 / 1,344 | NR | NR |
Threshold entries report GPU-hours / cumulative environment rollouts required to first reach each smoothed on-policy training-success threshold. NR indicates that the threshold was not reached.
Under Last-2, π-StepNFT reaches 90% on Object-0 with 336 rollouts and 2.38 GPU-hours, whereas PPO does not reach 60%. On Long-5, π-StepNFT reaches 70% with 256 rollouts and 0.91 GPU-hours, compared with 1,344 rollouts and 4.74 GPU-hours for PPO, and reaches 80% while PPO does not. Compared with its default Random-1 configuration, π-StepNFT Last-2 reduces the rollouts required to reach 90% on Object-0 from 1,280 to 336 and 80% on Long-5 from 2,752 to 1,024 (training curves). These results suggest that the proposed actor loss has the potential to utilize multiple observed denoising transitions from each collected rollout more effectively, improving environment-data utilization in the evaluated single-task settings.
Thank you to the authors for their detailed responses and additional experiments. The clarifications and experimental results address part of my concerns and confirm some of my initial assumptions. I will keep my score unchanged, as I believe it already appropriately reflects my overall assessment of the paper.
Thank you again for your time and for considering our clarifications and additional experiments. We respect your assessment and would like to better understand the remaining concerns so that we can address them more precisely.
We acknowledge that PPO retains higher overall LIBERO averages. Our response highlights the observed-transition ranking objective, its compatibility with critic-derived temporal credit, and the accompanying robustness and transition-utilization results.
Without a critic, Table 1 of the original submission reports higher LIBERO average success than Flow-SDE GRPO for both backbones: 92.0% vs. 90.0% for and 94.0% vs. 91.5% for , corresponding to gains of 2.0 and 2.5 percentage points.
With critic-derived feedback, the added ManiSkill experiment shows that π-StepNFT + Critic achieves 91.0% ID success, comparable to 90.9% / 89.7% for Flow-SDE / Flow-Noise PPO, while obtaining higher visual success (76.1% vs. 66.2% / 69.8%) and overall OOD success (61.2% vs. 54.2% / 58.3%). These results show that the visual advantage can coexist with PPO-comparable ID performance.
The additional Last-2 diagnostic also provides evidence of potential improvements in denoising-transition utilization. On Long-5, π-StepNFT reaches the 70% smoothed training-success threshold after 256 rollouts, compared with 1,344 for Flow-SDE PPO under its Last-2 configuration. The corresponding updates are included in our revised manuscript.
If time permits, could you briefly clarify whether the remaining concern primarily relates to the LIBERO performance gap or to the proposed objective and interpretation of the additional results? It would also be helpful to understand which initial assumptions you consider confirmed. We would appreciate your guidance in addressing these points and improving the work.
We thank the reviewer for recognizing the motivation and mechanistic analysis of π-StepNFT. We clarify its main advantages in flow-native actor optimization, compatibility with critic-based feedback, and denoising-transition utilization. For your convenience, please also refer to our revised manuscript for the corresponding updates.
A1 (W1, Q1): Main advantages over Flow-SDE PPO (Part 1)
Flow-native actor optimization. π-StepNFT directly optimizes the flow-matching velocity field through contrastive ranking of mirrored branches on observed xt→xt− SDE transitions. In contrast, πRL Flow-SDE PPO uses advantage-weighted current-to-old transition-policy probability ratios, while Flow-Noise PPO additionally retains the multi-step denoising solver chain during actor optimization. By mapping rollout feedback directly to local velocity-field updates, π-StepNFT avoids PPO-style importance ratios and backpropagation through the full denoising chain. This provides a direct velocity-prediction optimization interface for flow-based policies.
Standalone actor loss and critic compatibility. π-StepNFT introduces an actor loss that operates independently with binary success/failure feedback and can also incorporate critic-derived temporal credit without changing the ranking objective. This allows environment-time credit assignment to complement denoising-time supervision, as demonstrated by the following results.
Success rates are in %. OOD Avg. averages the nine shared visual, semantic, and execution variants, excluding ID.
Critic-derived feedback improves performance across multiple settings. On LIBERO, GAE improves the overall averages from 92.0% / 94.0% to 93.9% / 96.0% for π0/π0.5, with substantial gains on Long (86.7% → 90.0% and 79.8% → 88.4%, respectively). On ManiSkill, ID success increases from 84.3% to 91.0%, reaching PPO-level performance. Semantic success improves from 30.5% to 34.4%, exceeding both PPO baselines (33.1% / 33.3%), while execution success increases from 42.2% to 50.8%, surpassing Flow-SDE PPO (45.4%). The overall OOD average rises from 58.0% to 61.2%, exceeding both PPO baselines, while preserving the visual-shift advantage (76.1% vs. 66.2% / 69.8%). These results demonstrate that critic-derived temporal credit complements the proposed actor loss, improving long-horizon performance and robustness across the evaluated distribution shifts.