acceptodds
Withdrawn from ICLR 2027

DRIVE: Dense Reward-Informed Action-Value Learning from Mixed Experience for Flow-Based VLA Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) policies are predominantly trained through imitation learning on demonstration data, while the rich execution signals contained in autonomous failures and human interventions remain underexploited. Such mixed execution experience contains both effective and low-value actions, making direct imitation unsuitable for policy refinement. We propose DRIVE, a dense reward-informed action-value learning framework for refining flow-based VLA policies from mixed robot experience. DRIVE first adapts a task-conditioned visual-language progress estimator on successful trajectories, then uses it to quantify advancement, stagnation, and regression across expert demonstrations, autonomous successes, autonomous failures, and successful human-intervention trajectories. The resulting dense supervision is distilled into an action-chunk critic that estimates long-horizon values for candidate action chunks. Under the same task condition, DRIVE compares candidate values and converts their relative differences into soft refinement labels. These labels guide a negative-aware forward flow-matching objective that reinforces high-value action chunks while explicitly suppressing inferior alternatives. Importantly, the progress estimator and critic are used only during training, so the refined policy incurs no additional inference-time computation. Experiments on LIBERO and four real-world manipulation tasks show consistent improvements across multiple flow-based VLA backbones, achieving 98.25% average success on LIBERO and 62.5% on real-world tasks, with a 40.0-percentage-point improvement over the corresponding base policy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Reinforcement Learning, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qzLYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DRIVE, a post-training framework designed to refine flow-based VLA policies using mixed robot experience including expert demonstrations, autonomous successes, autonomous failures, and human interventions. The method adapts a vision-language progress estimator on successful rollouts to provide dense chunk-level rewards, which are distilled into an action-chunk critic via implicit Q-learning. Under the same state condition, the critic evaluates candidate action chunks to produce relative soft labels that guide a negative-aware flow-matching objective, reinforcing high-value actions while suppressing inferior ones. The critic then provides a negative-aware objective that reinforces high-value actions and penalizes low-value alternatives Experiments report 98.25% average success on LIBERO and 62.5% on four real-world tasks.

Critical strengths

The paper solves the problem of how to effectively utilizing mixed robot execution experience especially failures and human interventions. It avoids the problem of negative weight in SFT will cause the model to collapse. The method is simple and effective, and the experiments show that it can significantly improve the performance of the policy.

Critical weaknesses
  • some core components are not novel, such as the vision-language progress estimator and the negative-aware fine-tuning.
  • The effectiveness of the method is heavily dependent on the quality of the vision-language progress estimator and the critic.
Questions for authors
  • The paper mentions that the method is post-training, but unlike other online RL methods, it does not update the critic during training. Will this cause potential reward hacking?
  • The dense reward is generated by a learned progress estimator, which may be inaccurate. How does the quality of the progress estimator affect the performance of the method?
  • How is the failure and human intervention data collected in LIBERO? The base model of DRIVE is already 97.2% on LIBERO, so the benchmark is a bit saturated.
Reviewer_vXHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DRIVE is an offline RL method for VLAs that uses mixed-quality robot data to fine-tune a VLA policy. It trains a chunk-level critic on the rewards from this data and uses it to score candidate actions. Using the scored candidates, it fine-tunes the VLA with Negative-Aware Forward Flow Refinement, which encourages actions with high Q-values while suppressing those with low Q-values. Experiments on LIBERO and four real-world manipulation tasks show consistent improvements across multiple flow-based VLA backbones.

Critical strengths
  1. The paper tackles an important problem in robot learning: improving VLA policies with suboptimal data and deployment experience.
  2. Thorough evaluation across VLA backbones. DRIVE is applied to four flow-based VLAs (Evo-1, π0, π0.5, and MolmoAct2) and improves every one on real-world tasks.
Critical weaknesses
  1. The LIBERO gain is small and tuned on the test set. The improvement over the reproduced base policy (96.75%) is only about 1.5 points. The hyperparameter sweeps were run on the evaluation benchmark itself.
  2. Baselines are missing. The LIBERO comparison includes no offline RL method. The closest prior work, π0.6* [1], is not compared. Neither is value-guided action selection at inference or any RL fine-tuning method.

[1] Physical Intelligence et al. "π*0.6: a VLA That Learns From Experience." arXiv:2511.14759, 2025.

Questions for authors
  1. How does DRIVE compare with standard offline RL baselines on the same data and critic, such as IQL with AWR policy extraction, Diffusion-QL/FQL, or RECAP-style advantage conditioning?
  2. Can you measure how accurately the critic ranks candidate actions from the same state, for example against rollout outcomes? Does a time-only predictor reach similar correlations to those in Fig. 3?
  3. What happens if the frozen critic is used for value-guided action selection at inference instead of fine-tuning the policy?
Reviewer_rqRPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DRIVE is a training-time refinement framework for flow-based VLA policies. It first fine-tunes a base policy on expert demonstrations and collects autonomous successes, autonomous failures, and successful human-intervention trajectories. A progress estimator adapted on successful trajectories assigns dense chunk-level rewards to all mixed experience. These rewards are then used to train an IQL-style action-chunk critic Q(c,A)Q(c,A) and state-value function V(c)V(c). During refinement, DRIVE retains the logged action chunk for each condition and samples alternative chunks from a reference policy. The critic scores these candidates, and their within-condition relative values are converted into soft quality labels. These labels guide an NFT-style negative-aware flow-matching objective that reinforces high-value actions and suppresses inferior ones. The auxiliary progress estimator, critic, and candidate evaluation are discarded after training, so deployment uses only the refined VLA policy.

Critical strengths
  1. The RL method for flow matching models similar to \pi is an important issue.

  2. The paper provides explicit objectives and algorithms connecting dense progress rewards, action-value learning, candidate evaluation, and flow refinement.

Critical weaknesses
  1. Limited novelty. DRIVE mainly combines existing components: process reward modeling from Robo-Dopamine, IQL-style learning from IQL, and NFT-style flow refinement from DiffusionNFT. π-StepNFT had already applied NFT-style training to flow-based VLAs.

  2. Insufficient validation of the progress estimator. The estimator is adapted only on successful trajectories but is used to score failures, stagnation, and intervention states. The reported critic–progress correlation is not an independent validation because the critic was trained using those same progress labels.

  3. Incomplete ablations. The paper combines several components without performing systematic component-wise ablations. The main component and data-source ablations are conducted only on real-robot tasks; the LIBERO experiments mainly report hyperparameter sensitivity rather than simulation component ablations.

  4. No direct comparison with existing RL methods. The paper does not provide matched comparisons with current RL methods such as π-StepNFT, making it difficult to determine whether the gains come specifically from DRIVE’s value-guided refinement.

Questions for authors

see weakness above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.