acceptodds
Under review as a conference paper at ICLR 2027

RedFlow: Redirect Failure into Action-level Corrections for Flow-matching VLA Policy

67 views

Abstract

Reinforcement learning (RL) can improve Vision-Language-Action (VLA) policies through deployment experience, but reward- and preference-based RL primarily identifies desirable behaviors without specifying how to correct failed actions, leaving failure trajectories underutilized and limiting sample efficiency. Can such corrections be derived from fixed rollouts? Our key insight is that rollouts with different outcomes may contain action chunks executed in similar states, allowing higher-quality chunks to serve as locally supported corrective targets. Building on this insight, we introduce **RedFlow**, an offline post-training method for flow-matching VLA policies. *Execution-Context Matching* groups chunks using a compact representation of estimated task progress and the robot's proprioceptive state. *Quality-Guided Action Redirection* assigns signed quality scores to chunks and aggregates higher-quality chunks into corrective targets, reinforcing high-quality chunks, suppressing low-quality chunks, and redirecting correctable chunks toward their targets. RedFlow requires neither human intervention nor online data collection during post-training. Across four LIBERO suites, RedFlow improves the average success rate from 56.2% to 68.2%, outperforming the strongest evaluated offline baseline, AWR (62.3%), by 5.9 percentage points. Across three real-robot tasks, it improves the average success rate from 56.7% to 74.7%. On LIBERO-Spatial, RedFlow achieves a success rate of 75.8% with 1,536 fixed rollouts, while the evaluated online methods require 8.7–16 times as many fresh post-training rollouts to reach the same threshold.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Physical Intelligence, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_3mSkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RedFlow, an offline post-training framework for flow-matching VLA policies. It matches action chunks with similar task progress and robot proprioceptive states, estimates their quality using local progress changes and trajectory outcomes, and constructs corrective targets from higher-quality actions. The policy is trained to reinforce good actions, suppress low-quality actions, and redirect correctable actions toward these targets. Experiments on four LIBERO suites and three real-robot manipulation tasks demonstrate improved success rates and sample efficiency without additional human intervention or online data collection during post-training.

Critical strengths
  1. RedFlow learns from mixed-quality rollout data without requiring additional human interventions or online data collection during post-training, improving the utilization of existing experience.
  2. The method is evaluated across four LIBERO suites and three real-robot tasks, with consistent improvements over the base policy. Ablation studies further examine the contributions of context matching and action-level correction.
Critical weaknesses
  1. The method matches action chunks based on task progress and robot proprioception, without accounting for object poses or contact states. However, similar task progress and robot configurations do not necessarily imply similar corrective actions, as differences in object positions or contact conditions may require different actions. Averaging higher-quality actions within a matched cluster may produce invalid corrective targets.
  2. FlowPRO (https://arxiv.org/pdf/2606.05468) already studies offline post-training of flow-matching VLAs using paired successful and failed trajectories, while VINE(https://arxiv.org/pdf/2512.03913) leverages mixed-quality demonstrations for failure-aware planning. Although RedFlow introduces context-matched corrective targets and action-level redirection, the paper could better establish how these components provide a sufficiently distinct methodological advance beyond existing approaches.
Questions for authors
  1. By weighting and averaging the high-quality actions within the same matching cluster, why can this lead to effective error correction actions? If these actions are very different from each other, even corresponding to different operation methods, will the averaged action still be executable?
  2. The paper only uses the task progress and the robot's own state for matching. If the positions or contact conditions of the two objects are different, how can the method ensure that the matched action is still suitable as a reference for error correction?
  3. The paper assesses the quality of actions based on the changes in the progress of local tasks and the success or failure of the entire trajectory. If a trajectory ultimately fails, but some of the actions within it are actually correct and can be learned, how can the method avoid mistakenly labeling these actions as of low quality?
1 comment

Thank you again for your thoughtful feedback. If you have any remaining concerns or further suggestions, we would greatly appreciate hearing them while there is still time to make revisions, as we hope to address your questions as fully as possible.

Reviewer_rdyHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RedFlow improves a flow matching π0 policy through offline training on a fixed buffer of its own rollouts. Within each task, HDBSCAN groups action chunks by normalized proprioception and smoothed task progress from a frozen Robo-Dopamine 2.0 model. Each chunk receives a signed quality score based on local progress gains and the episode outcome. Training combines three terms: quality weighted flow matching, a margin penalty that discourages reproducing negative chunks, and an L2 correction that pulls their predicted actions toward a quality weighted average of positive actions in the same cluster. Correction is applied only when the cluster contains positive support. RedFlow raises average success from 56.2% to 68.2% on LIBERO, compared with 62.3% for AWR, and from 56.7% to 74.7% across three real robot tasks.

Critical strengths
  1. RedFlow provides explicit corrective targets rather than only changing how strongly observed actions are imitated. It uses better actions from comparable execution contexts to guide selected negative chunks, without requiring human corrections or further interaction during post training.

  2. The empirical results support both the proposed mechanism and its practical usefulness.

Critical weaknesses
  1. The empirical advantage over simpler offline training is not fully established. AWR uses a single reported temperature of 3, without a sensitivity study or a clearly defined, tuned filtered BC baseline. The improvement from 65.7% to 72.5% supports redirection beyond the authors’ particular weighting scheme, but leaves open whether stronger weighting or filtering could close the gap. Co-RFT and ARFM are also cited but not compared. Only one training seed is reported, and the Wilson intervals exclude training and buffer variability. These gaps limit confidence that the added machinery provides a reliable advantage over well tuned alternatives.

  2. The effectiveness of the corrective targets needs more direct validation. Matching uses proprioception and estimated progress but omits object and contact state, while all negative chunks in a cluster share the same positive action centroid. The target comparisons and distance diagnostics are useful, but they do not show that executing these targets from the corresponding failure states improves the outcome. Since a mismatched reference could reinforce an unsuitable action, this leaves a central assumption insufficiently tested. Controlled recovery experiments from recorded failure states would help distinguish effective corrections from a useful form of local action smoothing.

Questions for authors

Q1. Could you compare against AWR with stronger weighting and plain flow matching trained only on successful rollouts or positive quality chunks, using the same data and update budget? The current ablation supports redirection beyond the chosen weighting scheme, but these controls would clarify whether it also improves over simpler ways of excluding poor actions.

Q2. Could you report at least three training seeds for RedFlow, the strongest offline baseline and the main loss ablations on Spatial and Goal? Please report the mean and standard deviation and state whether the rollout buffer is held fixed. The current Wilson intervals capture evaluation uncertainty but leave training variability unresolved.

Q3. Can you evaluate the corrective targets from a representative set of their corresponding failure states, comparing subsequent progress or recovery success with the original actions? A comparison that also applies the centroid loss to positive chunks would help distinguish failure correction from general local action smoothing.

Q4. In the ablation without correctability filtering, what targets are assigned to outliers and clusters containing no positive chunks? Does this variant only remove the filtering condition, or does it also introduce a different target construction rule? This is important for interpreting the reported drop to 61.0%.

Q5. What drives useful policy improvement for negative chunks without positive support in their context cluster? For these chunks, correction is disabled, while quality weighted imitation still attracts predictions toward the recorded action and suppression discourages reproducing it. How should these competing terms produce a better alternative, and what assumptions about rollout coverage are needed for this approach to work?

3 comments

Thank you again for your thoughtful feedback. If you have any remaining concerns or further suggestions, we would greatly appreciate hearing them while there is still time to make revisions, as we hope to address your questions as fully as possible.

Thank you for the clarifications and additional experiments. I appreciate the value of this work and the authors’ efforts to address my concerns. The responses have resolved most of my concerns. I am therefore raising my rating from 2 to 3.

Thank you for taking the time to review our additional experiments and clarifications. We’re glad they addressed most of your concerns. We appreciate how your suggestions helped us strengthen the evaluation, and we’d be happy to discuss if you have any remaining questions.

Comments on the paper

4 comments

Meta Response

We thank the reviewers for their constructive feedback. Our rebuttal adds experiments and clarifications addressing three shared concerns: the strength of simpler baselines, training robustness, and the validity of corrective supervision.

Stronger baseline comparisons and robustness.

We added an AWR temperature sweep and Success-only FM under the same rollout buffer and update budget. RedFlow achieves 72.5% average success across Spatial, Object, and Goal, compared with 65.3% for the best tested AWR setting and 64.5% for Success-only FM. We also evaluated RedFlow, AWR, and the main loss ablations across three training seeds on Spatial and Goal, observing consistently higher mean performance for RedFlow with the buffer held fixed.

Direct validation of corrective targets.

From 100 saved failure states on LIBERO-Goal, matched targets improve recovery success from 53% to 67% under identical continuation conditions. Extending centroid loss to positive chunks instead reduces task success from 71.2% to 68.6%, supporting selective failure correction over broader centroid attraction.

Clarification of matching and unsupported failures.

Progress incorporates visual information indirectly, but matching does not guarantee identical object or contact states. Corrective targets therefore serve as locally supported training references. Negatives without positive support receive suppression alone, with both attraction and correction disabled; we will make this mask explicit in the formulation. We also clarified the nearest-positive fallback used in the no-filtering ablation and added LIBERO-Long coverage and dispersion statistics to characterize its limitations.

Together, these additions strengthen the empirical support for learning selective action-level corrections from unpaired, mixed-quality rollouts, while clarifying the assumptions and limits of context matching.

Thank you again for your thoughtful feedback. If you have any remaining concerns or further suggestions, we would greatly appreciate hearing them while there is still time to make revisions, as we hope to address your questions as fully as possible.

Follow-up on corrective targets, reward choice, and LIBERO-Long

Thank you again for your thoughtful review. We have just completed more additional experiments on reward choice and would like to share the results. Following your suggestions, we also revisited the individual recovery outcomes and examined what the LIBERO-Long diagnostics tell us about its smaller improvement.

1. Recovery outcomes from matched corrective targets

We examined all 100 paired trials from the saved LIBERO-Goal states. From each state, we executed either the original action chunk or its matched corrective target, then continued with the same policy and remaining execution budget. The matched target led to recovery in 18 of the 47 states where the original chunk failed. Conversely, it led to failure in 4 of the 53 states where the original chunk succeeded. This gives 14 additional recoveries overall and shows how often the outcome changed in each direction. The four reversals warrant attention, although their outcomes alone do not identify whether object pose or contact differences contributed.

Table 5. Paired recovery outcomes on 100 saved LIBERO-Goal states.

Outcome with original chunkRecovery with matched targetFailure with matched targetTotal
Recovery49453
Failure182947
Total6733100

2. Sensitivity of the baseline ranking to reward choice

Our earlier Success-only FM experiment tested whether selecting successful trajectories was sufficient. To address your question about ranking, we compared RedFlow, AWR, and IQL on Spatial, Object, and Goal after removing the trajectory-outcome term from Eq. (9). All three methods then used the same local progress difference as their quality signal. AWR used it to weight action chunks at its original temperature of 3.0, and IQL used it as the per-chunk reward. Within each suite, we held the rollout buffer, training seed (42), and update budget fixed across the two settings.

Table 6. Task success (%) under two training signals.

Training signalMethodSpatialObjectGoalAvg.
Progress + outcomeRedFlow75.870.471.272.5
Progress + outcomeAWR71.266.857.865.3
Progress onlyRedFlow71.466.264.467.3
Progress onlyAWR68.266.854.663.2
Outcome onlyAWR62.857.844.254.9

With the original score, RedFlow outperformed AWR on all three suites. With progress alone, RedFlow retained its lead on Spatial and Goal and achieved higher average success (67.3% vs. 63.2%), while AWR scored slightly higher on Object (66.8% vs. 66.2%).

For AWR, outcome-only weighting reduced success on every suite relative to progress-only weighting, with the largest drop on Goal (54.6% to 44.2%). One possible explanation is that the episode outcome assigns the same weight to every chunk in a trajectory. For example, a successful grasp followed by a failed placement receives the failure weight for both actions. Local progress can distinguish these stages, allowing AWR to give greater weight to useful actions even in trajectories that ultimately fail.

3. Corrective support on LIBERO-Long

On Long, 10.1% of negative chunks received a corrective target, close to the coverage on Object (10.5%) and Goal (10.7%), and below Spatial (13.7%). Long also had the highest within-cluster positive-action dispersion at 0.122, compared with 0.074 on Spatial, 0.085 on Object, and 0.102 on Goal. Thus, Long and Goal have similar corrective coverage, but the positive actions contributing to each target vary more on Long. This suggests one possible reason for the smaller gain on Long: even when a target is available, its supporting actions may give a less consistent corrective direction.

We appreciate your suggestions and will include these results and their scope in the revised manuscript.

Dear Reviewers and Area Chair,

Thank you for your careful reading of our paper and for the suggestions that guided our additional experiments. We would like to clarify RedFlow’s methodological contribution and summarize what we learned during the discussion.

1. RedFlow’s methodological contribution. A failed rollout records the action the policy took, while the action that might have helped at that state is unobserved. RedFlow’s contribution is to construct a corrective training target for selected failure states from the policy’s own fixed buffer of unpaired rollouts. It matches chunks by execution context, identifies positive actions observed in comparable contexts, and uses their quality-weighted centroid to guide the flow policy at the original negative observation. This goes beyond reweighting the recorded action by providing a direction in which to change it. Relative to FlowPRO, RedFlow addresses settings where human intervention-and-rollback pairs are unavailable. Relative to VINE, it addresses low-level execution errors that can arise even when the high-level plan is appropriate.

2. Direct evidence for selective correction. Removing the correction loss reduced average success across Spatial, Object, and Goal from 72.5% to 63.5%. To test the references themselves, we compared the original chunk and its matched target from each of 100 saved LIBERO-Goal states, using the same continuation policy and remaining budget. The target led to recovery in 18 states where the original chunk failed, and the reverse occurred in 4 states, increasing overall recovery from 53% to 67%. In a separate 500-episode evaluation, extending the centroid loss to positive chunks achieved 68.6%, compared with 71.2% for standard RedFlow. These results support applying corrective supervision selectively and show why an individual reference should be treated as a training guide rather than a guaranteed recovery action.

3. Simpler baselines and sensitivity to the training signal. Across Spatial, Object, and Goal, RedFlow achieved 72.5% average success, compared with 65.3% for AWR at its best tested temperature and 64.5% for flow matching trained only on successful rollouts. With local progress as the sole quality signal, RedFlow retained a higher average than AWR (67.3% vs. 63.2%), although AWR scored slightly higher on Object (66.8% vs. 66.2%). Using episode outcome alone for AWR yielded 54.9%. We will limit our reward-sensitivity conclusions to the methods and settings evaluated.

4. Robustness across training seeds. We evaluated RedFlow, AWR, and the main loss ablations with three training seeds while holding the rollout buffer fixed. RedFlow achieved 74.6 ± 1.1% on Spatial and 71.5 ± 0.8% on Goal (mean ± sample standard deviation), with the highest mean success rate among the tested variants on both suites.

5. Limits of context matching. We clarified that unsupported negative chunks receive margin-based suppression alone. The 61.0% no-filter ablation extends correction to these chunks using a nearest-positive fallback. We also found that corrective coverage on Long was similar to Goal (10.1% vs. 10.7%), while positive actions within matched clusters were more dispersed (0.122 vs. 0.102). This suggests that identifying a consistent corrective direction may be harder on Long even when positive support is available. We will report these diagnostics and make the limitations of matching under different object poses and contact conditions explicit in the revision.

We appreciate the time you devoted to the discussion. Your feedback helped us test RedFlow’s central idea more directly and clarify the conditions under which corrective supervision is useful.

The RedFlow Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.