acceptodds
Withdrawn from ICLR 2027

Learning Flow Policies from Unary Feedback for Real-World Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In real-world robot manipulation, flow policies can perform complex tasks after supervised fine-tuning (SFT), yet still make recurring errors during deployment. Expanding the expert demonstration dataset to correct these errors can be costly. Deployment failures provide another source of information for policy refinement. For flow policies, however, direct optimization through action generation can produce unstable gradients, and action log-probabilities are costly to evaluate. We propose Flow-KTO, an offline post-training method for flow policies. It learns from success and failure feedback on recorded execution clips. Flow-KTO accounts for asynchronous inference latency to align execution feedback with the corresponding action trajectories. The method replaces direct computation of policy log-likelihood ratios with a surrogate, avoiding backpropagation through action generation. A policy drift estimate modulates the strength of feedback updates at each training step. Across four real-robot tasks, Flow-KTO reduces the average target-mistake recurrence rate by 33.3% relative to the strongest baseline, while improving average success rate by 13.3% and increasing task throughput by 17.7% on average.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Flow Matching, Offline Policy Post-training, Deployment Feedback, Learning from Failures.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fFXAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method, called Flow-KTO, for leveraging deployment feedback for policy improvement. To method actually applies KTO-style objective on the positive/negative rollouts, combined with a BC loss and a drift estimate for stabilizing. Moreover, the method consider the asynchronization between action generation and observation in the real-world. The method is evaluated in real-wolrd deployment, and demonstrates improvement by leveraging on manipulation tasks.

Critical strengths
  1. The method demonstrate performance gain compared to existing methods on real-world tasks.
Critical weaknesses
  1. This paper is hard to follow. There are lots of equations in the method section while the notations are not well explained.
  2. The evaluation is limited. And the details about experimental settings are missing.
Questions for authors
  1. Can the author provide more justification on some experimental designs, such as the fraction of initial policy failed rollouts in the training set?
  2. There is an assumption that the offset between the generated action chunk and the observation is fixed. Is this a strong assumtion in real-world tasks? If so, does it have a large influence on the results?
Reviewer_BYfQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flow-KTO, an offline post-training method for improving pretrained flow policies using binary feedback on autonomous robot executions. The method substitutes differences in ELBO-weighted conditional flow-matching losses for policy log-likelihood ratios within a KTO objective. It additionally introduces latency-aware feedback alignment, estimates a reference point using samples from the current policy, and incorporates a behavior-cloning anchor. Experiments on four real-world manipulation tasks report an increase in macro-average success rate from 60% to 68% and a reduction in target-mistake recurrence from 30% to 20% relative to the fixed-data FPO baseline.

Critical strengths
  1. The problem setting is practically relevant. Learning from successful and failed autonomous executions could reduce dependence on additional expert corrections. Unary feedback also avoids collecting alternative actions under precisely matched physical conditions, which is difficult in real-world manipulation.

  2. The real-robot results are encouraging. The complete method improves success rate, target-mistake recurrence, and successful-task throughput across all four evaluated tasks. The negative-feedback ablation further supports the usefulness of explicitly learning from undesirable executions beyond positive-only fine-tuning.

Critical weaknesses
  1. The temporal definition of the training data is inconsistent with, or at least insufficient to support, the proposed latency-alignment rule. The paper defines aia_i as “the HH-step action chunk beginning at frame ii” (line 267) and pairs it with oio_i. Taken literally as execution-time indexing, this definition does not encode the claimed observation-to-execution delay. To account for that delay, a natural interpretation is that ii indexes the observation time and the associated action sequence begins execution at i+ki+k. Under this interpretation, requiring the entire HH-step execution window to lie within an annotation [τ,T][\tau,T], where T=τ+nT=\tau+n, gives i∈[τ−k,T−k−H]i\in[\tau-k,T-k-H], up to discrete endpoint conventions. However, Eq. (13) admits starts up to min⁡(T−H,T−k)\min(T-H,T-k), thereby including additional windows whose execution extends beyond the annotation endpoint.
  2. The connection between the proposed reference-point mechanism and its stated goal of avoiding excessive policy drift is insufficiently justified. With gradients stopped through zz, the feedback-gradient coefficient is proportional to σ′(β[gθ−z])\sigma'(\beta[g_\theta-z]). It therefore decreases with distance from the batch reference point, rather than monotonically with the magnitude of the reference-relative score. Consequently, a sample with gθ≈0g_\theta\approx0, indicating little change in its surrogate log-likelihood relative to the reference policy, can receive a small gradient weight when zz is sufficiently far from zero. Conversely, a sample with a substantially positive score can receive maximal weight if its score is close to zz. Although individual scores are not themselves distributional drift measures, these properties show that the mechanism does not directly implement a monotonic restriction on departure from the reference policy. The paper should explain why this particular weighting rule is expected to prevent excessive drift, beyond its observed benefit to task performance.
Questions for authors
  1. The gradient weight becomes small when β∣gθ−z∣\beta|g_\theta-z| is large, rather than when ∣gθ∣|g_\theta| is large. How does this support the stated objective of avoiding excessive drift?
  2. Should ii be interpreted as the observation timestamp, with oio_i paired with an action sequence executed from i+ki+k? If so, full containment within [τ,T][\tau,T] appears to require i≤T−k−Hi\le T-k-H. Why does Eq. (13) instead permit starts up to min⁡(T−H,T−k)\min(T-H,T-k), including windows whose execution extends beyond TT?
  3. Is aia_i a stored prediction or a reconstructed sequence of executed actions? Please specify the control frequency, inference frequency, prediction horizon, number of actions executed before replanning, and delay kk.
  4. Can you provide a matched direct KTO/ELBO-KTO adaptation to isolate the proposed modifications? Please also clarify how the non-integer-percentage ablation rates, such as 56.9% (line 460), 21.3% (line 461), 24.1% (line 461) and 43.2% (line 469), were obtained under the stated 25-trial protocol.
Reviewer_n97oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use success and failure feedback from deployment to reduce recurring errors in flow policies for real-world robot manipulation. The authors propose Flow-KTO, which refines a pretrained policy offline using recorded execution clips with independent positive and negative labels. The method first accounts for asynchronous inference latency to align execution feedback with the corresponding action chunks. It then uses the difference between the current and frozen reference policies' ELBO-weighted CFM losses as a likelihood-ratio surrogate and applies a bounded KTO objective to positive and negative samples, avoiding backpropagation through the action-generation ODE. Policy-drift estimation adjusts the feedback updates, while a BC anchor helps retain existing behavior. Across four real-robot tasks, Flow-KTO achieves an average success rate of 68%, compared with 60% for FPO, while reducing the average target-mistake recurrence rate from 30% to 20%.

Critical strengths
  1. Improving robot policies using success and failure clips from existing deployment data has practical value. The method uses independent positive and negative feedback, without requiring paired preference data under the same conditions or backpropagation through the action-generation ODE.

  2. Comparisons on four real-robot tasks show improved success rates and reduced target-mistake recurrence rates. Ablations of negative feedback, latency alignment, policy-drift estimation, and the BC anchor also provide evidence for their roles in the evaluated settings.

Critical weaknesses

W1. Using success and failure feedback from robot deployment to improve policies has been studied in many works, as the authors discuss in Related Work. Overall, the contribution focuses on integrating existing techniques and validating them on real robots, rather than introducing a new optimization algorithm. The main components also come from existing techniques: the bounded positive and negative feedback objective comes from KTO (Ethayarajh et al., 2024), the use of CFM loss differences as a likelihood-ratio surrogate builds on McAllister et al. (2025) and related work, and the BC anchor uses a standard behavior-cloning objective. The more specific design appears to be the integration of these components for offline post-training of real-robot flow policies, with latency-aware feedback alignment and policy-drift estimation. These designs appear fairly straightforward, and the methodological novelty is relatively limited.

W2. The paper uses the difference between the current and reference policies' ELBO-weighted CFM losses as a likelihood-ratio surrogate for positive and negative feedback updates and policy-drift estimation. This approach has a theoretical basis in prior work, and the authors explicitly acknowledge the approximations caused by the variational-gap difference and the flow-time cutoff. However, the paper does not directly verify whether the surrogate reliably reflects changes in policy probabilities during training, nor does it analyze how the choice of cutoff affects final performance. The ablations show the roles of the components in the current setting, but they are insufficient to assess sensitivity to these approximations. In addition, the paper reports small-scale experiments on only four real-robot tasks, without larger-scale quantitative evaluation in simulation. Evidence for the overall method's effectiveness across broader tasks and state distributions therefore remains limited.

Questions for authors
  1. For Weakness 1, could the authors clarify the technical innovations? Please also correct any misunderstanding in my assessment.

  2. For Weakness 2, experimental evidence would be welcome if feasible. If time does not permit additional experiments, a well-supported theoretical analysis would also be acceptable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.