acceptodds
Under review as a conference paper at ICLR 2027

Flowing With Purpose: Latent Action Guided Flow Matching Policies For Robotic Manipulation

Abstract

Flow matching has recently become a new standard for behavior cloning in robotic manipulation. However, state-of-the-art flow matching policies suffer from a systematic structural mismatch: they rely on a globally fixed isotropic source distribution despite the strongly fragmented and heteroscedastic structure of robotic action spaces. This agnostic initialization forces the model to learn highly entangled vector fields, bottlenecking training efficiency and limiting overall policy performance. To address this limitation, we introduce Latent Action Guided Flow Matching (LAFM), a novel framework that replaces the monolithic Gaussian with an adaptive library of learned prior distributions. By grounding these distributions using a latent action model, LAFM maps current observations to discrete motion primitives, selecting a specialized base distribution that provides an informed, structurally aligned initialization for the denoising process. This dynamic adaptivity naturally accommodates heteroscedasticity in human demonstrations and makes transport trajectories shorter and less entangled, reducing the number of training steps required to reach the final error of standard flow matching by 32.5%. Empirically, LAFM substantially outperforms standard flow matching formulations, increasing task success rates by 23.4 percentage points in real-world robotic deployments, 10.4 points on LIBERO-90, and 12 to 15 points on the more challenging LIBERO-Plus and RoboTwin 2.0 benchmarks. Code and model weights will be publicly released upon acceptance of the publication.

Primary area:
applications to robotics, autonomy, planning
Keywords:
flow matching, robotic manipulation, imitation learning, latent action models, learned priors, source distribution design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p3JEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Latent Action Guided Flow Matching (LAFM), which replaces the fixed isotropic Gaussian source in robotic flow-matching policies with latent-action-conditioned Gaussian priors to better capture multimodal and heteroscedastic action distributions. A pretrained latent action model discretizes visual dynamics into motion primitives, and the predicted latent action selects the corresponding learnable prior, with a proposition showing that mode-aligned priors can reduce expected transport distance. Experiments on real-world Franka manipulation and LIBERO show substantial gains over standard flow matching, including +23.4 points in real-world success rate and +10.4 points on LIBERO-90.

Critical strengths
  1. This paper presents a clear and well-motivated idea by replacing the fixed Gaussian source with latent-action-conditioned priors, providing a simple yet intuitive way to better align the source with multimodal and heteroscedastic robotic action distributions.
  2. LAFM demonstrates strong empirical performance, substantially improving over standard flow matching on LIBERO-90 and real-world manipulation tasks while remaining competitive with much larger pretrained VLA models.
  3. This paper includes useful ablations and qualitative analyses, including latent-action conditioning, learned prior visualization, and transport trajectory analysis, which help support the effectiveness of structured source distributions beyond auxiliary latent-action supervision.
Critical weaknesses
  1. The novelty positioning against recent structured-source, source-reparameterization, and learned-coupling methods is incomplete. Recent works (CSFM [1], COT-FM [2], CAR-Flow [3], VCT [4]) have already explored modifying the source distribution or source-target coupling in flow-based generative models. These works are highly relevant to LAFM's core motivation, yet the current comparison mainly focuses on standard FM and FM+LAC.
  2. Proposition 1 relies on the assumption μk≈Eot[mk(ot)]\mu_k \approx \mathbb E_{o_t}[m_k(o_t)] where mk(ot)m_k(o_t) is the mean of the target action mode and μk\mu_k is the mean of the learned source prior. However, although source and target are represented in the same Euclidean ambient space, there is no reason that the learned source mean should naturally lie close to the corresponding target-mode mean. This is a very strong but potentially problematic assumption.
  3. The respective contributions of the learned mean and variance are not disentangled. LAFM attributes its benefit both to mode alignment and to modeling heteroscedasticity, but the current experiments jointly learn mean and variance. Mean-only and variance-only ablations would clarify whether the improvement mainly comes from relocating the source modes or from modeling mode-dependent uncertainty.
  4. Standard LIBERO appears close to saturation and may not sufficiently stress the proposed method [5]. A more challenging robustness-oriented benchmark such as LIBERO-PRO [6] would better test whether latent-action priors capture transferable motion structure rather than only improving in-distribution fitting.

[1] CSFM: https://arxiv.org/abs/2602.05951 (NeurIPS)
[2] COT-FM: https://arxiv.org/abs/2603.13395 (CVPR 2026)
[3] CAR-Flow: https://arxiv.org/abs/2509.19300 (NeurIPS 2025)
[4] VCT: https://arxiv.org/abs/2502.18197 (ICML 2025)
[5] https://arxiv.org/abs/2606.04233
[6] https://arxiv.org/abs/2510.03827

Questions for authors
  1. Please discuss CSFM [1], COT-FM [2], CAR-Flow [3], and VCT [4] in the related work section. While CSFM can be considered concurrent work, COT-FM is a closely related prior method that also introduces structured source distributions for flow matching and evaluates on LIBERO. A direct comparison with COT-FM is therefore important for clarifying the empirical advantage and novelty of LAFM.
  2. Could the authors empirically verify whether the two assumptions in Proposition 1 are satisfied by the learned priors, i.e., μk≈Eot[mk(ot)]\mu_k \approx \mathbb{E}_{o_t}\left[m_k(o_t)\right] and Tr⁡(Σk)≤Tr⁡(I)\operatorname{Tr}(\Sigma_k) \leq \operatorname{Tr}(I)? This would help assess whether the proposition is applicable to the actual learned model.
  3. For the remaining weaknesses, I encourage the authors to provide additional experiments and analyses addressing the concerns above. If the revised evidence is sufficiently comprehensive and resolves these issues, I would be willing to reconsider my current score upward.
4 comments
AuthorsOpenReview ↗

Response to Reviewer p3JE (Part 1)

We thank the reviewer for the detailed review and for the precise list of related works and requested analyses. We address each point below. In short: the four works are now discussed in the paper; the conditions of Proposition 1 are verified empirically; the mean-only and variance-only ablations are running; and results on two less saturated benchmarks are in Appendix A.

1. Positioning against structured-source and coupling methods

We thank the reviewer for these references. All four are now discussed in a new subsection of the related work (Section 4.2). We also compare against robotics methods with condition-dependent sources (BRIDGER, Cocos, A2A) in our response to Reviewer mX8F, including a direct comparison with Cocos on LIBERO.

CSFM, CAR-Flow, VCT.

  • CSFM learns one Gaussian per condition for text-to-image generation. Its authors report that the benefit fades when a single condition maps to a highly multimodal target. This is the typical regime of robotic manipulation, which LAFM addresses by first selecting a discrete motion primitive.
  • CAR-Flow learns a per-class shift of the source but keeps its variance fixed, since learning the scale admits collapsed solutions. LAFM has no class label at inference, so it predicts the prior index from the observation, and it learns per-dimension variances, stabilized by a KL regularizer.
  • VCT learns a data-dependent coupling during training, but sampling still starts from a single isotropic Gaussian. LAFM changes the source used at both training and inference.

COT-FM is the closest prior method: it also assigns Gaussian sources to clusters of the target distribution. The two methods differ in how the sources are obtained and in what they are for.

  • COT-FM is applied to an already trained model. Sources are estimated by integrating a pretrained flow backward, and the model is then fine-tuned with optimal transport couplings within each cluster. LAFM learns its prior library jointly with the policy, from scratch.
  • For robot policies, where a new observation cannot be assigned to a fixed cluster, COT-FM replaces the cluster library with a continuous observation-conditioned Gaussian trained in an additional RL stage. LAFM keeps a discrete library and selects the prior by predicting a latent motion primitive.
  • COT-FM targets few-step sampling: it straightens the flow of a trained model. LAFM targets learning: it structures the source from the start of training, and its benefits are convergence speed and final success rate.

On LIBERO, COT-FM is applied to the pretrained FLOWER policy (~964M parameters) and reported on the Spatial and Long suites:

PolicyParamsPretrainedNFESpatialLong
FLOWER~964MYes497.193.5
FLOWER~964MYes194.287.3
FLOWER + COT-FM~964MYes196.194.5
FM (ours)110MNo1096.191.2
LAFM (ours)110MNo1097.292.3

These numbers show the difference in purpose. COT-FM recovers with one function evaluation what FLOWER reaches with four. LAFM improves the policy itself over its FM baseline. With roughly 9 times fewer parameters and no pretraining, LAFM is 1.1 points above COT-FM on Spatial and 2.2 points below on Long. As we show on Appendix I, both FM and LAFM are quite robust to the number of function evaluations, keeping its success rate with one step in LIBERO-90 (Figure 9). We see the two methods as complementary, one proposing an adaptation of trained policies to reduce NFE and another improving training efficency.

A close controlled comparison is given in Appendix F, where the prior library is indexed by clusters computed in action space (K-means over action chunks) instead of latent actions, with the same architecture and training. This reaches 89.5% on LIBERO-90, against 82.6% for FM and 93.0% for LAFM. Clustering the targets helps, as COT-FM argues, and latent actions give a better partition because the mode must be predicted from the observation alone.

Since CSFM, CAR-Flow, and VCT are evaluated on image generation, an empirical comparison would require reimplementing them for robot policies. The closest control is FM + LP (Table 3, added in the revision), which learns a single Gaussian source jointly with the policy. It reaches 84.3% on LIBERO-90, and its training collapses without the KL regularizer, consistent with the degenerate solutions described by CAR-Flow.

AuthorsOpenReview ↗

Response to Reviewer p3JE (Part 2)

2. Assumptions of Proposition 1

We agree with the reviewer that nothing guarantees a priori that a learned source mean lies close to the mean of its action mode. We have made two changes.

The proposition no longer assumes exact alignment. The revised Proposition 1 gives the reduction in expected transport distance as the sum of a mean-alignment contribution and a variance-reduction contribution, and holds whenever this sum is non-negative. The mean-alignment term is positive as soon as the prior mean is closer to the mode mean than the origin is. Appendix K gives the full decomposition, including the case where the selected mode differs from the action mode.

Both conditions are verified on the learned priors (Section 3.3), on a held-out validation set of LIBERO-90:

  • Mean alignment: for 95.1% of time steps, the selected prior mean is closer to the mean of the corresponding action mode than the origin, ∥m^k−μk∥2<∥m^k∥2\|\widehat{\mathbf{m}}_k - \boldsymbol{\mu}_k\|^2 < \|\widehat{\mathbf{m}}_k\|^2.
  • Source variance: the average variance per dimension is E^[Tr(Σk)/d]=0.13\widehat{\mathbb{E}}[\text{Tr}(\boldsymbol{\Sigma}_k)/d] = 0.13, well below the value of 1 of the isotropic source.

The alignment is therefore an outcome of training rather than an assumption. It is also what the objective favors: the prior parameters receive gradients from the flow matching loss, whose regression target is At−ϵk\mathbf{A}_t - \epsilon_k, so that shorter transport is easier to fit, while the KL regularizer prevents the priors from collapsing. Its effect on the learned flow is measurable: the kinetic energy drops by 67% (Section 3.3).

3. Contributions of the learned mean and variance

We agree that this ablation is needed. We are training two variants on LIBERO-90 with the same architecture and configuration as LAFM:

  • mean-only: learned μk\boldsymbol{\mu}_k, with Σk=I\boldsymbol{\Sigma}_k = \mathbf{I} fixed;
  • variance-only: learned Σk\boldsymbol{\Sigma}_k, with μk=0\boldsymbol{\mu}_k = \mathbf{0} fixed.

We will post the results as soon as the runs finish.

4. Benchmark saturation

We agree that the four standard LIBERO suites are close to saturation, and say so in Section 3.1.2. For this reason the paper also reports LIBERO-90 and, in Appendix A, two less saturated benchmarks:

BenchmarkFMLAFMGain
LIBERO (Spatial, Object, Goal, Long)95.4 ± 0.396.5 ± 0.3+1.1
LIBERO-9082.6 ± 0.293.0 ± 0.2+10.4
RoboTwin 2.053.7 ± 0.366.0 ± 0.2+12.3
LIBERO-Plus50.4 ± 0.265.5 ± 0.1+15.1

LIBERO-Plus is a robustness benchmark of the same family as LIBERO-PRO. It perturbs the four LIBERO suites along seven dimensions (object layout, camera viewpoint, robot initial state, language instructions, lighting, background, and sensor noise), for 10,030 task instances unseen during training. The same LAM, trained on a different robot (Fractal), is used for all benchmarks. LIBERO-PRO perturbs objects, initial states, instructions, and environments, which LIBERO-Plus also covers.

Summary of revisions

  • added Section 4.2, discussing CSFM, COT-FM, CAR-Flow, and VCT;
  • revised Proposition 1 so that it no longer assumes exact mean alignment, and verified its two conditions on the learned priors (Section 3.3);
  • added the FM + LP control (single learned source) to Table 3;
  • added discussion and comparison with other methods (BRIDGER, Cocos, A2A).

Still running, to be posted during the discussion period: the mean-only and variance-only ablations. We hope these changes address the reviewer's concerns, and we are glad to provide any further analysis.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in clarifying the relationship with prior work, revising Proposition 1, and providing results on more challenging benchmarks. The revised theoretical analysis and robustness evaluations address several of my major concerns. However, I believe the mean-only and variance-only ablations are important for understanding the respective contributions of mode alignment and heteroscedasticity. I look forward to the results of these ablations before making a final decision on my score.

We thank the reviewer for the follow-up. The mean-only and variance-only ablations are now complete and have been added to the paper (Appendix H). Both variants keep the latent action model, classifier, architecture, and training configuration of LAFM, and differ only in which prior parameter is learned. Results on LIBERO-90, over 3 seeds:

PolicyLearned meanLearned varianceSuccess rate (%)
FM––82.6 ± 0.2
FM + LAC––85.4 ± 0.1
LAFM, mean only✓x87.7 ± 0.5
LAFM, variance onlyx✓88.3 ± 0.2
LAFM✓✓93.0 ± 0.2

Each contribution identified by Proposition 1 is useful on its own: learning either the means (mode alignment) or the variances (heteroscedasticity) improves over FM + LAC, which shares the same latent action classifier, and the two variants are within seed variability of each other. However, neither recovers the performance of LAFM. The two parameters are therefore complementary: a prior is most useful when it is both centered on its motion mode and scaled to its spread.

We have also added the controlled comparison with BRIDGeR and A2A reimplemented in our architecture (Appendix G) and added both methods, using their original implementation, as baselines on Section 3.1.2.

We hope these results address the reviewer's remaining concern.

Reviewer_XPLWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LAFM replaces the fixed Gaussian source in an action flow-matching policy with a Gaussian selected by a predicted latent action. The idea is to start each generated action closer to the right motion mode. The paper evaluates this in simulation and on four Franka tasks, and offers a transport-based explanation for the gain.

Critical strengths

The first strength is the controlled comparison in Table 3. With the same policy architecture, FM reaches 82.6% on LIBERO-90, FM + LAC reaches 85.4%, and LAFM reaches 93.0%. The middle control matters: a latent-action signal alone does not reproduce the full gain. It gives me some confidence that changing the source distribution, rather than merely adding an auxiliary code, helps this policy.

The second is that the effect is not limited to one favorable table. LAFM also improves on LIBERO-Plus (65.5% versus 50.4%) and RoboTwin 2.0 (66.0% versus 53.7%) in Appendix A, Table 4. Those benchmarks are less saturated than the four standard LIBERO suites, where the gain is only 1.1 points. That pattern makes the result more interesting than a small improvement on an already near-ceiling benchmark.

The third is the same-architecture real-robot comparison: LAFM reaches 86.7% versus 63.3% for FM across four Franka tasks (Table 1). I do not think the trial count supports a precise estimate of the margin, but it is valuable that the paper checks whether the simulation gain survives contact with hardware. Together with the controlled LIBERO-90 result, this keeps me on the weak-accept side.

The action-clustering controls in Appendix F are a useful check that the gain is not tied to one arbitrary latent-code construction.

Critical weaknesses

My main concern is that the mechanistic explanation is stronger than the evidence. Proposition 1 assumes perfect mode coupling, yet the deployed code predictor agrees with video-derived labels only 73.1% of the time (Appendix E). There is no source-to-target distance analysis under predicted codes, especially wrong or uncertain ones. The 67% kinetic-energy reduction may follow simply from putting the source closer to targets; it does not show that the learned conditional field is less entangled. The idealized proposition therefore does not establish what happens in the deployed policy.

A second missing piece is a single learned Gaussian source. The main FM baseline starts from a fixed standard Gaussian, while LAFM learns latent-conditioned sources. A learned but unconditioned source would tell us how much of the gain comes from adapting the source distribution at all, versus splitting it into motion modes. This is a fairly direct control for the paper's central design choice, not just an extra ablation for completeness.

The third concern is how much to conclude from the broader comparisons. The physical result uses 15 trials per task, with no intervals or paired-reset breakdown (Table 1). Against π0, Appendix I uses batch sizes of 448 for LAFM/FM and 8 for π0 over the same number of steps, so the amount of target-task data seen is not comparable. I find the FM comparison persuasive, but I would not use these experiments to claim a robust real-world margin or a general advantage over π0.

Questions for authors
  1. Can you report transport distance and success split by correct versus incorrect predicted latent codes? I want to know whether the claimed advantage survives the 26.9% of cases where the classifier disagrees with its video label.
  2. What happens with one learned Gaussian source? This is the most direct missing control for the claim that mode-specific sources matter.
  3. Were real-robot initial conditions paired across FM and LAFM, and can you provide per-task counts or intervals?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XPLW (Part 1)

We thank the reviewer for the careful and constructive review, and for three precise questions. We answer each below.

1. Mechanistic explanation and mode coupling

Transport under wrong codes. We split the held-out LIBERO-90 time steps by whether the predicted code matches the LAM-derived one, and computed the kinetic energy of the learned flow on each subset:

SubsetShare of stepsKinetic energy, FMKinetic energy, LAFM
All steps100%136.0844.97
Codes agree73.1%135.2143.13
Top-1 errors26.9%137.3550.47
Top-5 errors2.6%138.5060.62

The advantage survives mispredictions. Kinetic energy rises from 44.97 to 50.4 when the predicted code is wrong, and to 60.6 in the rare cases where the label is not even among the five most likely codes. This remains less than half the kinetic energy of FM for the corresponding sub-sets.

Imperfect coupling in the analysis. We agree that the original proposition only covered the perfectly coupled case. In the revision, Proposition 1 states the reduction as the sum of a mean-alignment and a variance-reduction contribution, and Appendix K now treats imperfect coupling explicitly. When the selected mode jj differs from the action mode kk, the cost reduction over the isotropic source becomes Δmean+Δvar−P\Delta_{\text{mean}} + \Delta_{\text{var}} - P, where the penalty PP depends only on how much farther the selected center μj\boldsymbol{\mu}_j is from the action mode than μk\boldsymbol{\mu}_k, and on the difference between the two source variances. The reduction therefore persists whenever this penalty is smaller than the gain under perfect coupling.

Why the penalty is small in practice. This matches the measurements above:

  • Wrong codes are mostly nearby codes, so the selected center is only slightly farther. Top-5 accuracy is 97.4%, and the KL divergence between the selected and the labeled prior is on average 9.3 times smaller than between a random pair of priors (Appendix E).
  • The variance contribution is kept under any selection. With a wrong code it becomes d−Tr(Σj)d - \text{Tr}(\boldsymbol{\Sigma}_j), which stays large because every prior in the library has a much smaller variance than the isotropic source (0.13 per dimension on average, Section 3.3).

The priors are also trained with the policy's own predictions rather than with the LAM labels (no teacher forcing, Appendix E), so they adapt to the partition the classifier actually induces. Consistently, selecting the prior with the LAM-derived code instead of the predicted one does not reduce the action error (0.00735 against 0.00719, Appendix E).

Success split. A closed-loop success split is not measurable: the LAM-derived code requires the future frame, which only exists after the policy has acted using the very prior under study. We therefore report the open-loop split above.

AuthorsOpenReview ↗

Response to Reviewer XPLW (Part 2)

2. Single learned Gaussian source

We agree that this is the most direct control, and have added it as FM + LP (Table 3, Section 3.2). It replaces the fixed Gaussian with a single learned Gaussian, trained jointly with the policy under the same KL regularizer and weight as LAFM. On LIBERO-90:

PolicySuccess rate (%)
FM82.6 ± 0.2
FM + LP (single learned prior)84.3 ± 0.2
FM + LAC (latent action conditioning)85.4 ± 0.1
LAFM93.0 ± 0.2

Adapting the source without partitioning it recovers 1.7 of the 10.4 points. The gain therefore comes mainly from splitting the source into motion modes. In addition, the single learned prior is only stable with the KL regularizer and collapses without it (Section 3.2), whereas LAFM still reaches 91.7% with λ2=0\lambda_2 = 0 (Appendix D).

3. Real-world evaluation and comparison with π0\pi_0

Pairing. Initial conditions were not paired, since it is hard to guarantee the exact same conditions for several policies across several trials. Object positions were randomized independently for every trial, while the number of objects per episode was matched across models (Appendix J). Nevertheless, the positions come from the same distribution for all evaluated policies, which follows the same distribution of the training data, meaning that the objects are scatered randomly in the same region of the table, to guarantee a fair comparison.

Counts and intervals. Successes out of 15 trials per task, with 95% Wilson intervals on the pooled success rate:

PolicyPlatesDrawerCansBowlsTotalSuccess rate (95% CI)
π0\pi_0139101143/6071.7% [59.2, 81.5]
ACT629724/6040.0% [28.6, 52.6]
FM117101038/6063.3% [50.7, 74.4]
LAFM1512131252/6086.7% [75.8, 93.1]

LAFM succeeds at least as often as FM on every task. Pooled over tasks, the difference between LAFM and FM is significant (Fisher's exact test, p=0.006p = 0.006). We agree that 15 trials per task do not support a precise estimate of the margin on any single task: the 95% interval on the pooled difference is [7.9, 37.5] points.

Comparison with π0\pi_0. We do not claim an advantage over π0\pi_0: the difference in Table 1 is not significant (p=0.07p = 0.07). π0\pi_0 is included as a reference point, to show that a 0.11B policy trained from scratch can reach results comparable to those of a much larger pretrained VLA.

We agree that the two models do not see the same amount of target-task data. We believe, however, that no protocol makes this comparison strictly fair: π0\pi_0 is a 3.3B model with large-scale robot pretraining (700k steps) that is only fine-tuned, while LAFM is trained from scratch. We therefore chose to match the compute budget. Every model is trained on a single GPU for the same number of gradient steps, with the largest batch size that fits in memory (Appendix I.3). Matching the batch size instead would require several GPUs in parallel for π0\pi_0 alone. This budget is also generous for fine-tuning: 200k steps is well above the roughly 50k steps commonly used to fine-tune a VLA.

Our claims rest on the comparison with FM, which shares the architecture, hyperparameters, batch size, and number of steps of LAFM. If the reviewer believes it would add value to the paper, we are glad to run additional real-world evaluations for π0\pi_0 and LAFM.

Summary of revisions

  • added FM + LP (single learned prior) to Table 3 and Section 3.2;
  • restated Proposition 1 as a mean-alignment and a variance-reduction contribution, and extended Appendix K to imperfect mode coupling;
  • added empirical values to support Proposition 1 in Section 3.3;
  • minor adjustments to the text.

We hope these clarifications address the reviewer's concerns and, we are glad to provide any further analysis.

Thanks to the author for adding the experiments and explanations. The new single-Gaussian prior control experiment answered one of my main questions, the analysis of prediction encoding errors, and the number of successes and confidence intervals in real robot experiments, which also makes the results more convincing. However, taking the author's reply and the opinions of other reviewers into account, I still have some concerns. The current experiments do a good job of showing LAFM's improvements over standard FM, but compared to existing structured prior methods, its unique advantages aren't very clear: there are differences in model and training settings when compared with Cocos and COT-FM, and results for BRIDGER and A2A haven't been provided yet. Also, the K-means version has already achieved quite good results, so it still needs to be explained more fully what irreplaceable benefits the added trained latent action model actually brings, and whether it's worth adding this complexity. It's valuable that the same LAM can be used for different robots, but this alone doesn't fully answer the question. Regarding mechanism explanation, it's a positive result that there's still low kinetic energy when prediction coding goes wrong, but this might mostly come from shorter transmission distances or reduced source distribution variance, and doesn't directly indicate that the entanglement of the vector field is reduced. It's also unclear whether this advantage translates to higher success rates in closed-loop tasks. Looking forward to further baseline comparisons and ablation experiments on learning mean and variance separately to clarify these points. Overall, I acknowledge the additional work by the author, but I still have reservations about exactly where the performance improvements come from and which explanations are fully supported by experiments.

We thank the reviewer for the follow-up and are glad the single learned prior, the coding-error analysis, and the real-robot statistics were helpful. The remaining points are addressed below, mostly with results that have since been completed.

Comparison with structured-prior methods. In addition to adding BRIDGeR and A2A as baselines to Section 3.1.2, we have also reimplemented both methods in our architecture, with the same encoder, decoder, hyperparameters, and training budget (Appendix G). The results on LIBERO-90 are given below and discussed on the paper.

PolicySourceSuccess Rate (%)
FMFixed Isotropic Gaussian82.6 ± 0.2
FM + BRIDGeRCVAE Source Policy87.4 ± 0.1
FM + A2AProprioceptive History Embedding84.5 ± 0.5
LAFMLatent Action Conditioned Priors93.0 ± 0.2

K-means and the value of the LAM. We refer to our response to Reviewer mX8F: the K-means prior is an instance of our framework rather than a competitor, the LAM partition is its strongest instantiation, and its performance gain is not negligible (89.5% to 93.0%).

Entanglement. We agree with the reviewer. Proposition 1 and our measurements support shorter transport, not reduced entanglement, so we have removed the entanglement claim from the paper.

Closed-loop success. The closed-loop benefit is what our main results measure: +10.4 points on LIBERO-90, +12.3 on RoboTwin, +15.1 on LIBERO-Plus, and +23.4 points on the real robot. As explained in our earlier response, a closed-loop split by code correctness is not measurable, since the reference code requires the future frame produced by the policy's own action, but we have shown across several different settings that LAFM do achieve a higher success rate than its standard Flow Matching counterpart.

Mean and variance. Both ablations are complete (Appendix H, see details in our response to Reviewer p3JE). Learning only the means reaches 87.7%, only the variances 88.3%, and both 93.0%. Each contribution helps on its own, and the two are complementary.

We thank the reviewer again for the constructive discussion.

Reviewer_mX8FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Action Guided Flow Matching (LAFM), which replaces the standard Gaussian source distribution in flow-matching policies with a set of learnable Gaussian priors. The authors first train a latent action model (LAM) to extract discrete motion representations from videos. During policy training, a classifier predicts the latent action from the current observation and selects the corresponding Gaussian prior to initialize the flow-matching process. The idea is to provide a more informative starting distribution and simplify action generation.

Critical strengths
  1. The paper is easy to follow, with clear and well-designed figures that make the method easy to understand.

  2. Using a more informative, condition-dependent source distribution to guide flow matching is a reasonable idea. The motivation makes sense, and the experimental results show that it can be effective.

  3. I appreciate that the authors provide a theoretical argument for why the proposed priors can reduce transport cost, rather than relying only on intuition. The experiments also support the effectiveness of LAFM, making the overall argument more convincing.

Critical weaknesses
  1. The novelty is relatively limited, and the paper misses several closely related works. Using a condition-dependent or more informative source distribution for generative robot policies has already been studied extensively. For example, BRIDGER (RSS 2024, https://arxiv.org/abs/2402.16075)) uses informative source policies instead of Gaussian noise; Cocos (NeurIPS 2025, https://arxiv.org/abs/2505.11123) directly introduces condition-dependent Gaussian sources for flow matching; and Action-to-Action Flow Matching (2026, https://arxiv.org/abs/2602.07322) uses previous actions to initialize the flow. More recently, The Gaussian Is Enough (2026, https://arxiv.org/abs/2609.27070) systematically compares different source priors, including BRIDGER and Cocos, and finds that their benefits are limited when fine-tuning pretrained robot policies. LAFM is closely related to these approaches. However, the paper neither discusses these works nor compares against them. While LAFM achieves good results, I find it difficult to identify its actual novelty or contribution without these comparisons.

  2. I am not convinced by the paper's main motivation regarding multimodal action distributions. The authors argue that starting from a single isotropic Gaussian is problematic because it cannot capture the multimodal structure of robotic actions. However, the ability to model complex and multimodal distributions is precisely one of the main reasons for using flow matching in the first place. A standard Gaussian source does not prevent the learned flow from generating multimodal outputs. In fact, the authors' own Figure 5 shows that standard flow matching can transport Gaussian noise into different action modes. I think the authors need to explain more clearly why a structured source distribution is necessary, rather than simply making the transport process easier to learn.

  3. The practical value of LAFM is not sufficiently demonstrated. The real-world experiments consist of four task-specific manipulation settings, with policies trained on demonstrations of the same tasks. These experiments show that LAFM works, but tell us little about its generalization to new tasks or environments. More importantly, Figure 7 shows that LAFM achieves only a 3.5-percentage-point improvement over a much simpler K-means-based prior (93.0% vs. 89.5%). K-means requires very little computation, while LAFM requires training an additional latent action model, which takes approximately 40 hours on an H100 according to Appendix G. Given this additional complexity and training cost, I am not sure why practitioners would choose LAFM over a simple K-means prior. I would like to see stronger evidence that the latent action model provides enough additional value to justify its cost.

Questions for authors

Please see the weaknesses above. My main suggestion is to include comparisons with closely related works, particularly existing condition-dependent source distribution methods. I think this is necessary to clarify what LAFM contributes beyond previous approaches and why its latent-action-based prior is a better choice in practice.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mX8F (Part 1)

We thank the reviewer for their careful reading and for recognizing the clarity of the paper, the theoretical grounding of our transport argument, and the effectiveness of LAFM. We address the three concerns below.

1. Novelty and comparison with condition-dependent source methods

We thank the reviewer for these references. What distinguishes LAFM from them is the combination of three properties: the source is a discrete library of priors indexed by motion primitives, each prior has a learned mean and learned per-dimension variances, and the prior is selected from the current observation and trained jointly with the policy from scratch, with no external source policy, pretrained flow, or RL stage.

Two ablations on LIBERO-90 (Table 3, Section 3.2) show that these choices, and not source adaptation or latent action supervision alone, drive the gain:

  • FM + LP, a new ablation added in the revision (requested by Reviewer XPLW), replaces the fixed Gaussian with a single learned Gaussian, under the same KL regularizer. It reaches 84.3%, against 82.6% for FM and 93.0% for LAFM. Adapting the source without partitioning it recovers only 1.7 of the 10.4 points.
  • FM + LAC keeps the standard Gaussian and gives the same predicted latent action to the decoder as conditioning. It reaches 85.4%. The gain therefore comes from structuring the source, not from the auxiliary supervision.

We have added a dedicated subsection to the related work (Section 4.2) covering the works below. In short:

  • BRIDGER transports actions from a fixed, externally provided source policy (a heuristic or a separately trained CVAE) using stochastic interpolants. LAFM needs no external source policy: its prior library is learned jointly with the policy.
  • Cocos centers a Gaussian source on a compressed vision-language embedding, with fixed variance, to prevent the policy from ignoring hard-to-distinguish conditions. LAFM priors are grounded in motion rather than in observation semantics, form a discrete library, and have learned per-dimension variances that capture the heteroscedasticity of demonstrations (Appendix C.1).
  • A2A replaces the noise with a latent of the proprioceptive history, which makes the source essentially deterministic and tied to the ongoing motion. LAFM keeps a stochastic source and can switch priors when the motion changes, since the latent action is predicted from the visual observation.

Comparison with Cocos. Cocos reports results on the four standard LIBERO suites, now included in Table 2. LAFM, a single model trained jointly on the four suites, outperforms the four suite-specific Cocos models by 1.7 points on average (96.5% vs. 94.8%). The gains are on Goal (+3.4) and Long (+3.6); Object is at parity (+0.1) and Cocos is slightly ahead on Spatial (97.5% vs. 97.2%).

BRIDGER and A2A. Neither reports results on our benchmarks, so we are training both on LIBERO and LIBERO-90 from the original code. We will post these results as soon as the runs finish.

The Gaussian Is Enough studies a different regime from ours. Its negative result concerns swapping the source of a policy pretrained under N(0,I)\mathcal{N}(0, I) at fine-tuning time (their Section VI). We would not expect LAFM to help there either: the pretrained vector field is coupled to the isotropic source, and a pretrained model has already paid the cost of learning the entangled field. LAFM is designed and evaluated for training from scratch, where the authors notes that learned priors may remain valuable (their Section VIII-A). They do not test structured priors used throughout pretraining and kept during fine-tuning, which we identify as future work.

AuthorsOpenReview ↗

Response to Reviewer mX8F (Part 2)

2. Motivation

We agree with the reviewer: flow matching can represent multimodal action distributions from a single Gaussian source, and Figure 5 shows it. Our claim concerns how easily the policy is learned, not what it can represent. We have reworded the introduction to make this unambiguous.

An isotropic source gives every action mode the same location and the same amount of noise. The vector field alone must then route nearby noise samples toward distant modes, and contract the noise for consistent motions while spreading it for variable ones. LAFM provides this structure as an inductive bias, and the learned priors reflect it: on held-out LIBERO-90 data, the selected prior mean is closer to the mode mean than the origin for 95.1% of time steps, the average source variance per dimension is 0.13 (values added per request of reviewer p3JE to Section 3.3), and larger-range movements receive larger variances (Appendix C.1).

This easier transport is measurable in the figure the reviewer cites. On the same batch, FM reaches the modes with a kinetic energy of 139.7 and an L1 error of 0.54, against 44.9 and 0.45 for LAFM. Over the validation set, kinetic energy drops by 67%. LAFM matches the final error of FM after 135k of 200k steps and ends 20% below it. At equal training budget, this translates into +10.4 points on LIBERO-90.

3. Practical value and generalization

K-means. The K-means prior in Appendix F is not a baseline from prior work: it is our multi-prior framework with a different source of mode labels. Its +6.9 points over FM (89.5% vs. 82.6%) is therefore evidence for the core contribution, and practitioners who prefer it can use it within LAFM. The LAM partition is the strongest instantiation we tested. Going from 89.5% to 93.0% removes a third of the remaining failures (10.5% to 7.0%), on a benchmark where fine-tuned π0\pi_0 reaches 92.1%.

The margin depends on the LAM training data, as Appendix H reports: across six LAM training sets, LAFM ranges from 88.1% to 93.0%, always above FM, and above K-means for Fractal, OxE, and OxE + Ego4D.

Cost. The LAM is trained once (about 40 hours on one GPU) and reused unchanged for every experiment in the paper. We will release its weights with our code, so using LAFM requires no LAM training.

Why latent actions. K-means partitions live in the action space of one robot: they need action labels and cannot be shared across embodiments. Latent actions are inferred from visual dynamics alone. Our experiments already rely on this: a single LAM trained on Fractal, a different robot and visual domain, labels LIBERO, bimanual RoboTwin 2.0, and our real-world Franka data. This is what makes the approach relevant for multi-embodiment pretraining, which we identify as future work.

Generalization. We follow the standard protocol in which policies are trained on demonstrations of the target tasks, as are all baselines including fine-tuned π0\pi_0. Our contribution concerns the generative process of the policy and is orthogonal to the data and pretraining that drive zero-shot generalization. Within this protocol:

  • LIBERO-Plus (Appendix A) applies perturbations unseen during training: camera viewpoint, lighting, background, sensor noise, object layout, robot initial state, and language. LAFM improves over FM by 15.1 points (65.5% vs. 50.4%).
  • RoboTwin 2.0 applies strong domain randomization to 50 bimanual tasks. LAFM improves over FM by 12.3 points.
  • In the real world, with object positions randomized in every trial, LAFM trained from scratch (0.11B parameters) reaches 86.7% success, against 63.3% for FM and it is competitive with fine-tuned π0\pi_0 (3.3B).

Summary of revisions

  • added discussion on BRIDGER, Cocos, and A2A to the related work (Section 4.2);
  • added Cocos to Table 2;
  • added the FM + LP ablation (single learned prior) to Table 3 and Section 3.2;
  • reworded the introduction to make clear that our motivation concerns learning difficulty, not expressivity.

Results for BRIDGER and A2A will be posted as soon as they are available. We hope these clarifications address the reviewer's concerns about the novelty and practical value of LAFM, and we are glad to provide any further analysis.

As promised in our rebuttal, we have added BRIDGeR and A2A as baselines, trained from their original implementations with language conditioning. They are now included in Tables 2 and 3:

PolicyLIBERO (avg. of 4 suites)LIBERO-90
BRIDGeR76.8 ± 0.883.8 ± 0.6
A2A89.2 ± 0.160.3 ± 0.1
FM95.4 ± 0.382.6 ± 0.2
LAFM96.5 ± 0.393.0 ± 0.2

To separate the effect of the source distribution from that of the architecture, we also reimplemented both methods in our own architecture, as an additional ablation (Appendix G, see our response to Reviewer XPLW). LAFM outperforms both methods in either setting.

We hope these results address the reviewer's remaining concern about the comparison with condition-dependent source methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions made during the discussion period

We thank all reviewers for their constructive feedback, which has substantially improved the paper. We summarize below the changes made during the discussion period.

New experiments

  • Comparison with condition-dependent source methods. BRIDGeR and A2A were added as baselines on LIBERO and LIBERO-90 (Tables 2 and 3, Section 3.1.2), and Cocos on LIBERO (Table 2). (mX8F, p3JE)
  • Controlled comparison of source distributions (Appendix G). BRIDGeR and A2A were reimplemented in our architecture, so that only the source distribution and generative objective differ from FM. (mX8F, XPLW, p3JE)
  • Single learned prior (FM + LP, Table 3, Section 3.2). A single Gaussian source learned jointly with the policy, isolating the effect of partitioning the source into motion modes. (XPLW)
  • Mean-only and variance-only priors (Appendix H). Separates the contributions of mode alignment and heteroscedasticity. (p3JE)

Theory

  • Proposition 1 now expresses the reduction in transport cost as the sum of a mean-alignment and a variance-reduction contribution, and no longer assumes exact mean alignment. Section 3.3 verifies both conditions on the learned priors. (p3JE)
  • Appendix K extends the analysis to imperfect mode coupling, when the selected prior differs from the action mode. (XPLW)

Positioning and claims

  • Related work. Section 4.2 now discusses BRIDGeR, Cocos, A2A, CSFM, COT-FM, CAR-Flow, and VCT. (mX8F, p3JE)
  • Motivation. We have reworded the introduction to make unambiguous that our claim concerns how easily the policy is learned, not what a single Gaussian source can represent. (mX8F)
  • Entanglement. We removed the claim that LAFM yields less entangled vector fields, and now claim only shorter transport, which our analysis and measurements support. (XPLW)
  • Comparison with π₀. We have updated the text to make clear that we do not claim superiority over π₀, but rather that we achieve comparable results with a much smaller model without pre-training. (XPLW)

Reproducibility

  • Baseline adaptation. Appendix K describes how BRIDGeR and A2A were implemented and give the hyperparameters used. (mX8F)

We thank the reviewers again for their time and engagement throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.