acceptodds
Under review as a conference paper at ICLR 2027

MOSS: Mode-Oriented Source Shaping for Robot Flow Matching Policies

Abstract

In robotic manipulation, the same observation often admits multiple feasible actions. Generative policies such as diffusion and flow matching naturally capture such multimodal behaviors, yet which mode is executed is left to random sampling and cannot be actively controlled. We propose Mode-Oriented Source Shaping (MOSS), which turns mode selection into an explicit choice of where generation starts. We first reveal, both empirically and theoretically, that deterministic flow transport induces mode-dependent structure in the source space. Under ideal deterministic ODE transport, distinct behavioral modes have distinct source-space preimages, while practical approximation errors can blur their boundaries due to imbalanced data, limited model capacity, and few inference steps. Motivated by this source-space structure, MOSS constructs semantically aligned mode-specific Gaussian sources that encode mode identity in the initial noise. A modality-strength estimator predicts the local degree of multimodality, which adaptively controls the geometry of the mode-specific sources. It contracts them toward a shared Gaussian where behaviors are similar and separates them near behavioral branching points. During deployment, the desired behavior is selected at inference simply by sampling from the corresponding component. Across simulated and real-world manipulation tasks, MOSS steers to the target mode up to % of the time with high task success, recovers minority modes at over % guidance rate under a imbalance dataset, and surpasses mode-conditioned flow policy by points of guidance rate on real robots.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot learning, imitation learning, flow matching, generative policies, multimodal behavior

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qSfbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOSS (Mode-Oriented Source Shaping), a framework for controllable multimodal behavior generation in flow-matching-based robotic policies.

The key idea is to control the generated behavior by modifying the initial source distribution rather than explicitly conditioning the velocity field on a desired behavior mode. The authors first analyze the implicit mode-dependent partitioning of the source space under deterministic flow transport and discuss how practical approximation errors can weaken mode separation.

Based on this observation, MOSS constructs mode-specific Gaussian source distributions whose centers are aligned with semantic differences between behavioral modes. A modality-strength estimator predicts the local degree of behavioral multimodality and adaptively adjusts the separation and covariance of these distributions. A single shared flow-matching network is then trained to generate actions from the corresponding sources.

The method is evaluated on six simulated and three real-world robotic manipulation tasks. The experiments demonstrate improved mode controllability, particularly under imbalanced demonstrations, while maintaining competitive task success rates.

Critical strengths
  1. Effective and practical multimodal behavior control.

The paper addresses an important problem in generative robotic policies: reliably selecting a desired behavioral mode instead of relying on stochastic sampling. MOSS provides a relatively simple solution by encoding mode information into the initial source distribution while retaining a shared velocity field. This formulation avoids introducing an additional mode condition directly into the flow network and provides a useful alternative to conventional conditional generation.

  1. Encouraging experimental results across simulation and real-world tasks.

The paper evaluates the proposed method on six simulation tasks and three real-world robotic manipulation tasks. The real-world experiments show substantial improvements in average guidance rate over mode-conditioned FM (84.2% vs. 40.8%), together with higher average task success (85.3% vs. 73.3%). The experiments on imbalanced demonstrations are also encouraging, suggesting that the shared flow network can leverage common motion structures to improve minority-mode learning. Furthermore, the ablation results show that adaptive source shaping consistently improves controllability over fixed source separation on the evaluated simulation tasks.

Critical weaknesses
  1. Limited methodological novelty relative to existing source-shaping approaches.

My primary concern is the novelty of the proposed framework. Controlling generative behavior by manipulating the initial noise or source distribution is not a fundamentally new idea and has been explored in prior diffusion- and flow-based generative models. In the context of flow matching, related approaches such as SL-FM, LAFM, and Gaussian-mixture-based source modeling have already investigated structured source distributions for multimodal generation.

The main methodological distinction of MOSS appears to be the combination of semantically aligned Gaussian components and modality-dependent adaptive separation. While this is a reasonable design, the underlying idea remains closely related to existing source-space control methods. The paper would benefit from a more convincing demonstration of what cannot be achieved by simpler existing source-shaping strategies.

I also find the theoretical contribution somewhat limited. Theorem 1 largely follows from the properties of preimages under deterministic mappings and the exact pushforward condition. Theorem 2 establishes local separation under additional regularity assumptions. However, these results do not directly justify the particular Gaussian parameterization or the adaptive separation schedule proposed in MOSS. Consequently, the connection between the theoretical analysis and the specific methodological contribution is not sufficiently strong.

  1. Insufficient evidence isolating the contribution of adaptive source shaping.

Although the overall experimental results are promising, it remains unclear how much of the improvement comes from the proposed modality-adaptive mechanism rather than simply using mode-specific Gaussian sources.

The ablation experiments in Table 3 show that fixed mode-specific Gaussian separation already achieves relatively strong controllability, and adaptive shaping provides an additional improvement. However, these comparisons are limited to three simulation tasks. The real-world experiments and imbalanced-data evaluations do not include the corresponding fixed-separation baseline, making it difficult to determine whether the most significant reported improvements genuinely require the adaptive component.

Furthermore, the GMM-sourced baseline does not explicitly incorporate semantic mode supervision, whereas MOSS uses known mode labels to construct its sources. This difference in supervision complicates the interpretation of the comparison. A more direct comparison with semantically aligned but non-adaptive source distributions, using the same supervision, would better establish the incremental contribution.

Finally, the real-world evaluation contains only 20 trials per requested mode, without reported confidence intervals or repeated training-seed evaluations. Additional statistical evidence would help establish the robustness of the reported improvements.

Questions for authors
  1. Novelty relative to existing source-shaping methods.

Could the authors clarify the fundamental methodological difference between MOSS and existing approaches that control flow matching through structured or learned source distributions, particularly SL-FM?

Beyond explicitly associating Gaussian components with semantic mode labels, what is the main new technical mechanism that distinguishes MOSS from a supervised mixture-of-Gaussians source model?

A more direct comparison with existing source-shaping methods under matched supervision would significantly strengthen the novelty claim.

  1. Contribution of adaptive separation.

Table 3 shows that fixed mode-specific Gaussian separation already provides strong controllability. Could the authors extend this comparison to the three real-world tasks and the 10:90 imbalanced-data experiments?

In particular, how much of the reported improvement over mode-conditioned FM comes from semantic source separation itself, and how much comes specifically from the modality-strength estimator and adaptive source geometry?

It would also be informative to compare the proposed action-aligned Gaussian centers with simpler alternatives, such as fixed or randomly assigned separated Gaussian centers, to clarify the importance of the proposed geometric construction.

  1. Theoretical justification.

The theoretical analysis establishes that deterministic flow transport induces mode-dependent source-space preimages. However, it is less clear how these results motivate the specific Gaussian source parameterization in Section 4.

Could the authors clarify whether the theory provides any guarantee or quantitative insight into the effectiveness of the proposed adaptive separation strategy, beyond establishing the existence of mode-dependent source regions?

In particular, how does the theoretical analysis inform the choice of source-center displacement and covariance along the mode-discriminative subspace?

  1. Robustness and baseline comparisons.

The real-world results show substantial differences between MOSS and mode-conditioned FM, but the latter exhibits severe mode collapse in some tasks. How sensitive are these results to the conditioning implementation, training balance, and classifier-free guidance scale?

Although Appendix F explores stronger guidance, a more systematic comparison and statistical uncertainty estimates would help determine whether the observed advantage is robust across different baseline configurations and training runs.

Additionally, how sensitive is MOSS to inaccuracies in the estimated modality strength, especially when the local neighborhood statistics become unreliable under limited or imbalanced demonstrations?

9 comments
AuthorsOpenReview ↗

Response to Reviewer qSfb (1/5)

We sincerely thank reviewer qSfb for the thoughtful and constructive feedback, as well as for recognizing the practical value of MOSS in multimodal behavior control and real-robot tasks. Your insightful suggestions on methodological distinctions, supervision, and adaptive gains have been invaluable in helping us strengthen our work. Following your advice, we have carefully conducted additional experiments, including comparisons of source designs, fixed-separation experiments under 10:90 data imbalance, SL-FM semantic calibration, conditioning variants, and CFG sweeps. We sincerely appreciate your time and consideration, and hope that our revisions adequately address your concerns. Our detailed responses are provided below.

Q1: What distinguishes MOSS technically from SL-FM, LAFM, and existing structured-source methods? Beyond associating semantic labels with Gaussian components, how does it differ from a supervised Gaussian-mixture source?

Answer: We sincerely appreciate the your suggestion to better distinguish our work from prior studies. We fully acknowledge that structured source distributions have been explored in FM, Rectified Flow, and optimal transport research [3–5]. However, MOSS directly associates user-specified behavioral modes with source components and adapts the source distribution to the degree of behavioral branching for the first time.

MOSS comprises three designs:

  1. Semantic Pairing: Each demonstration's mode label determines its Gaussian component during training. Users select the same component to request the mode at inference, providing an explicit control interface.
  2. Action Alignment: Differences between modes' mean action chunks at branching determine source-center displacement directions and a low-rank subspace, aligning separation with actual action differences.
  3. Adaptive Separation: Local modality strength adjusts center distances and covariance, reducing separation during shared motion and increasing it at branching.

Comparison with SL-FM: SL-FM constructs intervenable source handles through orthogonal source lifting, but its base training does not directly associate them with user-specified semantics. Section 5.4 uses CEM search and a PPO high-level selector for target-route control[1]. MOSS establishes mode–component correspondence during training, enabling direct mode selection without additional search or selector training.

Comparison with LAFM: LAFM learns motion primitives through a latent action model and selects learned source priors from observations, aiming to better match sources to action structure[2]. MOSS addresses user-specified multimodal behavior control: modes can be actively selected at inference, while separation adapts to task progress. The objectives and source designs therefore differ.

Comparison with ordinary Gaussian-mixture sources: An ordinary GMM source could not necessarily establish stable component–semantics correspondence. We observed, for example, one component strongly favoring one behavior while another still generated both behaviors randomly. MOSS establishes supervised mode–Gaussian correspondence, organizes source geometry using action differences, and adapts separation to local multimodality. Its design therefore extends beyond attaching semantic labels to components.

Following your advice, we also added a direct SL-FM comparison. After training, we fixed each of source handles and ran 20 rollouts per handle to establish semantic correspondence. We then froze the mapping and evaluated each target mode for 50 trials:

TaskMode 1 GR / SRMode 2 GR / SRAvg. GR / SR
Push Cube74% / 74%92% / 92%83% / 83%
Grasp Eyeglasses98% / 100%62% / 98%80% / 99%
Stack Cube90% / 100%34% / 98%62% / 99%
Put Wine46% / 88%42% / 74%44% / 81%

SL-FM achieves high SR on some tasks but shows substantial per-mode GR differences, particularly 34% for Stack Cube Mode 2. MOSS provides more consistent control of user-requested modes.

To distinguish the contributions, we also added fixed-source and random-center ablations, both retaining semantic pairing, to test adaptive separation and action alignment respectively. Results appear in Q2 and Q3.

Revisions: Section 2 - distinctions from related methods; Table 1 and Appendix E.2 - SL-FM implementation and results.

AuthorsOpenReview ↗

Response to Reviewer qSfb (2/5)

Q2: How much does adaptation improve over fixed semantic Gaussian sources? Can you extend the comparison to 10:90 data and three real-robot tasks?

Answer: Thank you for this suggestion. Original Table 3 compares fixed semantic Gaussian sources with adaptive MOSS on three balanced simulation tasks; Appendices H.1–H.3 provide training curves and source–action displacement statistics. Following your advice, we added fixed-source ablations for 10:90 data and three real-robot tasks.

1. 10:90 imbalance

Mode 1 is the minority (10%) and Mode 2 the majority (90%). Fixed-separation results are:

TaskMode 1 GR / SRMode 2 GR / SR
Push Cube44% / 84%98% / 100%
Grasp Eyeglasses56% / 100%98% / 98%
Stack Cube70% / 96%98% / 98%

Fixed separation reaches only 44%–70% GR for minority modes. Adaptive MOSS provides better minority-mode control in the original 10:90 experiments, indicating that semantic pairing alone does not resolve imbalance and adaptive separation further improves minority-mode generation.

2. Real-robot fixed-source ablation

we have included additional fixed-separation experiments conducted previously but not reported in the original manuscript:

TaskMethodSRGR1GR2
Place CupsFixed Separation0.980.851.00
MOSS0.980.951.00
Pick VegFixed Separation0.700.600.50
MOSS0.750.800.65
Open DrawerFixed Separation0.800.600.70
MOSS0.830.900.75

MOSS improves GR in five of six modes and ties in the remaining one, showing that adaptation improves real-robot control even with the same semantic pairing. Balanced simulation, 10:90 experiments, and real-robot ablations all support adaptive separation.

Unfortunately, the original platform is no longer available for further tests; we have fully reported the existing results and evaluation settings in the revision.

Revisions: Table 3 and Appendix G - fixed-source ablations.

Q3: What do action-aligned centers contribute? Can fixed or random centers separate the roles of semantic pairing and adaptive geometry?

Answer: Thank you for this suggestion. We agree that the original manuscript did not clearly distinguish these components, and we appreciate the opportunity to clarify their respective roles.

Following your advice, we added random centers: given MOSS's mode axis uu, we sample r∼N(0,Id)r\sim\mathcal N(0,I_d) with a fixed seed and construct a random direction orthogonal to uu. This retains semantic pairing but removes action alignment. Fixed Aligned separates along uu; full MOSS additionally adapts centers and covariance using local modality strength.

Results on three simulation tasks are:

TaskSource DesignMode 1 GR / SRMode 2 GR / SRAvg. GR
Push CubeRandom Centers62% / 100%42% / 96%52%
Fixed Separation86% / 100%85% / 98%85.5%
MOSS99% / 99%100% / 100%99.5%
Grasp EyeglassesRandom Centers50% / 100%38% / 100%44%
Fixed Separation86% / 100%90% / 100%88%
MOSS100% / 100%100% / 100%100%
Stack CubeRandom Centers60% / 96%52% / 94%56%
Fixed Separation78% / 90%88% / 96%83%
MOSS98% / 98%96% / 96%97%

Replacing random centers with fixed aligned sources increases average GR by 33.5, 44, and 27 percentage points; adaptation adds 14, 12, and 14 points, respectively.

Source design affects both initial samples and the interpolation paths and velocity supervision in FM [3]. Overlapping paths with different target velocities can create conflicting supervision for a shared network. Rectified Flow and OT-based FM also show that source–target coupling and transport geometry affect learning [4,5]. MOSS therefore separates centers along action differences to reduce inter-mode velocity conflicts, while adaptation preserves sharing during common motion.

These comparisons distinguish the contributions: semantic pairing enables explicit selection, action alignment organizes source geometry, and adaptive separation further improves mode control.

Revisions: Appendix H - random-center comparison and action-alignment analysis.

AuthorsOpenReview ↗

Response to Reviewer qSfb (3/5)

Q4: How does the theory support Gaussian parameterization, center shifts, and subspace covariance? Does it guarantee or quantitatively explain adaptation?

Answer: We sincerely appreciate reviewer qSfb for raising this important question. We agree that the connection between our theoretical analysis and method design could have been explained more clearly. While the specific design of MOSS is not directly derived from the theory, our analysis provides insights into the relationship between initial noise and final behavioral modes in FM, which serves as the motivation for our method.

Under assumptions including continuity and exact transport, deterministic FM associates different behavioral modes with distinct source-space preimages. Theorem 1 relates these regions to target-mode probabilities; Theorem 2 analyzes mode preservation under source perturbations. Although based on established mathematical tools, these results connect continuity and pushforward properties to behavioral modes in robotic policies, identifying initial noise as an entry point for control.

We first conducted a simple source-shift experiment on 2D Navigation. Keeping standard FM weights fixed, we adjusted only the Gaussian mean using initial-noise statistics for the target mode and drew 100 samples per setting, as shown in Appendix A:

Sampling settingPath 1Path 2Path 3Path 4Stuck
Standard Gaussian141932269
Mean shifted to −0.3030436227821

Shifting the noise mean increases Path 1 occurrences from 14 to 62 without retraining. This shows that the initial source distribution influences final behavioral modes and directly motivates MOSS's adaptive source separation.

Building on this observation, MOSS uses action-aligned centers to encode inter-mode action differences and subspace covariance to control overlap along mode-discriminative directions. Adaptive separation then adjusts this geometry according to local modality strength: sources remain close during shared motions and separate near behavioral branches, as illustrated by Stack Cube.

This design is motivated by the theory and preliminary experiment, but its effectiveness is established empirically rather than theoretically guaranteed. Table 3, the 10:90 imbalance experiments, and real-robot results demonstrate the benefits of adaptive separation.

We sincerely appreciate your constructive suggestion and have made corresponding clarifications in the revised manuscript to better explain this distinction.

Revisions: Section 4.1 - connection between theory and method design.

Reviewer_Yh19OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper wants to let a user pick which behavior mode a flow-matching policy executes (e.g. go left vs. right). Instead of conditioning the velocity network on a mode label, MOSS gives each mode its own Gaussian source and picks the mode by choosing where the noise is sampled from. How far apart these sources are is set by a separate modality-strength estimator (kNN-based targets plus a small MLP): they collapse to a shared Gaussian when the modes behave alike and separate near branching points. There's some theory showing that ideal deterministic FM already splits the source space into mode-wise regions, and that approximation error blurs them. Experiments cover 6 sim tasks and 3 real tasks on a Galaxea R1 Lite, comparing against mode-conditioned FM, a GMM-source baseline, and several unconditional policies. There's also an imbalanced-data (10:90) study and ablations on the source design.

Critical strengths

The idea is simple and nice: steer the mode just by choosing where generation starts, with no change to the velocity network. It also naturally works with one-step / few-step samplers where CFG isn't available. The real-robot results are the most convincing part. Mode-conditioned FM basically ignores the label for one mode on two of the three tasks, while MOSS steers both modes reasonably well.

Critical weaknesses

I'm not sure the main baseline is strong enough. In simulation, mode-conditioned FM is already about as good as MOSS (and better on Put Wine mode 2), and in the 10:90 imbalance study (Fig. 7) it keeps up with MOSS on Grasp Eyeglass and Stack Cube; only Push Cube shows a clear gap. So the real-world gap carries most of the argument. But there it collapses completely on one mode (GR 0.00), with 20 trials per mode and no mention of multiple training seeds. That looks more like an under-tuned conditioning setup than a fundamental limitation. I'd like to see stronger conditioning variants (e.g. FiLM-style injection, label dropout/CFG tuned properly) and a direct comparison to SL-FM, which is the closest prior work, rather than only a generic GMM-source reimplementation.

The "why" isn't really nailed down, and the theory doesn't help much. Shifting the source is still a form of conditioning, since the network sees x_t, which carries the source offset (especially at small t). So "the mode is never given to the policy" is a bit of a semantic distinction. The paper never really explains why this route is more robust than a label, beyond a hand-wavy shortcut argument. Theorems 1-2 are mostly direct consequences of continuity and pushforward, and the global bi-Lipschitz bound needs sigma_min > 0, while the method uses sigma_min = 0 (only a local version is given). Meanwhile the adaptive estimator adds quite a bit of machinery (kNN on ResNet features, calibration, persistence, many hyperparameters) for a 12-14 point GR gain over fixed separation in Table 3; Fig. 8 shows more like 5-13 points depending on task and checkpoint, and Table 3's fixed-separation numbers don't fully match Appendix H.2 (e.g. Push Cube: 0.86/0.85 in Table 3 vs. a 0.90 mean in H.2).

Questions for authors
  1. How was mode-conditioned FM tuned for the real-robot tasks (label embedding, injection point, CFG dropout rate)? Does it still collapse with a stronger conditioning design? This matters a lot for my score.
  2. The mode axis U comes from mean action chunks in one branch-critical region. How does this behave when the mode difference isn't a roughly fixed direction in action space, e.g. Stack Cube, where the right action depends on object positions?
  3. Can you compare against SL-FM directly, or explain why it's not applicable?
  4. Do the real-world numbers hold across more trials or another training seed?
8 comments
AuthorsOpenReview ↗

Response to Reviewer Yh19 (1/5)

We sincerely thank Reviewer Yh19 for the thoughtful and constructive feedback, as well as for recognizing the key idea of mode selection through the initial source. Your insightful comments have been invaluable in helping us improve both the technical rigor and clarity of our work. In response, we have conducted additional conditioning and guidance experiments, included a direct SL-FM comparison, and carefully clarified our methodology and evaluation protocol. We hope these revisions adequately address your concerns and sincerely appreciate your time and consideration.

Q1: How are label representation, injection, label dropout, and CFG configured in real-robot Mode-conditioned FM? Would collapse persist with stronger conditioning?

Answer: Thank you for requesting these details. The original Mode-conditioned FM combines one-hot mode labels with visual and robot-state features into a global condition, which is injected through AdaLN in each DiT block [1]. During training, mode labels are dropped with probability 0.2, while observation conditions remain unchanged. The same architecture and training configuration are used for simulation and real robots, with platform-specific input and action dimensions.

Independent mode AdaLN. To test stronger, separate mode conditioning, we added an independent label MLP that generates additional AdaLN scale/shift parameters while retaining the original observation, time, and residual-gate pathways. Under identical settings (50 trials per mode), we obtained:

TaskConditioningMode 1 GR / SRMode 2 GR / SRAvg. GR / SR
Push CubeOriginal Mode-conditioned FM86% / 96%94% / 100%90% / 98%
Push CubeIndependent mode AdaLN84% / 98%92% / 98%88% / 98%
Grasp EyeglassesOriginal Mode-conditioned FM100% / 100%100% / 100%100% / 100%
Grasp EyeglassesIndependent mode AdaLN100% / 100%98% / 98%99% / 99%
Stack CubeOriginal Mode-conditioned FM96% / 96%94% / 94%95% / 95%
Stack CubeIndependent mode AdaLN88% / 88%90% / 92%89% / 90%

Independent AdaLN does not improve performance over the original conditioning.

CFG sweep. We also evaluated CFG scales 1–4 using the original baseline's fixed checkpoint [2]:

TaskCFG scaleMode 1 GR / SRMode 2 GR / SRAvg. GR / SR
Push Cube186% / 96%94% / 100%90% / 98%
Push Cube290% / 96%94% / 96%92% / 96%
Push Cube390% / 92%92% / 92%91% / 92%
Push Cube454% / 56%92% / 92%73% / 74%
Grasp Eyeglasses1100% / 100%100% / 100%100% / 100%
Grasp Eyeglasses2100% / 100%100% / 100%100% / 100%
Grasp Eyeglasses3100% / 100%100% / 100%100% / 100%
Grasp Eyeglasses4100% / 100%100% / 100%100% / 100%
Stack Cube196% / 96%94% / 94%95% / 95%
Stack Cube296% / 96%92% / 92%94% / 94%
Stack Cube394% / 94%92% / 92%93% / 93%
Stack Cube492% / 92%90% / 90%91% / 91%

Stronger guidance does not consistently improve performance. Push Cube improves slightly at scale 2 but degrades substantially at scale 4, while the other tasks already achieve near-ceiling performance at scale 1. Excessive guidance may move trajectories away from the training distribution, consistent with prior findings [3–5].

Real-robot CFG. We had also previously evaluated CFG on both modes of the real-robot tasks. Table 12 reported the weaker modes; additional scale-3 evaluations for the remaining modes yielded:

TaskModeCFG scaleGR
Place CupsMode 210.85
Place CupsMode 230.90
Pick VegMode 210.60
Pick VegMode 230.60
Open DrawerMode 110.85
Open DrawerMode 130.80

Stronger CFG provides no consistent improvement. Even when the requested mode is followed, grasping or placement failures can still limit GR.

AuthorsOpenReview ↗

Response to Reviewer Yh19 (2/5)

Failure analysis and limitations. Real-robot failures may arise from differences in effective training windows, branching-state coverage, and visual conditions. We observed trajectories switching modes during execution or failing after approaching the target. Thus, stronger conditioning alone may not resolve these failures.

For all methods, checkpoints are selected by validation loss and evaluated with three test seeds, reporting the best result. Unfortunately, the original real-world evaluation platform is no longer available for further deployment or repeated evaluations. We have therefore reported all available CFG results and apologized for this limitation.

Revisions: Appendix E.2 - baseline implementation; Appendix F - CFG experiments.

Q2: The mode axis U is constructed from mean action chunks in branching regions. How does this apply when actions depend on object positions and mode differences have no fixed direction, as in Stack Cube?

Answer: Thank you for raising this question. We apologize that the original manuscript insufficiently explained the construction and role of UU.

UU denotes directions distinguishing modes in the initial source space, rather than fixed directions of physical arm motion. Actual actions are generated by the FM velocity field conditioned on current visual and robot-state observations, allowing adjustment to object positions.

To construct UU, we identify the stage with the strongest mode separation in the training data, compute each mode's mean action chunk, remove the shared mean, and use the remaining differences. For two modes, the direction is the difference between their mean action chunks:

{\|\operatorname{vec}(\bar A_1-\bar A_2)\|_2}.$$ For example, if Stack Cube targets lie at $x=1$ and $x=-1$, their mean action chunks differ mainly along $x$ when the arm must choose left or right, and $U$ mainly reflects this difference. For a training layout with targets at $y=1$ and $y=-1$, recomputing $U$ would instead mainly capture the $y$-direction difference. Thus, $U$ organizes the initial source using the most prominent mode differences; it does not require identical action directions across object positions. Within a evaluation layout, observation-conditioned FM transport determines the specific actions as positions vary. **Revisions:** Section 4.3 and Appendix D.1 - construction of $U$ and the Stack Cube example. **Q3: Can you directly compare against the closest method, SL-FM, beyond a generic GMM-sourced implementation?** **Answer:** We sincerely appreciate this valuable suggestion. Following your constructive advice, we have implemented an additional baseline based on SL-FM [6]. SL-FM handles enable behavioral intervention but are not assigned target semantics during training. The paper selects handles through CEM/PPO for target routes and discusses incorporating semantic information [6]. To evaluate our predefined modes, we performed post-training handle semantic calibration: we fixed each of handles, ran 20 rollouts per handle, and counted the modes of successful behaviors. We selected handles for the two target modes, froze the mapping, and evaluated each requested mode for 50 trials. Calibration establishes the semantic correspondence absent from SL-FM's original handles. Results are: | Task | Mode 1 GR / SR | Mode 2 GR / SR | Avg. GR / SR | |---|---:|---:|---:| | Push Cube | 74% / 74% | 92% / 92% | 83% / 83% | | Grasp Eyeglasses | 98% / 100% | 62% / 98% | 80% / 99% | | Stack Cube | 90% / 100% | 34% / 98% | 62% / 99% | | Put Wine | 46% / 88% | 42% / 74% | 44% / 81% | Overall, SL-FM achieves high task success on several tasks but shows weaker control over the requested semantic modes, particularly on Stack Cube and Put Wine. These results highlight MOSS's advantage in reliably aligning source components with behavioral modes. **Revisions:** Table 1 - SL-FM results; Appendix E.2 - SL-FM implementation settings.
AuthorsOpenReview ↗

Response to Reviewer Yh19 (3/5)

Q4: Can real-robot results be validated with more trials or another training seed?

Answer: We sincerely appreciate the reviewer’s thoughtful concern regarding the statistical reliability of our real-robot evaluations. Our evaluation protocol of 20 trials per mode is consistent with several established works: Diffusion Policy uses 20 trials for several tasks [7], OneDP evaluates 20 predefined initial positions [9], and Consistency Policy reports 10–20 trials [8]. Nevertheless, we fully agree that additional trials and training seeds would further strengthen the statistical confidence of our results. Regrettably, due to the unavailability of the original robotic platform, we are currently unable to conduct further experiments. To address this concern as thoroughly as possible, we have included two previously conducted experiments that were not fully reported in the original manuscript. We sincerely hope these additional results help provide further evidence of our method's reliability.

(1) Fixed separation ablation. We evaluated MOSS(w/o adaptive) on three real-robot tasks with 20 trials per mode:

TaskSRGR₁GR₂
Place Cups0.980.851.00
Pick Veg0.700.600.50
Open Drawer0.800.600.70

Compared with adaptive MOSS, fixed separation achieves lower GR in five of six modes and ties in one, supporting the benefit of adaptive source separation.

(2) Additional real-robot CFG results. Beyond the weaker modes reported in Table 12, we previously evaluated CFG scale 3 on the remaining modes:

TaskModeCFG=1 GRCFG=3 GR
Place CupsMode 20.850.90
Pick VegMode 20.600.60
Open DrawerMode 10.850.80

Stronger CFG does not consistently improve performance. Together with the independent AdaLN simulation results in Q1, this suggests that real-robot failures may not be resolved by stronger conditioning alone. Differences in effective training windows, branching-state coverage, and visual conditions may also contribute.

Revisions: Table 3 - fixed separation ablation; Section 5.4 - ablation analysis; Appendix F - real-world evaluation CFG sweep.

Q5: Shifting the source is also conditioning. Why is it more reliable than labels? Is the claim that modes are never provided to the policy accurate?

Answer: We sincerely appreciate the reviewer for highlighting this important distinction. We fully agree that selecting a source distribution is itself a form of conditioning. We apologize for the imprecise wording and have clarified that mode labels are used to select the initial source distribution, rather than being directly fed into the FM velocity network.

The key distinction is how mode information is introduced. Direct conditional FM uses mode labels to modulate the velocity field from a shared Gaussian source. In contrast, MOSS pairs mode-specific actions with corresponding source components during training and selects the desired component at inference, while all modes share the same velocity network.

This design allows MOSS to share common motions while maintaining distinct behaviors at branching states. For example, Stack Cube modes share approach and grasping motions but differ during placement. MOSS adapts source separation to local modality strength, keeping components closer during shared motions and separating them at branching. This also benefits imbalanced datasets, where minority modes can exploit common motions learned from majority-mode demonstrations. Our 10:90 imbalance experiments and real-robot results support these advantages.

At inference, MOSS directly selects the desired source component, without additional conditioning branches or CFG evaluations. Unlike sampling according to training-mode frequencies, this allows users to explicitly request minority modes. The conditional FM, independent AdaLN, and CFG results in Q1 further show that stronger label conditioning does not necessarily yield better mode control.

Finally, because MOSS introduces no additional guidance evaluations, its source-selection mechanism could potentially be combined with few-step methods such as OneDP, Consistency Policy, and FlowPolicy [8-10].

Revisions: Section 2 - clarification of mode conditioning.

Reviewer_RNQWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the inability of multimodal flow-matching (FM) robot policies to explicitly select which behavioral mode is executed. The authors first argue, empirically (Fig. 2, Table 4) and theoretically (Thm. 1–2, Prop. 3), that under ideal deterministic ODE transport, each behavioral mode has a disjoint, mass-preserving preimage in the Gaussian source space, and that approximation error blurs these regions near their boundaries. Building on this, they propose MOSS:

  • Using mode-labeled demonstrations, it builds one Gaussian source component per mode. The source centers lie in the subspace spanned by the centered mode-mean action chunks (Eq. 8–10).
  • A separately trained modality-strength estimator qq, frozen after training, adjusts the spacing and variance of the components (Eq. 11–12). Where behaviors are similar, the components contract into an approximately shared Gaussian; at behavioral branching points, they separate.
  • The velocity field is shared across all modes and never receives the mode label. At deployment, a mode is selected by sampling x0x_0 from the corresponding component. Experiments cover 6 simulated and 3 real-robot tasks. Baselines include unconditional policies, mode-conditioned FM, and GMM-sourced FM. There is also a 10:90 mode-imbalance study and ablations of the source design.
Critical strengths

A simple control mechanism that only modifies the source distribution, and that still works on real-robot tasks where direct conditioning fails.

  • MOSS changes only the source distribution. It requires no architectural changes and no extra network evaluations at inference, and training is standard FM (Eq. 5, Alg. 1). All tasks share one set of source hyperparameters (Table 6).
  • On all three real-robot tasks, both modes can be controlled (GR 0.65–1.00, Table 2). In contrast, mode-conditioned FM drops to GR = 0.00 on one mode in both Pick Veg and Open Drawer. Raising the CFG scale to 3 only partially mitigates this and also lowers SR (Table 9).
  • This is the main evidence that where mode information enters the generative process matters, and it is the strongest argument for acceptance.
Critical weaknesses

1.The decisive real-robot result depends on a baseline of uncertain strength. In simulation, MOSS is only marginally better than mode-conditioned FM (Table 1). The case for source-space control therefore rests mainly on the real-robot results in Table 2, where mode-conditioned FM collapses to GR = 0.00nd in Open Drawer. However, the real-robot data is roughly balanced (51–55 episodes per mode, App. E.4), so the paper's "imbalance-induced shortcut" explanation (L049–052) cannot account for this collaps how the label is injected, whether labeldropout was used during training (which CFG requires), or how the baseline was tuned. Each method also appears to have been trained only once. Without theset that the real-robot gap reflects anunder-tuned baseline rather than an advantage of source-space control. 2.The closest comparisons are missing. There is no direct comparison with SL-FM (Zhang et al., 2026), which also separates behavior branches in source space with a shared velocity field (L136–138). A simple inference-time baseline is also missing: sample from unconditional FM and use a mode classifier to keep action chunks belonging to the requested mode. For the K>2K>2 tasks (Fig. 6), only MOSS's own results are reported.

Questions for authors
  1. How is the mode label injected in mode-conditioned FM (token, FiLM, or adaLN)? Was label dropout used during training? How was the baseline tuned? Given that the real-robot data is balanced, what explains GR = 0.00 on Pick Veg mode 1 and Open Drawer mode 2?
  2. Could you provide a direct comparison with SL-FM (or an explanation of why it is not applicable), or an "unconditional FM + mode-classifier rejection sampling" baseline under a matched inference budget?
7 comments
AuthorsOpenReview ↗

Response to Reviewer RNQW (1/3)

Thank you for your detailed and constructive review, and for recognizing the simplicity of MOSS's control mechanism and its practical value on real robots. We would like to express our sincerest gratitude for the time and efforts spent on handling and reviewing our manuscript. All your comments have been taken into account thoroughly during the revision process.

Q1: How are labels injected into Mode-conditioned FM? Is label dropout used? How are baselines selected and tuned?

Answer: Thank you for pointing out the missing implementation details. We have clarified them in the appendix.

Mode injection. The original Mode-conditioned FM combines categorical mode labels with visual and robot-state features into a global condition, which is injected through AdaLN in each DiT block [1]. To examine whether a separate conditioning pathway improves control, we additionally tested independent mode AdaLN, where a separate MLP generates mode-specific scale/shift parameters while retaining the original observation, time, and residual-gating pathways.

TaskConditioningMode 1 GR / SRMode 2 GR / SRAvg. GR / SR
Push CubeOriginal Mode-conditioned FM86% / 96%94% / 100%90% / 98%
Push CubeIndependent mode AdaLN84% / 98%92% / 98%88% / 98%
Grasp EyeglassesOriginal Mode-conditioned FM100% / 100%100% / 100%100% / 100%
Grasp EyeglassesIndependent mode AdaLN100% / 100%98% / 98%99% / 99%
Stack CubeOriginal Mode-conditioned FM96% / 96%94% / 94%95% / 95%
Stack CubeIndependent mode AdaLN88% / 88%90% / 92%89% / 90%

The original conditioning performs comparably or better across all three tasks, suggesting that its performance is not limited by the shared conditioning pathway.

We also explored direct language conditioning on real robots, using separate prompts for different modes. Preliminary results were worse than discrete-mode conditioning, potentially because the prompts share similar semantics and differ mainly in a few words. This is consistent with reported difficulties in distinguishing compositional relations in vision-language models [6] and reduced influence of language instructions when robotic policies encounter visually similar scenes [7].

Label dropout and CFG. During training, we drop the mode condition with probability 0.2. At inference, we apply classifier-free guidance [2]:

vs=vθ(xt,t,o,∅)+s[vθ(xt,t,o,k)−vθ(xt,t,o,∅)].v_s=v_{\theta}(x_t,t,o,\varnothing)+s\left[v_{\theta}(x_t,t,o,k)-v_{\theta}(x_t,t,o,\varnothing)\right].

Using the original baseline's selected checkpoint, we evaluated CFG scales from 1 to 4:

TaskCFG scaleMode 1 GR / SRMode 2 GR / SRAvg. GR / SR
Push Cube186% / 96%94% / 100%90% / 98%
Push Cube290% / 96%94% / 96%92% / 96%
Push Cube390% / 92%92% / 92%91% / 92%
Push Cube454% / 56%92% / 92%73% / 74%
Grasp Eyeglasses1100% / 100%100% / 100%100% / 100%
Grasp Eyeglasses2100% / 100%100% / 100%100% / 100%
Grasp Eyeglasses3100% / 100%100% / 100%100% / 100%
Grasp Eyeglasses4100% / 100%100% / 100%100% / 100%
Stack Cube196% / 96%94% / 94%95% / 95%
Stack Cube296% / 96%92% / 92%94% / 94%
Stack Cube394% / 94%92% / 92%93% / 93%
Stack Cube492% / 92%90% / 90%91% / 91%

Stronger guidance does not consistently improve performance. Push Cube reaches 92% average GR at scale 2 but drops to 73% at scale 4, while Grasp Eyeglasses and Stack Cube already achieve near-ceiling performance at scale 1. Excessive guidance may amplify prediction errors and shift trajectories away from the training distribution, consistent with prior observations [3–5].

Baseline selection and evaluation. For all methods, we select the checkpoint with the lowest validation loss, evaluate it using three different test seeds, and report the highest result. Real-robot experiments follow the same training configuration, with inputs and action dimensions adapted to the platform.

Revisions: Appendix E - baseline implementation and training details; Appendix F - CFG sweep.

AuthorsOpenReview ↗

Response to Reviewer RNQW (2/3)

Q2: Real-robot demonstrations are roughly balanced across modes. Why do Pick Veg Mode 1 and Open Drawer Mode 2 still have GR=0? Is an imbalance-induced shortcut a sufficient explanation?

Answer: Thank you for highlighting this issue. Although we collected equal numbers of episodes per mode, this does not guarantee balanced training data. Differences in trajectory length, state coverage, and observation conditions can still affect learning.

Since training samples are drawn uniformly from trajectory windows, longer episodes contribute more samples. If mode kk contains Wk,eW_{k,e} windows in episode ee, its sampling probability is

Pr⁡D(k)=∑eWk,e∑j∑eWj,e.\Pr_{\mathcal D}(k)=\frac{\sum_e W_{k,e}}{\sum_j\sum_e W_{j,e}}.

Even with equal episode counts, the mode with longer average collection times can therefore contribute more optimization samples. Coverage of critical branching states can also differ even when overall window proportions are similar. Together with visual variation, this makes real-robot demonstrations a noisy dataset with imperfect balance in windows and state coverage. The observed control degradation indicates that the original conditional baseline remains sensitive to these practical collection differences.

What's more, GR=0 does not necessarily indicate complete mode collapse. It means no evaluated rollout both succeeded and followed the requested mode. In our observations, some trajectories initially approached the correct target but switched modes during execution, while others failed after approaching.

We therefore do not attribute these failures solely to data imbalance. Flow matching with a shared velocity field may also struggle to maintain distinct behaviors near branching states, as suggested by prior analyses of multimodal transport and mode coverage [8–10]. MOSS addresses this by bringing source components closer for shared behaviors while separating them at branching states, facilitating both shared action structure and mode-specific control.

Revisions: Section 5.2 - real-robot data characteristics.

Q3: Can you directly compare against SL-FM and against unconditional FM + a mode classifier under the same inference budget?

Answer: Thank you for these suggestions. We added both baselines to our experiments.

(1) SL-FM. SL-FM [10] is closely related to our work. Since its code is unavailable, we implemented it following the paper. However, unlike MOSS, SL-FM learns intervenable source handles without directly assigning predefined semantic meanings to them. We therefore calibrated its handles using 20 rollouts per handle, selected distinct handles for the two target modes, and froze this mapping. We then evaluated each mode over 50 trials.

TaskMode 1 GR / SRMode 2 GR / SRAvg. GR / SR
Push Cube74% / 74%92% / 92%83% / 83%
Grasp Eyeglasses98% / 100%62% / 98%80% / 99%
Stack Cube90% / 100%34% / 98%62% / 99%
Put Wine46% / 88%42% / 74%44% / 81%

(2) Unconditional FM + mode classifier. We freeze the pretrained unconditional FM and train a classifier cψ(k∣A)c_\psi(k\mid A) to predict the mode of an action chunk AA, using cross-entropy loss and AdamW with validation-based early stopping.

At inference, we generate N=64N=64 candidate action chunks and sample one according to classifier-based weights:

wi=cψ(k∣Ai)γ∑j=1Ncψ(k∣Aj)γ.w_i=\frac{c_\psi(k\mid A_i)^\gamma}{\sum_{j=1}^{N}c_\psi(k\mid A_j)^\gamma}.

The results are:

TaskMode 1 GR / SRMode 2 GR / SRAvg. GR / SR
Push Cube90% / 100%76% / 90%83% / 95%
Grasp Eyeglasses100% / 100%96% / 100%98% / 100%
Stack Cube74% / 96%76% / 96%75% / 96%
Put Wine42% / 74%60% / 72%51% / 73%

Both baselines achieve meaningful mode control but remain below MOSS overall. Moreover, classifier-based selection requires generating and scoring 64 candidates per query, followed by an additional decoding pass in our implementation (approximately 65 sampling paths in total), whereas MOSS requires only one. Thus, MOSS achieves stronger mode control with substantially lower inference cost.

Revisions: Table 1 - additional baselines; Appendix E.2 - implementation details.

AuthorsOpenReview ↗

Response to Reviewer RNQW (3/3)

Q4: Only MOSS is reported for tasks with K>2. Can you add comparison methods?

Answer: Thank you for this helpful suggestion. In the original manuscript, we focused on demonstrating that MOSS can generalize beyond two modes. We agree that comparisons with additional baselines would provide a more comprehensive evaluation. Following your suggestion, we have added Mode-conditioned FM and GMM-Sourced FM results on 2D Navigation and Place Cube:

2D Navigation:

MethodMode 1 GR / SRMode 2 GR / SRMode 3 GR / SRMode 4 GR / SRAvg. GR / SR
Mode-conditioned FM96% / 100%88% / 98%98% / 100%92% / 100%93.5% / 99.5%
GMM-sourced FM97% / 99%90% / 97%83% / 96%99% / 100%92.25% / 98%
MOSS94% / 94%100% / 100%100% / 100%84% / 84%94.5% / 94.5%

Place Cube:

MethodMode 1 GR / SRMode 2 GR / SRMode 3 GR / SRAvg. GR / SR
Mode-conditioned FM82% / 82%80% / 84%86% / 86%82.67% / 84%
GMM-sourced FM38% / 90%8% / 88%60% / 90%35.33% / 89.33%
MOSS90% / 90%84% / 100%96% / 96%90% / 95.33%

On the 2D Navigation task, all three methods achieve high controllability, with MOSS obtaining the highest average GR (94.594.5%). On the Place Cube task, MOSS achieves the highest average GR (90.090.0%) and SR (95.395.3%), demonstrating its effectiveness in more challenging multi-modal manipulation settings.

Revisions: Table 2 - comparisons on tasks with K>2K > 2 modes; Section 5.2 - descriptions of compared methods and results.

References:

[1] W. Peebles et al. Scalable Diffusion Models with Transformers. ICCV, 2023. arXiv:2212.09748.

[2] J. Ho et al. Classifier-Free Diffusion Guidance. arXiv:2207.12598, 2022.

[3] H. Chung et al. CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models. arXiv:2406.08070, 2024.

[4] S. Lin et al. Common Diffusion Noise Schedules and Sample Steps are Flawed. arXiv:2305.08891, 2023.

[5] T. Karras et al. Guiding a Diffusion Model with a Bad Version of Itself. NeurIPS, 2024. arXiv:2406.02507.

[6] T. Thrush et al. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. CVPR, 2022. arXiv:2204.03162.

[7] Y. Fang et al. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs. arXiv:2602.17659, 2026.

[8] A. Salmona et al. Can Push-forward Generative Models Fit Multimodal Distributions? NeurIPS, 2022. arXiv:2206.14476.

[9] L. Mazza et al. Understanding Multimodality in Generative Behavioral Cloning. arXiv:2605.22493, 2026.

[10] H. Zhang et al. Source-Lifted Flow Matching for Intervenable Multimodal Imitation. arXiv:2607.10206, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and the major changes of revised manuscript

Dear Reviewers,

We would like to express our sincere gratitude to all three reviewers for your valuable time, careful evaluations, and thoughtful suggestions. We are truly grateful for your recognition of the strengths of our work. In particular, Reviewer RNQW highlighted the simplicity of our source-based control mechanism and its effectiveness on real robots where direct conditioning struggles. Reviewer Yh19 appreciated the intuitive idea of selecting behavioral modes through the initial source without modifying the velocity network, and found the real-robot results particularly convincing. Reviewer qSfb recognized the practical value of MOSS for multimodal behavior control, the encouraging results across simulation and real-world tasks, and the improvements under imbalanced demonstrations. We greatly appreciate these encouraging assessments, which have strengthened our confidence in the practical relevance of this research.

We are equally grateful for the reviewers' constructive concerns regarding baseline comparisons, methodological novelty, adaptive source shaping, theoretical justification, and experimental robustness. These suggestions have been invaluable in helping us identify areas for improvement. Following your advice, we have made substantial efforts to strengthen our experiments and clarify the presentation. We have uploaded a revised manuscript, with the major additions and modifications highlighted in blue for ease of reference. We respectfully summarize the main revisions below:

  1. Expanded Baseline Comparisons: Added direct comparisons with SL-FM and unconditional FM with classifier-based mode selection, including implementation details and semantic calibration. We also extended comparisons with Mode-Conditioned FM and GMM-Sourced FM to multi-mode tasks.

  2. Stronger Conditioning and CFG Evaluations: Clarified the implementation and training configurations of Mode-Conditioned FM, including label injection, dropout, and checkpoint selection. We further evaluated independent mode AdaLN and conducted systematic CFG scale sweeps on both simulation and real-robot tasks.

  3. Adaptive Source Shaping Ablations: Extended fixed-separation comparisons to 10:90 imbalanced demonstrations and all three real-robot tasks, providing further evidence for the contribution of adaptive source shaping beyond semantic source separation alone.

  4. Source Geometry and Robustness Analysis: Added comparisons with randomly assigned source centers, further analyzed action-aligned source geometry, and conducted noise-sensitivity experiments to examine robustness to inaccuracies in the estimated modality strength.

  5. Theoretical and Methodological Clarifications: Further clarified the relationship between our theoretical analysis and method design, the construction of mode-discriminative subspaces, and the distinctions between MOSS and existing source-based methods. We also explicitly clarified that source selection is a form of conditioning, while mode labels are not directly provided to the velocity network.

  6. Evaluation Protocol and Experimental Details: Expanded the descriptions of real-robot demonstrations, baseline configurations, and checkpoint selection. We also included previously conducted real-robot fixed-separation experiments and clarified the limitations of the current evaluation.

We sincerely hope that these additional experiments and clarifications help address the reviewers' concerns and provide a more comprehensive understanding of our contributions. We fully acknowledge that there remains room for improvement, and we are deeply grateful for the reviewers' guidance in strengthening both the technical rigor and clarity of our manuscript.

Thank you once again for your time, patience, and thoughtful consideration.

Best regards,

Authors of Submission 19746

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.