- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Source-Lifted Flow Matching for controllable multimodal imitation. It introduces selectable source components and auxiliary coordinates to preserve source identity during generation, allowing behavior to be influenced through the starting point of a shared flow model.
Critical strengths
-
The paper addresses a meaningful problem: generating diverse behaviors does not necessarily make those behaviors easy to select.
-
The proposed source handles offer a potentially useful interface between high-level decisions and continuous action generation, without requiring manual behavior labels.
-
The matched-prefix intervention setup is a useful way to examine whether changing a handle actually changes subsequent behavior.
Critical weaknesses
-
I am not fully convinced why control needs to be restricted to the source. The auxiliary coordinates still carry information about (z), and directly conditioning a shared network on (z) would also be a natural choice. Although this baseline is included, the practical benefit of the source-only restriction remains unclear. The restriction feels somewhat self-imposed, and its added complexity needs a clearer justification.
-
I also wonder whether the learned action groups correspond to meaningful behavior strategies. Differences in local actions do not necessarily translate into consistent long-term intentions. The connection between selecting a source and choosing a coherent strategy needs more discussion.
-
The paper includes ablations, but they could focus more directly on controllability. It would be useful to see how removing lifting or the responsibility floor, or using non-orthogonal tags, affects successful behavior switching. Task success and route entropy alone do not fully establish whether these components preserve branch identity as intended.
Questions for authors
see weaknesses.
2 comments
W3: Ablations that more directly evaluate controllability
We agree that more direct measures are needed to assess whether the model has learned effective modes. However, as you noted above, task-level evaluation often reflects coherent behavioral strategies.
We primarily compared task success and route entropy in the paper because these are commonly used metrics and allow us to assess whether our method maintains task performance.
We have also been considering the issue you raised. The paper reports route-change rates, both-success rates, and successful route-switching rates under same-prefix interventions (Table 2), as well as mutual information between handles and route choices (Figure 5) and associations between handles and subtasks in Kitchen (Figure 3(b)). In addition, with the low-level policy frozen, a high-level CEM/PPO selector achieves 91.7% joint route-and-task success over 24 target routes from a fixed initial state (Figure 3(c)). These results provide evidence from interventions, correlations, and downstream usability, respectively. Among them, successful route switching under same-prefix interventions most directly addresses your ablation question.
Following your suggestion, we supplemented the experiments with same-prefix intervention evaluations of the three modifications:
- Without lifting: We use the SL-FM w/o lift configuration from the paper, retaining the source handle while removing the auxiliary coordinates.
- Without the responsibility floor: We set the responsibility floor to zero during training.
- Nonorthogonal tags: We retain the lifting coordinates and use preset unit-norm tags with pairwise cosine similarity 0.5. The original method instead learns state-dependent tags regularized toward orthogonal anchors.
| Configuration | Route-code change (including failures, %) | Both-success rate (%) | Successful full-route switching (%) |
|---|---|---|---|
| SL-FM | 91.1 | 63.9 | 57.8 |
| Without lifting | 93.8 | 63.0 | 58.3 |
| Without responsibility floor | 99.0 | 44.8 | 44.3 |
| Nonorthogonal tags | 80.2 | 57.8 | 40.6 |
One notable observation is that removing the responsibility floor can produce a very high route-code change rate, meaning that the behavioral mode “certainly” changes, while the both-success rate drops substantially, from 63.9% to 44.8%. The value of the responsibility floor therefore needs to balance behavioral differentiation and accuracy.
We consider the nonorthogonal-tag ablation particularly important. Together with the ablation that removes lifting, it provides two complementary validations of the necessity of orthogonality. We will prioritize adding this ablation to the main text. We sincerely appreciate this suggestion! We surely believe that including this ablation will help address this gap in our evaluation and strengthen the paper.
W1: Why restrict control to the source?
Thank you for your comments!
We designed our method this way and distinguish it from z-conditioning because the augmentation serves two purposes:
The main benefit is that lifting substantially alleviates velocity conflicts in the flow field. For example, as illustrated in Figure R2-1 and Figure R2-2, provided in our response to Reviewer 2, the distinction between z-conditioning and SL-FM is whether lifting is used. The z-conditioned model produces less straight trajectories with greater curvature, whereas SL-FM produces straighter trajectories with fewer conflicts after lifting. This is consistent with the second purpose described above.
Reducing these conflicts lowers the complexity of denoising and enables the model to fit the target distribution more accurately. It also improves the interpretability of the flow field, making its properties easier to analyze.
W2: Do the action groups correspond to meaningful long-term behavioral strategies?
Your question addresses the most important issue in moving from the algorithm to practical downstream applications.
Our initial goal was to observe differences in overall behavior styles. However, as you pointed out, identifying and making use of long-term intentions is a difficult problem. If this can be achieved, FM would be well suited to learning and invoking multiple skills. However, identifying coherent behavioral strategies requires introducing the temporal dimension into clustering. Questions such as how to segment time and identify when to switch handles require further in-depth study. As shown in Figure 7 of our paper, our current solution is straightforward: we specify an execution window and then switch handles to achieve local behavioral changes. However, this approach is highly unreliable for complex tasks.
We are also exploring whether a metric can indicate when the current state permits a skill switch, but we do not yet have a stable method. We will explicitly acknowledge this limitation and discuss it further in the appendix.