- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies flow-matching policies for language-conditioned multi-task robot manipulation. Existing methods often use one shared Gaussian source and one shared linear transport path for all tasks, forcing the policy to recover very different action distributions from task-independent noise. TOPS first reverses a warm-up flow policy to map expert action chunks back to the source space, then fits a task-specific Gaussian mixture with low-rank covariance as the source prior. It compares the source covariance with the expert-action covariance and uses a Gaussian optimal transport map to identify directions that require expansion. TOPS applies a slow-then-fast interpolation schedule along these directions while keeping the policy backbone weights unchanged. Experiments on LIBERO, LIBERO-Plus, CALVIN, a π0.5 transfer setting, and real robots show strong results, especially for few-step inference and data efficiency.
Critical strengths
- The joint adaptation of task-specific sources and transport schedules is a clear and useful view. Task-dependent sources or priors have been studied in methods such as PriorGrad, MM-FM, and LeaP. TOPS combines a reverse-flow-derived low-rank Gaussian mixture source with a task-specific schedule derived from source-target covariance mismatch, without changing the policy backbone. This gives a clear distinction from methods that only change the prior, coupling, or a global path or schedule.
- The method and experiments form a complete chain. The reverse-flow-derived task source, low-rank Gaussian mixture, OT expansion directions, and schedule-aware inference are connected into one method. The paper evaluates them on ID and OOD settings, data efficiency, π0.5 transfer, and real robots. On LIBERO-10, TOPS improves success from 53.44% with Standard FM to 69.67% while using much less inference computation, showing practical value.
Critical weaknesses
- TOPS depends on the language task selector, and its out-of-distribution robustness is not sufficiently analyzed. The selector does not merely provide an additional semantic condition. It directly determines which task-specific source and schedule are used. The learned classifier in Table 11 achieves 98.6% top-1 accuracy on held-out instruction formulations from 34 known tasks, which still means that selection can fail. Ambiguous instructions or unseen noisy instructions may cause the selector to fail, potentially causing the whole method to fail. A selector error changes both the initial generation distribution and the transport path.
Questions for authors
- On the robustness of the language task selector: How does the method perform when the selector is correct, randomly incorrect, or exposed to ambiguous or noisy instructions and unseen task identities? In particular, how do task success, generated trajectories, and the selected source and schedule change under these conditions?
6 comments
Response to reviewer yxod (2/4)
2. From task recognition to consistent execution
Thank you for asking specifically about selected sources, schedules, and generated trajectories. This prompted us to investigate both whether correct recognition preserves the generation process and whether improving recognition produces a measurable execution benefit.
Consistency after recognition. We compared predictions from actual instructions against oracle task conditioning for the same policy, observation, and source randomness. On 76 correctly routed fixed-observation queries, the selected source and schedule, language condition, initial samples, solver states, and action chunks all match the oracle. In closed loop, all 711 correctly routed cases produce action sequences bitwise identical to their oracle counterparts. These checks follow the effect of language through the generation process itself: once equivalent expressions are assigned to the same correct task, their wording introduces no further changes to the tested trajectories. We have added the consistency checks and their design to Appendix A.8.2.
Connecting better recognition to better execution. The preceding evaluation combines augmentation and canonical conditioning. To distinguish their contributions, we added a controlled experiment that fixes the policy, source/schedule library, and canonical interface, and replaces only S0 with S2 on paired language inputs and initial states. Routing errors fell from 70 to 32, while success on native language variants increased from 67.3% to 72.2%, a gain of 4.9 pp [2.1, 7.7]. Because the action-generation components remain fixed, this improvement directly connects better task identification to stronger closed-loop performance. We have included this selector-replacement experiment and its analysis in Appendix A.8.3, Table 18.
These studies distinguish unfamiliar wording from an unspecified objective. Paraphrases and spelling variants retain a definite intended goal; genuinely ambiguous requests require clarification of that goal. Once identified, canonical conditioning provides a consistent interface for its execution.
Response to reviewer yxod (3/4)
3. Accommodating new objectives
Thank you for raising the question of unseen task identities. Your suggestion motivated two complementary studies during rebuttal: whether a new objective can reuse an existing generative configuration, and how new demonstrations can expand the library. These examine the distinction between the task associated with a source and the objective specified to the policy.
Reusing an existing configuration for a new goal. We evaluated two illustrative π₀.₅ cases from LIBERO-Goal whose objectives were excluded from downstream training and source construction. The policy, selector, and library remain frozen. The selector retrieves an existing source/schedule, while the VLA receives the original held-out-goal instruction, preserving the new objective. Cheese-to-bowl reuses cheese-to-basket and achieves 60% success versus FM's 46%; bowl-to-stove reuses bowl-from-stove-to-plate and achieves 94% versus 98%. These examples show that the prior supplies generative structure while current observations and language guide its use toward the requested goal. A source's original task therefore need not determine the executed objective. We have added the cases and their evaluation design to Appendix A.13.1, Table 21.
Growing the library with demonstrations. We also studied four independent LIBERO-10→11 expansions, each introducing a LIBERO-90 objective absent from the original training set. TOPS constructs a new source/schedule entry from 20 demonstrations, retains existing entries, and adapts the shared policy with new-task data and old-task replay. FM receives the same data and adaptation budget. Evaluating both the new objective and the expanded task set tests new-skill acquisition alongside performance across the existing repertoire. Success percentages below are shown as FM / TOPS.
| Added objective | New task | All eleven tasks |
|---|---|---|
| Milk → basket | 38.0 / 48.0 | 38.9 / 55.7 |
| Book → front compartment | 44.0 / 42.0 | 34.5 / 53.8 |
| Turn on stove and place pan | 36.0 / 56.0 | 42.4 / 48.7 |
| Turn off stove | 40.0 / 64.0 | 51.8 / 57.2 |
| Mean of displayed cases | 39.5 / 52.5 | 41.9 / 53.9 |
TOPS achieves stronger expanded-set performance in every displayed case, averaging 53.9% versus 41.9% for FM, with higher new-task success in three of four cases. The same strategy extends to π₀.₅: with 20 demonstrations per new LIBERO-90 goal, TOPS achieves 76% and 98% on book-to-left-compartment and pudding-to-left-of-plate, compared with 70% and 88% for FM. Together, reuse and supervised expansion provide concrete ways to accommodate objectives beyond the initial task library. We have included the expansion results in §5.3, with the four LIBERO-10 cases in Table 2 and their experimental design in Appendix A.13.
Response to reviewer yxod (1/4)
Thank you for recognizing TOPS's joint source-and-schedule design and its empirical effectiveness. Your observation that the selector determines both the starting distribution and transport schedule motivated us to examine the full connection between language recognition and robot behavior. During rebuttal, we added targeted studies of unfamiliar language, generation consistency, task-library expansion, and continuous conditioning. Together, these investigate how to recognize varied expressions, preserve execution after recognition, and accommodate objectives beyond a fixed task lookup. We have incorporated the experiments and their designs into the revised paper, with specific locations provided below.
1. Robustness to unseen wording and spelling variations
Thank you for highlighting unfamiliar and noisy instructions. Your question encouraged us to test whether the selector recognizes the intended goal beyond the original annotation pool. We therefore added independently generated paraphrases and spelling perturbations during rebuttal, separating phrasing families across training and evaluation. Each variant preserves the goal while changing its expression, so the experiment directly examines linguistic generalization for known tasks.
To identify what improves recognition, we compared official-text training (S0), semantic-paraphrase augmentation (S1), and additional spelling augmentation (S2), keeping the encoder, classifier, and training budget fixed. The paraphrases vary wording, sentence structure, and conversational expressions; spelling augmentation introduces character-level changes. S1 raises accuracy from 92.83% to 94.38% on LIBERO and from 58.96% to 75.59% on CALVIN; S2 further improves these results to 97.58% and 84.13%, respectively. Both stages contribute, showing that broader linguistic coverage helps the same classifier recognize task meaning across unfamiliar expressions. We have added this evaluation and its augmentation design to Appendix A.8.1, Table 16, for detailed reference.
Does improved recognition preserve the execution advantage? Equivalent instructions can also produce different policy-language embeddings. We therefore explored a canonical-conditioning interface in which the predicted task selects its source, schedule, and training-time language embedding. This lets the selector handle wording variation while the policy receives a consistent representation of the recognized objective. FM uses the same selector and canonical-conditioning rule, with its own task-independent source and linear path. Matched text-driven rollouts then compare action generation after the same language-processing step.
TOPS retains a 9.2–11.7 percentage-point advantage across all four instruction sets, with all paired confidence intervals above zero. Since the methods share the language frontend, this advantage reflects the effectiveness of TOPS's action generation when conditions are selected from actual text. Its success remains within 1.7–5.8 points of oracle conditioning, which supplies the correct task identity to the same policy. We have included this new closed-loop evaluation and the interface design in Appendix A.8.2, Table 17, for detailed reference.