acceptodds
Under review as a conference paper at ICLR 2027

TOPS: Reshaping Flow Source and Transport for Multi-Task Robot Manipulation

Abstract

Flow-based policies for language-conditioned multi-task manipulation commonly share a single source prior and a common linear transport path across tasks. However, reliably executing multi-task with distinct motion requirements remains challenging for a shared policy. We introduce (ask-riented riors and chedules), which adapts the flow source prior and interpolation schedule to each task without modifying the network architecture. TOPS first constructs multimodal task sources from reverse-mapped expert action sequences, allowing language instructions to select where generation begins. Guided by source-target covariance mismatch, it then slows the transition toward expert actions early in denoising and accelerates it later along directions requiring expansion. Under a endpoint model, our analysis shows that this schedule adjustment can lower an upper bound on velocity-prediction risk. Experiments on LIBERO, LIBERO-Plus, and CALVIN demonstrate improved performance across in-distribution, perturbation and cross-environment settings, alongside better data efficiency. Using one integration step, TOPS raises LIBERO-10 success from 53.44% to 69.67% compared with ten-step Standard Flow Matching (FM), while reducing inference time to approximately one-sixth of the baseline. TOPS also yields consistent gains when transferred to the large-scale flow-based policy across all four evaluated settings. Real-world experiments further demonstrate improved task success, including an increase from 20% with standard FM to 80% with TOPS on upper-drawer opening task.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Flow Matching, Multi-Task Learning, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yxodOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies flow-matching policies for language-conditioned multi-task robot manipulation. Existing methods often use one shared Gaussian source and one shared linear transport path for all tasks, forcing the policy to recover very different action distributions from task-independent noise. TOPS first reverses a warm-up flow policy to map expert action chunks back to the source space, then fits a task-specific Gaussian mixture with low-rank covariance as the source prior. It compares the source covariance with the expert-action covariance and uses a Gaussian optimal transport map to identify directions that require expansion. TOPS applies a slow-then-fast interpolation schedule along these directions while keeping the policy backbone weights unchanged. Experiments on LIBERO, LIBERO-Plus, CALVIN, a π0.5 transfer setting, and real robots show strong results, especially for few-step inference and data efficiency.

Critical strengths
  1. The joint adaptation of task-specific sources and transport schedules is a clear and useful view. Task-dependent sources or priors have been studied in methods such as PriorGrad, MM-FM, and LeaP. TOPS combines a reverse-flow-derived low-rank Gaussian mixture source with a task-specific schedule derived from source-target covariance mismatch, without changing the policy backbone. This gives a clear distinction from methods that only change the prior, coupling, or a global path or schedule.
  2. The method and experiments form a complete chain. The reverse-flow-derived task source, low-rank Gaussian mixture, OT expansion directions, and schedule-aware inference are connected into one method. The paper evaluates them on ID and OOD settings, data efficiency, π0.5 transfer, and real robots. On LIBERO-10, TOPS improves success from 53.44% with Standard FM to 69.67% while using much less inference computation, showing practical value.
Critical weaknesses
  1. TOPS depends on the language task selector, and its out-of-distribution robustness is not sufficiently analyzed. The selector does not merely provide an additional semantic condition. It directly determines which task-specific source qτq_\tau and schedule KτK_\tau are used. The learned classifier in Table 11 achieves 98.6% top-1 accuracy on held-out instruction formulations from 34 known tasks, which still means that selection can fail. Ambiguous instructions or unseen noisy instructions may cause the selector to fail, potentially causing the whole method to fail. A selector error changes both the initial generation distribution and the transport path.
Questions for authors
  1. On the robustness of the language task selector: How does the method perform when the selector is correct, randomly incorrect, or exposed to ambiguous or noisy instructions and unseen task identities? In particular, how do task success, generated trajectories, and the selected source and schedule change under these conditions?
6 comments
AuthorsOpenReview ↗

Response to reviewer yxod (1/4)

Thank you for recognizing TOPS's joint source-and-schedule design and its empirical effectiveness. Your observation that the selector determines both the starting distribution and transport schedule motivated us to examine the full connection between language recognition and robot behavior. During rebuttal, we added targeted studies of unfamiliar language, generation consistency, task-library expansion, and continuous conditioning. Together, these investigate how to recognize varied expressions, preserve execution after recognition, and accommodate objectives beyond a fixed task lookup. We have incorporated the experiments and their designs into the revised paper, with specific locations provided below.

1. Robustness to unseen wording and spelling variations

Thank you for highlighting unfamiliar and noisy instructions. Your question encouraged us to test whether the selector recognizes the intended goal beyond the original annotation pool. We therefore added independently generated paraphrases and spelling perturbations during rebuttal, separating phrasing families across training and evaluation. Each variant preserves the goal while changing its expression, so the experiment directly examines linguistic generalization for known tasks.

To identify what improves recognition, we compared official-text training (S0), semantic-paraphrase augmentation (S1), and additional spelling augmentation (S2), keeping the encoder, classifier, and training budget fixed. The paraphrases vary wording, sentence structure, and conversational expressions; spelling augmentation introduces character-level changes. S1 raises accuracy from 92.83% to 94.38% on LIBERO and from 58.96% to 75.59% on CALVIN; S2 further improves these results to 97.58% and 84.13%, respectively. Both stages contribute, showing that broader linguistic coverage helps the same classifier recognize task meaning across unfamiliar expressions. We have added this evaluation and its augmentation design to Appendix A.8.1, Table 16, for detailed reference.

Does improved recognition preserve the execution advantage? Equivalent instructions can also produce different policy-language embeddings. We therefore explored a canonical-conditioning interface in which the predicted task selects its source, schedule, and training-time language embedding. This lets the selector handle wording variation while the policy receives a consistent representation of the recognized objective. FM uses the same selector and canonical-conditioning rule, with its own task-independent source and linear path. Matched text-driven rollouts then compare action generation after the same language-processing step.

Language inputSelector accuracyFM successTOPS success
Unseen wording98.3%63.3%75.0%
2% spelling perturbation96.7%63.3%73.3%
5% spelling perturbation91.7%61.7%70.8%
Native LIBERO-Plus language variants96.1%62.7%72.2%

TOPS retains a 9.2–11.7 percentage-point advantage across all four instruction sets, with all paired confidence intervals above zero. Since the methods share the language frontend, this advantage reflects the effectiveness of TOPS's action generation when conditions are selected from actual text. Its success remains within 1.7–5.8 points of oracle conditioning, which supplies the correct task identity to the same policy. We have included this new closed-loop evaluation and the interface design in Appendix A.8.2, Table 17, for detailed reference.

AuthorsOpenReview ↗

Response to reviewer yxod (2/4)

2. From task recognition to consistent execution

Thank you for asking specifically about selected sources, schedules, and generated trajectories. This prompted us to investigate both whether correct recognition preserves the generation process and whether improving recognition produces a measurable execution benefit.

Consistency after recognition. We compared predictions from actual instructions against oracle task conditioning for the same policy, observation, and source randomness. On 76 correctly routed fixed-observation queries, the selected source and schedule, language condition, initial samples, solver states, and action chunks all match the oracle. In closed loop, all 711 correctly routed cases produce action sequences bitwise identical to their oracle counterparts. These checks follow the effect of language through the generation process itself: once equivalent expressions are assigned to the same correct task, their wording introduces no further changes to the tested trajectories. We have added the consistency checks and their design to Appendix A.8.2.

Connecting better recognition to better execution. The preceding evaluation combines augmentation and canonical conditioning. To distinguish their contributions, we added a controlled experiment that fixes the policy, source/schedule library, and canonical interface, and replaces only S0 with S2 on paired language inputs and initial states. Routing errors fell from 70 to 32, while success on native language variants increased from 67.3% to 72.2%, a gain of 4.9 pp [2.1, 7.7]. Because the action-generation components remain fixed, this improvement directly connects better task identification to stronger closed-loop performance. We have included this selector-replacement experiment and its analysis in Appendix A.8.3, Table 18.

These studies distinguish unfamiliar wording from an unspecified objective. Paraphrases and spelling variants retain a definite intended goal; genuinely ambiguous requests require clarification of that goal. Once identified, canonical conditioning provides a consistent interface for its execution.

AuthorsOpenReview ↗

Response to reviewer yxod (3/4)

3. Accommodating new objectives

Thank you for raising the question of unseen task identities. Your suggestion motivated two complementary studies during rebuttal: whether a new objective can reuse an existing generative configuration, and how new demonstrations can expand the library. These examine the distinction between the task associated with a source and the objective specified to the policy.

Reusing an existing configuration for a new goal. We evaluated two illustrative π₀.₅ cases from LIBERO-Goal whose objectives were excluded from downstream training and source construction. The policy, selector, and library remain frozen. The selector retrieves an existing source/schedule, while the VLA receives the original held-out-goal instruction, preserving the new objective. Cheese-to-bowl reuses cheese-to-basket and achieves 60% success versus FM's 46%; bowl-to-stove reuses bowl-from-stove-to-plate and achieves 94% versus 98%. These examples show that the prior supplies generative structure while current observations and language guide its use toward the requested goal. A source's original task therefore need not determine the executed objective. We have added the cases and their evaluation design to Appendix A.13.1, Table 21.

Growing the library with demonstrations. We also studied four independent LIBERO-10→11 expansions, each introducing a LIBERO-90 objective absent from the original training set. TOPS constructs a new source/schedule entry from 20 demonstrations, retains existing entries, and adapts the shared policy with new-task data and old-task replay. FM receives the same data and adaptation budget. Evaluating both the new objective and the expanded task set tests new-skill acquisition alongside performance across the existing repertoire. Success percentages below are shown as FM / TOPS.

Added objectiveNew taskAll eleven tasks
Milk → basket38.0 / 48.038.9 / 55.7
Book → front compartment44.0 / 42.034.5 / 53.8
Turn on stove and place pan36.0 / 56.042.4 / 48.7
Turn off stove40.0 / 64.051.8 / 57.2
Mean of displayed cases39.5 / 52.541.9 / 53.9

TOPS achieves stronger expanded-set performance in every displayed case, averaging 53.9% versus 41.9% for FM, with higher new-task success in three of four cases. The same strategy extends to π₀.₅: with 20 demonstrations per new LIBERO-90 goal, TOPS achieves 76% and 98% on book-to-left-compartment and pudding-to-left-of-plate, compared with 70% and 88% for FM. Together, reuse and supervised expansion provide concrete ways to accommodate objectives beyond the initial task library. We have included the expansion results in §5.3, with the four LIBERO-10 cases in Table 2 and their experimental design in Appendix A.13.

Reviewer_GC4qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TOPS (Task-Oriented Priors and Schedules), a method for improving language-conditioned multi-task robot manipulation with flow-matching policies. Instead of modifying the policy backbone, TOPS adapts the source distribution and interpolation schedule for each task. The method first reverses expert action trajectories through a pretrained flow model and fits task-specific Gaussian mixture sources using MPPCA. It then identifies expansion directions from source-target covariance mismatch using Gaussian optimal transport and applies slow-then-fast interpolation along these directions. Experiments on LIBERO, LIBERO-Plus, CALVIN, and real-world manipulation tasks demonstrate improved task success, data efficiency, and inference speed.

Critical strengths
  1. The method proposed by the paper doesn't require modifying the policy backbone, and only requires modifying the generative process instead of augmenting the existing components. Therefore, this paper propose a method that could be practical;
  2. The paper presents both strong empirical validation and theoretical proofs;
  3. The paper has shown strong improvement to flow matching baseline and the performance could be transferred to Pi 0.5;
  4. There are a few good case studies on multimodal task output and long horizon, which is helpful to understand the boundary of the method.
  5. The paper is overall well-written and present good intuition on the theoretical concept
Critical weaknesses

1.The notion of "task" and "task distribution" is central to the paper, but not well-defined throughout. To many generalist VLA framing (which is essentially the multi task setting in the paper), the task should not be a closed set, or otherwise this might be a strong assumption that the interpolation / extrapolation between the tasks should be able to supported by the paper but understudied. 2. The contribution of interpolation scheduling is not fully separated from the specialized inference solver. In Table 1, source-only methods use explicit-midpoint integration, whereas TOPS uses its schedule-aware solver The paper’s central claim is that reshaping source and transport is what helps, but the ablation says schedule and integrator “jointly” account for the increase - this might require further understanding and study.

Questions for authors
  1. This is to the limitation 2- how does the proposed method depends on specific solvers? I was trying to understand from the paper and Table 6 referred by main text ablation, but the table 6 seems to suggest that the paper can be sensitive to solvers?
  2. Could authors provide more theoretical intuition that how does the proposed method handle multimodal action distribution and long horizon task (such as tasks in Task and Motion Planning (TAMP) problem class)?
5 comments
AuthorsOpenReview ↗

Response to reviewer GC4q (1/3)

Thank you for your careful review and for recognizing TOPS's practical design, theoretical motivation, and empirical effectiveness. We conducted additional experiments during the rebuttal period specifically to address your questions about task-set extensibility, contribution attribution, and behavioral multimodality. We have added these experiments and their supporting details to the revised paper; below, we explain the findings and indicate where you can examine each addition.

CW1: Task definitions and task-set extensibility

Thank you for asking us to clarify task definitions and the openness of the task set. Your question helped us distinguish task objectives from generative priors more clearly and motivated direct tests of how the construction extends to new objectives.

A task is a language-specified manipulation objective, such as placing a book in a particular compartment. Its demonstrations define an expert action distribution; TOPS fits a separate generative source to their reverse-mapped actions. A task's source can contain multiple mixture components, and the policy continues to condition action generation on the current observation and instruction. We have clarified these distinctions in §§3.1 and 4.2 of the revised paper.

To directly examine the openness of this construction, we added three complementary studies during rebuttal.

1. Learning new objectives. We added unseen LIBERO-90 objectives to policies trained on LIBERO-10. TOPS appends a source/schedule entry and adapts the shared policy; FM receives the same demonstrations and adaptation budget. Evaluating the expanded task set connects acquisition of the new objective to performance on existing skills. Entries below are success percentages, shown as FM / TOPS.

Added objectiveNew taskExisting tenAll eleven
Milk → basket38.0 / 48.039.0 / 56.538.9 / 55.7
Book → front compartment44.0 / 42.033.5 / 55.034.5 / 53.8
Turn on stove and place pan36.0 / 56.043.0 / 48.042.4 / 48.7
Turn off stove40.0 / 64.053.0 / 56.551.8 / 57.2
Mean of displayed cases39.5 / 52.542.1 / 54.041.9 / 53.9

TOPS achieves a higher eleven-task macro-average in every displayed expansion, with an average 13.0-point gain on new tasks and 12.0-point gain across the expanded set. The same strategy extends to π₀.₅: with 20 demonstrations per new goal, TOPS achieves 76% and 98% on book-to-left-compartment and pudding-to-left-of-plate, compared with 70% and 88% for FM. Thus, task-specific modeling supports a library that grows as demonstrations become available. We have added these expansion results to §5.3, with the four LIBERO-10 cases in Table 2 and their experimental design in Appendix A.13.

2. Reusing existing sources for new goals. We also examined two illustrative π₀.₅ goals held out from downstream training and source construction. With the policy and library frozen, the selector retrieves an existing source/schedule while the VLA receives the original new-goal instruction. Reusing cheese-to-basket for cheese-to-bowl yields 60% success versus FM's 46%; reusing bowl-from-stove-to-plate for bowl-to-stove yields 94% versus 98%. These examples show that a source's training objective need not fix the executed goal: observation and language conditioning can guide an existing prior toward a different objective. We have included the new case studies and their evaluation design in Appendix A.13.1, Table 21, for your reference.

3. Replacing discrete indexing with continuous conditioning. Your concern about a closed task set also prompted us to examine whether discrete indexing is essential to TOPS. To test this, we implemented a continuous-conditioning prototype during rebuttal: shared density heads predict source and target statistics directly from visual and language context, allowing the source and OT schedule to be constructed without task-ID lookup or per-task EM. We then examined whether the scheduling benefit survives this change in representation. With the continuous source held fixed, adding the conditional schedule and its matched update raises success from 58.5% to 69.5%. These results show that TOPS's geometric construction can operate directly on context while retaining a benefit from scheduling. We have added the prototype design and controlled comparison to Appendix A.14, Table 22, with a summary in §5.3, for your reference.

AuthorsOpenReview ↗

Response to reviewer GC4q (2/3)

CW2/Q1: Joint design of scheduling and integration, and their respective contributions

Thank you for highlighting the relationship between scheduling and integration. Your question prompted us to examine their contributions more directly and make the overall design clearer. TOPS jointly designs the training path and its associated inference update: the schedule-aware integrator is an integral component of our method, derived from the temporal structure of the training path. It analytically incorporates the known schedule while using network evaluations to refresh the estimated displacement, connecting the training construction to efficient action generation.

The original Table 6 was designed to evaluate the contribution of this inference component. It holds the TOPS checkpoint, source distributions, and learned velocity field fixed, and compares integration rules at equal network-evaluation budgets. The observed differences demonstrate the importance of exploiting the schedule’s structure for few-evaluation inference. This experiment therefore provides direct evidence for the role of the associated integrator within TOPS’s joint design.

Your question also motivates a complementary investigation: how much does scheduling improve learning before numerical integration is involved? During rebuttal, we added solver-free regression experiments to answer this question, together with fixed-field numerical analyses and closed-loop stage interventions to explain the integrator’s contribution. These studies distinguish the learning benefit of the path from the numerical and execution benefits of its associated update.

Scheduling improves prediction before any inference solver is involved. We compared linear, slow-then-fast, and reversed schedules in matched velocity-regression problems without observation inputs or ODE integration. Slow-then-fast scheduling reduces prediction error by 22.5% on expansion directions and 14.3% across the full action chunk, with improvements on all ten tasks. The reversed schedule instead increases expansion-subspace error by 28.6%. The opposite effects establish a learning-side contribution from the timing of expansion, independently of the inference algorithm. We have added this comparison and its experimental design to Appendix A.9.2, Table 19.

The integrator exploits known structure in that path. Writing St=(1−t)e−tKS_t=(1-t)e^{-tK} and Gt=−∂tStG_t=-\partial_tS_t, a velocity query estimates displacement as Gt−1vθG_t^{-1}v_\theta. Integrating the known schedule over a subinterval gives xb=xa+(Sa−Sb)Ga−1vθ(xa,a,c).x_b=x_a+(S_a-S_b)G_a^{-1}v_\theta(x_a,a,c).

TOPS uses two consecutive half-interval updates, refreshing the displacement estimate at the intermediate state. To make this connection easier to follow, we have brought it into §4.4 of the revised paper; the full derivation remains in Appendix A.10.

To quantify how this update improves numerical accuracy, we fixed the learned field and compared numerical endpoints against a converged reference. At the same small budget, the schedule-aware update reduces median endpoint RMS error from 0.06079 with midpoint to 0.01956, a 67.8% reduction; Euler's error is 0.09063. This comparison shows how the schedule-aware update converts the path’s known temporal structure into greater numerical accuracy at the same evaluation budget. Together with Table 7, it connects the integrator’s numerical advantage to its contribution to task execution. We have added the new numerical analysis and reference-solution comparison to Appendix A.4.1, Table 8.

We then crossed the intermediate query construction and endpoint update to examine their roles in execution:

Second-query positionEndpoint updateSuccessful episodes
Euler half-stepMidpoint0/20
Schedule-aware half-stepMidpoint5/20
Euler half-stepSchedule-aware11/20
Schedule-aware half-stepSchedule-aware18/20

Both replacements recover successful executions, and the complete combination performs best. On states encountered during control, median endpoint RMS error on the executed action slice also falls from 0.0572 to 0.0159; mean position-command changes decrease from 1.64 to 0.81 cm. These diagnostics connect the numerical improvement to more consistent manipulation behavior. We have included the new stage interventions and execution analysis in Appendix A.4.1, Tables 9–10, together with their experimental descriptions.

Together, these results establish the complementary contributions of TOPS’s joint path–integrator design: scheduling improves velocity prediction before any inference solver is used, and the associated update realizes the learned transport more accurately with few network evaluations. The schedule-aware update exploits its known structure to deliver efficient few-evaluation generation. We have made this connection explicit in §4.4 and incorporated the additional evidence into Appendices A.9.2 and A.4.1.

AuthorsOpenReview ↗

Response to reviewer GC4q (3/3)

Q2: Multimodal actions and long-horizon execution

Thank you for asking about multimodal behavior and long-horizon tasks. Your question encouraged us to connect the source-and-schedule mechanism to executable behavior more directly, through new multimodal experiments and a clearer explanation of sequential execution.

A structured source can preserve multiple behaviors under the same condition. For a conditional flow map Φθ,c\Phi_{\theta,c} and mixture source qτ=∑mπτmqτmq_\tau=\sum_m\pi_{\tau m}q_{\tau m},

= \sum_m \pi_{\tau m} (\Phi_{\theta,c})_{\\#} q_{\tau m}.$$ Different source samples can therefore produce different action sequences under identical observation and language conditioning. A nonlinear flow can transform variation within each component into multiple behaviors, so components need not correspond one-to-one to behavioral modes. Slow-then-fast scheduling complements this structure by delaying expansion until intermediate states carry more information about the target endpoint. To directly address your question about multimodal actions, **we added two 2D closed-loop experiments during rebuttal**, with balanced demonstrated routes and no route label in the policy inputs. In the two-route obstacle task, **TOPS repeatedly completes both routes in every test scene**, demonstrating conditional multimodality. The four-opening task then tests whether richer behavior modes are covered fully and evenly. | Clearance | Four-route coverage: FM | Four-route coverage: TOPS | Balanced coverage: FM → TOPS | | --- | --- | --- | --- | | Wide | 0/20 | **8/20** | 0.015 → **0.145** | | Medium | 0/20 | **9/20** | 0.015 → **0.160** | | Narrow | 0/20 | **10/20** | 0.015 → **0.160** | Repeated coverage requires every route to succeed at least twice; balanced coverage measures the least represented successful route, with all attempts retained in the denominator. At the same few-evaluation budget, TOPS covers all four modes repeatedly in **8–10 scenes**, while FM meets this criterion in none. The balanced-coverage gains are positive across all three clearances. The new trajectory visualization makes this difference visible: FM's successful trajectories concentrate in the central openings, while TOPS also covers both outer routes. We have added both experiments, their metric definitions, and the trajectory comparison to **Appendix A.15, Table 23 and Figure 11**, so you can inspect how the measured coverage corresponds to executed behavior. **Long-horizon execution combines chunk structure with repeated feedback.** Source covariances span action chunks, representing coordinated variation across time and action dimensions. The controller then regenerates actions from updated observations, and successive instructions select the source and schedule for each subtask. This connects task-specific generation to sequential execution from states produced by earlier actions. The existing CALVIN results support this intuition: with π₀.₅, TOPS raises average chain length from **3.033 to 3.744** and five-task completion from **41.2% to 61.1%**. The supporting chain-depth results are reported in **Appendix A.1.1, Table 4**. Within a future TAMP system, a higher-level planner could select and order subgoals, while TOPS supplies the conditional action-generation module. We appreciate your questions for motivating these experiments and clarifications. The revision addresses each aspect as follows: - **CW1:** New expansion, prior-reuse, and continuous-conditioning studies demonstrate ways to extend task-specific modeling. They are included in **§5.3 and Appendices A.13–A.14**, alongside clarified definitions in **§§3.1 and 4.2**. - **CW2 / Q1:** New regression and fixed-field experiments separate scheduling's learning benefit from the integrator's numerical and execution benefits. They appear in **Appendices A.9.2 and A.4.1**, with the path–update connection clarified in **§4.4**. - **Q2:** New 2D tasks and trajectory visualizations demonstrate conditional multimodality and more balanced valid-mode coverage in **Appendix A.15**. The long-horizon explanation above connects these mechanisms to the existing CALVIN evidence in **Appendix A.1.1**.
Reviewer_aFN6OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses multi-task robot manipulation policies trained via flow matching. Standard flow-matching policies typically share a single isotropic Gaussian source prior and a uniform linear interpolation path across all tasks, which struggles with heterogeneous task action distributions and necessitates multi-step ODE integration during inference. The authors introduce TOPS (Task-Oriented Priors and Schedules), a framework that adapts both the source prior and the interpolation schedule to individual tasks without altering the core policy network architecture. Specifically, TOPS: (1) constructs multimodal task-specific source priors by fitting Mixtures of Probabilistic PCA (MPPCA) to reverse-mapped expert action sequences generated by a warm-up policy; (2) analyzes the source-to-target covariance mismatch using a Gaussian Optimal Transport (OT) map to identify "expansion directions" where expert actions have larger variance than source samples; (3) designs a direction-dependent "slow-then-fast" interpolation schedule along expansion directions while maintaining linear paths elsewhere, mathematically showing that this schedule lowers an upper bound on velocity-prediction risk under a Gaussian endpoint model; and (4) introduces a schedule-aware two-stage midpoint integrator enabling single-step action generation (NFE=2). Evaluated on LIBERO-10, LIBERO-Plus, and CALVIN (D->D and ABC->D), TOPS consistently outperforms standard flow matching and existing prior/schedule baselines (raising LIBERO-10 success from 53.44% to 69.67% while cutting inference time to ~1/6th). It also demonstrates successful transfer to the large-scale π0.5 model and validates improved performance on 8 physical tasks using a Galaxea R1 Lite robot arm.

Critical strengths
  1. Principled Theoretical Formulation and Substantial Inference Acceleration: The paper provides a rigorous mathematical bridge between Optimal Transport geometry, velocity-prediction risk bounds, and flow-matching policy scheduling. Rather than relying on empirical heuristics, the authors demonstrate analytically that slowing interpolation along expansion directions directly mitigates irreducible velocity-prediction errors (Section 4.3 and Appendix A.9). Coupled with the schedule-aware midpoint integrator, TOPS enables 1-step action generation (NFE=2) that reduces inference latency to ~1/6th of standard FM (from 49.4 ms to 8.2 ms on LIBERO) while outperforming 10-step Euler integration, resolving a critical real-time bottleneck for flow-based robot policies.

  2. Comprehensive Benchmark Evaluation and Large-Scale Foundation Policy Transfer: The empirical validation is thorough and multifaceted. Beyond standard multi-task benchmarks (LIBERO-10, CALVIN D->D and ABC->D), the paper rigorously investigates out-of-distribution robustness on LIBERO-Plus (2,100 variants), demonstrates superior data efficiency (achieving better results with 30% demonstrations than baselines with 50%), successfully scales onto the pretrained π0.5 foundation model with consistent gains (Figure 4), and validates physical deployment on an actual Galaxea R1 Lite manipulator across 8 distinct tasks with 160 real rollouts.

Critical weaknesses
  1. Disconnect Between Theory and Empirical Gains on OOD Benchmark (LIBERO-Plus): A critical discrepancy emerges when examining the OOD results in Table 1. On LIBERO-Plus, the ablation baseline modifying only the source distribution (Source only with fixed M) achieves 35.21 ± 0.66%, while the full TOPS (incorporating both task source and non-linear expansion scheduling) reaches 35.29 ± 0.08%. The net gain provided by the direction-dependent schedule under OOD perturbations is merely 0.08 percentage points, which is entirely within random evaluation noise. While the schedule provides a dramatic +6.5% boost in-distribution (LIBERO-10), the theoretical advantage of expansion scheduling does not materialize under environmental perturbations. The paper does not adequately discuss why scheduling benefits disappear out-of-distribution.

  2. Reliance on Predefined Task Clusters and Discrete Routing Bottleneck: TOPS requires a non-trivial offline pipeline: pre-training a warm-up policy for 300 epochs, running ODE reversal to populate noise banks, fitting per-task MPPCA mixtures, and training a text-based instruction selector (MiniLM-MLP) to route language commands to discrete task IDs. This framework fundamentally assumes a closed set of cleanly partitioned task identities. In open-vocabulary robot manipulation, ambiguous language prompts, or long-horizon continuous tasks where skills transition smoothly without clear task boundaries, mapping to a static, discrete task source becomes non-trivial and may create routing bottlenecks.

Questions for authors
  1. Lack of Scheduling Gains on LIBERO-Plus: In Table 1, full TOPS (35.29%) outperforms the Source-only variant (35.21%) by only 0.08 percentage points on LIBERO-Plus, whereas on LIBERO-10 the gain is +6.5% (69.67% vs 63.17%). Why does the expansion schedule fail to provide meaningful improvements under visual and dynamic perturbations? Does the source-target covariance mismatch shift under environmental shifts, rendering the offline OT eigenvectors suboptimal?

  2. Failure Modes of the Instruction Selector: How robust is the MiniLM-MLP instruction selector when encountering paraphrased, open-vocabulary, or ambiguous user instructions? If an instruction is misclassified to an incorrect task prior, how severely does rollout performance degrade compared to standard task-agnostic flow matching?

  3. End-to-End Amortization: Could the task-oriented prior and schedule parameters (μ_τ, Σ_τ, ρ_τ) be parameterized continuously as functions of language/visual embeddings, thereby bypassing the need for discrete task indexing and offline EM fitting?

3 comments
AuthorsOpenReview ↗

Response to reviewer aFN6 (1/3)

Thank you for your thoughtful review and for recognizing TOPS's theoretical motivation, inference efficiency, and empirical evaluation. To address your questions directly, we conducted additional experiments during the rebuttal period on OOD mechanisms, language-driven execution, and alternatives to discrete task conditioning. We have added these experiments and their supporting details to the revised paper, with specific locations provided below for your reference.

CW1 / Q1: Why is the scheduling gain small on LIBERO-Plus?

Thank you for highlighting this contrast. Your question prompted us to investigate more closely how the source and schedule contribute under distribution shift. The small incremental gain on LIBERO-Plus also reveals the contribution of TOPS's other core mechanism: the task-specific source prior accounts for nearly all of the aggregate OOD improvement, raising success from 27.11% to 35.21%, compared with 35.29% for full TOPS. Once selected, this prior supplies task-relevant structure that does not directly change with brightness or background appearance.

Visual conditioning can dominate the benefit of improved path geometry. To investigate the mechanism behind your OOD observation, we added a controlled visual intervention during rebuttal, perturbing only the images while fixing the source sample, expert action, and schedule. Under brightness ×0.4, the converged ODE endpoint's squared deviation from the expert action rises from 0.0086 to 0.2643, whereas the few-step solver's squared error relative to that reference rises only from 0.0041 to 0.0081, both measured per dimension in the scheduled subspace. Blur produces the same pattern. The solver continues to follow the learned field closely, but the field generates a different endpoint when its visual condition changes. We have added this diagnostic, its metric definitions, and the full results to Appendix A.12, Table 20, for your reference.

This provides a concrete explanation for the small OOD increment: improving the transport path cannot by itself correct an observation-conditioned prediction that moves away from the desired action. Regarding your covariance question, perturbations that change the required actions can also change the appropriate OT directions. Our visual intervention identifies an additional mechanism that operates even with fixed action geometry. The Gaussian risk analysis addresses the velocity-prediction problem induced by the path; the new diagnostic explains why its benefit need not translate proportionally into task success under observation shifts.

The scheduling mechanism remains effective in controlled comparisons. To further address the connection between theory and empirical gains, we added a velocity-regression comparison without observation inputs or inference integration. With the source and regression setup matched, slow-then-fast scheduling reduces expansion-subspace error by 22.5% and full-space error by 14.3%, improving both metrics on all ten tasks. Reversing the timing instead increases expansion-subspace error by 28.6%. These opposite effects show why the direction of scheduling matters. The new comparison and its experimental design are included in Appendix A.9.2, Table 19.

The generation results complete this picture. In the existing 2D toy experiment, slow-then-fast scheduling with its associated integrator raises target-region capture from 84.16% to 98.57% at the same evaluation budget; this comparison and its setup are documented in Appendix A.7, Table 14. We additionally tested numerical accuracy on a fixed robot-policy field during rebuttal: the schedule-aware solver reduces median endpoint RMS error from midpoint's 0.06079 to 0.01956. We have added this analysis to Appendix A.4.1, Table 8. Together, these results explain the complementary contributions: the task-specific source preserves most of the reported LIBERO-Plus gain, while scheduling improves velocity prediction and, with its matched update, few-step generation.

AuthorsOpenReview ↗

Response to reviewer aFN6 (2/3)

CW2 / Q2: Language robustness and the flexibility of task conditioning

Thank you for raising the practical questions of instruction handling and task-library flexibility. Your feedback helped us connect language recognition more directly to policy execution and skill acquisition. We designed new language and task-expansion experiments during rebuttal to examine both new expressions of known goals and new objectives.

New wording can be handled without requiring fixed instruction strings. We find that keeping the selector architecture unchanged, semantic and spelling augmentation raises held-out identification accuracy from 92.83% to 97.58% on LIBERO and from 58.96% to 84.13% on CALVIN. We have added the independent language evaluation, augmentation design, and results to Appendix A.8.1, Table 16, so you can examine this test of unfamiliar wording in detail.

To connect recognition to execution, we tested a canonical-conditioning interface: the predicted task selects its source, schedule, and training-time language embedding. FM uses the same language frontend. This lets us evaluate TOPS after actual text-based selection while keeping the frontend comparison matched.

Instruction setFM successTOPS successGain
Unseen wording63.3%75.0%+11.7 pp
2% spelling perturbation63.3%73.3%+10.0 pp
5% spelling perturbation61.7%70.8%+9.2 pp
Native LIBERO-Plus language variants62.7%72.2%+9.5 pp

As results showedTOPS retains its execution advantage across all four instruction sets, with all paired confidence intervals above zero. Moreover, every correctly routed rollout in these tests reproduces its oracle-conditioned action sequence exactly. Thus, the interface preserves generation behavior once the intended task is identified. We have included the new closed-loop evaluation and trajectory-consistency checks in Appendix A.8.2, Table 17, together with the interface description.

Incorrect routing can disrupt execution by changing both the source and schedule. We therefore tested whether better recognition recovers performance while keeping the policy and interface fixed. Replacing the original selector with the augmented version more than halves routing errors and raises success on native language variants from 67.3% to 72.2%. This connects improved language recognition directly to a closed-loop benefit. Genuinely ambiguous requests require clarification of the intended goal. Selection runs once per instruction, with measured CPU overhead of approximately 9 ms, or 0.2 ms with a cached embedding; source fitting and static OT remain offline. We have added the selector-replacement experiment and overhead analysis to Appendix A.8.3, Table 18.

The task library can also grow. We added unseen LIBERO-90 objectives to a LIBERO-10 policy using new demonstrations and shared-policy adaptation, with matched data and update budgets for FM and TOPS. Success rates below are percentages, shown as FM / TOPS.

Added objectiveNew taskAll eleven tasks
Milk → basket38.0 / 48.038.9 / 55.7
Book → front compartment44.0 / 42.034.5 / 53.8
Turn on stove and place pan36.0 / 56.042.4 / 48.7
Turn off stove40.0 / 64.051.8 / 57.2
Mean of displayed cases39.5 / 52.541.9 / 53.9

TOPS achieves stronger expanded-set performance in all four displayed cases, demonstrating a practical route for adding skills to the library. This also extends to π₀.₅: with 20 demonstrations per new goal, TOPS achieves 76% and 98% on placing a book in the caddy's left compartment and pudding to the left of a plate, versus 70% and 88% for FM. We have incorporated these new expansion results into §5.3, with the four LIBERO-10 expansions in Table 2 and their experimental design in Appendix A.13. These results address both flexibility in expressing a known task and supervised expansion to new objectives.

AuthorsOpenReview ↗

Response to reviewer aFN6 (3/3)

Q3: Can the source and schedule be conditioned continuously?

Thank you for suggesting continuous parameterization. This encouraged us to investigate how TOPS's geometric construction can operate directly on context. In direct response, we implemented and evaluated a continuous-conditioning prototype during rebuttal. Shared density heads predict source and expert-action distributions directly from visual and language context. Their conditional moments determine the OT schedule, replacing task-ID lookup and per-task EM. The density heads are fitted first and held fixed during policy adaptation.

With the same continuous source, adding the conditional schedule and its matched update raises LIBERO-10 success from 58.5% to 69.5%, a gain of 11.0 pp with a paired 95% CI of [4.0, 17.5]. This demonstrates that the scheduling benefit persists when the geometric statistics come from shared conditional models. Per-context OT computation increases latency from 12.9 to 23.8 ms. We have added the prototype construction, controlled comparison, and latency results to Appendix A.14, Table 22, and summarize the extension in §5.3.

Thank you again for motivating these additions. The revised paper addresses the three aspects of your review as follows:

  • CW1 / Q1: The new visual diagnostic explains how observation shifts can limit scheduling gains, while regression and numerical analyses establish its prediction and generation benefits. These additions appear in Appendices A.12, A.9.2, and A.4.1.
  • CW2 / Q2: New language evaluations connect recognition to reliable execution, and supervised expansion demonstrates a growing task library. These studies are included in Appendices A.8.1–A.8.3 and A.13, and §5.3.
  • Q3: The continuous prototype demonstrates source-and-schedule conditioning without task-ID lookup. Its design and results are included in Appendix A.14 and §5.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and the major changes of revised manuscript

Dear Reviewers,

Thank you for your careful reviews and constructive suggestions. We particularly appreciate your recognition of TOPS’s theoretical foundation, design, and practical value: reviewer aFN6 highlighted “Principled Theoretical Formulation and Substantial Inference Acceleration”; reviewer GC4q noted that “The paper presents both strong empirical validation and theoretical proofs”; and reviewer yxod stated that “The joint adaptation of task-specific sources and transport schedules is a clear and useful view.” We also appreciate your recognition of the backbone-preserving design, broad empirical evaluation, and transfer to π₀.₅ and real robots.

Your suggestions encouraged us to investigate the contributions of TOPS’s components more closely and examine its flexibility across language variations, new objectives, and multimodal behaviors. During the rebuttal period, we conducted targeted experiments and mechanism analyses specifically to address these questions. We have incorporated the key findings into the main text and supporting experiments into the appendix, with additions and revisions highlighted in blue. The principal changes are summarized below:

  1. Task definitions and library extensibility. We clarified task and source definitions and added four LIBERO-10→11 expansions, π₀.₅ new-goal adaptation, and configuration-reuse examples. Across the four expansion cases, TOPS achieves 53.9% expanded-set success versus 41.9% for FM, demonstrating task-library growth with additional demonstrations. These studies appear in §5.3 and Appendix A.13.

  2. Continuous source and schedule conditioning. We implemented a prototype that constructs sources and schedules directly from visual and language context. With the same continuous source, scheduling improves success from 58.5% to 69.5%, demonstrating that the mechanism remains effective without discrete task indexing. Details appear in §5.3 and Appendix A.14.

  3. Scheduling and integration contributions. New solver-free regression and fixed-field analyses show 22.5% lower expansion-subspace prediction error from scheduling and 67.8% lower median endpoint error relative to midpoint from the associated update. Their mathematical connection and supporting analyses appear in §4.4, Appendices A.9.2 and A.4.1.

  4. Mechanistic interpretation of OOD performance. We clarified the complementary roles of source and schedule and added visual interventions with fixed action geometry. These distinguish observation-conditioned prediction changes from integration error, deepening the interpretation of LIBERO-Plus performance. The discussion and analysis appear in §5.1 and Appendix A.12.

  5. Language robustness and closed-loop execution. New wording, spelling-perturbation, and selector-replacement experiments connect improved recognition to execution. Under a shared language frontend, TOPS exceeds FM by 9.2–11.7 percentage points across all four instruction sets. These studies appear in Appendices A.8.1–A.8.3.

  6. Multimodal behavior coverage. New two-route and four-opening tasks test distinct executable behaviors under identical conditioning. The results and trajectory visualizations demonstrate fuller and more balanced valid-route coverage with TOPS. These additions appear in Appendix A.15, Table 23, and Figure 11.

Thank you again for your time and thoughtful feedback. Your questions helped us turn important design considerations into directly testable experiments, strengthening the revised paper’s explanation and empirical support for TOPS’s mechanisms, flexibility, and practical value.

Best regards,
Authors of Submission 14907

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.