acceptodds
Under review as a conference paper at ICLR 2027

MOSAIC: Multi-task Orchestration via Sparse Atomic Interpolated Control

15 views

Abstract

Multi-task robot learning is not limited only by policy capacity, but also by the structure through which tasks share control. Dense sharing couples incompatible task dynamics in a common pathway, while expert isolation reduces transfer and scales poorly. We introduce MOSAIC, a sparse basis-sharing framework for scalable multi-task manipulation. MOSAIC learns a shared low-coherence dictionary of atomic control bases and infers, for each context, a sparse set of active bases with continuous coefficients through a deterministic expected spike-and-slab bottleneck. The resulting active basis span allows related task phases to reuse control factors while reducing update coupling among incompatible dynamics. To avoid mode averaging in continuous mixtures, MOSAIC aligns dictionary bases with temporal options and uses continuous skill coordinates to modulate low-level control. Across Meta-World MT-10 and MT-50, MOSAIC achieves strong multi-task performance, improves hard contact-rich tasks, and preserves single-pass closed-loop execution efficiency. Mechanism-level analyses, including ablations, sparse-routing diagnostics, update-coupling measurements, and basis interventions, support that the gains arise from structured selective sharing rather than increased capacity or task-specific parameter isolation. All code, dataset, and reproducibility scripts will be released.

Primary area:
reinforcement learning
Keywords:
Multi-task reinforcement learning, robotic manipulation, sparse representation learning, compositional skills, hierarchical reinforcement learning, negative transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wh3zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Multi-task Orchestration via Sparse Atomic Interpolated Control (MOSAIC), an approach to multi-task learning that attempts, to automatically learn options (sequences of actions) that can be re-used across a multi-task learning setup, minimizing negative transfer and encouraging positive transfer across tasks with a shared structure or with similar requirements (for example, two tasks both require swiping from left-to-right, so the model could learn this atomic skill that can be activated at the right time for both tasks, while separately learning a right-to-left swipe that doesn't conflict or overwrite previous behaviors).

Critical strengths
  • MOSAIC is compared comprehensively to existing strategies, such as naive multi-task learning, or gradient regularization techniques. The results show that MOSAIC learns much more robust policies that achieve significantly more consistent and reliable performance gains relative to prior work.
  • The paper is clearly written and easy to understand, despite containing a fair degree of complexity for the final system. I feel that most readers will be able to follow this work, and to therefore benefit from and built off of it.
  • The experiments and results present findings both on final performance in terms of success rates and also on related metrics like skill overlap and runtime efficiency. This helps to create a more complete picture of how MOSAIC performs relative to baselines, and the metrics generally show that MOSAIC performs on-par or better than all baselines while staying very competitive in terms of runtime and efficiency.
  • An ablation study helps to clearly demonstrate the contributions of each component in isolation.
Critical weaknesses
  • MOSAIC is a fairly complex system, so much so that significant portions of the paper have been relegated to the Appendix. While some of these elements may be part of the "usual RL pipeline", it can leave the paper in an awkward spot. For example, much of Figure 2 is actually not represented in the main text (the entire bottom half of the figure). While I finished the paper and had a good sense for what happened and how it all worked together, I would struggle to actually reproduce the results from the main text alone. The authors state that code will be released, which does help mitigate this concern.
    • A related weakness of the paper is the complexity itself. The forefront of robotics is centered around scale and VLAs, and a quite complex bespoke system is unlikely to have significant impact on that forefront, as it is very unclear how such a system would apply to these state-of-the-art policies and architectures. I do not believe this is sufficient cause for rejection, but it does limit the impact of the work.
  • The basis and option usage diagnostics are not very well explained, and it is not clear how to read these results. The metrics are not introduced, the rows are not really introduced, separated out, or explained. These results seem very helpful for contextualizing the claims of the paper around things like separation of bases, re-usability of skills, and contextualization of skills, so the table deserves more explanation.
  • The paper uses "spike-and-slab" repeatedly in the first few pages, but this phrase is not defined or explained until page 5. I don't know if it is strictly necessary everywhere that it is currently used, and the paper could be easier to read with more readily understandable text.
Questions for authors
  • Section 3 introduces the task definition vector as a language instruction, but states that it could also be a task index. A task index would likely be much more computationally demanding for learning the relevant bases, and would make generalization very implausible. Are there any experiments on MT-10 or MT-50 with task indices, or was the entire paper done with language vectors? And how were the language vectors generated?
Reviewer_J52DOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MOSAIC addresses interference in multi-task robot learning through a shared dictionary of control bases, context-dependent sparse coefficients, and basis-aligned temporal options. The paper reports strong performance on Meta-World MT-10 and MT-50, supported by component ablations and routing diagnostics.

Critical strengths
  • The approach is principled and well motivated. Selective sharing directly addresses the tension between reusing common control structure and avoiding destructive interference. The theoretical analysis provides a relevant rationale for the architecture, although its correspondence to the complete implementation needs clarification.
  • The empirical analysis is relatively comprehensive, covering benchmark comparisons, component ablations, basis-usage patterns, and execution efficiency. The reported performance improvements are substantial.
  • The hierarchy-only + FiLM baseline is a useful control. It tests whether the gains come simply from adding temporal options and feature modulation, helping assess the specific contribution of sparse dictionary routing.
Critical weaknesses
  • Scaling to hundreds or thousands of tasks is unclear. A central expectation is that overlapping tasks increasingly reuse existing bases, allowing dictionary size to grow sublinearly and potentially stabilize within related task families. Using 12 bases for MT-10 and 60 for MT-50 does not demonstrate this benefit.
  • Scene complexity is insufficiently explored. Clutter, occlusion, and multiple interacting objects could make perception and routing bottlenecks, even if dictionary capacity is sufficient.
  • The termination formulation appears problematic. Eqs. 16 and 21 make the termination advantage always negative for positive temperature, so Eq. 22 encourages termination at every step, conflicting with temporal persistence.
  • Some reported metrics require reconciliation. Table 2’s 99.1% success and 7.5% interference gap appear incompatible under the stated definitions and common aggregation. Table 8’s aggregate standard deviations also appear to average per-task deviations rather than measure variation in aggregate performance across seeds.
  • The coupling analysis is incomplete for the implemented architecture. Eq. 8 omits the additional gradient path through the selected basis supplied directly to FiLM.
  • References are duplicated and inconsistently formatted. GradNorm and Hard Tasks First each have both an incomplete entry and a separate entry with publication details.
Questions for authors
  • Dictionary scalability. Would you expect dictionary size to grow sublinearly with task count as more tasks reuse existing bases? What would enable scaling to hundreds or thousands of tasks, and what would limit this reuse?
  • Complex scenes and task conditioning. How would clutter and occlusion affect routing when relevant control factors are difficult to infer from images? Can task conditioning distinguish different tasks in the same scene? An explanation or preliminary evidence would help.
  • Temporal persistence. What prevents the termination objective from encouraging termination at every step, and what option durations are learned?
  • Metric consistency. Can the authors reconcile the reported interference gaps with their definition and clarify how aggregate standard deviations were computed?
  • Coupling analysis. Does the conclusion in Eq. 8 still hold when the direct basis-to-FiLM gradient path is included?
Reviewer_VVb8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MOSAIC, a reinforcement learning framework for multi-task robotic manipulation. It aims to address two well-known challenges in multi-task learning: dense parameter sharing tends to induce negative inter-task transfer, while independent expert architectures fail to adequately enable knowledge reuse. The authors introduce a method combining a shared skill dictionary, sparse skill composition, and hierarchical control, allowing the robot to infer context-dependent sparse continuous coefficients over a shared dictionary of atomic skill bases across different tasks and execution phases, and to generate actions through a hierarchical policy.

Critical strengths
  1. The research problem is well-motivated. Sharing representations across tasks without causing destructive interference is a valuable, long-standing question.
  2. The paper includes detailed mechanistic analysis experiments.
Critical weaknesses
  1. The paper repeatedly emphasizes that related task phases should reuse overlapping bases, whereas incompatible dynamics are encouraged to induce low-overlap active supports. Although Eq. (8) connects coefficient overlap to dictionary-update coupling, the structural regularizers do not explicitly encode which contexts are compatible or should share bases.
  2. Regarding writing quality: the presentation is notably unclear. The full loss formulation and optimization details are deferred to Appendices A and B.1, making the joint optimization of the modules difficult to follow from the main text. Additionally, there are inconsistencies between symbols used in figures and those in the text. For example, the language encoder is denoted by gθg_\theta in Eq. (2) but by geg_e in Figure 2.
Questions for authors
  1. Does the near-orthogonality regularizer primarily encourage separation among directions in the learned dictionary, or does it also lead to distinct functional control roles after nonlinear modulation and policy decoding? The near-orthogonality regularizer acts on the learned dictionary, yet subsequent nonlinear control mappings may re-entangle these directions. The existing analyses (task–basis usage heatmaps, task–task support-overlap matrices, and temporal basis-activation visualizations) primarily demonstrate that different tasks exhibit structured basis usage patterns. They do not directly verify that these basis vectors possess distinct, interpretable, and reusable control roles, nor do they sufficiently establish that meaningful compositional relationships exist among them.
  2. The paper defines a basis-aligned option set in Section 3.4, with options in one-to-one alignment with the dictionary columns. However, each option comprises a specialized low-level policy and a termination condition, with option-specific control implemented through FiLM modulation of a shared backbone, and its action generation depends on both the corresponding skill basis and a skill representation composed of multiple basis elements. It therefore remains unclear whether each option truly learns an atomic control function aligned with its corresponding skill basis, or whether it may nevertheless learn complex behaviors spanning multiple control phases. Beyond index alignment and FiLM conditioning on the selected basis, what mechanisms or functional evidence support an interpretable alignment between individual bases and option behaviors?
  3. In the main results on MT-10, MOSAIC's success rate curves appear remarkably smooth across most tasks, whereas the Pick-Place task exhibits noticeable fluctuations. Although the paper reports the evaluation protocol and aggregation across random seeds, it does not clearly specify whether temporal smoothing was applied to the learning curves or whether the same plotting settings were used across tasks. These omissions make it difficult to assess training variability.
  4. A concern regarding the experimental setup: the paper introduces a near-orthogonality regularizer in Section 3.2. However, according to Table 6, MT-10 uses 12 skill basis vectors of dimension 10. Since the number of dictionary columns exceeds the dimensionality, these basis vectors are mathematically incapable of being mutually orthogonal, and the regularization objective therefore cannot be fully satisfied. Although the paper motivates this term as a low-coherence regularizer and reports stable performance in overcomplete settings in Appendix C.1, it remains unclear how effectively the learned dictionary achieves basis separation under this configuration. Could the authors report the attained orthogonality loss or the off-diagonal coherence of the learned dictionary to clarify the behavior of this regularizer under this overcomplete dictionary setting?

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly (if you haven't done so).

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.