acceptodds
Under review as a conference paper at ICLR 2027

Mixture of Primitives: Fine-Grained Expert Sharing for Generative Visual Perception in Unified Multimodal Models

Abstract

Unified multimodal models (UMMs) increasingly support dense visual perception through image generation, an emerging setting that we refer to as Generative Visual Perception (GVP). While GVP tasks share a common generative interface, they require distinct visual transformations to produce task-specific outputs, motivating task-dependent specialization while preserving shared representations. Mixture-of-experts (MoE) provides a natural mechanism for introducing this conditional specialization through routed experts, but conventional MoE parameterization couples routing decisions with expert-level parameter ownership, requiring each expert to maintain an independent copy of its full transformation. Such coupling restricts parameter ownership to the expert level. We propose Mixture of Primitives (MoP), a fine-grained expert-sharing mechanism that decouples routing identity from parameter ownership by selectively sharing internal channel-group primitives across experts, enabling fine-grained parameter sharing while preserving route-specific transformations. We mainly validate MoP on Cheers, a pretrained UMM, forming Vision-Cheers, which achieves competitive performance with task-specific models while retaining image generation and multimodal understanding capabilities, and further validate MoP on Bagel for cross-architecture generality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Model, Vision Perception

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7fiuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Mixture of Primitives (MoP), a parameter-sharing procedure for routed residual experts in unified multimodal models. After learning three disjoint experts, it aligns channel groups using response similarity, selects shared gate/up blocks using gradient compatibility, and recovers the consolidated model with task training and local distillation. Controlled Cheers and Bagel experiments report improved dense prediction with fewer residual parameters; a larger Vision-Cheers checkpoint also supports generation and understanding.

Critical strengths
  1. Separating routing identity from parameter ownership is a useful design perspective. The construction preserves sparse inference and permits sharing only selected internal blocks rather than requiring entire experts to coincide.

  2. The principal continuation comparison is substantially controlled: common source checkpoint, data, optimization, trainable modules, routing and distillation. In Table 2, MoP improves RefCOCO cIoU from 56.6 to 62.2 over continued disjoint training while reducing residual parameters from 1.734B to 1.414B. Random MoP has the same parameter budget and performs much worse.

Critical weaknesses

W1. The compatibility mechanism lacks evidence that its coordinates support the proposed merge. Appendix D, pp. 16–17, explicitly states that experts are independently initialized and that Hungarian matching only permutes whole 1,120-channel groups, retaining their internal order. Equations 3–4 then compare parameter-gradient coordinates, and consolidation averages those coordinates elementwise. Similar group outputs do not establish neuron correspondence inside a group: jointly permuting gate/up rows and down columns leaves a primitive's function unchanged but generally changes the gradient-vector comparison and its average with another expert. This does not prove that MoP fails empirically; it limits the interpretation of the selector as identifying reusable computations.

W2. The controls do not yet establish the value of the proposed granularity over informed sharing. Random MoP is informative, but Fixed Partial shares every group with a predetermined pair topology and has a different budget (Table 7). Table 12 varies granularity only through response statistics at one layer, not downstream performance. Thus the paper establishes an advantage over these particular assignments, but does not isolate the benefit over a function-informed whole-expert or simpler group-sharing alternative. This matters because expert alignment/merging is established prior work, including the already cited MC-SMoE, ICLR 2024.

W3. Several consequential protocol details remain unresolved. Table 4 gives global batch size 800 for both final-model stages, whereas Appendix J.4 gives 500 and explicitly reports 500,000 recovery sample presentations. These imply different training exposure. Moreover, NYUv2 uses ground-truth scale-and-shift alignment in disparity space (Appendix H.2); the external Table 1 depth scores therefore need protocol labels before being interpreted as comparable. The matched internal results remain useful. Random topology results are averaged over three seeds, but corresponding dispersion and MoP calibration/recovery variability are not reported.

Questions for authors

Please see weakness section.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 7fiu

We thank the reviewer for the detailed and insightful comments. Your feedback helped us clarify the methodological assumptions, strengthen the experimental comparisons, and improve the transparency of our evaluation protocols.

W1. Within-group coordinates and the basis for merging

Thank you for highlighting this distinction. Hungarian matching establishes group-level correspondence without guaranteeing within-group neuron alignment, so gradient comparison and parameter averaging remain coordinate-dependent.

However, group-level matching is only the starting point. During recovery, task supervision and Stage-1 residual distillation progressively adapt the tied channels to support compatible representations across routes.

We have clarified this interpretation in Section 3.4.1, Figure 2, and the appendix.

W2. Informed sharing baselines and granularity

(a) Topology variants and parameter budgets

Random MoP matches MoP's 1.414B residual parameter budget by preserving its sharing-pattern counts, whereas Fixed Partial uses predefined pairwise sharing across all groups, resulting in 1.349B parameters. Both serve as topology controls rather than granularity controls.

To assess fine-grained sharing against informed coarse-grained sharing, we add a whole-expert sharing baseline with a comparable parameter budget (1.404B vs. 1.414B). MoP outperforms this baseline on all four metrics, supporting the effectiveness of selective primitive-level sharing.

MethodResidual params (B)RefCOCO cIoU ↑RefCOCO gIoU ↑NYUv2 AbsRel ↓NYUv2 δ₁ ↑
Random MoP1.41438.738.410.788.5
Fixed Partial1.34927.227.711.387.4
expert-level sharing1.40458.059.311.586.8
MoP (G=4G=4)1.41462.263.310.388.9

(b) Downstream granularity experiments

Due to time constraints, we added downstream experiments with G=1,4,8G=1,4,8 to examine representative coarse, default, and finer grouping configurations. Among these settings, G=4G=4 achieves the best downstream performance.

GRefCOCO cIoU ↑RefCOCO gIoU ↑NYUv2 AbsRel ↓NYUv2 δ₁ ↑
159.161.210.588.7
462.263.310.388.9
857.057.811.386.9

(c) Informed expert-level and group-level controls

We added an MC-SMoE-inspired whole-expert sharing baseline. It first aligns groups by their responses, then uses full-expert output similarity on common calibration tokens to select 16 of the 28 layers, fairly matching MoP’s parameter budget (1.404B vs. 1.414B).

At each selected layer, it shares the most similar expert pair while retaining all three routing identities. The baseline retains 1.404B unique residual parameters, close to MoP's 1.414B. In contrast to whole-expert sharing, MoP selectively shares primitive gate/up parameters while retaining route-specific down projections.

MoP achieves RefCOCO cIoU/gIoU of 62.2/63.3 versus 58.0/59.3 for whole-expert merging, and NYUv2 AbsRel/δ1\delta_1 of 10.3/88.9 versus 11.5/86.8. These results show better downstream performance for MoP at a comparable parameter budget.

MethodResidual params (B)RefCOCO cIoU ↑RefCOCO gIoU ↑NYUv2 AbsRel ↓NYUv2 δ₁ ↑
Expert-level params sharing1.40458.059.311.586.8
MoP (gradient-based topology)1.41462.263.310.388.9

Parameter counts are unique residual parameters in billions; metrics are in percent.

AuthorsOpenReview ↗

Rebuttal for Reviewer 7fiu—Continued

W3. Experimental reporting and evaluation protocols

(a) Training batch size and sample exposure

Thank you for identifying this typo error. Table 4 is correct: both stages of the final Vision-Cheers model use a global batch size of 800. We have corrected Appendix J.4 to report a batch size of 800 and 800,000 recovery sample presentations, replacing 500 and 500,000, respectively.

(b) Comparability of depth evaluation protocols

Thank you for the suggestion. Our NYUv2 evaluation uses per-image least-squares scale-and-shift alignment to ground truth in disparity space before computing AbsRel and δ1\delta_1. We have labeled the evaluation protocols in Table 1(c). Vision-Cheers shares the disparity-space alignment protocol used by Depth Anything and Lotus-2, while supporting perception, generation, and understanding within a single generalist model.

Table 1(c): Monocular depth estimation on NYUv2

MethodAbsRel ↓δ₁ ↑
Task-specific models
Depth Anything†4.398.1
Depth Anything V2†4.597.9
Metric3D‡8.394.4
WorDepth‡8.893.9
DICEPTION§6.196.0
Lotus-2 (FLUX)†4.197.6
Generalist models
Vision-Cheers†8.593.2
Vision Banana‡8.194.8

† GT-aligned disparity-space evaluation; ‡ metric-depth evaluation; § GT-aligned depth-space evaluation.

(c) Random seeds and result variability

Thank you for pointing out the missing variability statistics. The table below reports the mean and sample standard deviation across three Random MoP runs. We have included these statistics in the manuscript.

MethodRefCOCO cIoU ↑RefCOCO gIoU ↑NYUv2 AbsRel ↓NYUv2 δ₁ ↑
Random MoP38.7 ± 1.138.4 ± 0.810.7 ± 0.288.5 ± 0.1

Mean ± sample standard deviation over three runs; all metrics are in percent.

For MoP, we use three random seeds to independently repeat calibration and recovery from the same Stage-1 checkpoint under identical settings. The results below report their variability.

MethodRefCOCO cIoU ↑RefCOCO gIoU ↑NYUv2 AbsRel ↓NYUv2 δ₁ ↑
MoP61.47 ± 0.6462.73 ± 0.5110.30 ± 0.1089.37 ± 0.50
Reviewer_7fiuOpenReview ↗

Thank you for the rebuttal.

Thank you for the response and additional experiments. The informed expert-level baseline at a comparable parameter budget and the downstream granularity study substantially address my concerns about the sharing comparisons. The batch-size correction, depth-protocol labels, and variability statistics also address the main reporting issues. Overall, the additional evidence strengthens the empirical contribution. I maintain my weak acceptance recommendation. Please incorporate these results and clarifications into the revised manuscript.

Reviewer_e7mCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Mixture of Primitives (MoP) shares weights across residual experts in a unified multimodal model. Stage 1 trains disjoint experts with separate weights. Stage 2 aligns their channel groups, called primitives, and uses update compatibility to select gate/up weights to share. Down projections stay separate, and recovery training adapts the model to sharing. On Cheers and Bagel, MoP improves segmentation and depth over continued-disjoint training while storing fewer parameters. The full Vision-Cheers model also supports image generation and multimodal understanding.

Critical strengths
  1. Table 2 compares MoP with continued-disjoint training from the same checkpoint, using the same data, training budget, and distillation. MoP improves segmentation and depth with fewer stored parameters. It also substantially improves segmentation over Random MoP, supporting the choice of which weights to share.

  2. The controlled improvement also holds on Bagel. The full Vision-Cheers model supports perception and multimodal understanding, and scores higher on GenEval than the published Cheers baseline. This is a useful result for extending a model to more tasks.

Critical weaknesses
  1. MoP improves perception quality, but its parameter saving is modest at the full-model level. Sharing also does not reduce inference FLOPs. The practical gain is therefore better perception with a modest reduction in stored parameters.
Questions for authors
  1. Equation 3 compares experts on the same calibration tokens, but shared weights are updated using tokens routed to any of their owners. What motivates using common-token gradients rather than routed-token gradients for ownership selection?
1 comment
AuthorsOpenReview ↗

Rebuttal for Reviewer e7mC

We thank the reviewer for the thoughtful feedback. Your suggestions helped us clarify the practical efficiency benefits of MoP and the motivation for common-token gradient probing.

1.Critical Weaknesses

Thank you for distinguishing parameter savings from inference cost. We have clarified the efficiency claims in the “Induced Ownership Topology” subsection: better perception performance and a 7.4% reduction in whole-model unique parameters in the 1-M analysis model relative to the disjoint-expert baseline, with unchanged nominal residual inference FLOPs.

2.Questions For Authors

We probe gradients on common tokens to compare candidate primitives' local update compatibility under identical inputs and upstream task gradients. Since the sharing topology remains fixed during recovery while the router continues to adapt, routed-token gradients depend on current token assignments and may be unavailable for experts receiving no tokens.

To examine this issue, we analyzed 198 calibration samples and found that 52 (26.3%) have a zero-token expert in at least one layer, as shown below. Although aggregating approximately 50 samples makes all routed cosines computable, common-token probing provides consistent comparison conditions independent of routing assignments.

StatisticObservation
Routed layers28
Valid calibration samples198
Samples with ≥1 zero-token expert52 / 198 (26.3%)
Zero-coverage sample-layer-expert positions0.505%
Undefined routed cosine after subset aggregation0%
Reviewer_xfZmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies parameter sharing for generative visual perception in unified multimodal models. It introduces Mixture of Primitives (MoP), which decouples expert routing identity from parameter ownership by selectively sharing internal channel groups across experts. The method first trains independently parameterized residual experts, then identifies compatible primitives for sharing and performs recovery training. Experiments on Cheers and Bagel show improvements in visual perception performance with fewer residual parameters.

Critical strengths

1)MoP provides a compelling approach to fine-grained sharing across experts. By preserving expert-level routing while allowing selected internal primitives to share parameters, it offers a flexible way to balance shared computation and expert specialization. The controlled experiments show improved visual perception performance with fewer residual parameters.

2)The design has potential relevance beyond a single model architecture. Experiments on both Cheers and Bagel show consistent improvements over continued training with disjoint experts while reducing residual parameters, providing preliminary evidence of cross-architecture applicability.

Critical weaknesses

1)There is a lack of direct validation for the selection criteria based on gradient compatibility. The authors use gradient similarity to determine whether primitives are suitable for sharing, with the aim of reducing update conflicts during joint training. However, existing experiments primarily compare random sharing against fixed sharing; they have not yet reported a controlled experiment that evaluates the more direct approach of selecting components to share based on output similarity.

Questions for authors

1)Why opt for a two-stage training process—learning independently first, followed by the selection of shared primitives? Would it be possible to specify the primitive-sharing relationships from the outset and train the MoP (Mixture of Primitives) directly?

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer xfZm

We thank the reviewer for the constructive feedback and valuable suggestions. Your comments helped us strengthen the experimental validation of our sharing criterion and two-stage training design.

1.Critical Weaknesses

We added an output-similarity-based ownership selection baseline with the same parameter budget (see the table below). Gradient-based MoP performs better on all four metrics across RefCOCO and NYUv2, supporting the downstream value of gradient compatibility over output similarity for sharing decisions.

We have included this comparison in Section 4.4 of the revised manuscript. Gradients show how parameters should change to reduce the task loss, making them a more direct signal for deciding which parameters to share (Yu et al., NeurIPS 2020).

Reference: Yu T, Kumar S, Gupta A, et al. Gradient Surgery for Multi-Task Learning. Advances in Neural Information Processing Systems, 33: 5824–5836, 2020.

Selection criterionResidual params (B)RefCOCO cIoU ↑RefCOCO gIoU ↑NYUv2 AbsRel ↓NYUv2 δ₁ ↑
Output similarity1.455.557.711.387.5
Gradient compatibility (MoP)1.462.263.310.388.9

2.Questions For Authors

Yes, MoP can be trained directly with a predefined sharing topology. We use two-stage training to first learn task-dependent residual transformations and then determine which primitives to share. At initialization, untrained residual branches lack the stable, task-informed signals needed for reliable ownership selection, motivating our two-stage design.

To validate this design, we added Direct MoP, trained from initialization with the same sharing topology, parameter budget, and training budget to isolate the effect of two-stage training. As shown below, two-stage MoP achieves better downstream performance.

Training strategyResidual params (B)RefCOCO cIoU ↑RefCOCO gIoU ↑NYUv2 AbsRel ↓NYUv2 δ₁ ↑
Direct MoP1.446.851.511.288.0
Two-stage MoP1.462.263.310.388.9

Thank you for the response. It addresses most of my concerns. I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Rebuttal Summary and Manuscript Revisions

We sincerely thank all reviewers for their thoughtful evaluations and constructive feedback. We have carefully addressed the concerns and revised the manuscript accordingly.

All corresponding experiments, clarifications, and corrections have been incorporated into the revised manuscript.

We sincerely appreciate the reviewers' suggestions, which have helped us strengthen the clarity, rigor, and experimental validation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.