- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Mixture of Primitives (MoP), a parameter-sharing procedure for routed residual experts in unified multimodal models. After learning three disjoint experts, it aligns channel groups using response similarity, selects shared gate/up blocks using gradient compatibility, and recovers the consolidated model with task training and local distillation. Controlled Cheers and Bagel experiments report improved dense prediction with fewer residual parameters; a larger Vision-Cheers checkpoint also supports generation and understanding.
Critical strengths
-
Separating routing identity from parameter ownership is a useful design perspective. The construction preserves sparse inference and permits sharing only selected internal blocks rather than requiring entire experts to coincide.
-
The principal continuation comparison is substantially controlled: common source checkpoint, data, optimization, trainable modules, routing and distillation. In Table 2, MoP improves RefCOCO cIoU from 56.6 to 62.2 over continued disjoint training while reducing residual parameters from 1.734B to 1.414B. Random MoP has the same parameter budget and performs much worse.
Critical weaknesses
W1. The compatibility mechanism lacks evidence that its coordinates support the proposed merge. Appendix D, pp. 16–17, explicitly states that experts are independently initialized and that Hungarian matching only permutes whole 1,120-channel groups, retaining their internal order. Equations 3–4 then compare parameter-gradient coordinates, and consolidation averages those coordinates elementwise. Similar group outputs do not establish neuron correspondence inside a group: jointly permuting gate/up rows and down columns leaves a primitive's function unchanged but generally changes the gradient-vector comparison and its average with another expert. This does not prove that MoP fails empirically; it limits the interpretation of the selector as identifying reusable computations.
W2. The controls do not yet establish the value of the proposed granularity over informed sharing. Random MoP is informative, but Fixed Partial shares every group with a predetermined pair topology and has a different budget (Table 7). Table 12 varies granularity only through response statistics at one layer, not downstream performance. Thus the paper establishes an advantage over these particular assignments, but does not isolate the benefit over a function-informed whole-expert or simpler group-sharing alternative. This matters because expert alignment/merging is established prior work, including the already cited MC-SMoE, ICLR 2024.
W3. Several consequential protocol details remain unresolved. Table 4 gives global batch size 800 for both final-model stages, whereas Appendix J.4 gives 500 and explicitly reports 500,000 recovery sample presentations. These imply different training exposure. Moreover, NYUv2 uses ground-truth scale-and-shift alignment in disparity space (Appendix H.2); the external Table 1 depth scores therefore need protocol labels before being interpreted as comparable. The matched internal results remain useful. Random topology results are averaged over three seeds, but corresponding dispersion and MoP calibration/recovery variability are not reported.
Questions for authors
Please see weakness section.
3 comments
Rebuttal for Reviewer 7fiu—Continued
W3. Experimental reporting and evaluation protocols
(a) Training batch size and sample exposure
Thank you for identifying this typo error. Table 4 is correct: both stages of the final Vision-Cheers model use a global batch size of 800. We have corrected Appendix J.4 to report a batch size of 800 and 800,000 recovery sample presentations, replacing 500 and 500,000, respectively.
(b) Comparability of depth evaluation protocols
Thank you for the suggestion. Our NYUv2 evaluation uses per-image least-squares scale-and-shift alignment to ground truth in disparity space before computing AbsRel and . We have labeled the evaluation protocols in Table 1(c). Vision-Cheers shares the disparity-space alignment protocol used by Depth Anything and Lotus-2, while supporting perception, generation, and understanding within a single generalist model.
Table 1(c): Monocular depth estimation on NYUv2
| Method | AbsRel ↓ | δ₁ ↑ |
|---|---|---|
| Task-specific models | ||
| Depth Anything† | 4.3 | 98.1 |
| Depth Anything V2† | 4.5 | 97.9 |
| Metric3D‡ | 8.3 | 94.4 |
| WorDepth‡ | 8.8 | 93.9 |
| DICEPTION§ | 6.1 | 96.0 |
| Lotus-2 (FLUX)† | 4.1 | 97.6 |
| Generalist models | ||
| Vision-Cheers† | 8.5 | 93.2 |
| Vision Banana‡ | 8.1 | 94.8 |
† GT-aligned disparity-space evaluation; ‡ metric-depth evaluation; § GT-aligned depth-space evaluation.
(c) Random seeds and result variability
Thank you for pointing out the missing variability statistics. The table below reports the mean and sample standard deviation across three Random MoP runs. We have included these statistics in the manuscript.
| Method | RefCOCO cIoU ↑ | RefCOCO gIoU ↑ | NYUv2 AbsRel ↓ | NYUv2 δ₁ ↑ |
|---|---|---|---|---|
| Random MoP | 38.7 ± 1.1 | 38.4 ± 0.8 | 10.7 ± 0.2 | 88.5 ± 0.1 |
Mean ± sample standard deviation over three runs; all metrics are in percent.
For MoP, we use three random seeds to independently repeat calibration and recovery from the same Stage-1 checkpoint under identical settings. The results below report their variability.
| Method | RefCOCO cIoU ↑ | RefCOCO gIoU ↑ | NYUv2 AbsRel ↓ | NYUv2 δ₁ ↑ |
|---|---|---|---|---|
| MoP | 61.47 ± 0.64 | 62.73 ± 0.51 | 10.30 ± 0.10 | 89.37 ± 0.50 |
Thank you for the rebuttal.
Thank you for the response and additional experiments. The informed expert-level baseline at a comparable parameter budget and the downstream granularity study substantially address my concerns about the sharing comparisons. The batch-size correction, depth-protocol labels, and variability statistics also address the main reporting issues. Overall, the additional evidence strengthens the empirical contribution. I maintain my weak acceptance recommendation. Please incorporate these results and clarifications into the revised manuscript.
Rebuttal for Reviewer 7fiu
We thank the reviewer for the detailed and insightful comments. Your feedback helped us clarify the methodological assumptions, strengthen the experimental comparisons, and improve the transparency of our evaluation protocols.
W1. Within-group coordinates and the basis for merging
Thank you for highlighting this distinction. Hungarian matching establishes group-level correspondence without guaranteeing within-group neuron alignment, so gradient comparison and parameter averaging remain coordinate-dependent.
However, group-level matching is only the starting point. During recovery, task supervision and Stage-1 residual distillation progressively adapt the tied channels to support compatible representations across routes.
We have clarified this interpretation in Section 3.4.1, Figure 2, and the appendix.
W2. Informed sharing baselines and granularity
(a) Topology variants and parameter budgets
Random MoP matches MoP's 1.414B residual parameter budget by preserving its sharing-pattern counts, whereas Fixed Partial uses predefined pairwise sharing across all groups, resulting in 1.349B parameters. Both serve as topology controls rather than granularity controls.
To assess fine-grained sharing against informed coarse-grained sharing, we add a whole-expert sharing baseline with a comparable parameter budget (1.404B vs. 1.414B). MoP outperforms this baseline on all four metrics, supporting the effectiveness of selective primitive-level sharing.
(b) Downstream granularity experiments
Due to time constraints, we added downstream experiments with G=1,4,8 to examine representative coarse, default, and finer grouping configurations. Among these settings, G=4 achieves the best downstream performance.
(c) Informed expert-level and group-level controls
We added an MC-SMoE-inspired whole-expert sharing baseline. It first aligns groups by their responses, then uses full-expert output similarity on common calibration tokens to select 16 of the 28 layers, fairly matching MoP’s parameter budget (1.404B vs. 1.414B).
At each selected layer, it shares the most similar expert pair while retaining all three routing identities. The baseline retains 1.404B unique residual parameters, close to MoP's 1.414B. In contrast to whole-expert sharing, MoP selectively shares primitive gate/up parameters while retaining route-specific down projections.
MoP achieves RefCOCO cIoU/gIoU of 62.2/63.3 versus 58.0/59.3 for whole-expert merging, and NYUv2 AbsRel/δ1 of 10.3/88.9 versus 11.5/86.8. These results show better downstream performance for MoP at a comparable parameter budget.
Parameter counts are unique residual parameters in billions; metrics are in percent.