- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates whether specialization can emerge among independently trained vision models without a shared central router or cross-agent gradients. The authors propose DISCO, which enables agents to select helpers through local routers and collaborate via stop-gradient K/V communication. Experiments on masked feature reconstruction across multiple visual domains demonstrate emergent specialization and some benefits from collaboration.
Critical strengths
-
Convincing evidence of emergent specialization. DISCO achieves strong specialization without domain labels. Section 4.5 shows that independent local routers agree on the expert for most of inputs, supporting the paper's central claim.
-
Useful analysis of the communication mechanism. The K/V interventions in Table 3 suggest that collaboration benefits from meaningful helper representations rather than simply increasing the attention context length.
Critical weaknesses
-
Limited practical benefit of collaboration and insufficiently controlled performance comparisons. DISCO's collaboration score (0.653) only matches the solo model and remains below its own delegation score (0.662) and learned central routing (0.678). Although DISCO achieves the highest R-Top5-Worst, indicating improved standalone performance of weaker agents, this does not establish an advantage over direct delegation. Compute-matched comparisons with standard Top-K MoE or larger single-model baselines are also missing.
-
Different training objectives partially confound the collaboration comparisons. DISCO explicitly optimizes helped reconstruction, while the other regimes are primarily trained for delegation. Its superior collaboration performance is therefore partly expected. Table 9 further shows that delegation rewards combined with communication training achieve comparable collaboration performance. Comparisons under identical communication-training objectives would better isolate the contribution of improvement-based routing.
-
Potential optimization interference between solo and helped modes. The router reward is approximately L_solo - L_helped, but the requester explicitly optimizes only L_helped. Both modes share parameters, including query projections, while attending to different K/V contexts. Although stop-gradient does not directly encourage degradation of solo performance, shared parameter updates may still affect it. The interventions in Table 3 do not fully resolve this concern.
Questions for authors
- Why does distributed delegation (0.665) underperform learned central routing (0.678), despite approximately 98% expert-selection agreement? What accounts for the remaining performance gap?
- Could the authors report the training trajectories of solo and helped losses and compare solo-only, helped-only, and joint training? What happens when the helper's K/V replaces, rather than augments, the requester's K/V?
- The authors are encouraged to evaluate emergent specialization on additional datasets or under different task paradigms to better establish the generality of the conclusions.
5 comments
Rebuttal (2/5)
Q2: How does the approach compare with a compute-matched Top-K MoE or a larger single model? (Weakness 1)
A2: Specialization improves on the single model both at matched computation per sample and at equal capacity per agent, while spreading parameters over agents without specialization does not help; the MoE reference of our setting is the learned central router, a top-1 gate over whole agents with the same active parameters as the solo model.
The learned central router performs exactly the computation of the solo model per training and test sample, one agent pass plus a 0.6M-parameter MLP (Table 6), and improves R-Top5 from 0.653 to 0.678, also as a realized delegation score. Random routing, with the same compute and the same total parameters, does not improve (0.650). Training several generalists on the full mixture does not close the gap either: four independently trained solo models, using four times the training samples of a population, reach 0.637 even with an oracle that picks the best of them for every input, below every specialized population of the same budget (0.645–0.652; new Table 11 in Appendix A.11).
A larger single agent improves as well, but so does a population of equally large agents: with LoRA rank 256 instead of 64, the solo model rises from 0.633 to 0.653 and a learned-central population from 0.652 to 0.670, so the specialized population stays ahead at equal capacity per agent (new Table 11).
A token-level Top-K MoE answers a different question: its experts are sub-modules that share attention layers, a gate and all gradients, and none can process an input alone, so the expert identity on an input, on which our metrics and the delegation/collaboration operating points are built, is not defined. Where expert specialization inside a single network has been measured, experts tend to follow token position or denoising timestep, or form a standing committee of always-active experts, rather than the content of the input (Fei et al., 2024; Wang et al., 2026). DISCO itself costs more training passes (three forward and two backward per sample) because the requester reconstructs with and without the message; this is the price of removing the central router and the cross-agent gradients, and we do not claim that it beats a monolith at equal FLOPs.
Q3: Can comparisons under identical communication-training objectives isolate the contribution of improvement-based routing? (Weakness 2)
A3: We agree that this is the right comparison, and the reward-versus-channel cell of Table 9 is exactly this experiment. Under identical communication training, the improvement reward gives the same helped performance, higher specialization in every seed, routers that agree with the expert (98% instead of 82%), and a routing signal that is local to each agent.
The cell "delegation reward + communication training" trains every requester on its helped reconstruction, as DISCO does, and changes only the router reward from the requester's improvement to the helper's loss. The helped scores are then the same (collaboration 0.631 vs. 0.631–0.632 for the three DISCO seeds, Worst+help 0.624 vs. 0.624–0.625), so we agree with you, and Section 4.6 states it, that the collaboration gain comes from learning to read the message. The improvement reward changes the division of labor: specialization is higher in every DISCO seed (Sp 0.919–0.948 vs. 0.855), and the routers select the loss-defined expert on 98% of the inputs instead of 82%, although they never observe a helper's loss. It is also the only reward that an agent can compute from its own losses: the delegation reward requires the helper to evaluate its reconstruction of the requester's sample against the target and report it back, while the improvement reward requires nothing but the message.
The comparison you ask for is therefore in the paper, and its conclusion is that the improvement reward contributes locality and expert-aligned routing at no cost in helped performance; the revision makes this explicit in Appendix A.8.
Rebuttal (3/5)
Q4: Can the helped training of shared parameters degrade the requester's solo performance? (Weakness 3)
A4: Thank you for this careful question. The router's choice cannot be biased by the solo term, and the data show the opposite of degradation: DISCO's agents are the best standalone reconstructors of all specialized populations at every point of training.
The solo term does not affect which helper is preferred. For a given sample, the reward is , and its first term does not depend on the chosen helper j. It acts as a per-sample baseline: whatever happens to the solo loss shifts the rewards of all helpers equally, so the ranking of the helpers, and therefore the policy the router learns, is determined by alone. A degraded solo mode could inflate the size of the reward, but not change which helper it favours.
The solo mode is not degraded. If helped training interfered with the solo mode, DISCO's agents would be worse on their own than the agents of distributed delegation, whose requesters never receive the helped loss. They are better throughout training. After an initial decrease, the average standalone loss of an agent rises during fine-tuning in every specialized population, because specialists lose the domains they no longer serve, and it rises least in DISCO: at the end of fine-tuning it is 0.120, against 0.136 for distributed delegation and 0.148 for the learned central router (new Figure 10a). In R-Top5, a DISCO requester alone scores 0.618 against 0.602 for delegation (Table 3; 0.614–0.620 vs. 0.602–0.608 in the three seeds), and DISCO has the highest Worst of the specialized populations (0.568). The shared parameters thus transfer what the requester learns from the helper's message into its standalone reconstruction, rather than interfering with it.
The solo term cannot bias the routing, and helped training improves rather than degrades the solo mode; the revision adds the loss trajectories as the new Figure 10.
Q5: Why does distributed delegation trail learned central routing despite 98% agreement? (Question 1)
A5: This question led us to a precise decomposition, which we added to the revision. The gap of 0.013 has two parts: the experts themselves are about 0.006 weaker, and about 0.008 comes from the inputs on which the randomly drawn requester is itself the expert and must hand the input to another agent. Whenever the requester is not the expert, distributed routing reaches the expert envelope of its own population, as central routing does.
Routing among the other agents is near-perfect. When the requester is not the expert on an input, its router sends the input to the expert in 98.7% of cases, and the resulting score (0.672) equals the expert envelope of the population (0.672); the learned central router reaches 98.9%.
The cost of the requester exclusion. With four agents, the random requester is itself the expert on 25% of the inputs. Since a requester may not select itself, it has to hand these inputs to another agent, which scores 0.642 on them. This accounts for the drop from 0.672 to 0.665 (before rounding, the two parts are 0.0057 and 0.0077, and the central router itself loses 0.0003 to routing), whereas the central router loses almost nothing to routing (0.678 for both its envelope and its delegation score). If the expert requester kept its own inputs, delegation would reach 0.672. The self-selection ablation shows that routers learn exactly this when allowed: they keep 91% of the inputs on which their own agent is the expert, and the gap between delegation and Best shrinks to 0.001 (Table 9).
The weaker experts. The expert envelope of distributed delegation is 0.672 against 0.678 for central routing: its population leaves one agent almost idle (U = 0.85 vs. 0.995) and pairs the six domains into three niches, so fewer agents share the work (Figure 8).
The 98% agreement and the 0.013 gap are thus consistent: the routers find the expert, and the gap reflects one agent fewer in use plus the exclusion rule at evaluation time. The decomposition is in the new Appendix A.12.
Rebuttal (1/5)
Thank you for the thoughtful review. We appreciate that you find the evidence of emergent specialization without domain labels convincing, in particular that independent local routers agree on the expert for most inputs (Section 4.5), and that you consider the K/V interventions of Table 3 a useful analysis showing that collaboration relies on meaningful helper representations rather than on a longer attention context. Your questions led to several new analyses, summarized below; all new material in the revised PDF is shown in blue.
Q1: What is the practical benefit of collaboration over direct delegation? (Weakness 1)
A1: In DISCO, collaboration is the robust operating point: it is the best of all regimes on data outside the specialization mixture, it degrades far less when routing fails or an agent leaves the population, and its agents remain more capable on their own than those of the other specialized populations. In distribution, with a correct router and all agents present, handing the input to the expert is better, as the paper states.
Outside the specialization mixture. On the ImageNet100 validation set, DISCO with communication reaches 0.642, above its own delegation (0.639; paired difference +0.003, interval [0.0027, 0.0032]), the solo model (0.636), distributed delegation (0.638 without and 0.634 with communication) and the learned central router (0.634 and 0.631), whose in-distribution advantage disappears here (Figure 2). DISCO's collaboration is also the highest ImageNet100 score in every seed of Appendix A.8.
When routing fails. With a randomly chosen helper, a DISCO requester still reaches 0.636 by reading its keys and values, whereas handing the input to the same random agent gives 0.618; a wrong helper costs collaboration 0.017 but delegation 0.044.
When an agent leaves. Removing one agent from the trained population, without retraining, costs DISCO's collaboration 0.010 on average (0.653 to 0.643), which then stays above DISCO's own delegation (0.638), while every delegation-based rule loses 2.3 to 3.5 times as much (0.024 for DISCO's own delegation, 0.030 for distributed delegation, 0.033 for the learned central router, 0.035 for K-means): the requester still contributes its own reconstruction and reads another agent instead of being replaced by it.
Every agent stays useful. As you note, DISCO has the highest R-Top5-Worst of the specialized populations (0.568), and its agents are better on their own (0.618 vs. 0.602 for delegation, Table 3), because requesters are trained on their helped reconstructions of all domains.
The value of collaboration is therefore robustness rather than peak in-distribution accuracy, and the new Appendix A.12 reports these analyses (Table 12).