acceptodds
Under review as a conference paper at ICLR 2027

Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradients

Abstract

Consider a population of neural networks with no shared gate and no gradients flowing between them: each network has its own copy of the weights and is trained on its own, all see the same heterogeneous data, and any network may ask another for help through a forward pass. Does a division of labor emerge in such a population, and is it of any use? This is a candidate regime for training populations of models at scale, and it reverses how machine learning has organized a division of labor so far: in conventional approaches, such as mixtures of experts, one gate trained jointly with all experts decides who processes what, and specialization is typically assumed rather than measured. We answer the question in a controlled, small-scale proxy for predictive visual pretraining. Initially identical agents are finetuned with one shared objective, masked prediction of frozen DINOv3 features, on an unlabeled mixture of six visual domains, with no roles or central router. We measure specialization (does the best agent for an input align with the latent domain?) and utilization (is responsibility spread across the population?) along a ladder of regimes that removes central control step by step, ending in DISCO (DIStributed COllaboration), a fully local protocol in which each agent chooses a helper, reads its internal state through a gradient-free channel, and rewards its own router only for the local improvement the help produces. Specialization emerges, and it is useful: (i) a randomly routed population of the same size is worse than a single generalist while semantically routed populations are better, so specialization rather than population size is what helps; (ii) specialization survives the removal of the central router; (iii) the gradient-free exchange makes emergent expertise usable by non-experts: in DISCO alone a randomly chosen agent, helped by the expert, matches the solo generalist, while the experts themselves surpass it, and the effect persists on data outside the specialization mixture; (iv) local routers trained only on their own improvement select the emergent expert for 98% of inputs; (v) the effects persist across population size, agent capacity, data imbalance, and fine-tuning seeds. These results provide small-scale, measurable evidence for the population dynamics that decentralized predictive pretraining would require.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
emergent specialization, decentralized / multi-agent learning, self-supervised learning, division of labor, gradient-free communication, learned routing, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ymyZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whether specialization can emerge among independently trained vision models without a shared central router or cross-agent gradients. The authors propose DISCO, which enables agents to select helpers through local routers and collaborate via stop-gradient K/V communication. Experiments on masked feature reconstruction across multiple visual domains demonstrate emergent specialization and some benefits from collaboration.

Critical strengths
  1. Convincing evidence of emergent specialization. DISCO achieves strong specialization without domain labels. Section 4.5 shows that independent local routers agree on the expert for most of inputs, supporting the paper's central claim.

  2. Useful analysis of the communication mechanism. The K/V interventions in Table 3 suggest that collaboration benefits from meaningful helper representations rather than simply increasing the attention context length.

Critical weaknesses
  1. Limited practical benefit of collaboration and insufficiently controlled performance comparisons. DISCO's collaboration score (0.653) only matches the solo model and remains below its own delegation score (0.662) and learned central routing (0.678). Although DISCO achieves the highest R-Top5-Worst, indicating improved standalone performance of weaker agents, this does not establish an advantage over direct delegation. Compute-matched comparisons with standard Top-K MoE or larger single-model baselines are also missing.

  2. Different training objectives partially confound the collaboration comparisons. DISCO explicitly optimizes helped reconstruction, while the other regimes are primarily trained for delegation. Its superior collaboration performance is therefore partly expected. Table 9 further shows that delegation rewards combined with communication training achieve comparable collaboration performance. Comparisons under identical communication-training objectives would better isolate the contribution of improvement-based routing.

  3. Potential optimization interference between solo and helped modes. The router reward is approximately L_solo - L_helped, but the requester explicitly optimizes only L_helped. Both modes share parameters, including query projections, while attending to different K/V contexts. Although stop-gradient does not directly encourage degradation of solo performance, shared parameter updates may still affect it. The interventions in Table 3 do not fully resolve this concern.

Questions for authors
  1. Why does distributed delegation (0.665) underperform learned central routing (0.678), despite approximately 98% expert-selection agreement? What accounts for the remaining performance gap?
  2. Could the authors report the training trajectories of solo and helped losses and compare solo-only, helped-only, and joint training? What happens when the helper's K/V replaces, rather than augments, the requester's K/V?
  3. The authors are encouraged to evaluate emergent specialization on additional datasets or under different task paradigms to better establish the generality of the conclusions.
5 comments
AuthorsOpenReview ↗

Rebuttal (1/5)

Thank you for the thoughtful review. We appreciate that you find the evidence of emergent specialization without domain labels convincing, in particular that independent local routers agree on the expert for most inputs (Section 4.5), and that you consider the K/V interventions of Table 3 a useful analysis showing that collaboration relies on meaningful helper representations rather than on a longer attention context. Your questions led to several new analyses, summarized below; all new material in the revised PDF is shown in blue.

Q1: What is the practical benefit of collaboration over direct delegation? (Weakness 1)

A1: In DISCO, collaboration is the robust operating point: it is the best of all regimes on data outside the specialization mixture, it degrades far less when routing fails or an agent leaves the population, and its agents remain more capable on their own than those of the other specialized populations. In distribution, with a correct router and all agents present, handing the input to the expert is better, as the paper states.

Outside the specialization mixture. On the ImageNet100 validation set, DISCO with communication reaches 0.642, above its own delegation (0.639; paired difference +0.003, interval [0.0027, 0.0032]), the solo model (0.636), distributed delegation (0.638 without and 0.634 with communication) and the learned central router (0.634 and 0.631), whose in-distribution advantage disappears here (Figure 2). DISCO's collaboration is also the highest ImageNet100 score in every seed of Appendix A.8.

When routing fails. With a randomly chosen helper, a DISCO requester still reaches 0.636 by reading its keys and values, whereas handing the input to the same random agent gives 0.618; a wrong helper costs collaboration 0.017 but delegation 0.044.

When an agent leaves. Removing one agent from the trained population, without retraining, costs DISCO's collaboration 0.010 on average (0.653 to 0.643), which then stays above DISCO's own delegation (0.638), while every delegation-based rule loses 2.3 to 3.5 times as much (0.024 for DISCO's own delegation, 0.030 for distributed delegation, 0.033 for the learned central router, 0.035 for K-means): the requester still contributes its own reconstruction and reads another agent instead of being replaced by it.

Every agent stays useful. As you note, DISCO has the highest R-Top5-Worst of the specialized populations (0.568), and its agents are better on their own (0.618 vs. 0.602 for delegation, Table 3), because requesters are trained on their helped reconstructions of all domains.

The value of collaboration is therefore robustness rather than peak in-distribution accuracy, and the new Appendix A.12 reports these analyses (Table 12).

AuthorsOpenReview ↗

Rebuttal (2/5)

Q2: How does the approach compare with a compute-matched Top-K MoE or a larger single model? (Weakness 1)

A2: Specialization improves on the single model both at matched computation per sample and at equal capacity per agent, while spreading parameters over agents without specialization does not help; the MoE reference of our setting is the learned central router, a top-1 gate over whole agents with the same active parameters as the solo model.

The learned central router performs exactly the computation of the solo model per training and test sample, one agent pass plus a 0.6M-parameter MLP (Table 6), and improves R-Top5 from 0.653 to 0.678, also as a realized delegation score. Random routing, with the same compute and the same total parameters, does not improve (0.650). Training several generalists on the full mixture does not close the gap either: four independently trained solo models, using four times the training samples of a population, reach 0.637 even with an oracle that picks the best of them for every input, below every specialized population of the same budget (0.645–0.652; new Table 11 in Appendix A.11).

A larger single agent improves as well, but so does a population of equally large agents: with LoRA rank 256 instead of 64, the solo model rises from 0.633 to 0.653 and a learned-central population from 0.652 to 0.670, so the specialized population stays ahead at equal capacity per agent (new Table 11).

A token-level Top-K MoE answers a different question: its experts are sub-modules that share attention layers, a gate and all gradients, and none can process an input alone, so the expert identity on an input, on which our metrics and the delegation/collaboration operating points are built, is not defined. Where expert specialization inside a single network has been measured, experts tend to follow token position or denoising timestep, or form a standing committee of always-active experts, rather than the content of the input (Fei et al., 2024; Wang et al., 2026). DISCO itself costs more training passes (three forward and two backward per sample) because the requester reconstructs with and without the message; this is the price of removing the central router and the cross-agent gradients, and we do not claim that it beats a monolith at equal FLOPs.

Q3: Can comparisons under identical communication-training objectives isolate the contribution of improvement-based routing? (Weakness 2)

A3: We agree that this is the right comparison, and the reward-versus-channel cell of Table 9 is exactly this experiment. Under identical communication training, the improvement reward gives the same helped performance, higher specialization in every seed, routers that agree with the expert (98% instead of 82%), and a routing signal that is local to each agent.

The cell "delegation reward + communication training" trains every requester on its helped reconstruction, as DISCO does, and changes only the router reward from the requester's improvement to the helper's loss. The helped scores are then the same (collaboration 0.631 vs. 0.631–0.632 for the three DISCO seeds, Worst+help 0.624 vs. 0.624–0.625), so we agree with you, and Section 4.6 states it, that the collaboration gain comes from learning to read the message. The improvement reward changes the division of labor: specialization is higher in every DISCO seed (Sp 0.919–0.948 vs. 0.855), and the routers select the loss-defined expert on 98% of the inputs instead of 82%, although they never observe a helper's loss. It is also the only reward that an agent can compute from its own losses: the delegation reward requires the helper to evaluate its reconstruction of the requester's sample against the target and report it back, while the improvement reward requires nothing but the message.

The comparison you ask for is therefore in the paper, and its conclusion is that the improvement reward contributes locality and expert-aligned routing at no cost in helped performance; the revision makes this explicit in Appendix A.8.

AuthorsOpenReview ↗

Rebuttal (3/5)

Q4: Can the helped training of shared parameters degrade the requester's solo performance? (Weakness 3)

A4: Thank you for this careful question. The router's choice cannot be biased by the solo term, and the data show the opposite of degradation: DISCO's agents are the best standalone reconstructors of all specialized populations at every point of training.

The solo term does not affect which helper is preferred. For a given sample, the reward is Rij=lsolo(xi,aq)−lhelped(xi,aq,aj)R_{ij} = l_{solo}(x_i, a_q) − l_{helped}(x_i, a_q, a_j), and its first term does not depend on the chosen helper j. It acts as a per-sample baseline: whatever happens to the solo loss shifts the rewards of all helpers equally, so the ranking of the helpers, and therefore the policy the router learns, is determined by lhelpedl_{helped} alone. A degraded solo mode could inflate the size of the reward, but not change which helper it favours.

The solo mode is not degraded. If helped training interfered with the solo mode, DISCO's agents would be worse on their own than the agents of distributed delegation, whose requesters never receive the helped loss. They are better throughout training. After an initial decrease, the average standalone loss of an agent rises during fine-tuning in every specialized population, because specialists lose the domains they no longer serve, and it rises least in DISCO: at the end of fine-tuning it is 0.120, against 0.136 for distributed delegation and 0.148 for the learned central router (new Figure 10a). In R-Top5, a DISCO requester alone scores 0.618 against 0.602 for delegation (Table 3; 0.614–0.620 vs. 0.602–0.608 in the three seeds), and DISCO has the highest Worst of the specialized populations (0.568). The shared parameters thus transfer what the requester learns from the helper's message into its standalone reconstruction, rather than interfering with it.

The solo term cannot bias the routing, and helped training improves rather than degrades the solo mode; the revision adds the loss trajectories as the new Figure 10.

Q5: Why does distributed delegation trail learned central routing despite 98% agreement? (Question 1)

A5: This question led us to a precise decomposition, which we added to the revision. The gap of 0.013 has two parts: the experts themselves are about 0.006 weaker, and about 0.008 comes from the inputs on which the randomly drawn requester is itself the expert and must hand the input to another agent. Whenever the requester is not the expert, distributed routing reaches the expert envelope of its own population, as central routing does.

Routing among the other agents is near-perfect. When the requester is not the expert on an input, its router sends the input to the expert in 98.7% of cases, and the resulting score (0.672) equals the expert envelope of the population (0.672); the learned central router reaches 98.9%.

The cost of the requester exclusion. With four agents, the random requester is itself the expert on 25% of the inputs. Since a requester may not select itself, it has to hand these inputs to another agent, which scores 0.642 on them. This accounts for the drop from 0.672 to 0.665 (before rounding, the two parts are 0.0057 and 0.0077, and the central router itself loses 0.0003 to routing), whereas the central router loses almost nothing to routing (0.678 for both its envelope and its delegation score). If the expert requester kept its own inputs, delegation would reach 0.672. The self-selection ablation shows that routers learn exactly this when allowed: they keep 91% of the inputs on which their own agent is the expert, and the gap between delegation and Best shrinks to 0.001 (Table 9).

The weaker experts. The expert envelope of distributed delegation is 0.672 against 0.678 for central routing: its population leaves one agent almost idle (U = 0.85 vs. 0.995) and pairs the six domains into three niches, so fewer agents share the work (Figure 8).

The 98% agreement and the 0.013 gap are thus consistent: the routers find the expert, and the gap reflects one agent fewer in use plus the exclusion rule at evaluation time. The decomposition is in the new Appendix A.12.

Reviewer_L754OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a population of independent neural networks can spontaneously develop a meaningful division of labor (specialization) during self-supervised pretraining, without a shared central gate or cross-agent gradients. The study is a controlled, small-scale proxy: populations of 4 to 8 ViT agents (43M parameters for the base size), fine-tuned with LoRA on an unlabeled 36K-image mixture of six visual domains to reconstruct masked frozen DINOv3 patch features. Its proposal, DISCO, combines local improvement-based routing rewards with stop-gradient attention key/value sharing. Along a ladder of routing regimes, the authors show that emergent specialization aligns with the latent domains and lets non-expert agents recover generalist-level reconstruction performance through gradient-free communication.

Critical strengths
  1. Interesting Premise and Solid Proof of Concept: The core research question is well-motivated. Removing the central gate and cross-agent gradients is an appealing direction for asynchronous, decentralized pretraining.

  2. Rigorous Controlled Experiments: The step-by-step “ladder of training regimes” isolates the effect of removing the central router (Sp, Best, Deleg.), and the reward-versus-channel ablation in Appendix A.8 is a genuinely informative control, since it separates learning to read a message from learning whom to ask. The sweeps over population size (K = 4, 6, 8) against six latent domains also show that the partitioning adapts to whether there are fewer, as many, or more agents than domains (e.g., merging the six domains into three two-domain niches when K = 4).

Critical weaknesses
  1. No compute-matched baseline, and no control separating specialization from having four models. A.5 states that a parameter- or FLOP-matched single model is not part of the study. Two cheap controls would settle the central empirical question: (i) a single agent with 4× the LoRA rank trained for the same wall-clock budget; (ii) four agents each trained on the full mixture, routed by loss or an oracle, the only control that separates “specialization” from “four independently trained models”, since the random-routing control fixes the total budget and therefore gives each of its agents a quarter of the data.

  2. The asynchronous and dynamic-population premises that motivate removing the central gate are never tested. Section 1 motivates the setting by asynchronous updates, agents added or removed at will, and separate hardware. In the implementation every agent is trained in one synchronous loop, one requester is drawn per step, and each router is pulled toward a uniform marginal by a KL term without which the population collapses (Section 4.6). No experiment adds or removes an agent mid-training, updates routers at different rates, or simulates stale routing. As written, the paper shows that synchronous local routing performs close to a central gate in a static population, not that the asynchronous or open-population pretraining promised in Section 1 is feasible. A single experiment (a 5th agent added mid-run, or routers updated at different frequencies) would make the claim concrete; without it the practical case rests on a small collaboration gap (2.7 points, 0.653 vs. 0.626) and on the routing statistics (Sp, agreement), neither of which measures the benefit that motivates the setting. Relatedly, Table 1’s “Learned central” already is a sample-level gated-MoE reference, so the missing standard comparison is a token-level top-1 MoE at matched active parameters.

Questions for authors
  1. Can you report run-to-run variance (at least three full-budget seeds) for Table 2, at least for Collab. and Sp? If the collaboration advantage over Solo is +0.0002 at full budget and negative at a quarter of the budget, what claim do you intend the paper to support?

  2. Can you add the two controls listed in Weakness 3 (a single agent with 4× the LoRA rank; four agents trained on the full mixture and oracle-routed)? If not, can you state explicitly what the population comparison does and does not control for, and report wall-clock cost (6.1 vs. 5.3 vs. 4.6 h) alongside pass counts?

  3. What evidence can you provide that the protocol survives asynchronous or dynamic regimes — for example a 5th agent added mid-training, or routers updated at different frequencies with a fixed lag? A minimal version of either would be sufficient.

4 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

Thank you for the detailed review and the concrete, actionable suggestions. We appreciate that you find the premise well motivated and the proof of concept solid, that you see the ladder of training regimes as a rigorous control that isolates the removal of the central router, and that you single out the reward-versus-channel ablation and the population-size sweeps, in which the partition adapts to having fewer, as many, or more agents than domains, as informative. Your suggestions led to several new analyses, summarized below; all new material in the revised PDF is shown in blue.

Q1: Can you report run-to-run variance over full-budget seeds? (Question 1)

A1: The three paired training seeds we report replicate every comparison that the paper's conclusions rest on, and the training dynamics show that their quarter budget is conservative for the collaboration scores.

Appendix A.8 repeats the two distributed regimes with three new seeds, and the solo, random and learned-central controls with two to three seeds, at a quarter of the budget (Table 9). A new seed changes the LoRA and router initialization, the data order, the requester and helper draws, and the masks. In every seed, DISCO trades an expert Best within 0.002 of delegation for a higher Worst (0.585–0.601 vs. 0.565–0.571), higher collaboration (0.631–0.632 vs. 0.621–0.623) and higher Worst+help (0.624–0.625 vs. 0.607–0.608), and its routers agree with the expert on 98–99% of the inputs. Specialization emerges in every run (Sp = 0.92–0.95 for DISCO; collaboration has a standard deviation of 0.001 across seeds). What varies across seeds is the shape of the division of labor, i.e. utilization and whether an agent is left almost idle, and we report this explicitly.

The quarter budget understates the collaboration scores rather than overstating them. Along the full-budget run, the helped score of DISCO rises steadily (collaboration 0.638, 0.642, 0.646, 0.649, 0.651, 0.653 at fine-tuning epochs 150 to 400), whereas that of delegation stays flat at 0.623–0.626 (Table 7). This is consistent with collaboration being 0.002–0.003 below the solo model at a quarter of the budget and on par with it at the full budget: the requesters keep learning to read experts that keep sharpening.

The replicated comparisons are therefore exactly those that support our claims, and the revision states the seed variability of Collab. and Sp explicitly in Appendix A.8.

Q2: Which claim does the paper support, given that collaboration only matches the solo model? (Question 1, Weakness 2)

A2: Thank you for asking us to sharpen this. The paper does not claim that collaboration beats the solo model. It claims that (i) a division of labor aligned with the latent structure emerges without a central router, (ii) this specialization is useful at matched compute, and (iii) expertise that emerged in one agent can be used by the others through a gradient-free channel. Each claim is supported by a measurement that replicates across seeds.

(i) Emergence. Local routers trained only on their own improvement produce Sp = 0.93, close to the privileged K-means split (0.96), and select the loss-defined expert for 98% of the inputs, although they never see a helper's standalone loss (Sections 4.2 and 4.5).

(ii) Usefulness. The learned central router has exactly the per-sample compute of the solo model and improves R-Top5 from 0.653 to 0.678, whereas random routing, with the same compute and the same number of parameters, does not improve (0.650). Training more generalists does not close the gap either (A4 below).

(iii) Shareability. The same requester on the same input gains 0.035 from the selected helper's message (0.618 to 0.653), and 0.011–0.018 in every quarter-budget seed; a shuffled message costs 0.239 and a random helper's message is worth half as much (Table 3). The worst agent on an input, once helped, reaches 0.644 instead of 0.593 in delegation (paired difference +0.051, Appendix A.10). A specialized non-expert, which alone scores well below the generalist, is thereby brought back to the generalist's level, while the experts remain above it. "Matching the solo model" is thus the level reached by an agent that is not the expert, not the level of the population.

On the practical side, the new analyses of the revision show where collaboration outperforms delegation: on data outside the specialization mixture, where DISCO with communication is the best of all regimes (0.642, against 0.636 for the solo model and at most 0.638 for distributed delegation and 0.634 for the learned central router at either operating point), when routing fails, and when an agent leaves the population (A8 below).

AuthorsOpenReview ↗

Rebuttal (2/4)

Q3: Can you compare with a single agent with 4× the LoRA rank, trained for the same wall-clock budget? (Weakness 1, Question 2)

A3: Thank you for this concrete suggestion; we ran it together with the matching population. A single agent with four times the LoRA rank gains from its larger capacity, but a population whose agents have the same rank gains as well and stays ahead, so specialization adds to capacity rather than substituting for it.

We trained the solo model with LoRA rank 256 instead of 64 (α scaled to keep α/r fixed), which gives it 14.2M trainable adapter parameters, as many as the adapters of all four agents together, with the seed and data order of Table 9. It reaches 0.653 after 100 fine-tuning epochs and 0.657 after 133, the wall-clock ratio of DISCO to the solo model (6.1 vs. 4.6 hours). To separate capacity from specialization, we also trained a learned-central population whose four agents all have rank 256, for the same 100 epochs:

Agent capacity (LoRA rank)Single agentLearned-central population (4 agents)
64 (3.5M adapter parameters)0.6330.652
256 (14.2M adapter parameters)0.653 (0.657 at 133 epochs)0.670

Raising the rank improves the single agent by 0.020 and the population by 0.018, and at either rank the specialized population is 0.017–0.019 ahead of the single agent with the same computation per sample. A rank-256 agent uses all of its 14.2M adapter parameters on every input, so it corresponds to making every agent larger rather than to adding agents; the two axes are complementary, and specialization emerges at every capacity we tested (Sp 0.81 for the rank-256 population; 0.84–0.93 for S, B and L agents, Table 8). Spreading the same parameters over agents without specialization does not help at all: random routing, with the trainable parameters of every population, stays below the solo model (0.650 vs. 0.653 at the full budget, 0.628 vs. 0.633–0.634 at a quarter of it), and four independently trained generalists gain only 0.003–0.004 with a loss oracle (A4).

Both runs are in the new Appendix A.11 (Table 11).

Q4: Can you add agents that are each trained on the full mixture and routed by an oracle? (Weakness 1, Question 2)

A4: Thank you for this suggestion; it is the cleanest way to separate specialization from having several independently trained models, and we added it to the revision. Four generalists trained on the full mixture gain almost nothing from an oracle that picks the best of them for every input, and they stay below every specialized population trained with a quarter of their samples.

Four solo models are trained independently (seeds 1–4), each on the full mixture for 100 fine-tuning epochs, so together they have seen four times the training samples of a population. We treat them as a four-agent population and route every input to the one with the lowest loss, an oracle that no router could implement. They score 0.633–0.634 individually and 0.637 with the oracle. The specialized populations, trained for the same number of epochs on a quarter of the generalists' samples, reach 0.645 (DISCO), 0.646 (distributed delegation) and 0.652 (learned central routing), and the generalists show no division of labor at all (Sp = 0.004, U = 1.00). Specialization, not the number of independently trained models, is what improves the population.

This control complements the random-routing control: random routing matches the training budget and gives each agent a quarter of the data, while the generalists give each agent all the data at four times the budget; both stay below the specialized populations. The new comparison is in Appendix A.11 (Table 11).

AuthorsOpenReview ↗

Rebuttal (3/4)

Q5: What does the population comparison control for, and what is the wall-clock cost? (Question 2)

A5: All population regimes share the agents, their initialization, the LoRA budget per agent, the data order, and the number of training samples; they differ in total parameters, in agent passes per sample, and, as a consequence, in wall-clock time. We summarize this below:

SoloRandom / K-meansLearned centralDistr. delegationDISCO
Agents × trainable parameters1 × 3.5M4 × 3.5M4 × 3.5M + 0.6M4 × 4.1M4 × 4.1M
Training samples36K × 400 epochssamesamesamesame
Agent passes per training sample (fwd / bwd)1 / 11 / 11 / 12 / 13 / 2
Agent passes per test sample (Deleg. / Collab.)1 / –1 / 21 / 22 / 32 / 3
Wall-clock, 400 fine-tuning epochs, 4 GPUs4.6 h6.4 h / 6.3 h5.3 h5.0 h6.1 h

Wall-clock times are less informative than pass counts: they include the evaluation passes that every population regime runs at every epoch for logging, and the runs were trained on two different nodes, partly two at a time with shared data loading. This is why random routing and K-means, whose training step costs the same as that of the solo model, took longer than the learned central router. On the pass counts, the learned central router, which is the regime that shows the benefit of specialization most directly, costs the same as the solo model per sample. The comparisons do not control for total parameters, which is why the random-routing control (same parameters as every population, no specialization, below solo) and the new generalist control (A4) are part of the study.

Q6: How does the approach compare with a token-level top-1 MoE at matched active parameters? (Weakness 2)

A6: A token-level MoE does not have standalone experts, so the quantity our study measures, which agent is the expert on an input, is not defined for it; the matched-active-parameter MoE reference of our setting is the learned central router, which routes whole samples and activates one agent per sample.

Our specialization and utilization metrics compare the standalone losses of separate networks on the same input. In a token-level MoE, experts are feed-forward sub-modules that share the attention layers, the gate and all gradients, and no expert can process an image on its own, so neither Sp nor U, nor delegation or collaboration, is defined. This is also why token-level MoEs answer a different question: when their specialization has been measured, experts tend to organize around token position, noise level, or a standing committee of always-active experts rather than around the content of the input. The learned central router is the gate of a sample-level MoE: it is trained online with the agents, activates exactly one agent per sample, and has the same active parameters per sample as the solo model and as a top-1 MoE built from our agents. A token-level MoE built from our agents would also not yield the per-agent quantities that the study is about.

Q7: Does the dependence on the KL term undermine the claim that the division of labor emerges? (Weakness 2)

A7: No: the KL term balances how much each router sends to each agent, but it is indifferent to which inputs go where, and balance without semantics does not produce a division of labor; what emerges is the alignment of the balanced niches with the domains.

The penalty acts only on the marginal routing distribution of the active router over its batch (Appendix A.6), a quantity each router computes from its own outputs, so it requires no coordination between agents. Its role is the one load balancing plays in every MoE: without it, the population collapses onto one agent within four epochs (Section 4.6), exactly as gated MoEs collapse onto few experts. Two controls show that it does not create the specialization. Random routing is perfectly balanced (U = 1.00) and has no specialization (Sp = 0.01, Table 2). Random routing combined with communication training is equally balanced and also does not break the symmetry (Sp = 0.00, Table 9). The semantic content of the division of labor comes from routing tied to per-sample performance, and the requester exclusion is not needed for it either: with self-selection allowed, specialization still emerges (Sp = 0.81), and routers keep 91% of the inputs on which their own agent is the expert.

The KL term is thus a precondition, as in MoEs, but not the cause of specialization, which Section 4.6 and Appendix A.8 state.

AuthorsOpenReview ↗

Rebuttal (4/4)

Q8: Does the protocol survive asynchronous or dynamic regimes? (Weakness 2, Question 3)

A8: Thank you for proposing concrete tests; they led to the experiment we find most telling. When the data change, a newcomer added to a trained population takes over the new part of the data while the existing niches keep their experts; in addition, the routers can be updated rarely or with a long lag, messages from a helper that lags behind keep almost all of their value, and the population tolerates the loss of an agent without retraining.

Adding an agent when the data change. We took the trained four-agent DISCO population (seed 1) and added a new component to its mixture: 5,999 CLEVR images, synthetic scenes that none of the agents saw during pretraining or specialization, with the same budget as each of the six domains. Training then continued for 40 epochs, either with the four agents alone or with a newcomer that starts exactly as the original agents did (pretrained weights, a fresh adapter and a fresh router, with a fifth output added to the existing routers). We first tried adding a newcomer to the unchanged mixture: there, all niches are taken by trained experts, and in the 27 epochs before we stopped that run the newcomer remained almost idle. When the data change, the outcome is the one a decentralized population needs (seven-component validation set):

SpAgreementBest, 6 old domainsBest, CLEVRExpert on CLEVR
Before the new component0.86885%0.6450.844two agents (63%, 27%)
Four agents continue0.84187%0.6510.889three agents (47%, 35%, 10%)
A newcomer joins0.93996%0.6520.902the newcomer (>99.9%)

The newcomer becomes the expert on 99.95% of the CLEVR images, all four existing routers send every CLEVR input to it, and each of the six original domains keeps its expert (94–100% of its images, as before the new component). Without it, the new component is split across three agents, the division of labor becomes less aligned with the components, and CLEVR is reconstructed less well, while the six original domains are reconstructed equally well in both runs.

Stale routers. Once the niches have formed, routing decisions stop changing: the routers of the checkpoint after 150 fine-tuning epochs select the same helper as the final routers for 99.98% of the (requester, input) pairs. Applying routers that are 50 to 250 epochs out of date to the final agents leaves both operating points unchanged to the third decimal (delegation 0.662, collaboration 0.653), and they pick the current expert for 98.6% of the inputs. Routers can therefore be updated at a much lower frequency than the agents, or with a lag of tens of epochs, without any loss.

Stale messages. When the selected helper's keys and values are computed by its checkpoint from 50 fine-tuning epochs earlier (about 7,000 optimizer steps), while requester and router are current, collaboration reaches 0.651 instead of 0.653, and the message keeps 95% of its benefit over the requester alone (0.618).

Removing an agent. We removed each agent of the trained populations in turn, without retraining, and let the remaining agents route among themselves. DISCO's collaboration loses 0.010 on average (0.653 to 0.643), while every delegation-based routing loses 2.3 to 3.5 times as much (0.024 for DISCO's own delegation, 0.030 for distributed delegation, 0.033 for the learned central router, 0.035 for K-means), because a requester whose expert has left still contributes its own reconstruction and reads another agent.

What the implementation already decouples. No gradient crosses between agents, and each router is trained only on the batches of its own agent. The full systems setting, with separate hardware and asynchronous updates, is beyond the scope of this controlled study, and the paper describes these properties as permitted by the protocol rather than tested (Section 1).

These experiments make the open-population premise concrete: agents can be added when the data change, routers and messages tolerate staleness, and the loss of an agent costs little. They are reported in the new Appendix A.12 (Tables 12 and 13).

We are grateful for the review, which led to several new analyses in the appendix. We hope our answers address your concerns, and if so, we would be thankful if you would consider updating your rating. We would be happy to discuss any remaining questions.

Reviewer_CVXSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether independent vision models can develop specialized roles without a shared router or cross-model gradients. The authors propose DISCO, where each model chooses a helper based on the improvement to its own prediction and reads the helper’s intermediate features. The experiments show clear specialization across six visual domains. A weaker model can improve with help from an expert. However, collaboration remains worse than handing the task directly to the expert. The evaluation is also limited to reconstructing frozen DINOv3 features.

Critical strengths
  1. I like the research question. Letting models discover their own roles is more interesting than assigning them in advance.
  2. The controls are well designed. They help show that the model uses the helper's content and clarify where the gains come from.
Critical weaknesses
  1. The practical advantage is unclear. DISCO uses extra models and computation, but there is no comparison with a single larger model under the same budget. It is also unclear when collaboration is preferable to direct delegation.
  2. The evaluation is limited to frozen DINOv3 features and datasets with fairly clear domain differences. It is unclear whether the same behavior would appear in less structured settings or downstream tasks.
Questions for authors
  1. Would DISCO still offer an advantage over a single larger model with the same compute budget? When would collaboration be preferable to direct delegation?
  2. Does the learned specialization help on a downstream task, e.g., image classification? Would it still emerge if the training data had less distinct domain boundaries?
  3. Keeping the delegation reward and adding communication training already gives collaboration scores close to DISCO. What does the improvement-based reward add in practice, apart from better agreement on which model is the expert?
4 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for the careful and constructive review. We are glad that you find the research question interesting, letting models discover their own roles rather than assigning them in advance, and that you consider the controls well designed. Your questions prompted several new analyses, summarized below; all new material in the revised PDF is shown in blue.

Q1: Does DISCO offer an advantage over a single larger model with the same compute budget? (Weakness 1, Question 1)

A1: Thank you for raising this question, which helps us separate what the population buys from what it costs. At matched computation per sample, a specialized population is better than the single model; more capacity improves both, and at equal capacity per agent the specialized population stays ahead, whereas spreading parameters over agents without specialization does not help at all.

Same computation per sample. The learned central router runs exactly the computation of the solo model: one agent forward and backward pass per training sample, and one agent forward pass per test sample, plus a 0.6M-parameter MLP (Table 6 in Appendix A.5; router size in Appendix A.2). It raises R-Top5 from 0.653 to 0.678 (paired difference +0.026, interval [0.025, 0.026]; Appendix A.10), and since it sends 99% of the inputs to their expert, its realized delegation score is also 0.678 (Table 2). Random routing has the same computation and the same total parameters, yet it scores 0.650, below the solo model.

More generalists. Training more generalists does not reproduce the gain either. Four independently trained solo models, each fine-tuned on the full mixture and therefore together using four times the training samples of a whole population, reach only 0.637 when every input is sent to whichever of them has the lowest loss, an oracle no router could implement. At the same quarter budget, the specialized populations reach 0.645 (DISCO), 0.646 (delegation) and 0.652 (central), and the four generalists show no specialization at all (Sp = 0.004).

A larger single agent. Raising the LoRA rank of the solo model from 64 to 256 gives one agent the trainable capacity of a whole four-agent population, active on every input. It improves the single agent from 0.633 to 0.653 at a quarter of the budget (0.657 when trained for the wall-clock time of DISCO). The same increase of rank improves a learned-central population from 0.652 to 0.670, so at equal capacity per agent the specialized population stays 0.017–0.019 ahead (Table 11). Capacity and specialization add up: a population can grow by making its agents larger and by adding agents, and specialization emerges at every agent size we tested (S, B and L agents, Table 8).

What the extra passes of DISCO buy. DISCO trains with three agent forward and two backward passes per sample (Table 6), because the requester reconstructs once alone and once with the message. This overhead is the price of the setting we study: no component sees every sample, and no gradient crosses between agents. We do not claim that a decentralized population beats a monolith at equal FLOPs, and the paper does not make this claim.

Specialization therefore helps at matched computation and at every capacity, and it cannot be replaced by more generalists, while DISCO's overhead is what it costs to achieve it without central control. The revision adds the comparisons with independently trained generalists and with larger agents (new Appendix A.11, Table 11).

AuthorsOpenReview ↗

Rebuttal (2/3)

Q2: When is collaboration preferable to direct delegation? (Summary, Weakness 1, Question 1)

A2: In DISCO, collaboration is preferable to delegation whenever the selected agent is not a reliable expert for the input: outside the specialization mixture, when routing fails, and when an agent leaves the population. In distribution, with a correct router and all agents present, handing the input to the expert is better, as the paper states. The new analyses below use the existing checkpoints and the evaluation protocol of the paper.

Outside the specialization mixture. On the ImageNet100 validation set, where no agent is a trained expert, collaboration is the best operating point of all regimes: DISCO with communication reaches 0.642, against 0.639 for its own delegation (paired difference +0.003, interval [0.0027, 0.0032]), 0.636 for the solo model, 0.638 and 0.634 for distributed delegation without and with communication, and 0.634 and 0.631 for the learned central router (Figure 2). DISCO's collaboration is also the highest ImageNet100 score in every seed of Appendix A.8.

When routing fails. If the helper is chosen at random instead of by the router, a DISCO requester still reaches 0.636 by reading the helper's keys and values, whereas handing the input to the same random agent gives 0.618. A wrong helper costs collaboration 0.017, but it costs delegation 0.044.

When an agent leaves. We removed each agent from the trained populations in turn, without any retraining. DISCO's collaboration loses 0.010 on average (0.653 to 0.643) and ends above DISCO's own delegation (0.638), whereas every delegation-based rule loses 2.3 to 3.5 times as much: 0.024 for DISCO's own delegation, 0.030 for distributed delegation, 0.033 for the learned central router and 0.035 for K-means. The difference is concentrated on the inputs of the removed expert, where the requester still contributes its own reconstruction and reads another agent instead of being replaced by it.

Every agent stays more useful on its own. Because requesters are trained on their helped reconstructions of all domains, DISCO has the highest R-Top5-Worst of the specialized populations (0.568) and its agents are better on their own (0.618 vs. 0.602 for delegation, Table 3), so the population degrades gracefully instead of depending on a single specialist per domain.

Delegation is thus the operating point for clean, in-distribution inputs with a reliable router, and collaboration the robust one. The revision reports the removal and routing-failure analyses in the new Appendix A.12 (Table 12).

Q3: Does the learned specialization help on a downstream task such as classification? (Weakness 2, Question 2)

A3: Our agents are predictors rather than encoders, so the downstream task they serve is prediction itself; what specialization improves is the content of the hidden region at the right location, which is what R-Top5 measures and what an image-level label would not reveal.

Each agent fills in the 70% of the DINOv3 patch grid that it does not see, as a masked-feature or future-frame predictor does in predictive pretraining (Section 1). R-Top5 asks, for every hidden patch, whether the prediction retrieves the correct patch of the same image among its 196 patches, i.e. whether the right content is predicted at the right place; trivial predictors score 0.120 (copying the nearest visible patch) and 0.026 (chance), the agent before specialization 0.563, and the solo model 0.653 (Appendix A.10). Image classification depends on which semantic content is present rather than on where it is, and in a strong frozen feature space it can often be inferred from the visible part of the image alone, which every regime receives unchanged; it would therefore say little about the quality of the predicted region. We instead evaluate the prediction per domain (Table 10), on data outside the specialization mixture (Figure 2), and qualitatively (Figure 7), where the expert reconstructs whole structures that the non-expert misses, such as the pan and gripper in BridgeData V2 or the road boundary in KITTI, and the help restores them.

Using specialized predictors inside a downstream system, for instance as the predictive component of a world model used for planning, is the natural next step, and it requires a training setup beyond the controlled study of this paper.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q4: Would specialization still emerge with less distinct domain boundaries? (Weakness 2, Question 2)

A4: Yes: the population separates two domains that a clustering of privileged clean features does not tell apart, and it groups similar domains together when agents are scarce, so the division of labor follows the similarity of the data rather than only dataset identity.

BridgeData V2 and RT-1 are both tabletop robot-manipulation datasets with similar scenes, arms and objects, the least distinct pair of our mixture. With four agents, distributed delegation, DISCO and the K-means split all assign them to the same agent (Figure 8). With six agents, as many as there are domains, DISCO separates them: 81% of the BridgeData V2 images have one agent as their expert and 96% of the RT-1 images another. The K-means control, which clusters clean DINOv2 features of the uncorrupted images, does not: it merges the two robotics datasets into one cluster and splits the face images instead, so the emergent division of labor is more aligned with the domains than the privileged clustering (Sp 0.914 vs. 0.887). With eight agents, DISCO again keeps the two apart (93% and 96%). On the imbalanced mixture, where the dominant robotics domains are split differently, specialization remains clear (Sp = 0.75, U = 0.93, Section 4.4).

The six-domain mixture and the frozen DINOv3 target space are deliberate: measuring specialization requires a known latent factor, and the frozen feature space keeps a population trainable in hours, which is what makes the controlled sweeps possible (Section 3.2). The protocol itself is task-agnostic and applies unchanged to any transformer trained by prediction. Within our setting, the robotics case shows that the population resolves boundaries that are subtle in a strong pretrained feature space, and a new experiment shows that it also absorbs data it has never seen: when synthetic CLEVR scenes are added to the mixture of a trained population, a newly added agent takes them over while the six original domains keep their experts (Appendix A.12, Table 13).

The revision adds the robotics analysis as a new paragraph of Appendix A.9.

Q5: What does the improvement-based reward add beyond better agreement on the expert? (Question 3)

A5: This is an excellent question, and the reward-versus-channel ablation was designed to answer it. The improvement reward makes the routing signal local: it is the only reward an agent can compute from its own losses, and it achieves the same helped performance with routers that agree with the expert and with higher specialization.

Same helped performance, stronger division of labor. With the delegation reward and communication training, the population reaches the helped scores of DISCO (Table 9). With the improvement reward, however, specialization is higher in every DISCO seed (Sp 0.919–0.948 vs. 0.855) and the routers agree with the expert on 98% of the inputs instead of 82%.

Locality. The delegation reward is the helper's loss on the requester's sample: to compute it, the helper must evaluate its own reconstruction against the target and send the result back. The improvement reward is the difference between two losses of the requester itself, with and without the message, so the only thing that crosses between agents is the message. It is therefore the reward that makes DISCO the only regime of Table 1 in which every learning signal is computed from the agent's own loss, which is the property that a decentralized population needs.

Usefulness rather than standalone accuracy. The improvement reward asks "who helps me most?" rather than "who is best on their own?". That the two coincide in our populations (98% agreement, Section 4.5) is one of our findings, and it is what makes the division of labor visible to the agents themselves; in populations whose members differ in architecture or modality, the two need not coincide, and the improvement reward remains the correct signal.

The improvement reward therefore does not add helped accuracy, and we state this in Section 4.6; it provides locality and an agreed-upon division of labor at no cost in performance.

We are grateful for the review, which led to several new analyses in the appendix. We hope our answers address your concerns, and if so, we would appreciate it if you would consider raising your score. We would be glad to discuss any remaining questions.

Thank you for the response and additional experiments. They clarify several points, but my main concerns remain. The matched-compute results concern central routing rather than DISCO, and the evaluation is still limited to frozen-feature prediction, leaving its broader practical benefit unclear. I am therefore keeping my rating at 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

General Response to All Reviewers

We sincerely thank the reviewers for their thoughtful and constructive reviews. We are encouraged that the reviewers find the research question interesting and well motivated (CVXS, L754), the evidence of emergent specialization without domain labels convincing (ymyZ), the ladder of training regimes a rigorous, controlled design (L754), and the controls, the reward-versus-channel ablation and the message interventions informative (CVXS, L754, ymyZ).

We have uploaded a revised version of the paper, in which all changes are marked in blue. The main text is unchanged; following the reviewers' suggestions, the revision adds new experiments and analyses to the appendix:

  • larger agents and independently trained generalists: specialization stays ahead of a single agent with four times the LoRA rank at equal capacity per agent, and of four generalists trained on the full mixture (Appendix A.11, Table 11);
  • a new data component (CLEVR) added to a trained population: a newly added agent takes it over while the original domains keep their experts (Appendix A.12, Table 13);
  • the robustness of collaboration and delegation to routing failures, to the removal of an agent, and to stale routers and messages, and why distributed delegation trails central routing (Appendix A.12, Table 12);
  • the standalone and helped losses during training (Figure 10), the separation of two similar robotics domains (Appendix A.9), and the seed variability and the role of the improvement reward (Appendix A.8).

The individual responses answer each question and contain further clarifications and analyses. Each answer states its main conclusion in bold, followed by the supporting details. We would be happy to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.