acceptodds
Under review as a conference paper at ICLR 2027

Co-Evolving Query Routing and Agent Specialization in Mixture-of-Agents

Abstract

Mixture-of-Agents systems typically treat query routing and agent post-training as separate stages. A router fitted to a frozen pool cannot track later policy updates, and agents are not given competence-aware data that would induce complementary expertise. We study this coupling as co-evolution: a router and a population of independently updated LLM agents are optimized in the same reinforcement learning loop, so that allocation and specialization change together. Before generation, a predictive familiarity estimator maps intermediate hidden states to a familiarity score that estimates how well an agent matches a query, without full rollouts or external judges. Cumulative-threshold routing then activates the smallest score-ranked subset whose familiarity scores sum to at least a threshold, trading accuracy against compute. The same scores steer training queries toward agents that are becoming proficient on related inputs, shifting a pool of homogeneous generalists into specialists. Across multiple domains, the resulting system outperforms static-agent routers and fine-tuning pipelines that keep the collaboration workflow fixed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-agents, query routing, multi-agent LLMs, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t7A8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CERA-MoA, a Mixture-of-Agents framework where the router and agents are trained in the same reinforcement learning process. The router estimates how familiar each agent is with a query before generation, and uses a cumulative-threshold rule to decide how many agents to activate. The selected agents are then updated on the queries they receive, and their rewards are used to update the router. In this way, routing changes as the agents improve, and different agents can become specialized in different types of tasks. The paper evaluates CERA-MoA on shared-backbone and heterogeneous agent pools across math, code, instruction following, and general reasoning. The results show better ID and OOD performance than static routing and fixed-workflow training baselines, lower generation cost than fixed Top-2 routing, and different specialization patterns across agents.

Critical strengths

Strength 1: The paper studies an important problem by training the router and agents together instead of treating routing and agent post-training as separate stages.

Strength 2: The familiarity-based router can select agents before generation and avoids running every agent or using an external judge for routing.

Strength 3: The experiments cover both shared-backbone and heterogeneous agent pools, with results on ID, OOD, routing efficiency, and agent specialization.

Critical weaknesses

Weakness 1: The paper claims that the router tracks changes in agent capability during training, but the current experiments do not show this clearly. Since only selected agents are updated on a query, agents that are routed more often to a certain type of query also receive more training on that type. Their rewards may then improve, which can further increase their familiarity scores through Eq. (15). This creates a feedback loop between routing and training, so it is unclear how much of the observed routing behavior comes from tracking capability changes and how much comes from reinforcing earlier routing decisions.

Weakness 2: Each agent has its own predictor–target pair, and the predictor heads are updated with different numbers of queries during training. The paper then compares the resulting familiarity scores across agents as if they were on the same scale. It is unclear whether a score of 0.7 for one agent means the same thing as 0.7 for another agent, especially when some agents receive far more updates than others. Since these scores are used for both ranking and the cumulative threshold in Eq. (6).

Weakness 3: The paper reports one training run with seed 42 for each setting. This is a notable limitation for CERA-MoA because the routing decisions affect which data each agent sees, and those updates then affect later routing decisions. Small differences early in training can therefore lead to different specialization patterns and final results. The paper does not show whether the reported gains or the specialization in Figure 3 are stable across runs.

Questions for authors

Question 1: Can you directly test whether the familiarity score tracks changes in agent capability over time? For example, if an agent’s performance changes while its routing frequency is held fixed, does the router update its familiarity score accordingly?

Question 2: Are the familiarity scores calibrated across agents? Can you show that the scores remain comparable when the predictor heads have very different update counts, for example by reporting per-agent score distributions or calibration against actual rewards?

Question 3: Do the main performance gains and the specialization patterns in Figure 3 remain consistent across different random seeds? Please report results from multiple runs for the main setting.

3 comments
AuthorsOpenReview ↗

Follow-up on the additional seeds

The experiments in the manuscript are seed 42. Seeds 41 and 43 are two further training runs of the same Qwen3-4B pool (four agents, the default recipe). Averages are the unweighted means used in Tables 1 and 2.

In-distribution

SeedGSM8kMATHDAPOMBPPEurusTACOMAG.IFBBHAvg.
42 (manuscript)93.579.641.860.331.313.492.872.483.463.2
4192.677.437.461.131.913.894.273.280.862.5
4393.380.645.860.725.99.492.872.883.062.7
Mean93.179.241.760.729.712.293.372.882.462.8

Seed 41 is 0.7 below the reported ID average; seed 43 is 0.5 below. Across the three seeds the average stays in 62.5–63.2, and the mean is 62.8.

Out-of-distribution

SeedIFEvalHumanEvalAGIEvalARC-cLogicBenchOlympiadAvg.
42 (manuscript)83.082.571.392.765.641.672.8
4182.185.765.392.965.032.170.5
4383.079.272.093.065.844.172.9
Mean82.782.569.592.965.539.372.1

Seed 41 is 2.3 below the reported OOD average; seed 43 is 0.1 above. The three-seed mean is 72.1, and the range is 70.5–72.9.

All three seeds stay above the strongest Qwen3-4B baseline, AT-GRPO (61.0 / 71.5), on ID; the three-seed mean is 62.8 / 72.1. On OOD, seeds 42 and 43 remain above AT-GRPO. Seed 41 is slightly lower because OlympiadBench is slightly lower.

The gap is a trade between hard math and code. In seed 41 the hard-math agent receives fewer of those queries, so MATH, DAPO, and OlympiadBench are lower, while the code agent is slightly stronger. Seed 43 goes the other way: hard math is higher, and the code agent is slightly weaker. The seed only changes which specialist is stronger; across all three seeds the routed averages stay at the level reported in the maniscript.

Thank you for provide the rebuttal. My concerns have been addressed. I'll keep my score.

AuthorsOpenReview ↗

Response to Reviewer t7A8

Thank you again for the positive assessment and constructive comments

Reviewer_Y8qoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CERA-MoA to address the separation between query routing and agent post-training in Mixture-of-Agents systems. It optimizes a router and independently learning LLM agents within the same reinforcement learning loop. A predictive familiarity estimator uses intermediate hidden states to assess agent–query compatibility before generation, while cumulative-threshold routing adaptively selects agent subsets to balance performance and computational cost. The same scores guide training-data allocation to encourage complementary specialization as agent capabilities evolve. Experiments across multiple domains report improvements over static-agent routing and fixed-workflow fine-tuning baselines.

Critical strengths
  1. The coupling of routing and agent learning is meaningful. The router tracks changes in agent capabilities and adjusts both query allocation and update frequency. This extends MoA to adaptive agent training.

  2. The experimental evaluation is extensive. Experiments cover mathematical reasoning, code generation, instruction following, and general reasoning, using three shared-backbone configurations and a heterogeneous agent pool. CERA-MoA achieves the highest reported ID and OOD average scores in each configuration. Ablations examine familiarity estimation, allocation, exploration, and feature depth, with additional analyses of training dynamics and hyper-parameter sensitivity.

Critical weaknesses
  1. I cannot tell whether the gains come from bidirectional router–agent feedback. Training multiple models, adding adaptation parameters, or changing each agent's training data and update count could also explain the improvement. I suggest three control experiments to clarify this. First, compare capability-aware allocation with random or balanced allocation using the same model configurations, optimizer, and total training budget. Second, compare online co-evolution with a two-stage pipeline that trains experts on fixed semantic clusters before training the router. Third, keep CERA-MoA's final agent pool fixed and evaluate other routers, to measure how much the routing method adds given the same trained agents.

  2. Absolute agent performance and collaboration gains are unclear. Figure 3 shows relative capability profiles and query allocation. I suggest reporting each agent's overall and per-domain accuracy on the same held-out test set. Using the same trained checkpoints, compare the full system with individual agents. These results would show each agent's absolute performance and quantify the accuracy gain from collaboration over the single-agent baselines.

Questions for authors
  1. Use of RND: I am not very familiar with RND. Could the authors explain why a frozen random target is needed to estimate agent competence? Does this structure offer any advantages over a simple scoring model trained with the same reward supervision and margin loss?

  2. Training cost compared with static methods: Does jointly training the router and agents require substantially more compute than static methods? Appendix B.1 gives a rough training cost for CERA-MoA. Could you report a side-by-side training-cost comparison with the static baselines?

  3. Top-1/Top-2 settings: Did you use the same trained agents and router for Top-1 and Top-2 in Table 3, and just choose one or two agents at test time? Or did you retrain the agents and router for each setting? If you retrained them, please clarify how agents were selected during training and testing, and compare the training costs of Top-1, Top-2, and adaptive routing.

2 comments

Thank you for the response and the additional experiments. Your response addressed my questions, and I will keep my rating of 3 (Weak acceptance).

AuthorsOpenReview ↗

Response to Reviewer Y8qo

Thank you again for the positive assessment and constructive comments.

Reviewer_gX2oOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CERA-MoA, a framework that jointly trains a router and multiple LLM agents. The router assigns training queries based on each agent's estimated strengths, while the agents improve through reinforcement learning. The router is then updated using their rewards, allowing agents to gradually develop different skills. At inference time, it dynamically selects agents to balance accuracy and computational cost.

Critical strengths

The idea of jointly optimizing routing and agent training is interesting and intuitive. It allows agents to develop different skills without manually assigning roles.

The method achieves strong results across multiple benchmarks and model settings, while reducing inference costs compared to fixed routing.

The experiments include useful ablations that show the importance of the familiarity estimator and exploration.

Critical weaknesses

The familiarity score is a heuristic rather than a direct measure of agent performance. It is unclear why summing these scores is a reliable way to decide how many agents to activate.

The evidence for agent specialization is limited. Different routing patterns do not necessarily mean that agents have learned stronger or more specialized skills.

The experiments use only one random seed, and the paper acknowledges possible overlap between training and test data.

Questions for authors

When only one agent is selected, its familiarity is increased whenever it receives a positive reward, even if another unselected agent would perform much better. How can the router correct such overconfident routing decisions? Can the authors evaluate whether it can recover from initially incorrect familiarity estimates or changes in agent competence?

The router favors agents that already perform well on a query, but these agents may receive little useful policy gradient when all their rollouts have the same reward. Why should current competence be a good criterion for allocating training data? Would routing based on expected learning progress be more effective?

Fixed Top-3 training achieves nearly the same accuracy as CERA-MoA, despite using a different data allocation strategy. Does this suggest that the main benefit comes from more efficient inference routing rather than better agent specialization?

3 comments
AuthorsOpenReview ↗

Follow-up on the additional seeds

The experiments in the manuscript are seed 42. Seeds 41 and 43 are two further training runs of the same Qwen3-4B pool (four agents, the default recipe). Averages are the unweighted means used in Tables 1 and 2.

In-distribution

SeedGSM8kMATHDAPOMBPPEurusTACOMAG.IFBBHAvg.
42 (manuscript)93.579.641.860.331.313.492.872.483.463.2
4192.677.437.461.131.913.894.273.280.862.5
4393.380.645.860.725.99.492.872.883.062.7
Mean93.179.241.760.729.712.293.372.882.462.8

Seed 41 is 0.7 below the reported ID average; seed 43 is 0.5 below. Across the three seeds the average stays in 62.5–63.2, and the mean is 62.8.

Out-of-distribution

SeedIFEvalHumanEvalAGIEvalARC-cLogicBenchOlympiadAvg.
42 (manuscript)83.082.571.392.765.641.672.8
4182.185.765.392.965.032.170.5
4383.079.272.093.065.844.172.9
Mean82.782.569.592.965.539.372.1

Seed 41 is 2.3 below the reported OOD average; seed 43 is 0.1 above. The three-seed mean is 72.1, and the range is 70.5–72.9.

All three seeds stay above the strongest Qwen3-4B baseline, AT-GRPO (61.0 / 71.5), on ID; the three-seed mean is 62.8 / 72.1. On OOD, seeds 42 and 43 remain above AT-GRPO. Seed 41 is slightly lower because OlympiadBench is slightly lower.

The gap is a trade between hard math and code. In seed 41 the hard-math agent receives fewer of those queries, so MATH, DAPO, and OlympiadBench are lower, while the code agent is slightly stronger. Seed 43 goes the other way: hard math is higher, and the code agent is slightly weaker. The seed only changes which specialist is stronger; across all three seeds the routed averages stay at the level reported in Tables 1–2.

Thank you for the detailed rebuttal and additional experiments. The new results address several of my concerns and strengthen the paper. My remaining concern is still on agent specialization. While the authors provide a specialization evaluation of individual CERA-MoA agents, a similar specialization evaluation of agents trained with Fixed Top-3 routing is missing, which has nearly identical performance as CERA-MoA. Overall, I find the idea interesting and appreciate the additional evidence in rebuttal, so I have increased my score to 3.

AuthorsOpenReview ↗

Follow-up on the Top-3 agent performance

Thank you for the increased score, and for keeping the missing control this precise. All four Fixed Top-3 agents are now evaluated alone, with routing and voting off.

In-distribution

GSMMATHDAPOMBPPEurusTACOMAGIFBBHID
CERA Agent 190.877.637.458.021.65.785.865.674.657.5
CERA Agent 293.180.042.848.621.46.391.473.481.259.8
CERA Agent 392.374.432.057.630.813.893.667.275.459.7
CERA Agent 492.571.324.659.131.113.392.064.068.257.3
Top-3 Agent 191.676.935.459.527.911.791.864.876.659.6
Top-3 Agent 291.376.735.859.127.112.391.264.275.059.2
Top-3 Agent 391.076.832.460.728.112.492.065.677.059.6
Top-3 Agent 491.476.435.059.127.612.892.465.876.459.7

Out-of-distribution

IFEvalHumanEvalAGIEvalARC-cLogicBenchOlympiadOOD
CERA Agent 183.076.068.891.648.940.768.2
CERA Agent 281.970.171.192.665.740.370.3
CERA Agent 380.681.265.592.764.333.869.7
CERA Agent 478.987.062.991.734.130.064.1
Top-3 Agent 178.078.670.992.964.639.370.7
Top-3 Agent 278.678.669.991.865.339.070.5
Top-3 Agent 376.779.269.792.565.939.070.5
Top-3 Agent 477.179.970.191.765.739.670.7

With all agents updated on nearly every query, mixed-domain gradients conflict: the four Top-3 agents are broad but nearly identical, and none reaches the CERA-MoA peaks. So the gain comes from three-sample voting diversity rather than complementary expertise — at 817.97 tokens vs. 367.77 (Appendix E.4, Table 10). Thank you again for the constructive exchange.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.