acceptodds
Withdrawn from ICLR 2027

ChurnMAS: Adaptive Role-Model Routing for LLM-based Multi-Agent Systems with Evolving Model Pools

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-based dynamic **m**ulti-**a**gent **s**ystems (MAS) increasingly adopt heterogeneous backbones to exploit complementary model capabilities across agent roles. However, most existing approaches focus on dynamically organizing collaboration structures while assuming a *fixed model pool*, overlooking continual model releases and retirements during deployment. Such model churn can invalidate existing role-model assignments, requiring routing mechanisms capable of adapting to changing model pools. To tackle this problem, we propose **ChurnMAS**, a framework that formulates role-model routing as a context-aware adaptation problem over dynamic model pools. ChurnMAS introduces *role-conditioned model profiles*, which unify capability priors with continuously accumulated behavioral evidence to characterize how different LLMs fit specific agent roles. Based on these evolving profiles, ChurnMAS dynamically reconfigures role-model assignments according to task requirements and changing model availability, enabling continual adaptation without *updating routing parameters* during deployment. Experiments on MATH, MMLU, MedQA, and MBPP demonstrate that ChurnMAS outperforms both single-model baselines and dynamic multi-agent methods. Further analyses show that role-conditioned model profiles effectively capture model specialization and facilitate adaptation to newly introduced LLM backbones under model churn. The code will be open-sourced.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Language Model Agents, Agents, Agentic Systems, Reasoning, LLM-based Multi-Agent System

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_saKBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ChurnMAS, an adaptive role–model routing framework for multi-agent systems operating under evolving LLM model pools. It maintains role-conditioned model profiles that combine prior capability descriptions with behavioral evidence accumulated during execution. A reinforcement-learning-based router dynamically selects agent roles and assigns available models based on task context, while adapting to newly introduced or retired models without updating routing parameters.

Critical strengths
  • The combination of capability priors and continuously updated role-specific behavioral evidence provides a flexible mechanism for incorporating unseen models.
Critical weaknesses
  • W1. The contribution of behavioral evidence under model churn is not sufficiently isolated. Table 2 (p. 7) shows that removing behavioral evidence reduces performance by only 0.82 pp under the initial model pool, while larger differences appear under selected expanded pools. However, the paper does not compare against simpler alternatives such as periodically updating role-specific success statistics or using a frozen router with refreshed model profiles, making the necessity of the proposed mechanism unclear.

  • W2. The role suitability scores rely on an unvalidated LLM evaluator. Section 3.1 (p. 4) uses DeepSeek-V4-Flash to assign role fulfillment and suitability scores, which are then accumulated as behavioral evidence. Figure 5 (p. 8) visualizes these evaluator-derived scores as evidence of specialization, but no independent role-level ground truth or intervention validates whether higher suitability scores predict better downstream outcomes.

  • W3. Model churn adaptation is evaluated without measuring adaptation efficiency. Section 4.3 and Figure 4 (p. 7) test sequential additions of unseen models and report task accuracy after each addition. However, the paper does not show performance as a function of the number of observations collected for a new model, leaving its cold-start behavior, convergence speed, and adaptation costs unclear.

Questions for authors
  • Q1. Do the evaluator-generated role suitability scores predict actual downstream improvements when the same role is assigned to different models under controlled conditions?

  • Q2. How many executions are required for ChurnMAS to adapt to an unseen model, and how does its accuracy–cost trade-off compare with simple online routing baselines?

Reviewer_Th67OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies role-model routing in LLM-based multi-agent systems under model churn. The authors propose ChurnMAS and model task execution as a contextual MDP. Each model has a profile that combines natural-language capability priors with role-specific behavioral evidence. The Router selects roles based on the task state, then assigns models using role-conditioned representations and model profiles. REINFORCE trains the routing policy. During execution, a separate evaluator gives process rewards and feedback on role suitability to update the profiles. At deployment, the Router's parameters stay fixed while its context and behavior statistics change.

Critical strengths

The paper explains how model releases and retirements can make existing role-model assignments less useful, beyond the problem of choosing a collaboration structure. Updating profiles gives a clear way to learn from experience while keeping the Router's parameters fixed.

In addition to sequential model addition, Appendix D.3 tests model removal and replacement on MBPP. Table 2 also gives initial evidence for the effects of role selection, model selection, and behavioral evidence.

Critical weaknesses

Sections 3.1–3.3 use DeepSeek-V4-Flash throughout training and deployment. Table 1 matches the model pool but does not establish equal evaluator information, call counts, or total budgets.

Section 4.1 says evidence builds up within each dataset. It does not clearly state how profiles are initialized or reset between training, testing, and model-addition stages. Task order and reuse of test tasks are also unclear. Figures 4, 7, and 8 mainly show final accuracy for each pool.

The w/o pbeh ablation in Table 2 removes all behavioral evidence, so it does not isolate role conditioning. The suitability scores in Figures 5–6 also come from the same evaluator and do not independently confirm actual task contribution. Please compare profiles without role information and profiles with shuffled role labels, using the same amount of evidence.

Questions for authors

Need provide the actual capability-prior text for each model, how it was created, and when it was created. Tests that remove model names or change prior wording would help show how much unseen-model routing depends on human-written priors and model identity.

Reviewer_nXKFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ChurnMAS, a framework for adapting role–model assignments in LLM-based multi-agent systems (MAS) when the available LLM backbones change over time. The authors identify model churn, including the addition, removal, and replacement of LLMs, as an important practical challenge that existing dynamic MAS methods do not explicitly address.

Critical strengths

1.The role-conditioned model-profile design is intuitive and potentially useful. 2.The experimental evaluation covers multiple task domains.

Critical weaknesses

1.My main concern is the significance of this work. Although the problem may be novel, I don’t understand how it can be applied to the real world. A real-world motivating example is expected.

2.The technical novelty, to me, is limited. The research problem, basically, is a role-allocation problem, and the key component in the methodology is an RL-based optimization. This is an unsurprisingly application.

3.The proposed adaptation mechanism relies heavily on evaluator-generated feedback. So, the effectiveness of the framework depends heavily on the reliability of this evaluator. The adopted evaluator is DeepSeek-V4-Flash. I suggest testing different evaluators as an ablation study.

Questions for authors
  1. Figure 1 shows that model releases and retirements happen frequently. However, it is still unclear how much this problem affects real-world multi-agent systems. Could the authors provide a concrete deployment scenario where model churn significantly impacts system performance, and explain why existing routing approaches would be insufficient?

  2. The proposed method mainly combines role-conditioned model profiles with RL-based routing. What is the key technical advantage over simpler approaches, such as selecting models based on historical role-specific performance scores? Would a simpler non-RL routing strategy achieve similar results?

  3. DeepSeek-V4-Flash is used to generate both training rewards and behavioral feedback for model profiles. How sensitive are the results to the choice of evaluator? Would the framework still perform well with a different or less capable evaluator? Also, since DeepSeek-V4-Flash is included in the test-time model pool, could this introduce evaluation bias?

  4. The authors emphasize that ChurnMAS can adapt to newly introduced models without updating the routing parameters. However, the framework still needs to accumulate behavioral evidence through evaluator feedback. How many interactions are typically required before a new model can be effectively assigned to different roles, and what is the additional evaluation cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.