- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ChurnMAS, an adaptive role–model routing framework for multi-agent systems operating under evolving LLM model pools. It maintains role-conditioned model profiles that combine prior capability descriptions with behavioral evidence accumulated during execution. A reinforcement-learning-based router dynamically selects agent roles and assigns available models based on task context, while adapting to newly introduced or retired models without updating routing parameters.
Critical strengths
- The combination of capability priors and continuously updated role-specific behavioral evidence provides a flexible mechanism for incorporating unseen models.
Critical weaknesses
-
W1. The contribution of behavioral evidence under model churn is not sufficiently isolated. Table 2 (p. 7) shows that removing behavioral evidence reduces performance by only 0.82 pp under the initial model pool, while larger differences appear under selected expanded pools. However, the paper does not compare against simpler alternatives such as periodically updating role-specific success statistics or using a frozen router with refreshed model profiles, making the necessity of the proposed mechanism unclear.
-
W2. The role suitability scores rely on an unvalidated LLM evaluator. Section 3.1 (p. 4) uses DeepSeek-V4-Flash to assign role fulfillment and suitability scores, which are then accumulated as behavioral evidence. Figure 5 (p. 8) visualizes these evaluator-derived scores as evidence of specialization, but no independent role-level ground truth or intervention validates whether higher suitability scores predict better downstream outcomes.
-
W3. Model churn adaptation is evaluated without measuring adaptation efficiency. Section 4.3 and Figure 4 (p. 7) test sequential additions of unseen models and report task accuracy after each addition. However, the paper does not show performance as a function of the number of observations collected for a new model, leaving its cold-start behavior, convergence speed, and adaptation costs unclear.
Questions for authors
-
Q1. Do the evaluator-generated role suitability scores predict actual downstream improvements when the same role is assigned to different models under controlled conditions?
-
Q2. How many executions are required for ChurnMAS to adapt to an unseen model, and how does its accuracy–cost trade-off compare with simple online routing baselines?