- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a competence-aware framework that adapts multi-agent collaboration to both the current query and the underlying language model. A fixed set of 48 probe questions yields a reusable model-capability fingerprint, which is combined with query embeddings and trained using measured single-model success rates. The resulting representation controls role activation and communication-graph generation, while competence-dependent regularization manages the collaboration budget. Evaluation covers six reasoning and coding benchmarks, multiple backbone models, component ablations, and additional experiments on unseen models and tasks. The authors report a six-benchmark average score of 93.81%, higher accuracy than G-Designer, and lower deployment-time token usage for stronger backbone models. The central contribution is using explicit backbone-model capability as a control signal for automatic multi-agent orchestration.
Critical strengths
-
The paper addresses a practical limitation of task-only orchestration: the same query may require different collaboration budgets for different backbone models. Reusing a compact capability fingerprint is a pragmatic way to incorporate model-specific information.
-
The empirical study covers six benchmarks, several dynamic-structure baselines, input ablations, and tests on unseen backbone models. Additional GPQA and BBEH-mini experiments provide useful evidence beyond the primary benchmark suite.
Critical weaknesses
-
The incremental value over closely related automatic orchestration methods remains insufficiently demonstrated. Although G-Designer, MasRouter, and CARD are included, direct comparisons with ARG-Designer, MetaGen and so on would better establish the contribution relative to other automatic role and topology design methods. Matching inputs and inference budgets would help isolate the added value of explicit competence modeling.
-
The role-budget prior explicitly encourages average activation to vary with “one minus standalone success rate,” and the communication penalty also depends on that rate. However, weaker standalone performance does not necessarily imply greater benefit from cooperation among multiple copies of the same model. Existing input ablations neither directly validate this relationship nor compare against simpler competence-based budget rules under matched collaboration budgets.
-
The efficiency analysis emphasizes deployment tokens and amortization of the 48-question registration cost rather than total system cost. Offline success-rate collection requires repeated trials across 14 models, followed by orchestration-model training, yet these costs are excluded from comparisons.
Questions for authors
See Weaknesses.
My central concern remains unanswered. ARG-Designer, MetaGen, and even more recent relevant methods already automate the generation or adaptation of agent roles and communication structures. Against this background, emphasizing that competence data and a trained orchestrator can be reused does not explain what additional value the proposed framework provides. Reusability alone is not a compelling advantage when alternative methods can already construct collaboration structures automatically, including through training-free inference-time adaptation. The relevant question is why explicit competence-data collection and dedicated orchestrator training are needed, and what they deliver beyond these alternatives. Without direct comparisons under comparable models, tasks, and resource budgets, the response establishes a difference in implementation, not a convincing incremental contribution.