acceptodds
Under review as a conference paper at ICLR 2027

Competence-Aware Orchestration for LLM-Based Multi-Agent Systems

Abstract

LLM-based multi-agent systems (MAS) have emerged as a promising paradigm for enhancing complex reasoning through role specialization and collaboration, yet a key challenge remains: how can collaboration be adapted to a backbone's competence on the current query? This problem matters because more collaboration is not always better—richer coordination may help on difficult cases, but it also incurs additional token cost and redundant interactions. We propose a two-stage competence-aware framework for automatic MAS orchestration. It first models the LLM backbone's task-conditioned competence for the current query, and then uses this signal to adapt role activation and the communication graph. Across six benchmarks, our method achieves an average score of 93.81%. Experiments with six LLM backbones of varying competence levels further show that our method adjusts collaboration according to the inferred competence: for higher-competence backbones, it improves average accuracy from 87.22 to 89.97 while reducing token cost by 36.26%; for lower-competence backbones, it yields a further average gain of 4.58 points over existing methods. The same trend also holds on unseen backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-agent systems, competence-aware orchestration, adaptive collaboration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cPRyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a competence-aware framework that adapts multi-agent collaboration to both the current query and the underlying language model. A fixed set of 48 probe questions yields a reusable model-capability fingerprint, which is combined with query embeddings and trained using measured single-model success rates. The resulting representation controls role activation and communication-graph generation, while competence-dependent regularization manages the collaboration budget. Evaluation covers six reasoning and coding benchmarks, multiple backbone models, component ablations, and additional experiments on unseen models and tasks. The authors report a six-benchmark average score of 93.81%, higher accuracy than G-Designer, and lower deployment-time token usage for stronger backbone models. The central contribution is using explicit backbone-model capability as a control signal for automatic multi-agent orchestration.

Critical strengths
  1. The paper addresses a practical limitation of task-only orchestration: the same query may require different collaboration budgets for different backbone models. Reusing a compact capability fingerprint is a pragmatic way to incorporate model-specific information.

  2. The empirical study covers six benchmarks, several dynamic-structure baselines, input ablations, and tests on unseen backbone models. Additional GPQA and BBEH-mini experiments provide useful evidence beyond the primary benchmark suite.

Critical weaknesses
  1. The incremental value over closely related automatic orchestration methods remains insufficiently demonstrated. Although G-Designer, MasRouter, and CARD are included, direct comparisons with ARG-Designer, MetaGen and so on would better establish the contribution relative to other automatic role and topology design methods. Matching inputs and inference budgets would help isolate the added value of explicit competence modeling.

  2. The role-budget prior explicitly encourages average activation to vary with “one minus standalone success rate,” and the communication penalty also depends on that rate. However, weaker standalone performance does not necessarily imply greater benefit from cooperation among multiple copies of the same model. Existing input ablations neither directly validate this relationship nor compare against simpler competence-based budget rules under matched collaboration budgets.

  3. The efficiency analysis emphasizes deployment tokens and amortization of the 48-question registration cost rather than total system cost. Offline success-rate collection requires repeated trials across 14 models, followed by orchestration-model training, yet these costs are excluded from comparisons.

Questions for authors

See Weaknesses.

Reviewer_tSpQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes competence-aware orchestration for LLM-based multi-agent systems, adapting roles and communication structures to different model backbones. For example, it can use simple roles/structures for smarter models and more complex roles/structures for weaker models.

To achieve this, it first constructs a task-conditioned competence representation and uses it to further predict role activation and the communication graph.

The whole prediction network is trained on 40 queries from each dataset. In total, six datasets are used for evaluation (MMLU, GSM8K, AQuA, MultiArith, SVAMP, HumanEval), and the proposed methods achieved better performance (93.8% on average) compared to the closest baseline, G-Designer (90.1%)

Critical strengths
  • The idea is neat: we probably do not need complex roles/structures for more capable models on easy problems. The authors further ground it with competence estimation and conditional roles/graph activation.
Critical weaknesses
  • The approach does not seem to be compatible with more recent agent benchmarks such as SWE-Bench and OSWorld. I would imagine the task-conditioned competence modeling will be way harder and less reliable when the agentic task is no longer a single query but a combination of a query and the whole environment (e.g., the whole codebase).
  • The method is somewhat very complex, and I am not sure about how practical/generalizable it is. For example, there are multiple training stages; are they stable enough? Are they sensitive to hyperparameters? For the graph-based MAS generation, it also involves RL; how stable is it? There are some other alternatives; for example, I just let gpt-5.6-luna/gpt-6-luna decide the role activation and communication graph. Will that be easier?
Questions for authors

See weakness above.

Reviewer_w13dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage framework for competence-aware multi-agent orchestration. It learns a query-conditioned competence representation from a backbone’s capability fingerprint and task information, then uses this representation to guide role activation and communication graph generation. Experiments across multiple benchmarks and backbones report improved accuracy and reduced deployment token usage in selected settings.

Critical strengths

The authors propose a novel architecture and orchestration algorithm that achieve performance improvements in the reported evaluations.

Critical weaknesses
  1. The main six-benchmark comparison lacks per-method inference costs and matched-budget evaluations. While the appendix reports token savings over G-Designer in selected settings, the evidence on harder benchmarks remains limited to one unseen backbone, with relatively modest accuracy gains.

  2. The method uses standalone competence to determine collaboration budgets and semantic relevance to supervise role activation. However, neither assumption is directly validated against the actual marginal benefits of collaboration or role specialization. The existing ablations do not isolate these design choices.

Questions for authors

Can you provide compute-matched comparisons against single-agent and representative MAS baselines, reporting both accuracy and total inference tokens?

Comments on the paper

2 comments

My central concern remains unanswered. ARG-Designer, MetaGen, and even more recent relevant methods already automate the generation or adaptation of agent roles and communication structures. Against this background, emphasizing that competence data and a trained orchestrator can be reused does not explain what additional value the proposed framework provides. Reusability alone is not a compelling advantage when alternative methods can already construct collaboration structures automatically, including through training-free inference-time adaptation. The relevant question is why explicit competence-data collection and dedicated orchestrator training are needed, and what they deliver beyond these alternatives. Without direct comparisons under comparable models, tasks, and resource budgets, the response establishes a difference in implementation, not a convincing incremental contribution.

Thank you for the additional clarification. I understand that conducting these additional experiments within such a short timeframe is not easy, and I hope the authors will include these comparisons in the revised version. If you do so, I will raise my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.