acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Multi-agent Orchestration

Abstract

Learned orchestration can automatically construct effective language-model multi-agent systems, but existing approaches couple planning to fixed worker pools and train decomposition and collaboration from the same terminal outcome, limiting transfer and obscuring credit assignment. We introduce DeOrch, which separates worker-agnostic planning from concrete worker selection. Its two-stage planner first decomposes the task without worker information, then chooses collaboration operations using compact, worker-identity-free matchability feedback from the pool, enabling conditional credit assignment to decomposition and collaboration decisions. A lightweight matcher estimates worker suitability from behavior on a fixed probe set and adapts online with a contextual bandit, allowing new workers to be incorporated without retraining the planner or matcher. Across diverse in- and out-of-distribution tasks, DeOrch outperforms prior automatic MAS orchestration methods with fewer worker calls than competing learned orchestrators, remains effective when transferred to an entirely unseen worker pool without retraining, and shows consistent gains from both components.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent, orchestration, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TocYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DEORCH which separates task planning from worker selection. A planner first decomposes the task, then chooses collaboration operations using an identity-free summary of the worker pool. A probe-based matcher assigns workers, while nested rollouts provide separate training signals for decomposition and collaboration. The method improves results across eight benchmarks and transfers to an unseen worker pool without planner retraining. The matcher uses a fixed probe set to estimate worker suitability and complementarity. During training, multiple collaboration plans are evaluated under each fixed decomposition, so the planner can compare collaboration choices without changing the upstream task structure. The full method scores 51.5 versus 49.1 for MAS-Orchestra, while using slightly fewer worker calls. Online bandit updates are optional and disabled in the main comparison.

Critical strengths
  • The unseen-pool experiment directly tests the motivation. With online adaptation disabled, DEORCH scores 50.2, compared with 47.5 for Independent and 40.9 for Conductor (Table 2). This is useful evidence that separating the planner from worker identities helps transfer.
  • The training ablations are informative. Conditional credit raises the average before adding the cost penalty The appendix also includes rollout-budget matching, a counterfactual-credit control, and token accounting beyond worker calls.
  • The cost-aware objective also produces a sensible trade-off. Together with the token comparison against single-model refinement, this makes the result stronger than a claim based only on calling more models. The paper has already addressed several obvious compute confounds.
Critical weaknesses
  • The results leave me unsure how much of the benefit actually comes from multi-agent collaboration. In Table 3, decomposing the task and using a single worker per subtask already achieves 50.4, compared with 51.5 for the full method, while requiring 1.68 rather than 5.35 worker calls. This suggests that much of the gain may come from better decomposition and worker selection. The paper gives useful comparisons against other systems, but says less about what kinds of problems genuinely need its learned collaboration.
  • This also affects the novelty assessment. Learned orchestration and routing to unseen models already have precedents in some existing works like MAS-Orchestra and UniRoute (Generally we do not require direct comparisons with arXiv papers, but this is not a new idea). Combining these ideas is reasonable, but the paper needs a clearer account of what the combination makes possible beyond a good task decomposer with a capable router. The strongest simple variant is also missing from the unseen-pool comparison, where it would be particularly informative.
  • The central assumption behind decoupling deserves more scrutiny. Hiding worker identities makes sense for transfer, but DEORCH goes further: it chooses the task decomposition before seeing the current pool's capabilities. A useful decomposition may depend on who is available to execute it. A pool of narrow specialists could need a different division of work from a pool of strong generalists, and choosing collaboration operations afterward may not repair an unsuitable decomposition.
  • In transfer results, one replacement pool does not establish when this restriction helps or when it becomes a bottleneck. More fundamentally, independence from worker names does not require independence from worker capabilities. The paper has not made a convincing case that withholding capability information at the decomposition stage is the right trade-off, rather than simply a convenient way to make the interface transferable.
Questions for authors
  • Where does collaboration make a meaningful difference beyond decomposition and routing? Are the gains concentrated in tasks that require complementary workers, or are they small improvements spread across benchmarks? How does Decomposition + Single perform on the unseen pool?
  • Why should task decomposition be independent of the available capabilities? If a new pool has a very different balance of strengths, can the second-stage planner compensate without changing the decomposition? A concrete success or failure case would help clarify the limits of the design.
  • What is the main new lesson from DEORCH beyond combining a learned planner with a transferable router? Which result best shows a benefit that this simpler combination would miss?
7 comments

Dear reviewer TocY, We sincerely thank the reviewer for the comments and constructive feedback. We address each concern in detail below.

[W1 & Q1: Contribution of Learned Collaboration] We respectfully point out that the relatively modest gap between Decomposition + Single and DeOrch does not diminish the value of learned collaboration. (1) Table 3 shows that learned operation selection improves average performance from 50.4 to 51.5. This improvement is consistent across all eight benchmarks: MuSiQue (+1.1), TravelPlanner (+1.4), Omni-MATH-2 (+1.2), LiveCodeBench v6 (+0.8), BBEH (+0.6), NaturalPlan (+0.9), PlanBench-mystery (+1.0), and GAIA (+2.4). These consistent gains demonstrate that the benefits of collaboration are not confined to particular task types. This is consistent with prior findings that multi-agent collaboration, such as debate and ensemble, can improve performance across diverse tasks. While Decomposition + Single reduces worker calls from 5.35 to 1.68, the additional performance gain of full DeOrch can be valuable in accuracy-oriented scenarios where higher inference cost is acceptable. (2) Following the reviewer's suggestion, we additionally evaluated Decomposition + Single on the unseen worker pool without retraining. The results are as follows:

MethodMuSiQueTPOmniLCBBBEHNPPB-MGAIAAvg.
Decomposition + Single74.010.251.582.547.761.042.321.248.8
DeOrch75.511.352.383.049.762.344.323.050.2

DeOrch consistently outperforms Decomposition + Single across all eight benchmarks (+1.4 points on average), further demonstrating that learned operation selection remains effective when transferring to unseen worker pools. (3) To further illustrate when learned collaboration is beneficial, we provide a concrete case study from BBEH (zebra_puzzles-128), a challenging constraint puzzle with 142 clues. Both DeOrch and Decomposition + Single use the same two-subtask decomposition: solving the puzzle and extracting the final answer. However, DeOrch selects DEBATE-3 for the solving subtask, where three workers produce revised answers (3, 2, 3), leading to the correct final answer (3). In contrast, Decomposition + Single uses SINGLE-1 and incorrectly predicts 5. Although DeOrch requires more worker calls (7 vs. 2), this case demonstrates how learned operation selection can improve correctness without changing the task decomposition. We have also included the complete case study in the revised manuscript.

[W2 & Q3: Novelty and Technical Contributions] We respectfully believe that the novelty of DeOrch has been underestimated. While DeOrch addresses the same automatic MAS orchestration problem as Conductor and MAS-Orchestra, it introduces fundamentally different methods to overcome their limitations. Our core technical contributions are twofold: (1) Matchability-informed two-stage RL training. This enables more effective credit assignment for task decomposition and operation selection. To the best of our knowledge, we are the first to introduce this two-stage RL paradigm for disentangled credit assignment. (2) Decoupled orchestration architecture and a novel matcher. On paradigm novelty, DeOrch is the first learned MAS orchestration framework to decouple planning from learnable worker selection, eliminating the need to retrain the LLM-based orchestrator when transferring to unseen workers. On technical novelty, unlike existing routers such as UniRoute, our matcher combines density-corrected similarity-based matching with online contextual bandit adaptation and supports complementarity-aware multi-worker selection. In short, we propose a better planner, a better matcher combining offline and online routing, and an effective integration of the two through matchability feedback, rather than simply combining an existing learned planner with an existing transferable router. Table 3 demonstrates the effectiveness of our novel two-stage RL training framework (49.6 → 51.9), Tables 4–5 validate our matcher, and Tables 1–2 establish overall performance and transferability advantages.

[W3 & Q2: Worker-Agnostic Task Decomposition] We respectfully believe that worker-agnostic task decomposition is a reasonable and effective design choice. (1) Task decomposition primarily depends on the task's semantic structure and dependencies. While worker capabilities can help adjust subtask granularity and difficulty, they are not essential for producing an effective decomposition. (2) A natural way to make decomposition depend on worker capabilities would be to iteratively refine the decomposition based on matchability feedback. However, this would incur substantial additional training and inference costs. (3) Even when the initial decomposition is suboptimal for the current worker pool, Stage II can compensate by selecting stronger collaboration operations, such as DEBATE or ENSEMBLE, based on matchability feedback. (4) Even within our current framework, multi-round execution could allow the planner to implicitly adapt decomposition to worker capabilities by leveraging execution traces from previous rounds. Our experiments use a single reasoning round for fair comparison with baselines, while exploring multi-round reasoning is left for future work. (5) The case study provided in our response to W1 & Q1 illustrates this flexibility: given the same decomposition, DeOrch successfully resolves a subtask through DEBATE where single-worker execution fails. Together with our experimental results, this supports the effectiveness of the current design. We view capability-aware decomposition as an interesting future direction rather than a prerequisite for the contributions of this work.

[W4: Transfer Evaluation on a New Worker Pool] To address the reviewer's concern, we conducted an additional transfer experiment on a pool of four workers from entirely unseen model families (Falcon-H1-7B-Instruct, Phi-4, OLMo-3-7B-Instruct, and Llama-3.1-8B-Instruct), without retraining the planner or matcher. The results are shown below:

MethodMuSiQueTPOmniLCBBBEHNPPB-MGAIAAvg.
Best Single Agent48.42.815.830.915.821.20.56.717.8
Independent55.93.719.932.916.416.10.06.719.0
Frontier-Planner53.42.918.337.615.817.21.27.919.3
Conductor53.93.318.835.415.316.50.56.718.8
DeOrch60.26.423.039.417.822.12.711.522.9

Together with our original transfer experiment, these results demonstrate DeOrch's effectiveness across different unseen worker pools, including entirely unseen model families. For the rationale behind worker-agnostic decomposition, please refer to our response to W3 & Q2.

We sincerely thank the reviewer again for the constructive feedback, which has helped us improve the clarity and completeness of our paper. We hope our responses adequately address the concerns and welcome further discussion. Finally, we would like to reiterate our two core contributions: (1) On technical novelty, we are the first to introduce this two-stage RL paradigm for disentangled credit assignment. (2) On technical and paradigm novelty, we introduce a novel matcher combining offline probe-based matching with online bandit adaptation and supporting multi-worker selection, together with the first MAS orchestration framework to decouple planning from learnable worker selection. Both contributions are supported by extensive experiments across diverse benchmarks. We hope these clarifications and additional results will strengthen the reviewer's assessment of our work and encourage reconsideration of the overall rating.

Dear Reviewer TocY,

Thank you again for your insightful comments. We have provided additional experiments and detailed clarifications addressing your concerns, including the Decomposition + Single comparison on an unseen worker pool, a concrete collaboration case study, and further discussion of our technical contributions and design choices. As the discussion deadline is approaching, we would greatly appreciate any feedback on our rebuttal, particularly whether our additional evidence and explanations have addressed your concerns. We would be happy to clarify any remaining questions before the deadline.

Thank you for your time and consideration.

Reviewer_nH1XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces DeOrch, a method for orchestrating multiple language-model agents. Instead of jointly deciding task decomposition and worker assignments, it first produces a worker-agnostic decomposition. A matcher then summarizes how well the available workers support each subtask. Using that feedback, the planner chooses collaboration operations—such as single-worker execution, ensemble, debate, or review—and the matcher assigns concrete workers. They train the planner with separate learning signals for decomposition and collaboration, obtained by trying multiple collaboration plans for each decomposition. They evaluate using six workers ranging from roughly 4B to 14B parameters, across benchmarks including MuSiQue, math, coding, planning, and tool-use tasks. They compare against Conductor and other baselines, and test transfer to an unseen worker pool.

Critical strengths

There are two strenghts of the paper:

  • The paper presents a well-motivated idea: the separation of task decomposition, collaboration planning, and worker selection. Its conditional credit-assignment method provides distinct learning signals by comparing collaboration plans under a fixed decomposition and averaging their outcomes to evaluate decomposition. Transfer experiments with an unseen worker pool support the practical value of this design.

  • The evaluation spans diverse tasks and includes controlled ablations to assess individual components. The reported results demonstrate a useful accuracy–cost trade-off, with higher average performance and fewer worker calls and inference tokens than competing learned orchestrators.

Critical weaknesses

There are three main weaknesses of this paper.

  • The method presentation is difficult to follow. Section 3 introduces dense matcher details before clearly establishing the full procedure; a high-level algorithm block would help. Figure 1 is visually crowded, yet not very informative. and it does not clearly distinguish planner training from online matcher adaptation or explain how worker-pool information and matchability feedback enter the workflow. Section 3 overloaded notation, e.g. D for the decomposition graph versus and D(s) for capability breadth, and L for the number of subtasks and L(s) for spoilage mass, further reduce readability.

  • The experimental reporting is insufficient to assess reliability and reproducibility. Repeated-run results and uncertainty estimates are not reported, and important details about worker generation settings, collaboration protocols, and baseline configurations remain unclear.

  • The analysis provides limited insight into the orchestration strategies the planner learns, with no inspection of learned task graphs or collaboration choices. Limitations and future work receive little attention in the main text and the relevant discussion in Appendix B.8 is not referenced there.

Questions for authors
  1. Can the authors provide a overview of the training procedures, clarifying how planner training and online matcher adaptation differ, which components are updated or kept fixed, and how execution rewards are used in each phase?

  2. Can the authors report results across independent evaluation runs with repeated worker executions and include measures of variability, such as standard deviations?

  3. Can the authors discuss the cost more thoroughly and explain when learned collaboration justifies the additional expense? Table 3 shows 50.4 with 1.68 calls for “Decomposition + Single,” versus 51.5 with 5.35 calls for full DeOrch. Can task-level results or examples of learned strategies explain where collaboration helps?

5 comments

Dear reviewer nH1X, We thank the reviewer for the comments and constructive feedback. We respond point by point below.

[W1 & Q1: Method Presentation and Training Procedure] We are sorry that the reviewer found parts of our method difficult to follow. To improve clarity, we have made the following revisions and explanations: (1) We added a high-level algorithm block summarizing the complete training and deployment procedures. (2) We refined Figure 1 to reduce visual clutter and clarify the distinction between offline planner training and optional deployment-time online adaptation. We also expanded the caption to explain how worker-pool information and matchability feedback enter the workflow. (3) To further improve notation clarity, we made several minor symbol changes: D(s)D(s) (capability breadth) → H(s)H(s), L(s)L(s) (spoilage mass) → V(s)V(s), and BB (worker team) → T\mathcal{T}, making them more visually distinct from DD (decomposition DAG), LL (number of subtasks), and B(s)B(s) (best-worker strength), respectively. (4) We further clarify that online adaptation is an optional plug-in used only at deployment, independent of planner training. Specifically, we first train the planner without online adaptation: the probe-based matcher remains fixed, and terminal execution rewards are used to train the planner through RL. Once training is complete, the planner is frozen. During deployment, online adaptation can optionally be enabled as a plug-in, in which case execution feedback updates only the contextual bandit's posterior, while the planner and static matcher remain fixed.

[W2 & Q2: Experimental Reliability] (1) [On Standard Deviations] Our original evaluations use greedy decoding (temperature = 0) for both the planner and workers, making repeated executions under identical conditions largely deterministic. Moreover, DeOrch consistently achieves the best overall performance across eight diverse benchmarks and maintains strong performance on unseen worker pools, providing strong empirical evidence of its effectiveness and generalizability. To further address the reviewer's concern, we conducted three independent evaluation runs with stochastic planner decoding (temperature = 0.5), while keeping worker decoding deterministic (temperature = 0). The results (mean ± std) are shown below:

MethodMuSiQueTPOmniLCBBBEHNPPB-MGAIAAvg.
Frontier-Planner59.6 ± 0.73.7 ± 0.838.7 ± 0.271.3 ± 0.537.1 ± 0.446.1 ± 0.59.9 ± 0.527.7 ± 0.936.8 ± 0.3
Untrained Planner60.7 ± 0.53.6 ± 0.338.5 ± 0.270.9 ± 0.432.1 ± 0.350.2 ± 0.325.6 ± 0.826.9 ± 0.938.6 ± 0.3
Conductor70.7 ± 0.510.1 ± 0.541.3 ± 0.375.4 ± 0.438.1 ± 0.553.3 ± 0.238.2 ± 0.428.7 ± 0.344.5 ± 0.2
MAS-Orchestra73.7 ± 0.215.0 ± 0.349.2 ± 0.377.1 ± 0.745.2 ± 0.261.9 ± 0.441.7 ± 0.329.5 ± 0.749.2 ± 0.1
Decomposition + Single75.8 ± 0.216.3 ± 0.250.8 ± 0.178.3 ± 0.347.6 ± 0.164.2 ± 0.242.4 ± 0.328.9 ± 0.350.5 ± 0.0
DeOrch76.8 ± 0.617.8 ± 0.752.3 ± 0.379.5 ± 0.748.3 ± 0.265.1 ± 0.343.9 ± 0.531.3 ± 0.351.9 ± 0.2

These results show that DeOrch consistently outperforms all baselines across all eight benchmarks, including Decomposition + Single, even when accounting for variability across independent runs, further confirming the robustness of its performance advantages. (2) [On Worker Generation Settings] We have added the following worker generation settings to Appendix B.3. Frozen workers use greedy decoding (temperature = 0), with thinking disabled and a 40,960-token context window. Output limits are 2,048/4,096 tokens (intermediate/final) by default, increased to 4,096/8,192 for code tasks and 8,192/8,192 for TravelPlanner. ReAct workers are limited to 3,000 tokens per interaction. The same worker generation settings are applied to all baselines for fair comparison. (3) [On Collaboration Protocols] We have added detailed collaboration execution protocols in the Appendix, covering all five operations, intermediate result propagation, DAG scheduling, and terminal-node constraints. (4) [On Baseline Configurations] Section 4.1 and Appendix B.3 already provide the key baseline configurations, including worker-pool adaptations, training rollout budget matching, and differences from the original implementations, while otherwise following the original papers. To address the reviewer's concern, we have additionally included the relevant training hyperparameters from the original Conductor and MAS-Orchestra implementations in Appendix B.3. Both Conductor and MAS-Orchestra are trained on the same training data as DeOrch.

[W3: Learned Strategies and Limitations] To provide more insight into the learned orchestration strategies, we added a case study from BBEH (zebra_puzzles-128). The planner decomposes the task into two subtasks: solving a challenging constraint puzzle and extracting the final answer. It selects DEBATE-3 for the difficult solving subtask and SINGLE-1 for the final extraction. The debate produces three revised answers (3, 2, 3), leading to the correct final answer (3), whereas Decomposition + Single produces an incorrect answer (5) under the same decomposition. This illustrates how the planner assigns different collaboration operations based on subtask needs, highlighting the value of learned collaboration. We have also included the detailed case study in the revised manuscript. Regarding limitations and future work, we have already provided a detailed discussion in Appendix B.10. Due to space constraints, we placed this discussion in the Appendix and have now added an explicit reference in Section 5 (Conclusion).

[Q3: Cost and Benefits of Learned Collaboration] Table 3 shows that learned operation selection improves average performance from 50.4 to 51.5. This improvement is consistent across all eight benchmarks: MuSiQue (+1.1), TravelPlanner (+1.4), Omni-MATH-2 (+1.2), LiveCodeBench v6 (+0.8), BBEH (+0.6), NaturalPlan (+0.9), PlanBench-mystery (+1.0), and GAIA (+2.4). While Decomposition + Single reduces worker calls from 5.35 to 1.68, the additional performance gain of full DeOrch can be valuable in accuracy-oriented scenarios where higher inference cost is acceptable. Our case study (see our response to W3) further illustrates this value: both variants produce the same decomposition, but DeOrch selects DEBATE for a subtask and obtains the correct final answer, whereas Decomposition + Single fails (7 vs. 2 worker calls). This demonstrates how learned collaboration can justify additional worker calls by improving task correctness.

We sincerely thank the reviewer again for the constructive feedback, which has helped us improve the clarity, reproducibility, and completeness of our paper. Finally, we would like to reiterate our two core contributions: (1) On technical novelty, we are the first to introduce this two-stage RL paradigm for disentangled credit assignment. (2) On technical and paradigm novelty, we introduce a novel matcher combining offline probe-based matching with online bandit adaptation and supporting multi-worker selection, together with the first MAS orchestration framework to decouple planning from learnable worker selection. Both contributions are supported by extensive experiments and controlled ablations. We hope these clarifications and additional evidence will strengthen the reviewer's assessment of our work and encourage reconsideration of the overall rating. We welcome further discussion.

Dear Reviewer nH1X,

Thank you again for your detailed and constructive suggestions. In response, we have revised the method presentation, clarified implementation details, conducted additional evaluations with standard deviations, and provided a case study illustrating the benefits of learned collaboration. As the discussion deadline is approaching, we would greatly appreciate any feedback on our rebuttal and would be happy to address any remaining questions or concerns before the deadline.

Thank you for your time and consideration.

Reviewer_iAtsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DeOrch, a MAS orchestration method that addresses two limitations of existing approaches: they couple planning to a fixed worker pool, and they train task decomposition and collaboration decisions from the same terminal reward. DeOrch decouples worker-agnostic planning from concrete worker selection. Its planner works in two stages, first decomposing the task and then choosing collaboration operations; during RL training, multiple collaboration plans are sampled for each decomposition, so that separate credit can be assigned to the two stages from the same terminal reward. In addition, a lightweight matcher builds a behavioral signature for each worker from its outcomes on a fixed probe set and uses it to select suitable workers for each subtask, so that new workers can be added without retraining. Experiments show that DeOrch outperforms existing automatic MAS orchestration methods while using fewer worker calls than other learned orchestrators, and remains effective on an unseen worker pool without retraining.

Critical strengths
  • Through two-stage planning and nested conditional sampling, the method assigns separate credit to task decomposition and collaboration decisions: it compares different collaboration plans under the same decomposition, and compares the average downstream quality across decompositions. This yields separate learning signals for the two types of decisions from the same terminal reward, without requiring additional process rewards.

  • The matcher estimates each worker's suitability for a subtask from its success/failure signature on a fixed probe set, and summarizes the capability, complementarity, and probe support of the worker pool into six-dimensional, worker-identity-free matchability features for the planner. As a result, a new worker can be incorporated simply by evaluating it once on the probe set, without retraining the planner or the matcher. On an entirely unseen worker pool, the method still outperforms all baselines.

Critical weaknesses
  • The models in the new worker pool (gemma-4-26B-A4B, Qwen3-14B, granite-4.2-8b, and Ministral-3-8B) are not truly unseen by the trained planner, as they come from the same model families (gemma-4, Qwen3, granite-4.x, and Ministral-3) as those in the training pool. Since models from the same family may share similar strengths, the matchability features the planner receives are likely close to those seen during training, so this experiment provides only limited evidence of transferability.
Questions for authors
  • Could you evaluate the method on a worker pool containing models from unseen families, or models that are much stronger or weaker than those in the training pool?
  • Beyond selecting the best-matched worker, can the method account for the cost–performance trade-off between small and large models, e.g., assigning small models to simple subtasks to save cost and large models to complex subtasks to ensure performance?
3 comments

Dear reviewer iAts, We thank the reviewer for the insightful comments and constructive suggestions, which are valuable for improving our paper. We address the concerns below.

[W1 & Q1: Generalization to Unseen Worker Pools] (1) Although the new workers belong to the same model families as those in the training pool, differences in model sizes and versions can lead to distinct capabilities, resulting in different probe signatures, suitability estimates, and matchability features. (2) As shown in Table 2, Conductor's performance drops from 44.0 to 40.9 on the unseen pool, while DeOrch maintains strong performance (51.5 to 50.2). This demonstrates that changes in the worker pool pose a genuine challenge to learned orchestrators that depend on worker identities. (3) To further address this concern, we constructed an additional pool of four workers (Falcon-H1-7B-Instruct, Phi-4, OLMo-3-7B-Instruct, and Llama-3.1-8B-Instruct) from entirely unseen model families and evaluated transfer without retraining the planner or matcher. The results are shown below:

MethodMuSiQueTPOmniLCBBBEHNPPB-MGAIAAvg.
Best Single Agent48.42.815.830.915.821.20.56.717.8
Independent55.93.719.932.916.416.10.06.719.0
Frontier-Planner53.42.918.337.615.817.21.27.919.3
Conductor53.93.318.835.415.316.50.56.718.8
DeOrch60.26.423.039.417.822.12.711.522.9

DeOrch outperforms all baselines across all eight benchmarks, achieving an average of 22.9 compared with 19.3 for the strongest baseline. Together with our original unseen-pool experiment, these results provide strong evidence of DeOrch's transferability across different worker pools, including entirely unseen model families, without retraining.

[Q2: Cost–Performance Trade-off] DeOrch currently optimizes worker calls rather than worker-specific inference costs. This is a deliberate scope choice, as our primary contributions lie in improving MAS orchestration rather than developing a cost-aware LLM router. Importantly, DeOrch's modular architecture allows cost-aware routing methods to be incorporated into worker selection while preserving the planner-facing interface. We view this as an interesting direction for future work.

We sincerely thank the reviewer again for the constructive feedback. We would like to reiterate our two core contributions: (1) A novel matchability-informed two-stage RL training framework that enables more effective credit assignment for task decomposition and operation selection. (2) A novel decoupled orchestration architecture that integrates a worker-agnostic planner with a lightweight matcher combining offline probe-based matching and online bandit adaptation, enabling transferable worker selection and pool-aware collaboration through matchability feedback. Both contributions are supported by extensive experiments. We hope our clarifications and additional results will further strengthen the reviewer's assessment of our work and encourage reconsideration of the overall rating.

Dear Reviewer iAts,

Thank you again for your constructive feedback. We have provided additional experimental evidence addressing your main concern about transferability, including a new evaluation on a worker pool comprising entirely unseen model families. As the discussion deadline is approaching, we would greatly appreciate any feedback on our rebuttal, particularly whether our additional results have addressed your concerns. We would be happy to clarify any remaining questions before the deadline.

Thank you for your time and consideration.

Reviewer_iAtsOpenReview ↗

Response to the Authors

Thank you for the detailed response and the additional experiments. The new results on a worker pool drawn entirely from unseen model families address my concern about transferability, and I find the clarification on the scope of cost-aware routing reasonable. My questions have been resolved.

After considering the rebuttal, I believe my current score already reflects my overall assessment of the paper, so I will keep it unchanged. I encourage the authors to include the new unseen-family experiment in the revised version, as it substantially strengthens the claim of transferability without retraining.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely appreciate your time and efforts in handling our submission, and we fully understand the substantial workload involved in serving as an Area Chair. Nevertheless, given the highly marginal ratings of 3 (confidence 4), 2 (confidence 3), and 2 (confidence 2), we respectfully ask that you personally examine our paper and rebuttal and provide an independent assessment of its merits.

Regarding the rebuttal outcomes:

  1. Reviewer iAts (3, confidence 4) explicitly confirmed that our additional experiments resolved the transferability concern and that our clarification of cost-aware routing was reasonable, stating, "My questions have been resolved."
  2. Reviewer nH1X (2, confidence 3) has not participated in the discussion. We believe our additional repeated evaluations with standard deviations, detailed case study, and improvements to method presentation and reproducibility substantially address this reviewer's concerns.
  3. Reviewer TocY (2, confidence 2) acknowledged our additional evidence but retained reservations about worker-pool generalization and collaboration costs. We respectfully disagree with these remaining concerns. Our experiments on two substantially different unseen worker pools, including one with entirely unseen model families and much weaker capabilities, consistently demonstrate DeOrch's superiority over existing baselines such as Conductor without retraining. While no finite evaluation can cover every possible worker distribution, this does not diminish the demonstrated advancement over existing methods. Regarding collaboration costs, Decomposition + Single is itself a lightweight configuration of our framework, while additional accuracy through more intensive collaboration is a well-established and valuable trade-off in accuracy-prioritized settings.

We firmly believe DeOrch makes three significant contributions: (1) the first two-stage RL framework for disentangled credit assignment in orchestration; (2) a novel matcher integrating offline profiling, online adaptation, and complementarity-aware worker selection; and (3) a decoupled planner–matcher architecture enabling transfer to unseen worker pools without retraining.

Our broader vision is to enable effective orchestration of heterogeneous small language models, leveraging their complementary capabilities to achieve competitive performance. We firmly believe that DeOrch represents a substantial technical and architectural advance over existing automatic MAS orchestration methods. While we openly acknowledge its limitations in the paper, we believe that these limitations should be weighed against its demonstrated contributions, rather than overshadowing them. A single paper need not exhaustively solve every possible variation of a problem to make a meaningful advance.

We sincerely hope you will carefully consider the paper, the additional experimental evidence, and the full rebuttal discussion when forming your independent recommendation.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.