- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DEORCH which separates task planning from worker selection. A planner first decomposes the task, then chooses collaboration operations using an identity-free summary of the worker pool. A probe-based matcher assigns workers, while nested rollouts provide separate training signals for decomposition and collaboration. The method improves results across eight benchmarks and transfers to an unseen worker pool without planner retraining. The matcher uses a fixed probe set to estimate worker suitability and complementarity. During training, multiple collaboration plans are evaluated under each fixed decomposition, so the planner can compare collaboration choices without changing the upstream task structure. The full method scores 51.5 versus 49.1 for MAS-Orchestra, while using slightly fewer worker calls. Online bandit updates are optional and disabled in the main comparison.
Critical strengths
- The unseen-pool experiment directly tests the motivation. With online adaptation disabled, DEORCH scores 50.2, compared with 47.5 for Independent and 40.9 for Conductor (Table 2). This is useful evidence that separating the planner from worker identities helps transfer.
- The training ablations are informative. Conditional credit raises the average before adding the cost penalty The appendix also includes rollout-budget matching, a counterfactual-credit control, and token accounting beyond worker calls.
- The cost-aware objective also produces a sensible trade-off. Together with the token comparison against single-model refinement, this makes the result stronger than a claim based only on calling more models. The paper has already addressed several obvious compute confounds.
Critical weaknesses
- The results leave me unsure how much of the benefit actually comes from multi-agent collaboration. In Table 3, decomposing the task and using a single worker per subtask already achieves 50.4, compared with 51.5 for the full method, while requiring 1.68 rather than 5.35 worker calls. This suggests that much of the gain may come from better decomposition and worker selection. The paper gives useful comparisons against other systems, but says less about what kinds of problems genuinely need its learned collaboration.
- This also affects the novelty assessment. Learned orchestration and routing to unseen models already have precedents in some existing works like MAS-Orchestra and UniRoute (Generally we do not require direct comparisons with arXiv papers, but this is not a new idea). Combining these ideas is reasonable, but the paper needs a clearer account of what the combination makes possible beyond a good task decomposer with a capable router. The strongest simple variant is also missing from the unseen-pool comparison, where it would be particularly informative.
- The central assumption behind decoupling deserves more scrutiny. Hiding worker identities makes sense for transfer, but DEORCH goes further: it chooses the task decomposition before seeing the current pool's capabilities. A useful decomposition may depend on who is available to execute it. A pool of narrow specialists could need a different division of work from a pool of strong generalists, and choosing collaboration operations afterward may not repair an unsuitable decomposition.
- In transfer results, one replacement pool does not establish when this restriction helps or when it becomes a bottleneck. More fundamentally, independence from worker names does not require independence from worker capabilities. The paper has not made a convincing case that withholding capability information at the decomposition stage is the right trade-off, rather than simply a convenient way to make the interface transferable.
Questions for authors
- Where does collaboration make a meaningful difference beyond decomposition and routing? Are the gains concentrated in tasks that require complementary workers, or are they small improvements spread across benchmarks? How does Decomposition + Single perform on the unseen pool?
- Why should task decomposition be independent of the available capabilities? If a new pool has a very different balance of strengths, can the second-stage planner compensate without changing the decomposition? A concrete success or failure case would help clarify the limits of the design.
- What is the main new lesson from DEORCH beyond combining a learned planner with a transferable router? Which result best shows a benefit that this simpler combination would miss?
7 comments
[W3 & Q2: Worker-Agnostic Task Decomposition] We respectfully believe that worker-agnostic task decomposition is a reasonable and effective design choice. (1) Task decomposition primarily depends on the task's semantic structure and dependencies. While worker capabilities can help adjust subtask granularity and difficulty, they are not essential for producing an effective decomposition. (2) A natural way to make decomposition depend on worker capabilities would be to iteratively refine the decomposition based on matchability feedback. However, this would incur substantial additional training and inference costs. (3) Even when the initial decomposition is suboptimal for the current worker pool, Stage II can compensate by selecting stronger collaboration operations, such as DEBATE or ENSEMBLE, based on matchability feedback. (4) Even within our current framework, multi-round execution could allow the planner to implicitly adapt decomposition to worker capabilities by leveraging execution traces from previous rounds. Our experiments use a single reasoning round for fair comparison with baselines, while exploring multi-round reasoning is left for future work. (5) The case study provided in our response to W1 & Q1 illustrates this flexibility: given the same decomposition, DeOrch successfully resolves a subtask through DEBATE where single-worker execution fails. Together with our experimental results, this supports the effectiveness of the current design. We view capability-aware decomposition as an interesting future direction rather than a prerequisite for the contributions of this work.
[W4: Transfer Evaluation on a New Worker Pool] To address the reviewer's concern, we conducted an additional transfer experiment on a pool of four workers from entirely unseen model families (Falcon-H1-7B-Instruct, Phi-4, OLMo-3-7B-Instruct, and Llama-3.1-8B-Instruct), without retraining the planner or matcher. The results are shown below:
| Method | MuSiQue | TP | Omni | LCB | BBEH | NP | PB-M | GAIA | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Best Single Agent | 48.4 | 2.8 | 15.8 | 30.9 | 15.8 | 21.2 | 0.5 | 6.7 | 17.8 |
| Independent | 55.9 | 3.7 | 19.9 | 32.9 | 16.4 | 16.1 | 0.0 | 6.7 | 19.0 |
| Frontier-Planner | 53.4 | 2.9 | 18.3 | 37.6 | 15.8 | 17.2 | 1.2 | 7.9 | 19.3 |
| Conductor | 53.9 | 3.3 | 18.8 | 35.4 | 15.3 | 16.5 | 0.5 | 6.7 | 18.8 |
| DeOrch | 60.2 | 6.4 | 23.0 | 39.4 | 17.8 | 22.1 | 2.7 | 11.5 | 22.9 |
Together with our original transfer experiment, these results demonstrate DeOrch's effectiveness across different unseen worker pools, including entirely unseen model families. For the rationale behind worker-agnostic decomposition, please refer to our response to W3 & Q2.
We sincerely thank the reviewer again for the constructive feedback, which has helped us improve the clarity and completeness of our paper. We hope our responses adequately address the concerns and welcome further discussion. Finally, we would like to reiterate our two core contributions: (1) On technical novelty, we are the first to introduce this two-stage RL paradigm for disentangled credit assignment. (2) On technical and paradigm novelty, we introduce a novel matcher combining offline probe-based matching with online bandit adaptation and supporting multi-worker selection, together with the first MAS orchestration framework to decouple planning from learnable worker selection. Both contributions are supported by extensive experiments across diverse benchmarks. We hope these clarifications and additional results will strengthen the reviewer's assessment of our work and encourage reconsideration of the overall rating.
Dear Reviewer TocY,
Thank you again for your insightful comments. We have provided additional experiments and detailed clarifications addressing your concerns, including the Decomposition + Single comparison on an unseen worker pool, a concrete collaboration case study, and further discussion of our technical contributions and design choices. As the discussion deadline is approaching, we would greatly appreciate any feedback on our rebuttal, particularly whether our additional evidence and explanations have addressed your concerns. We would be happy to clarify any remaining questions before the deadline.
Thank you for your time and consideration.
Dear reviewer TocY, We sincerely thank the reviewer for the comments and constructive feedback. We address each concern in detail below.
[W1 & Q1: Contribution of Learned Collaboration] We respectfully point out that the relatively modest gap between Decomposition + Single and DeOrch does not diminish the value of learned collaboration. (1) Table 3 shows that learned operation selection improves average performance from 50.4 to 51.5. This improvement is consistent across all eight benchmarks: MuSiQue (+1.1), TravelPlanner (+1.4), Omni-MATH-2 (+1.2), LiveCodeBench v6 (+0.8), BBEH (+0.6), NaturalPlan (+0.9), PlanBench-mystery (+1.0), and GAIA (+2.4). These consistent gains demonstrate that the benefits of collaboration are not confined to particular task types. This is consistent with prior findings that multi-agent collaboration, such as debate and ensemble, can improve performance across diverse tasks. While Decomposition + Single reduces worker calls from 5.35 to 1.68, the additional performance gain of full DeOrch can be valuable in accuracy-oriented scenarios where higher inference cost is acceptable. (2) Following the reviewer's suggestion, we additionally evaluated Decomposition + Single on the unseen worker pool without retraining. The results are as follows:
DeOrch consistently outperforms Decomposition + Single across all eight benchmarks (+1.4 points on average), further demonstrating that learned operation selection remains effective when transferring to unseen worker pools. (3) To further illustrate when learned collaboration is beneficial, we provide a concrete case study from BBEH (zebra_puzzles-128), a challenging constraint puzzle with 142 clues. Both DeOrch and Decomposition + Single use the same two-subtask decomposition: solving the puzzle and extracting the final answer. However, DeOrch selects DEBATE-3 for the solving subtask, where three workers produce revised answers (3, 2, 3), leading to the correct final answer (3). In contrast, Decomposition + Single uses SINGLE-1 and incorrectly predicts 5. Although DeOrch requires more worker calls (7 vs. 2), this case demonstrates how learned operation selection can improve correctness without changing the task decomposition. We have also included the complete case study in the revised manuscript.
[W2 & Q3: Novelty and Technical Contributions] We respectfully believe that the novelty of DeOrch has been underestimated. While DeOrch addresses the same automatic MAS orchestration problem as Conductor and MAS-Orchestra, it introduces fundamentally different methods to overcome their limitations. Our core technical contributions are twofold: (1) Matchability-informed two-stage RL training. This enables more effective credit assignment for task decomposition and operation selection. To the best of our knowledge, we are the first to introduce this two-stage RL paradigm for disentangled credit assignment. (2) Decoupled orchestration architecture and a novel matcher. On paradigm novelty, DeOrch is the first learned MAS orchestration framework to decouple planning from learnable worker selection, eliminating the need to retrain the LLM-based orchestrator when transferring to unseen workers. On technical novelty, unlike existing routers such as UniRoute, our matcher combines density-corrected similarity-based matching with online contextual bandit adaptation and supports complementarity-aware multi-worker selection. In short, we propose a better planner, a better matcher combining offline and online routing, and an effective integration of the two through matchability feedback, rather than simply combining an existing learned planner with an existing transferable router. Table 3 demonstrates the effectiveness of our novel two-stage RL training framework (49.6 → 51.9), Tables 4–5 validate our matcher, and Tables 1–2 establish overall performance and transferability advantages.