acceptodds
Under review as a conference paper at ICLR 2027

EvoOrch: Evolving Orchestration for Dynamic Multi-Agent Systems

Abstract

Large language model-based multi-agent systems (MASs) solve complex, long-horizon tasks by decomposing them into subtasks and assigning these subtasks to specialized sub-agents. However, existing systems often rely on predefined or ad hoc orchestration structures, limiting their adaptability at both the intra-task and cross-task levels: 1) within a task, they struggle to dynamically reorganize sub-agent structures and correct execution failures in a timely manner, and 2) across tasks, they make limited use of previous trajectories to improve future orchestration. To address these limitations, we propose **EvoOrch**, a multi-agent orchestration framework that supports both intra-task and cross-task evolution. EvoOrch consists of a persistent Meta Agent and dynamically instantiated sub-agents. At the intra-task level, EvoOrch performs structural evolution by dynamically constructing and extending task-specific sub-agent structures, and configuration evolution by revising ineffective active sub-agents during execution. At the cross-task level, EvoOrch follows a two-stage pipeline that first hierarchically organizes completed multi-agent trajectories into structured trajectories, and then performs trajectory-driven analysis to distill reusable orchestration and verification insights, which are stored in an experience base and retrieved to guide future tasks without parameter updates. Experiments on diverse settings show that EvoOrch surpasses the state-of-the-art automated orchestration method by 7.9 points on average, while achieving a better Pareto trade-off between performance and cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Multi-Agent System

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GP36OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoOrch is a training-free framework for dynamically orchestrating LLM-based multi-agent systems. A Meta Agent creates task-specific agent structures and can revise active agents’ tools, instructions, context, models, or budgets. Across tasks, completed trajectories are summarized and distilled into reusable orchestration insights. These insights are retrieved from a frozen experience base during evaluation. Experiments across six benchmarks show improvements over several orchestration baselines, with additional analyses of intervention, experience retrieval, backbone allocation, and execution cost.

Critical strengths

The framework addresses a practical limitation of agent delegation: a worker may become ineffective after it has been created. Allowing the Meta Agent to revise that worker in place is a useful design choice, especially when tools, context, or execution budgets become bottlenecks.

The empirical evaluation is reasonably broad. Improvements appear across multiple benchmarks and backbones, and the main results combine higher task scores with competitive inference costs. For example, on GAIA with GPT-5, EvoOrch improves Avg@3 from AOrchestra’s 58.7 to 64.2 while reducing the reported cost from $0.81 to $0.56.

Several additional controls strengthen the evidence. Appendix A.2.7 gives retrieved insights to competing systems, while the cross-task analyses compare raw trajectories, distilled insights, self-review, and gold-assisted review. Freezing the experience base before evaluation also provides a clear separation between historical experience and test tasks.

Critical weaknesses

The intervention analysis does not fully establish the value of targeted diagnosis. The same-state comparison in Figure 2b is useful, but retaining the original configuration is a relatively limited control. A stuck agent might also recover through a budget-matched restart or by receiving the full tool set from the beginning. Since tool changes are the most frequent intervention, this distinction matters. The paper should also define “recovery success” precisely and report the number of evaluated states and uncertainty for each intervention type.

The structural ablation changes several factors together. Restricting the system to serial delegation removes parallel scheduling and structured information handoffs, and can change how much work is delegated within one Meta-Agent checkpoint. Table 2 therefore supports the usefulness of the complete graph-based delegation mechanism, but does not isolate the benefit of adapting dependencies during execution. A fixed parallel or dependency-based workflow with comparable worker and token budgets would provide a more informative comparison.

The cost advantage is measured after preparing the experience base. The main system uses trajectories from 400 tasks and Golden-Review distillation, but the reported costs describe inference-time token usage. The cost of collecting and reviewing those trajectories is not quantified. This does not invalidate the inference results, but it leaves the practical savings unclear for smaller workloads. The authors should report preparation costs and an amortized comparison. They should also distinguish training-free operation from the use of gold supervision during experience construction.

Questions for authors
  1. How many same-state intervention pairs were evaluated, and does “recovery success” mean completing the delegated subtask or solving the original task?
  2. How does EvoOrch compare with a fixed parallel/dependency workflow under matched total worker-step or token budgets?
  3. What is the total cost of collecting and distilling the 400 historical trajectories, and after how many evaluation tasks do the inference savings offset that cost?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GP36 (Part 1)

Thank you for your thoughtful comments and questions. For clarity, we have organized our responses around the weaknesses and questions raised in your original review.

Weakness 1: The intervention analysis does not sufficiently establish the benefits of targeted diagnosis. Moreover, the paper should clarify whether “recovery success” refers to completing the delegated subtask or solving the original task, and report the number of same-state intervention pairs evaluated, along with the uncertainty for each intervention type. (Original Weakness 1 and Question 1)

Response W1: Thank you for the valuable suggestions. We further clarify the recovery success rate evaluation in our revision.

1. Recovery Success Definition and Evaluation Reliability.

We define recovery success as whether an intervened sub-agent successfully completes its delegated subtask, rather than the original task. Specifically, a sub-agent is considered unsuccessful if it does not return a finish action or its completion status is not done. For sub-agents returning done, we use DeepSeek-V3.2 to assess whether their execution trajectories and outputs satisfy the delegated subtask requirements. Each evaluation is repeated three times, and the average judgment is used to compute the recovery success rate.

We evaluate 731 same-state intervention pairs collected from three independent GAIA runs, including 255 Tool, 172 Model, 157 Instruction, 120 Budget, and 27 Context interventions. Given the limited number of Context interventions, we repeat each same-state experiment three times to improve reliability. We also manually inspect 40 cases, finding that 37/40 LLM judgments agree with human assessments, while 35/40 interventions lead to observable behavioral changes. We will clarify these details in our revision.

2. Targeted Intervention vs. Full Tool Access.

Regarding your suggestion that a stuck agent might also recover through a budget-matched restart or by receiving the full tool set from the beginning, we agree that these are valuable controls. We further investigate one of these settings by providing the full available tool set at the intervention point. Specifically, we evaluate 100 sampled tool-intervention cases. Compared with targeted tool intervention, providing all tools only slightly improves recovery success from 58.3% to 59.3%, while increasing the average sub-agent cost from $0.0167
to $ 0.0208. This suggests that providing additional tools indiscriminately does not necessarily benefit sub-agent execution.

For tasks involving multiple tools, such as Travel Planning (9 tools), or more complex, long-horizon tasks, simply increasing the budget or providing the full tool set is not always Pareto-optimal. In contrast, decomposing agents into specific configuration dimensions facilitates more fine-grained adjustments and management, which we believe represents a more efficient perspective on configuration evolution.

AuthorsOpenReview ↗

Response to Reviewer GP36 (Part 2)

Weakness 2: The current structural ablation does not isolate the benefit of runtime structural adaptation. How does EvoOrch compare with a fixed parallel or dependency-based workflow under matched execution budgets? (Original Weakness 2 and Question 2)

Response W2: Thank you for your valuable suggestion. We conduct an additional Fixed-Structure ablation, where the Meta Agent generates a single task-specific DAG at the beginning of execution, with the agent structure remaining fixed throughout the task while other operations are retained. In the default EvoOrch setting, the Meta-Agent checkpoint budget is 10, the sub-agent budget is 30 steps. To allow a larger per-agent execution budget for the Fixed-Structure variant, we increase the maximum sub-agent steps from 30 to 100. The results are shown below.

MethodGAIA Avg@3 ↑GAIA Pass@3 ↑GAIA Cost ↓BrowseComp-Plus Avg@3 ↑BrowseComp-Plus Pass@3 ↑BrowseComp-Plus Cost ↓
EvoOrch65.580.60.1044.563.50.31
w/ Fixed Structure52.969.70.1435.650.50.37
w/o Stru. Evo.53.971.80.0837.655.50.25
w/o Config. Evo.60.275.80.1542.359.50.38
w/o Cross. Evo.62.477.60.1241.360.50.36

Compared with the Fixed-Structure variant, EvoOrch improves Avg@3 by 12.6 points on GAIA and 8.9 points on BrowseComp-Plus, while reducing inference costs. These results demonstrate the benefit of adapting agent structures based on intermediate execution results, beyond simply using predefined parallel or dependency-based delegation. Our main experiments also include predefined workflow methods such as OWL and MASFactory, providing further comparative evidence for the runtime agent structure evolution.

Weakness 3: The reported costs exclude the preparation of 400 historical trajectories and Golden-Review distillation. What is the total preparation cost, and how many evaluation tasks are needed to amortize it through inference savings? (Original Weakness 3 and Question 3)

Response W3: Thank you for the question. The costs reported in our main experiments cover inference-time execution only. Historical trajectory collection and experience distillation incur additional preparation costs, which we clarify below and further include them in our revision.

1. Experience Preparation. EvoOrch constructs its experience base from 400 historical trajectories using DeepSeek-V3.2 in Golden-Review. The average cost is approximately 0.12 per trajectory for execution and 0.05 for Golden-Review distillation, totaling $68 for experience preparation. The experience base is frozen before evaluation and reused across subsequent tasks. For N evaluation runs, the preparation cost is 68/N68/N per run. For example, in main experimentt cross 165 GAIA, 200 BrowseComp-Plus, and 240 Travel Planning tasks, each evaluated three times, (N = 1815), yielding an amortized cost of approximately 0.037 per run. This overhead decreases further as the experience base is reused.

2. Inference Cost Savings. As shown in Table 2, cross-task evolution improves GAIA Avg@3 from 62.4 to 65.5 while reducing inference cost from 0.12 to 0.10 per run. On BrowseComp-Plus, Avg@3 increases from 41.3 to 44.5, while cost decreases from 0.36 to 0.31. For BrowseComp-Plus, the inference savings of 0.05 per run slightly exceed the preparation cost of 0.037 per run, indicating that the preparation overhead can be offset under this setting. These results show that historical insights improve inference efficiency, while the one-time preparation overhead can be amortized over repeated use. We will clarify these costs and their amortization in the revised manuscript.

3. Training-free vs. Gold Supervision. EvoOrch requires no model parameter updates, but Golden-Review uses ground-truth answers during offline experience preparation. The experience base is frozen before evaluation, and no evaluation-task ground-truth answers are accessed during test-time execution. We further compare Golden-Review and Self-Review in our Cross-task Evolution Analysis, demonstrating the effectiveness of different review strategies without model training, thereby avoiding the computational overhead of model parameter updates.

We would be happy to provide further clarification and address any additional questions or concerns you may have.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Response

Dear Reviewer GP36,

Thank you for taking the time to review our paper and provide valuable feedback. We sincerely appreciate your time and effort, and apologize for the limited time available for further discussion.

As less than two days remain in the discussion period, we would be grateful if you could let us know whether our rebuttal has adequately addressed your concerns or if there are any remaining questions that we could further clarify.

Thank you again for your time and consideration. We look forward to hearing your feedback.

Best regards,

The Authors

Dear Reviewer GP36,

I hope this message finds you well.

As the discussion period is coming to a close, we would greatly appreciate it if you could revisit our rebuttal and let us know whether any concerns remain. We would be happy to provide further clarification or address any outstanding questions before the deadline.

Thank you again for your time and effort in reviewing our paper and considering our response.

Best regards,

The Authors

Reviewer_tsmQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoOrch, a training-free orchestration framework for dynamic multi-agent systems. It introduces three mechanisms: structural evolution, which dynamically creates sub-agents and organizes their dependencies; configuration evolution, which adjusts or terminates sub-agents based on execution feedback; and cross-task evolution, which distills reusable orchestration insights from historical trajectories to guide future tasks. The paper is clearly structured, and the experimental evaluation is relatively comprehensive.

Critical strengths

None

Critical weaknesses

However, several core mechanisms exhibit varying degrees of technical overlap with existing work. First, the dynamic sub-agent instantiation mechanism presented in Section 2.3 is closely related to AOrchestra (ICML 2026, https://proceedings.mlr.press/v306/ruan26e.html), which already introduces on-demand instantiation of specialized sub-agents and a unified agent configuration abstraction. Second, EvoOrch implements structural evolution during task execution by updating the nodes and dependency edges of the agent graph. This is related to GPTSwarm (ICML 2024, https://proceedings.mlr.press/v235/zhuge24a.html), which investigates the optimization of agent computation graphs and their connections. Its automated orchestration and execution-feedback mechanisms are also related to AFlow (ICLR 2025, https://proceedings.iclr.cc/paper_files/paper/2025/hash/5492ecbce4439401798dcd2c90be94cd-Abstract-Conference.html), which employs workflow search, code modification, and feedback-driven optimization. Furthermore, the hierarchical trajectory organization, reusable experience distillation, and retrieval mechanisms introduced in Section 2.4 are substantially related to EvolveR (ICML 2026, https://proceedings.mlr.press/v306/wu26bf.html), which distills reusable strategic principles from historical interaction trajectories and retrieves them to guide subsequent tasks.

Does EvoOrch introduce genuinely novel mechanisms, or does its primary contribution lie in integrating existing techniques for dynamic sub-agent instantiation, graph-structure optimization, runtime configuration adjustment, and cross-task experience reuse into a unified framework? The paper does not adequately clarify the specific algorithmic differences between EvoOrch and these prior approaches, nor does it provide targeted comparative experiments to establish synergistic benefits beyond a straightforward combination of existing techniques. The current ablation studies show that removing individual components degrades performance, but this alone does not establish the independent novelty of those components, let alone demonstrate that the overall framework is technically distinct from existing approaches. Consequently, the paper's claims of originality remain insufficiently substantiated. The authors should identify the specific technical contributions that distinguish EvoOrch from prior methods and provide corresponding empirical evidence.

The experimental evaluation also has several limitations:

  1. The paper does not clearly specify how fairness is controlled for baselines that may leverage cross-task memory or historical experience. In particular, it remains unclear whether comparable baselines are allowed to use equivalent retrieval or memory mechanisms. This ambiguity raises concerns about whether EvoOrch's performance gains are attributable, at least in part, to an advantage in access to historical experience rather than to its proposed orchestration mechanisms.

2.For GAIA and BrowseComp-Plus, the evaluation setup lacks sufficient detail regarding the judging protocols, such as whether final answers are assessed using deterministic rules, an LLM-as-a-judge approach, or a combination of both. Without this information, it is difficult to assess the reliability of the reported results and the comparability of performance differences across methods.

  1. Although the paper includes an interactive CLI benchmark, its evaluation provides limited coverage of a broader range of code-related, GUI-based, and multimodal tasks. The generalizability of EvoOrch beyond web research, deep research, and planning tasks therefore remains insufficiently established.
Questions for authors

as listed above

5 comments
AuthorsOpenReview ↗

Response to Reviewer tsmQ (Part 1)

Thank you for reviewing our paper and providing valuable feedback. We organize our responses around the concerns raised in your review.

Weakness 1: The paper does not sufficiently distinguish EvoOrch from existing methods or establish its novelty beyond integrating existing techniques. More explicit technical distinctions and targeted empirical evidence are needed.

Response W1: Thank you for highlighting these related studies. We would like to clarify the technical differences between EvoOrch and prior methods. Our main contribution is an execution-time multi-agent orchestration framework that continuously adapts both agent structures and active sub-agent configurations according to the evolving task state. The key distinction is that structural evolution and configuration evolution are performed during task execution: the Meta Agent progressively constructs sub-agent groups and dependencies based on intermediate results, while revising active sub-agent configurations timely when execution problems arise. We further introduce a trajectory-driven method to distill reusable orchestration and verification insights to guide future Meta-Agent decisions. For the work mentioned in your review, we have included them in related work and compared EvoOrch with them in the experiment in our submission. And we will clarify the differences from the referenced methods as follows.

1. AOrchestra:We acknowledge AOrchestra's contributions to unified agent abstraction and on-demand sub-agent instantiation, which we adopt as a foundation in Section 2.1.2. However, AOrchestra primarily supports single-agent instantiation and serial delegation, with limited support for explicit dependencies among multiple agents. In contrast, EvoOrch supports progressive structural evolution, dynamically constructing parallel and dependency-based agent groups as new subproblems emerge. EvoOrch also supports in-place configuration revisions of active sub-agents, including tools, instructions, context, backbone models, and budgets.

2. GPTSwarm and AFlow. Both methods investigate automated workflow optimization and are included in our experimental comparisons in Table 1 and 11. GPTSwarm optimizes agent computation graphs and connectivity, while AFlow searches over code-represented workflows using execution feedback. Their optimization primarily occurs on development tasks, with the resulting workflows applied to test tasks. In contrast, EvoOrch performs structural evolution during task execution, using intermediate results to determine subsequent sub-agents and dependencies. It also supports runtime configuration revisions of active sub-agents rather than relying solely on previously optimized workflows.

3. EvolveR. Like EvoOrch, EvolveR distills reusable principles from historical trajectories. However, EvolveR primarily focuses on agent reasoning and tool-use strategies, together with subsequent policy evolution. In contrast, EvoOrch focuses on multi-agent orchestration and verification. Our hierarchical trajectory organization preserves the relationship between Meta-Agent orchestration decisions and sub-agent outcomes enabling the distillation of insights that guide future evolution without parameter updates.

Empirical Evidence. We provide the following experimental evidence for EvoOrch and its three evolution mechanisms. 1. Main Results and Ablation Studies. Table 1 shows that EvoOrch outperforms existing multi-agent orchestration methods across benchmarks and backbones, improving over AOrchestra by 11.0 points on GAIA while maintaining competitive inference costs. Table 2 further shows that removing structural, configuration, and cross-task evolution reduces GAIA Avg@3 by 11.6, 5.3, and 3.1 points, respectively, with consistent degradation on BrowseComp-Plus.

2. Targeted Mechanism Analyses. 2.1 Structural Evolution: Figure 2a shows that EvoOrch adapts orchestration structures to task difficulty and execution stages. On challenging GAIA Level 3 tasks, EvoOrch improves Avg@3 from 34.6 (AOrchestra) to 51.3 (+16.7 points), with increasingly complex parallel and dependency-based structures. We additionally introduce a Fixed-Structure baseline that constructs a task-specific agent graph before execution. EvoOrch outperforms this baseline by 12.6 points on GAIA and 8.9 points on BrowseComp-Plus, with lower inference costs. This comparison isolates the benefits of runtime structural evolution beyond simply introducing parallel and dependency-based agent structures.

2.2 Configuration Evolution: Figure 2b evaluates five types of runtime interventions through same-state comparisons. Targeted configuration revisions improve weighted recovery success from 36.3% to 56.5% (+20.2 points). These results demonstrate the effectiveness of fine-grained configuration adaptation for recovering ineffective sub-agents. (Response W1 continues in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer tsmQ (Part 2)

2.3 Cross-task Evolution: Tables 5–6 and 13 examine the effects of trajectory representation, review signals, retrieval strategies, and cross-domain transfer. On BrowseComp-Plus, hierarchical trajectory distillation improves Avg@3 from 42.3 to 44.5 under out-of-domain Golden-Review, and from 44.8 to 48.5 under in-domain Golden-Review, compared with raw trajectories. Moreover, even insights distilled from out-of-domain TaskCraft tasks improve performance from 41.3 to 44.5, demonstrating the transferability of orchestration and verification strategies.

Overall, EvoOrch differs from prior methods in its execution-time structural evolution and configuration evolution, supported by cross-task orchestration and verification insights. The targeted analyses further demonstrate that its improvements arise not merely from introducing parallel and dependency-based structures or combining existing mechanisms, but from adapting agent structures and configurations throughout execution.

Weakness 2: The fairness of cross-task experience access across baselines is unclear, particularly whether they use comparable retrieval or memory mechanisms. This raises concerns that EvoOrch's gains may partly stem from privileged access to historical experience rather than its orchestration mechanisms.

Response W2: Thank you for raising this concern. We clarify the fairness of cross-task experience access through our baseline settings, two complementary controlled experiments and ablation studies.

1. Baseline Settings and Retrieved Insights. In our main experiments, we follow the original designs of the compared methods, including their existing memory and experience mechanisms. We further conduct a controlled experiment in which ReAct and AOrchestra are also provided with relevant historical insights, adapted to their respective execution and orchestration mechanisms (Appendix A.2.7, Table 12). The results show that retrieved insights improve ReAct from 47.8 to 48.5 Avg@3 and AOrchestra from 58.7 to 60.4 on GAIA. Nevertheless, EvoOrch achieves 64.2 Avg@3, outperforming AOrchestra with retrieved insights by 3.8 points at a lower inference cost (0.56 vs. 0.77). This indicates that providing historical insights to competing methods does not eliminate EvoOrch's performance advantage.

2. Cross-task Evolution Ablation Study. We conducted an ablation study to isolate the contribution of cross-task experience by disabling cross-task evolution while retaining structural and configuration evolution in our submissoon. As shown in Table 2 of Ablation Section, EvoOrch without cross-task evolution achieves 62.4 Avg@3 on GAIA and 41.3 on BrowseComp-Plus, outperforming AOrchestra by 7.9 and 5.3 points and EvoMAS by 4.2 and 4.1 points, respectively. Enabling cross-task evolution provides additional gains of 3.1 and 3.2 points. These results show that EvoOrch retains substantial performance improvements even without historical experience, highlighting the effectiveness of its intra-task structural and configuration evolution mechanisms.

Together, these experiments show that EvoOrch's performance gains are not solely attributable to access to historical experience, while cross-task insights provide further improvements.

Weakness 3: The evaluation protocol for GAIA and BrowseComp-Plus is insufficiently specified, particularly regarding rule-based matching and LLM-as-a-judge, raising concerns about evaluation reliability and fairness across methods.

Response W3: Thank you for the question. We have clarified the answer cleansing evaluation protocol in Appendix A.4.1 (Lines 1050–1056). Our evaluation follows the established protocols of GAIA, BrowseComp-Plus, DeepPlanning, and relevant benchmarks. Specifically, we use rule-based matching whenever applicable; otherwise, following AOrchestra, we use Gemini-3 for answer extraction and semantic consistency checking. The same judging procedure is consistently applied to all methods to ensure fair comparisons. We will further clarify these details in the revision.

AuthorsOpenReview ↗

Response to Reviewer tsmQ (Part 3)

Weakness 4: Despite including an interactive CLI benchmark, the evaluation has limited coverage of coding, GUI-based, and multimodal tasks, leaving EvoOrch's generalizability beyond research and planning insufficiently established.

Response W4: Thank you for the suggestion. Our benchmark selection primarily focuses on complex, long-horizon tasks that require dynamic task decomposition, multi-agent and adaptive orchestration, rather than any specific application domain. Accordingly, we evaluate EvoOrch across general assistant tasks (GAIA), deep research (BrowseComp-Plus), constrained planning (Travel Planning), broad information seeking (WideSearch), multilingual browsing (BrowseComp-Zh), and terminal tasks (Terminal-Bench 2.0).

According to your suggestion in coding task, we conduct additional experiments on SWE-bench Verified using DeepSeek-V3.2. Following AOrchestra's evaluation setting, we use the same 100 test cases and compare EvoOrch with ReAct and AOrchestra.

MethodAvg@3 ↑Pass@3 ↑
ReAct65.383.0
AOrchestra78.785.0
EvoOrch83.787.0

EvoOrch outperforms AOrchestra by 5.0 points in Avg@3 and 2.0 points in Pass@3, demonstrating its effectiveness on coding tasks beyond the original evaluation domains.

We would be happy to provide further clarification and address any additional questions or concerns you may have.

Reviewer_tWB1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoOrch, a training-free framework for dynamically orchestrating multi-agent systems. A persistent Meta Agent constructs task-specific dependency structures and revises the tools, instructions, context, models, or budgets of active sub-agents during execution. Across tasks, the framework organizes completed trajectories and distills reusable orchestration and verification insights, which are retrieved from a frozen experience base without updating model parameters. Experiments across deep research, planning, and terminal tasks report improved performance and competitive API inference costs, supported by ablations and analyses of configuration interventions, experience reuse, and model allocation.

Critical strengths
  • The paper is well organized, and the distinction between intra-task and cross-task adaptation makes the framework easy to follow.
  • Evaluations span multiple task types and model families, including supplementary experiments on Chinese browsing and terminal benchmarks.
  • The paper examines review signals, trajectory representations, retrieval strategies, and out-of-domain experience sources.
Critical weaknesses
  • The comparison omits MASFly, which already combines experience reuse, runtime supervision, and agent replacement. A clearer comparison is needed to establish the incremental contribution relative to this close approach.
  • Although the 65/100 development/test split is specified, the comparison in Table 11 does not sufficiently report method-specific optimization budgets and feedback settings for AFlow and GEPA.
  • The benchmarks cover complementary settings, but their connection to the three proposed mechanisms is not made sufficiently explicit.
  • The structural ablation restricts execution to serial delegation, removing parallel and dependency-group capabilities alongside structural adaptation. This makes it difficult to isolate the benefit of adapting the structure during execution.
Questions for authors
  • Which task conditions and failure modes in the selected benchmarks specifically test structural adaptation, configuration intervention, and cross-task experience reuse?
  • What are the key technical differences from MASFly, and can a comparison with matched backbones, tools, and budgets clarify their practical impact?
  • What development budgets, feedback signals, and stopping criteria were used for AFlow and GEPA in Table 11, and which settings were matched with EvoOrch?
  • How sensitive are the results to intervention timing, particularly the monitoring interval and repeated-failure threshold?
  • Do the reported costs include trajectory collection and experience distillation? Please report these costs separately and clarify how they are amortized across subsequent tasks.
6 comments
AuthorsOpenReview ↗

Response to Reviewer tWB1 (Part 1)

Thank you for the effort and time you dedicated to reviewing our paper. We have greatly benefited from your valuable feedback. For clarity, we have organized our responses around the weaknesses and questions raised in your original review.

Weakness 1: How does EvoOrch differ technically and empirically from MASFly under matched backbones, tools, and budgets? (Original Weakness 1 and Question 2)

Response W1: Thank you for the suggestion. We will include MASFly in our related work and report the additional baseline results in the revision. We follow MASFly's 10/110 training/test split on GAIA and use 50 historical cases and 200 evaluation cases on BrowseComp-Plus. EvoOrch is re-evaluated on the same test splits:

MethodGAIA Avg@3GAIA Pass@3GAIA CostBrowseComp-Plus Avg@3BrowseComp-Plus Pass@3BrowseComp-Plus Cost
EvoOrch (Full Evaluation in Paper)65.580.60.1044.563.50.31
EvoOrch (Matched Split)64.579.10.1043.862.00.30
MASFly51.571.00.1534.351.50.34

Under the matched evaluation splits, EvoOrch outperforms MASFly by 13.0 points on GAIA and 9.5 points on BrowseComp-Plus in Avg@3, with improvements of 8.1 and 10.5 points in Pass@3, respectively. EvoOrch also achieves lower inference costs on both benchmarks. The key technical differences are as follows:

1. Progressive structural evolution(runtime supervison):

MASFly retrieves historical SOPs to generate an initial task-specific multi-agent structure. During execution, its Watcher corrects abnormal behavior or reconfigures agents. In contrast, EvoOrch progressively constructs sub-agent dependency structures throughout task execution, allowing the Meta Agent to dynamically instantiate parallel and dependency-based sub-agent groups according to intermediate results and emerging subproblems. Structural evolution is driven by both task progression and execution failures, enabling flexible exploration and adaptation during execution. This advantage is particularly relevant to complex tasks requiring multiple rounds of exploration and coordination. As shown in our Structural Evolution Analysis (Section 4.1, Figure 2a), the benefits of dynamic orchestration become more pronounced on challenging GAIA Level 3 tasks.

2. Fine-grained Configuration Evolution(agent replacement):

MASFly primarily handles ineffective agents through Watcher-guided corrective feedback or agent replacement. In contrast, EvoOrch decomposes agent configurations into independent dimensions and diagnoses execution problems during runtime, enabling targeted revisions of tools, instructions, context, backbone models, and budgets. This avoids unnecessary agent reconstruction while preserving useful execution progress. As demonstrated in our Configuration Evolution Analysis (Figure 2b), these fine-grained interventions improve sub-agent recovery success through targeted configuration revisions.

3. Cross-task Experience Reuse(experience reuse):

Both MASFly and EvoOrch leverage historical experience to guide multi-agent orchestration, but differ in how experience is represented and utilized. MASFly primarily reuses historical SOPs and failure experiences for initial system instantiation and Watcher supervision. In contrast, EvoOrch distills orchestration and verification insights from historical multi-agent trajectories to guide the Meta Agent's subsequent orchestration decisions during task execution.

Weakness 2: Table 11 lacks details on AFlow/GEPA optimization budgets, feedback signals, stopping criteria, and which settings were matched with EvoOrch. (Original Weakness 2 and Question 3)

Response W2: Thank you for the suggestion. For both baselines, we use the same 65 development and 100 held-out test instances, with identical tools, backbones, and evaluation protocols. Both methods receive development-set correctness feedback during offline optimization, while the held-out test instances are used exclusively for final evaluation.

AFlow. We allow up to 20 optimization rounds, evaluating each candidate twice on all 65 development instances. Average scores and execution feedback guide workflow optimization, with convergence determined by the top-3 historical scores over five consecutive rounds.

GEPA. Following official settings, we split the 65 development instances into 32 feedback and 33 Pareto validation instances, with reflection minibatches of 3 and a maximum of 1,000 metric calls. Improved candidates undergo Pareto validation; we disable merging and stop when the budget is exhausted.

Unlike AFlow and GEPA, which require repeated offline evaluations to optimize workflows or prompts, EvoOrch distills reusable orchestration and verification insights from historical trajectories without iterative workflow search. These insights guide subsequent tasks without additional offline optimization.

AuthorsOpenReview ↗

Response to Reviewer tWB1 (Part 2)

Weakness 3: The connection between the benchmarks and the three evolution mechanisms is insufficiently explained. Which task conditions and failure modes evaluate structural adaptation, configuration intervention, and cross-task experience reuse? (Original Weakness 3 and Question 1)

Response W3: Thank you for the suggestion. We would like to clarify the reason behind our benchmark selection and its connection to the three proposed evolution mechanisms.

1. Benchmark Selection and Generality.

Following AOrchestra and prior agentic studies, we adopt representative benchmarks such as GAIA and Terminal-Bench, and extend our evaluation to deep research (BrowseComp-Plus), constrained planning (Travel Planning), broad information seeking (WideSearch), and multilingual browsing (BrowseComp-Zh). These benchmarks span diverse domains while sharing challenges in long-horizon execution, complex task decomposition, and multi-agent coordination.

Our goal is to develop a general-purpose multi-agent orchestration framework rather than a benchmark-specific solution. The main experiments evaluate its generality, while the case studies illustrate the three evolution mechanisms under different task conditions.

2. Ablation Studies and Mechanism Validation.

To further examine the contributions of the three proposed evolution mechanisms, we conduct ablation studies on GAIA and BrowseComp-Plus (Table 2). Removing structural, configuration, or cross-task evolution consistently reduces performance. We further analyze structural and configuration evolution in the Analysis Section.

3. Case Studies. To better address your question about benchmark conditions and failure modes, we further clarify the task conditions and failure modes that exercise each mechanism through three representative cases beyond the case study in the submission:

Structural Evolution — GAIA Level 3 (Case ID: 9e1fc53b-46ff-49a1-9d05-9e6faac34cc5). The task requires identifying five World of Warcraft character classes based on constraints involving team roles, armor types, abilities, and loot restrictions. Ambiguous clues, including a bear attacking the boss and a character using Metamorphosis, lead to conflicting class combinations. EvoOrch progressively adapts its agent structure as these conflicts emerge (||: parallel agents; →: explicit dependency):

  1. Parallel exploration → unresolved prerequisite. The Meta Agent creates armor researcher || loot researcher || ability researcher. The agents identify leather loot and associate holy spells with Paladin/Priest, the bear with Druid, and Metamorphosis with Demon Hunter. However, incomplete armor information and unresolved loot restrictions require further verification.

  2. Dependency-based solving → inconsistent interpretation. The Meta Agent constructs item verifier → constraint solver, ensuring that verified item information precedes constraint solving. The verifier reports a Rogue-only restriction, but the solver cannot satisfy the armor distribution (two plate and two cloth wearers) alongside the ability clues, indicating potentially incorrect assumptions.

  3. Parallel rechecking → reasoning bottleneck. The Meta Agent creates armor researcher || item verifier || puzzle solver. The armor researcher completes the armor mapping, and the item verifier overturns the Rogue-only restriction. However, the solver remains unsuccessful: assigning the bear to Druid and Metamorphosis to Demon Hunter introduces two leather-wearing classes, conflicting with the required armor distribution.

  4. Focused reinterpretation → consistent solution. The Meta Agent creates a specialized constraint-reinterpretation solver with the updated evidence and conflicts. It considers the bear as a Hunter's pet and Metamorphosis as a historical Warlock ability, then uses Python to enumerate combinations. The resulting classes—Death Knight, Hunter, Paladin, Priest, and Warlock—match the reference answer.

This case demonstrates execution-time structural adaptation: parallel evidence collection → dependency-based solving → conflict-driven rechecking → specialized hypothesis revision. Each delegation addresses a limitation revealed by intermediate execution results rather than following a predefined fixed workflow. (Response W3 continues in Part 3.)

AuthorsOpenReview ↗

Response to Reviewer tWB1 (Part 3)

Configuration Evolution — Travel Planning (Case ID: 117). The task involves planning a Chongqing–Zhengzhou trip for November 12–18, 2025, covering transportation, accommodation, meals, and sightseeing. The Meta Agent assigns a sub-agent to select the three highest-rated recommended attractions and the highest-rated natural-scenery attraction in Zhengzhou, initially providing only recommend_attractions. Since this tool returns descriptions and categories without ratings, the sub-agent cannot complete the ranking and reports the missing information through call_main_agent. The Meta Agent adds query_attraction_details to the same running sub-agent, enabling it to retrieve ratings such as 4.6 for the Yellow River Museum and 4.8 for Zijingshan Park. This intervention provides the information needed for the delegated subtask, although the overall itinerary does not pass all evaluation checks.

Cross-task Evolution — BrowseComp-Plus (Case ID: browsecomp_plus_681). The task asks for a venue's capacity after a 1980s expansion, requiring historical, geographical, and literary evidence. EvoOrch retrieves an orchestration insight from an out-of-domain TaskCraft trajectory (taskcraft_71), recommending parallel exploration followed by dependency-based verification. Guided by this insight, the Meta Agent investigates the venue's history and literary clues in parallel, then constructs a dependency-based verification chain to cross-check the area's establishment year, the play's production year, and the venue's capacity expansion. EvoOrch returns 6,187, matching the reference answer. This illustrates how historical orchestration and verification insights guide structural decisions on new tasks without transferring task-specific answers.

We will include the complete cases in the revised paper. These examples illustrate the mechanisms qualitatively; the ablation results provide the broader empirical comparison.

Weakness 4: The structural ablation restricts execution to serial delegation, removing parallel and dependency-group capabilities alongside structural adaptation. This makes it difficult to isolate the benefit of adapting the structure during execution.

Response W4: Thank you for the suggestion. We further conduct a controlled ablation to isolate the contribution of runtime structural evolution from the benefits of parallel and dependency-based delegation. 1. Fixed-Structure Ablation. We introduce a new variant, w/ Fixed Structure, in which the Meta Agent constructs a task-specific multi-agent structure and assigns the corresponding sub-agent configurations at the beginning of execution. The resulting structure remains fixed throughout the task, without creating additional agent groups or modifying their dependencies. In the default EvoOrch setting, the Meta-Agent checkpoint budget is 10, the sub-agent budget is 30 steps. To allow a larger per-agent execution budget, we increase the maximum sub-agent steps to 100 in the Fixed-Structure variant.

MethodGAIA Avg@3 ↑GAIA Pass@3 ↑GAIA Cost ↓BrowseComp-Plus Avg@3 ↑BrowseComp-Plus Pass@3 ↑BrowseComp-Plus Cost ↓
EvoOrch65.580.60.1044.563.50.31
w Fixed Structure52.969.70.1435.650.50.37
w/o Stru. Evo.53.971.80.0837.655.50.25
w/o Config. Evo.60.275.80.1542.359.50.38
w/o Cross. Evo.62.477.60.1241.360.50.36

Compared with the fixed-structure variant, EvoOrch improves Avg@3 by 12.6 points on GAIA and 8.9 points on BrowseComp-Plus, respectively. These results show that preconstructing a task-specific multi-agent structure is insufficient to achieve the same performance, even with a larger per-agent execution budget. Adapting the agent structure to intermediate execution results provides substantial additional benefits beyond predefined parallel and dependency-based delegation.

2. Comparison with Predefined Workflows. Our main experiments also include representative predefined workflow methods, such as OWL and MASFactory, which construct their primary collaboration structures before execution. EvoOrch consistently achieves stronger performance, providing complementary evidence for the benefits of runtime structural evolution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.