- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EvoOrch is a training-free framework for dynamically orchestrating LLM-based multi-agent systems. A Meta Agent creates task-specific agent structures and can revise active agents’ tools, instructions, context, models, or budgets. Across tasks, completed trajectories are summarized and distilled into reusable orchestration insights. These insights are retrieved from a frozen experience base during evaluation. Experiments across six benchmarks show improvements over several orchestration baselines, with additional analyses of intervention, experience retrieval, backbone allocation, and execution cost.
Critical strengths
The framework addresses a practical limitation of agent delegation: a worker may become ineffective after it has been created. Allowing the Meta Agent to revise that worker in place is a useful design choice, especially when tools, context, or execution budgets become bottlenecks.
The empirical evaluation is reasonably broad. Improvements appear across multiple benchmarks and backbones, and the main results combine higher task scores with competitive inference costs. For example, on GAIA with GPT-5, EvoOrch improves Avg@3 from AOrchestra’s 58.7 to 64.2 while reducing the reported cost from $0.81 to $0.56.
Several additional controls strengthen the evidence. Appendix A.2.7 gives retrieved insights to competing systems, while the cross-task analyses compare raw trajectories, distilled insights, self-review, and gold-assisted review. Freezing the experience base before evaluation also provides a clear separation between historical experience and test tasks.
Critical weaknesses
The intervention analysis does not fully establish the value of targeted diagnosis. The same-state comparison in Figure 2b is useful, but retaining the original configuration is a relatively limited control. A stuck agent might also recover through a budget-matched restart or by receiving the full tool set from the beginning. Since tool changes are the most frequent intervention, this distinction matters. The paper should also define “recovery success” precisely and report the number of evaluated states and uncertainty for each intervention type.
The structural ablation changes several factors together. Restricting the system to serial delegation removes parallel scheduling and structured information handoffs, and can change how much work is delegated within one Meta-Agent checkpoint. Table 2 therefore supports the usefulness of the complete graph-based delegation mechanism, but does not isolate the benefit of adapting dependencies during execution. A fixed parallel or dependency-based workflow with comparable worker and token budgets would provide a more informative comparison.
The cost advantage is measured after preparing the experience base. The main system uses trajectories from 400 tasks and Golden-Review distillation, but the reported costs describe inference-time token usage. The cost of collecting and reviewing those trajectories is not quantified. This does not invalidate the inference results, but it leaves the practical savings unclear for smaller workloads. The authors should report preparation costs and an amortized comparison. They should also distinguish training-free operation from the use of gold supervision during experience construction.
Questions for authors
- How many same-state intervention pairs were evaluated, and does “recovery success” mean completing the delegated subtask or solving the original task?
- How does EvoOrch compare with a fixed parallel/dependency workflow under matched total worker-step or token budgets?
- What is the total cost of collecting and distilling the 400 historical trajectories, and after how many evaluation tasks do the inference savings offset that cost?
4 comments
Response to Reviewer GP36 (Part 2)
Weakness 2: The current structural ablation does not isolate the benefit of runtime structural adaptation. How does EvoOrch compare with a fixed parallel or dependency-based workflow under matched execution budgets? (Original Weakness 2 and Question 2)
Response W2: Thank you for your valuable suggestion. We conduct an additional Fixed-Structure ablation, where the Meta Agent generates a single task-specific DAG at the beginning of execution, with the agent structure remaining fixed throughout the task while other operations are retained. In the default EvoOrch setting, the Meta-Agent checkpoint budget is 10, the sub-agent budget is 30 steps. To allow a larger per-agent execution budget for the Fixed-Structure variant, we increase the maximum sub-agent steps from 30 to 100. The results are shown below.
| Method | GAIA Avg@3 ↑ | GAIA Pass@3 ↑ | GAIA Cost ↓ | BrowseComp-Plus Avg@3 ↑ | BrowseComp-Plus Pass@3 ↑ | BrowseComp-Plus Cost ↓ |
|---|---|---|---|---|---|---|
| EvoOrch | 65.5 | 80.6 | 0.10 | 44.5 | 63.5 | 0.31 |
| w/ Fixed Structure | 52.9 | 69.7 | 0.14 | 35.6 | 50.5 | 0.37 |
| w/o Stru. Evo. | 53.9 | 71.8 | 0.08 | 37.6 | 55.5 | 0.25 |
| w/o Config. Evo. | 60.2 | 75.8 | 0.15 | 42.3 | 59.5 | 0.38 |
| w/o Cross. Evo. | 62.4 | 77.6 | 0.12 | 41.3 | 60.5 | 0.36 |
Compared with the Fixed-Structure variant, EvoOrch improves Avg@3 by 12.6 points on GAIA and 8.9 points on BrowseComp-Plus, while reducing inference costs. These results demonstrate the benefit of adapting agent structures based on intermediate execution results, beyond simply using predefined parallel or dependency-based delegation. Our main experiments also include predefined workflow methods such as OWL and MASFactory, providing further comparative evidence for the runtime agent structure evolution.
Weakness 3: The reported costs exclude the preparation of 400 historical trajectories and Golden-Review distillation. What is the total preparation cost, and how many evaluation tasks are needed to amortize it through inference savings? (Original Weakness 3 and Question 3)
Response W3: Thank you for the question. The costs reported in our main experiments cover inference-time execution only. Historical trajectory collection and experience distillation incur additional preparation costs, which we clarify below and further include them in our revision.
1. Experience Preparation. EvoOrch constructs its experience base from 400 historical trajectories using DeepSeek-V3.2 in Golden-Review. The average cost is approximately 0.12 per trajectory for execution and 0.05 for Golden-Review distillation, totaling $68 for experience preparation. The experience base is frozen before evaluation and reused across subsequent tasks. For N evaluation runs, the preparation cost is per run. For example, in main experimentt cross 165 GAIA, 200 BrowseComp-Plus, and 240 Travel Planning tasks, each evaluated three times, (N = 1815), yielding an amortized cost of approximately 0.037 per run. This overhead decreases further as the experience base is reused.
2. Inference Cost Savings. As shown in Table 2, cross-task evolution improves GAIA Avg@3 from 62.4 to 65.5 while reducing inference cost from 0.12 to 0.10 per run. On BrowseComp-Plus, Avg@3 increases from 41.3 to 44.5, while cost decreases from 0.36 to 0.31. For BrowseComp-Plus, the inference savings of 0.05 per run slightly exceed the preparation cost of 0.037 per run, indicating that the preparation overhead can be offset under this setting. These results show that historical insights improve inference efficiency, while the one-time preparation overhead can be amortized over repeated use. We will clarify these costs and their amortization in the revised manuscript.
3. Training-free vs. Gold Supervision. EvoOrch requires no model parameter updates, but Golden-Review uses ground-truth answers during offline experience preparation. The experience base is frozen before evaluation, and no evaluation-task ground-truth answers are accessed during test-time execution. We further compare Golden-Review and Self-Review in our Cross-task Evolution Analysis, demonstrating the effectiveness of different review strategies without model training, thereby avoiding the computational overhead of model parameter updates.
We would be happy to provide further clarification and address any additional questions or concerns you may have.
Follow-up on Our Rebuttal Response
Dear Reviewer GP36,
Thank you for taking the time to review our paper and provide valuable feedback. We sincerely appreciate your time and effort, and apologize for the limited time available for further discussion.
As less than two days remain in the discussion period, we would be grateful if you could let us know whether our rebuttal has adequately addressed your concerns or if there are any remaining questions that we could further clarify.
Thank you again for your time and consideration. We look forward to hearing your feedback.
Best regards,
The Authors
Dear Reviewer GP36,
I hope this message finds you well.
As the discussion period is coming to a close, we would greatly appreciate it if you could revisit our rebuttal and let us know whether any concerns remain. We would be happy to provide further clarification or address any outstanding questions before the deadline.
Thank you again for your time and effort in reviewing our paper and considering our response.
Best regards,
The Authors
Response to Reviewer GP36 (Part 1)
Thank you for your thoughtful comments and questions. For clarity, we have organized our responses around the weaknesses and questions raised in your original review.
Weakness 1: The intervention analysis does not sufficiently establish the benefits of targeted diagnosis. Moreover, the paper should clarify whether “recovery success” refers to completing the delegated subtask or solving the original task, and report the number of same-state intervention pairs evaluated, along with the uncertainty for each intervention type. (Original Weakness 1 and Question 1)
Response W1: Thank you for the valuable suggestions. We further clarify the recovery success rate evaluation in our revision.
1. Recovery Success Definition and Evaluation Reliability.
We define recovery success as whether an intervened sub-agent successfully completes its delegated subtask, rather than the original task. Specifically, a sub-agent is considered unsuccessful if it does not return a finish action or its completion status is not done. For sub-agents returning done, we use DeepSeek-V3.2 to assess whether their execution trajectories and outputs satisfy the delegated subtask requirements. Each evaluation is repeated three times, and the average judgment is used to compute the recovery success rate.
We evaluate 731 same-state intervention pairs collected from three independent GAIA runs, including 255 Tool, 172 Model, 157 Instruction, 120 Budget, and 27 Context interventions. Given the limited number of Context interventions, we repeat each same-state experiment three times to improve reliability. We also manually inspect 40 cases, finding that 37/40 LLM judgments agree with human assessments, while 35/40 interventions lead to observable behavioral changes. We will clarify these details in our revision.
2. Targeted Intervention vs. Full Tool Access.
Regarding your suggestion that a stuck agent might also recover through a budget-matched restart or by receiving the full tool set from the beginning, we agree that these are valuable controls. We further investigate one of these settings by providing the full available tool set at the intervention point. Specifically, we evaluate 100 sampled tool-intervention cases. Compared with targeted tool intervention, providing all tools only slightly improves recovery success from 58.3% to 59.3%, while increasing the average sub-agent cost from $0.0167
to $ 0.0208. This suggests that providing additional tools indiscriminately does not necessarily benefit sub-agent execution.
For tasks involving multiple tools, such as Travel Planning (9 tools), or more complex, long-horizon tasks, simply increasing the budget or providing the full tool set is not always Pareto-optimal. In contrast, decomposing agents into specific configuration dimensions facilitates more fine-grained adjustments and management, which we believe represents a more efficient perspective on configuration evolution.