- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes E³-Orch, a reinforcement learning framework for adaptive multi-agent orchestration. It introduces a milestone-plan-act workflow that decomposes complex tasks into intermediate objectives, constructs dependency-aware execution graphs, and enables parallel execution and dynamic replanning. To improve decision-level credit assignment, the method employs tree-structured rollouts that compare alternative orchestration decisions under shared execution histories, together with separately normalized rewards for task performance, execution cost, and planning completeness. Experiments across seven benchmarks spanning mathematical reasoning, embodied decision-making, and deep research demonstrate consistent improvements in task performance and token efficiency over existing baselines. Additional experiments and ablations support the effectiveness of the proposed components and their generalization to unseen executor configurations.
Critical strengths
-
Well-motivated and coherent framework. The milestone-plan-act workflow provides a meaningful intermediate abstraction for adaptive task decomposition, dependency-aware execution, and decision-level credit assignment. Integrating this workflow with tree-structured rollouts and multi-objective reinforcement learning offers a practical approach to improving both orchestration effectiveness and efficiency.
-
Consistent empirical improvements with supporting ablations. E³-Orch achieves the best task performance across seven benchmarks under two executor configurations, while also improving token efficiency. The ablation studies systematically examine the contributions of the milestone abstraction, tree-structured rollouts, and reward components, providing evidence that the proposed design choices contribute to the observed improvements.
Critical weaknesses
-
Insufficient justification of decision-level credit assignment. Although tree-structured rollouts are motivated by comparing alternative decisions under shared execution histories, the performance reward evaluates each decision against a zero-success reference posterior rather than its sibling decisions. Moreover, advantage normalization is performed across the rollout forest rather than within shared-history groups. This raises questions about whether the proposed formulation effectively isolates individual decision contributions, which is central to the paper's methodological motivation.
-
Insufficient isolation of the proposed RL framework's contribution. While the paper provides extensive ablations, it lacks a training-free baseline using the identical milestone-plan-act workflow. Consequently, it remains unclear whether the improvements primarily arise from the hierarchical orchestration structure or the proposed RL training procedure. This limits the evidence supporting the necessity of the learning framework.
Questions for authors
-
Credit Assignment: Why does the performance reward compare each decision against a zero-success reference posterior rather than sibling decisions sharing the same execution history? How does this design effectively isolate decision-level contributions?
-
Contribution of RL: How does E³-Orch compare against a training-free baseline using the identical milestone-plan-act workflow? This comparison is important for distinguishing improvements from the orchestration structure itself versus the proposed RL training procedure.
-
Executor Configuration: In Appendix A.7, model_a and model_b use the same Qwen2.5-7B-Instruct backbone but have different capability rankings and a 1:2 cost ratio. What empirical evidence supports these assignments, and how sensitive are the results to the assumed cost ratios?
3 comments
Response to Reviewer iof2 (Part 2 / 2)
W2 & Q2: Contribution of RL
Thank you for the suggestion. We add results for a training-free variant of E³-Orch, which uses the identical milestone-plan-act workflow and the same orchestrator backbone but no RL training. The results are shown in the table below. All rows except the training-free variant are reproduced from Table 1. We have added the experiments of RL in the Appendix B.6.
| Method | Training | AIME24/25 | MATH500 | ALFWorld | WebShop | GAIA | HLE |
|---|---|---|---|---|---|---|---|
| AutoGen | ✗ | 16.7 ± 0.3 / 14.8 ± 0.4 | 79.5 ± 0.5 | 52.4 ± 0.4 | 41.9 ± 0.6 | 8.3 ± 0.5 | 2.7 ± 0.4 |
| MAS-ZERO | ✗ | 29.5 ± 0.6 / 25.0 ± 0.5 | 85.1 ± 0.5 | 58.2 ± 0.6 | 49.7 ± 0.4 | 11.5 ± 0.5 | 3.0 ± 0.4 |
| E³-Orch (training-free) | ✗ | 18.2 ± 0.3 / 15.6 ± 0.4 | 81.4 ± 0.5 | 53.6 ± 0.4 | 44.1 ± 0.6 | 9.0 ± 0.5 | 4.3 ± 0.4 |
| Best Training Baseline | ✓ | 33.5 ± 0.4 / 28.0 ± 0.4 | 88.1 ± 0.4 | 68.8 ± 0.5 | 58.9 ± 0.4 | 17.2 ± 0.5 | 6.6 ± 0.3 |
| E³-Orch | ✓ | 35.0 ± 0.4 / 31.5 ± 0.5 | 90.4 ± 0.4 | 70.2 ± 0.5 | 62.7 ± 0.6 | 20.5 ± 0.4 | 7.8 ± 0.3 |
First, even without training, E³-Orch outperforms AutoGen on all benchmarks by 0.7–2.2 points, indicating that the milestone-plan-act workflow is a reasonable structure on its own. However, the training-free variant falls behind MAS-ZERO on all benchmarks except HLE. Note that MAS-ZERO iteratively designs, evaluates, and refines a multi-agent system for each problem at inference time and selects among candidate answers via self-verification, which uses additional test-time computation, whereas the training-free E³-Orch generates each milestone and plan in a single pass.
Second, RL training accounts for most of the improvement. Training improves E³-Orch by 3.5–18.6 points across benchmarks, enabling it to surpass MAS-ZERO by 4.8–13.0 points and the best training-based baseline on each benchmark by 1.2–3.8 points. These results indicate that the milestone-plan-act structure alone is insufficient for the best performance: the improvements come primarily from the proposed RL training framework, with the structure providing an effective foundation for it.
Q3: Executor configuration
The orchestrator never observes a 1:2 cost ratio. It receives only capability and cost rankings (intelligence_rank and cost_rank), not cost values (Appendix A.7). These rankings are specified in a YAML configuration file, a flexible interface that supports diverse rankings and measurements: executors can be ranked by performance, token cost, wall-clock time, or other metrics, and new ranking attributes can be added without modifying the method.
In our experiments, the two executors share the same backbone and differ in maximum generation length (2,048 vs. 4,096 tokens, the context_windows field in Appendix A.7). This follows commercial platforms such as ChatGPT and Claude, which offer the same model at different capability and cost levels through reasoning-effort settings (low/medium/high). Empirically, we evaluate these two settings on AIME24 / GPQA as following table, which demonstrates that model_b is strong than model_a.
| model_a | model_b | |
|---|---|---|
| AIME24 | 8.3 | 10.7 |
| GPQA | 29.8 | 34.2 |
The relative costs κ = (1.0, 2.0) in Appendix A.13 quantify the different sources of executor tokens. These weights apply solely to executor tokens and are excluded from the orchestrator's input, as the orchestrator conditions on rankings rather than raw values. We have a premise: since model_a is the cheaper executor, the cost weight assigned to model_b must exceed that of model_a. It then follows from the properties of weighted sums that, provided E³-Orch uses less of model_b than the baselines do, the resulting ordering remains invariant, as demonstrated in the following table, which lists the model_b usage of E³-Orch and the baselines across all evaluation runs.
| E³-Orch | Puppeteer | MasRouter | DAAO | |
|---|---|---|---|---|
| # Tokens of model_b | 54760.73k | 70902.3k | 60245.4k | 65328.2k |
Moreover, E³-Orch transfers to new executor configurations without retraining (Appendix B.3), including a three-executor pool with κ = (1.0, 2.0, 3.0) and enriched, domain-specific profiles. Importantly, when the cost ratio is κ = (1.0, 2.0, 3.0), E³-Orch still outperforms baselines on both performance and cost.
We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.
Dear Reviewer iof2,
Thank you for your positive assessment and for the time you have worked on our paper. With only one day left in the discussion period, we would like to kindly ask whether our response has addressed your concerns. We sincerely appreciate your consideration.
Best,
Authors
Response to Reviewer iof2 (Part 1 / 2)
Thank you for your thoughtful review and for recognizing the well-motivated design of our milestone-plan-act framework, its consistent improvements in performance and token efficiency across seven benchmarks, and our systematic ablation studies. We address your concerns point by point below.
W1 & Q1: Justification of credit assignment
Advantage normalization is not performed only at the forest level. As described in Lines 262–269, the cost reward is first normalized among siblings that share the same execution history (Equation (2)) and then across the forest (Equation (4)). Cost depends strongly on context (e.g., a difficult milestone requires more tokens), so forest-level normalization alone would attribute context-induced cost differences to the decision itself.
We do not normalize the performance reward within siblings, for two reasons:
Empirically, normalizing the performance reward within siblings reduces performance on all three task families (by 1.4, 3.1, and 3.9 points) at a similar token cost, as shown in the table below. We have added the new ablation study in Section 4.3.
[1] Williams, Ronald J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Machine Learning 8.3 (1992): 229–256.
[2] Sutton, Richard S., et al. "Policy gradient methods for reinforcement learning with function approximation." Advances in Neural Information Processing Systems 12 (1999).
[3] Shao, Zhihong, et al. "DeepSeekMath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).