- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates global orchestration as a way to reduce the inference overhead of enterprise agents. It introduces the AGENTORCH benchmark to compare complete workflow generation with step-by-step interaction under matched conditions. In the reported cost comparison, step-by-step interaction requires 23.2 times as many LLM calls, although it remains more accurate. The paper further shows that incremental workflow construction, revision using execution feedback, and targeted training can narrow this accuracy gap.
Critical strengths
- The benchmark evaluates both protocols on the same SOPs, tools, and execution criteria across multiple runtime variants, supporting a controlled comparison of accuracy and inference overhead.
- The REPLAN analysis shows that gains are concentrated in initially invalid workflows and that validation feedback alone recovers much of the improvement. This identifies argument and reference validity as an important bottleneck in global orchestration.
Critical weaknesses
- The benchmark focuses on structured Workato workflows and evaluates generated workflows in simulation. Generalization to other automation environments and less structured tasks remains unclear.
- The evaluation uses fixed interface schemas, leaving GLOBAL’s robustness to unexpected changes in tool output structure unclear. Such changes may require revising the generated workflow, and the resulting impact on accuracy and inference cost remains insufficiently evaluated.
Questions for authors
- How would GLOBAL recover from runtime failures that its predefined handlers cannot resolve, and how would any additional LLM calls or workflow regeneration affect its efficiency advantage over STEP-BY-STEP?
Rebuttal Summary
Dear AC,
We thank the AC and all three reviewers for their constructive feedback. During rebuttal, we added experiments and analyses on performance gaps, simulator fidelity, metric robustness, repair, inference costs, and generalization. We summarize them below.
TL;DR of Our Work
AgentOrch compares GLOBAL orchestration (generate one executable workflow upfront) with STEP-BY-STEP interaction (replan after each tool result) under matched conditions across 650 production-derived SOPs and 3,220 runtime variants. GLOBAL reduces inference overhead but remains less accurate; the gap has narrowed by ~79% across model generations. We identify three routes to improvement: incremental construction (+9.98 pp on average), execution-feedback revision (+6.13 pp on held-out variants), and targeted training (+10.29 pp SFT; +11.93 pp RL).
Reviewers’ Recognition
Reviewers recognize the value of our benchmark and controlled comparison:
Reviewers recognize the insights and rigor of our analysis:
How We Addressed Reviewers’ Concerns
1. What drives the gap? Workflow complexity and validity (MKNG). New subgroup analysis shows the gap increases with workflow length (26.3 → 38.3 pp) and nesting depth (25.3 → 36.5 pp), but remains nearly flat across application counts. 83.9% of workflows repaired by REPLAN initially fail schema, type, or reference checks, explaining the effectiveness of validation and correction (Section 4; Tables 16, 18).
2. Why enterprise SOPs, and how reliable is the data? (MKNG). We clarify the role of repeated execution, consequential side effects, and prescribed procedures. From 803 sanitized production Workato recipes, we retain 650 multi-step SOPs. Four authors reviewed descriptions over five rounds; every variant was executed and manually checked. We document tool coverage, workflow structure, and selection bias (Sections 1, 3; Appendices A.1, G).
3. Are simulator and metrics trustworthy? Real-connector and alternative-trace checks (MKNG, hPy3, HufH). We test model-generated GLOBAL workflows on real Slack, Snowflake, and Jira connectors: simulated Orchestration Accuracy differs by just 0.05–2.3 pp, with outcomes differing on at most 1.5% of variants and unchanged model rankings. Across 89 model–protocol runs, crediting valid alternative traces shifts the mean STEP-BY-STEP advantage from 32.6 to 33.3 pp, so trace conformity does not explain the gap (Section 3; Appendices D.2, J.1).
4. What causes REPLAN gains? Static repair vs. execution feedback (hPy3). Validation-only repair recovers 75.5% of the aggregate REPLAN gain in our three-model decomposition. Under a shared three-revision-call cap, execution feedback adds 1.05–1.49 pp on initially invalid workflows for those models. We report model-level gains and realized inference budgets (Section 4; Appendix J.2).
5. Does efficiency survive recovery? Amortization and limitations (hPy3, HufH). In the main comparison, STEP-BY-STEP uses 23.2× as many LLM calls and 3.64× as many completion tokens. GLOBAL reaches completion-token break-even after 0.94–3.49 executions for generation alone, or 5–8 executions including three full REPLAN rounds. We clarify how unhandled failures stop execution (Appendices F, H).
6. Does training generalize? Separate SFT/RL and unseen templates (hPy3). We separately report average accuracy gains of 10.29 pp (SFT) and 11.93 pp (RL), replacing the earlier best-of-two summary. On SOPs without matching training workflow templates, gains remain 5–19 pp (Section 4; Appendix J.3).
Overall, these additions directly address the major concerns and reinforce our central result: GLOBAL can become substantially more accurate while preserving inference-efficiency advantages on structured enterprise workflows. We sincerely thank the reviewers for their valuable feedback and respectfully invite the AC to consider the additional evidence and revisions in the final assessment of our work.