acceptodds
Under review as a conference paper at ICLR 2027

AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?

Abstract

Enterprise workflow agents typically rely on step-by-step interaction (Step-by-Step): the agent executes an action, observes its result, and invokes a large language model (LLM) again to decide the next step. Global orchestration (Global) instead generates a complete executable workflow in advance and executes it without further LLM calls, substantially reducing inference overhead, but it remains less accurate than Step-by-Step. We study how close Global can come to Step-by-Step and what narrows the remaining gap. We introduce AgentOrch, a benchmark that directly compares global orchestration with step-by-step interaction on 650 enterprise SOPs and 3,220 runtime variants under matched tools, environments, and evaluation. Step-by-Step remains more accurate for every evaluated model, but the gap has shrunk by about 79% from early models to the strongest recent model. We identify three routes to further narrowing this gap. Incremental workflow construction improves Global accuracy by an average of 9.98 percentage points (pp) without runtime execution feedback. Revising complete workflows with execution feedback improves all evaluated models, including 6.13 pp on held-out runtime variants. Targeted training on complete workflows improves accuracy by an average of 12.18 pp across all five trained models. Although a gap remains, global orchestration offers a promising path toward enterprise agents that deliver higher accuracy with fewer LLM calls.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, LLM Agents, Workflow Orchestration, Tool Use, Enterprise Automation, Agentic Workflows, Execution Feedback, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HufHOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates global orchestration as a way to reduce the inference overhead of enterprise agents. It introduces the AGENTORCH benchmark to compare complete workflow generation with step-by-step interaction under matched conditions. In the reported cost comparison, step-by-step interaction requires 23.2 times as many LLM calls, although it remains more accurate. The paper further shows that incremental workflow construction, revision using execution feedback, and targeted training can narrow this accuracy gap.

Critical strengths
  • The benchmark evaluates both protocols on the same SOPs, tools, and execution criteria across multiple runtime variants, supporting a controlled comparison of accuracy and inference overhead.
  • The REPLAN analysis shows that gains are concentrated in initially invalid workflows and that validation feedback alone recovers much of the improvement. This identifies argument and reference validity as an important bottleneck in global orchestration.
Critical weaknesses
  • The benchmark focuses on structured Workato workflows and evaluates generated workflows in simulation. Generalization to other automation environments and less structured tasks remains unclear.
  • The evaluation uses fixed interface schemas, leaving GLOBAL’s robustness to unexpected changes in tool output structure unclear. Such changes may require revising the generated workflow, and the resulting impact on accuracy and inference cost remains insufficiently evaluated.
Questions for authors
  • How would GLOBAL recover from runtime failures that its predefined handlers cannot resolve, and how would any additional LLM calls or workflow regeneration affect its efficiency advantage over STEP-BY-STEP?
Reviewer_hPy3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AgentOrch, a benchmark of 650 enterprise standard operating procedures and 3,220 runtime variants. It compares workflows generated in advance with agents that decide actions after observing each tool result. The authors report that step-by-step interaction remains more accurate, while global orchestration uses fewer LLM calls. They study incremental workflow construction, revision using execution feedback, and targeted training as ways to improve global orchestration.

The protocol comparison, benchmark construction, and main numerical results are generally clearly presented. Some headline efficiency and training summaries require qualifications from the appendix.

The benchmark and error decomposition provide useful evidence, although their implications are narrower than some of the headline claims.

The reported arithmetic is consistent with the tables checked, but important concerns remain about construct validity and attribution of gains.

Critical strengths

A controlled comparison on common SOPs, tool interfaces, and runtime variants is valuable. The benchmark includes executable workflows rather than relying only on textual judgments. The appendix provides meaningful details about simulator agreement, trace scoring, and the origins of revision improvements. These disclosures make the results easier to scrutinize.

Critical weaknesses

First, the main metric partly rewards agreement with a reference execution rather than independent task correctness. Orchestration Accuracy averages a five-component binary trace score with side-effect accuracy. A functionally valid alternative workflow can therefore lose substantial credit for using a different tool sequence, ordering, or control path. The paper includes semantic treatment for some queries and a separate side-effect check, but does not fully establish that reference-trace conformity is an unbiased proxy for successful orchestration. The simulator is validated against reference replays, yet this does not prove equivalent behavior for unusual or incorrect model-generated workflows.



Second, the paper's execution-feedback story is considerably weaker than its aggregate revision result might suggest. The authors report that 92.6% of the gain in their three-model decomposition comes from repairing initially invalid workflows, with initial failures detectable before execution; validation-only revision recovers 75.5% of the aggregate REPLAN gain. This suggests that much of the improvement comes from static validation and repair rather than information learned from executing workflows. The reported 23.2-times call difference and 3.64-times completion-token difference are not a complete cost or latency comparison, and use different reporting denominators. Finally, the 12.18-point training gain selects the better of SFT and RL per model. It is not the typical effect of either method separately, and the recipe-disjoint split does not test transfer to unseen tool ecosystems.

Questions for authors

How many workflows fail trace agreement while achieving all intended effects through a valid alternative execution? Can these cases be independently adjudicated? How well does the simulator agree with real connectors on model-generated failures and non-reference tool sequences? Under equal inference and validation budgets, what additional gain comes specifically from execution feedback over validation-only repair? What is the end-to-end cost and latency at different numbers of runtime variants per SOP? Can the authors report SFT and RL outcomes separately, and test workflow-template-disjoint generalization?

Reviewer_MKNGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentOrch, a benchmark for evaluating global workflow orchestration against step-by-step interaction in enterprise environments. Using 650 SOPs and 3,220 runtime variants, the authors systematically compare the two paradigms and investigate several approaches to improving global orchestration, including incremental workflow construction, execution-feedback-based revision, and targeted training. The experiments demonstrate that global orchestration can substantially reduce inference overhead, although a considerable performance gap remains.

Critical strengths

The paper introduces a benchmark grounded in enterprise workflows, providing a valuable resource for studying LLM-based workflow orchestration. By supporting both global orchestration and step-by-step interaction under consistent evaluation settings, AgentOrch could facilitate systematic comparisons and support future research in this direction.

Critical weaknesses
  1. Many of the experimental findings appear relatively intuitive and largely consistent with existing knowledge in the field. For instance, the benefits of incremental construction, execution-feedback-based refinement, and task-specific training are not particularly surprising. While the paper provides empirical evidence supporting these observations, it offers limited deeper insights into why these approaches work, under what conditions they are most effective, or what fundamentally determines the performance gap between the two orchestration paradigms. Consequently, the experimental contribution appears somewhat incremental. A more thorough analysis of the underlying mechanisms and failure modes would substantially strengthen the paper.
  2. Although the paper focuses on enterprise workflow orchestration, the motivation for selecting this particular setting is not sufficiently discussed. In particular, the paper does not clearly introduce the distinctive challenges of enterprise workflows compared with other agent environments, or explain how these challenges motivate the benchmark design. While the appendix briefly discusses the suitability of structured enterprise workflows for global orchestration, a more systematic characterization of their unique properties and challenges is needed. Such an analysis would help establish the significance of the chosen setting and clarify what new research questions the benchmark enables.
  3. For a benchmark-oriented contribution, data provenance and quality are essential to establishing the credibility of the evaluation. However, the main paper provides limited information about the origin, collection, and selection of the underlying enterprise workflows. Although Appendix A.1 and Appendix G provide additional details about the sanitized Workato recipes and their validation, this information should be more prominently presented and thoroughly discussed in the main paper. In particular, the authors should clarify how representative the collected workflows are of real-world enterprise tasks, how data quality is ensured, and what potential selection biases may exist. Without a sufficiently transparent characterization of the data, it is difficult to assess the reliability, representativeness, and generalizability of the benchmark.
  4. The evaluation relies heavily on simulated execution, yet the main paper provides limited technical details about how the simulator is implemented. Although Appendix A.2 describes the simulated connector backend and Appendix D reports simulator fidelity, important implementation details remain unclear, such as how tool behaviors, state transitions, and execution failures are modeled. This is particularly important for methods that explicitly leverage execution feedback, as their effectiveness may depend on the fidelity and characteristics of the simulation environment. The authors should provide a more detailed description of the simulation mechanism in the main paper and discuss how potential discrepancies between simulated and real execution might affect the conclusions.
Questions for authors

See the weakness part.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear AC,

We thank the AC and all three reviewers for their constructive feedback. During rebuttal, we added experiments and analyses on performance gaps, simulator fidelity, metric robustness, repair, inference costs, and generalization. We summarize them below.

TL;DR of Our Work

AgentOrch compares GLOBAL orchestration (generate one executable workflow upfront) with STEP-BY-STEP interaction (replan after each tool result) under matched conditions across 650 production-derived SOPs and 3,220 runtime variants. GLOBAL reduces inference overhead but remains less accurate; the gap has narrowed by ~79% across model generations. We identify three routes to improvement: incremental construction (+9.98 pp on average), execution-feedback revision (+6.13 pp on held-out variants), and targeted training (+10.29 pp SFT; +11.93 pp RL).

Reviewers’ Recognition

Reviewers recognize the value of our benchmark and controlled comparison:

  • MKNG: “The paper introduces a benchmark grounded in enterprise workflows, providing a valuable resource for studying LLM-based workflow orchestration.”
  • hPy3: “A controlled comparison on common SOPs, tool interfaces, and runtime variants is valuable.”
  • HufH: “The benchmark evaluates both protocols on the same SOPs, tools, and execution criteria across multiple runtime variants, supporting a controlled comparison of accuracy and inference overhead.”

Reviewers recognize the insights and rigor of our analysis:

  • HufH: “This identifies argument and reference validity as an important bottleneck in global orchestration.”
  • hPy3: “The benchmark and error decomposition provide useful evidence.”
  • hPy3: “The appendix provides meaningful details about simulator agreement, trace scoring, and the origins of revision improvements.”

How We Addressed Reviewers’ Concerns

1. What drives the gap? Workflow complexity and validity (MKNG). New subgroup analysis shows the gap increases with workflow length (26.3 → 38.3 pp) and nesting depth (25.3 → 36.5 pp), but remains nearly flat across application counts. 83.9% of workflows repaired by REPLAN initially fail schema, type, or reference checks, explaining the effectiveness of validation and correction (Section 4; Tables 16, 18).

2. Why enterprise SOPs, and how reliable is the data? (MKNG). We clarify the role of repeated execution, consequential side effects, and prescribed procedures. From 803 sanitized production Workato recipes, we retain 650 multi-step SOPs. Four authors reviewed descriptions over five rounds; every variant was executed and manually checked. We document tool coverage, workflow structure, and selection bias (Sections 1, 3; Appendices A.1, G).

3. Are simulator and metrics trustworthy? Real-connector and alternative-trace checks (MKNG, hPy3, HufH). We test model-generated GLOBAL workflows on real Slack, Snowflake, and Jira connectors: simulated Orchestration Accuracy differs by just 0.05–2.3 pp, with outcomes differing on at most 1.5% of variants and unchanged model rankings. Across 89 model–protocol runs, crediting valid alternative traces shifts the mean STEP-BY-STEP advantage from 32.6 to 33.3 pp, so trace conformity does not explain the gap (Section 3; Appendices D.2, J.1).

4. What causes REPLAN gains? Static repair vs. execution feedback (hPy3). Validation-only repair recovers 75.5% of the aggregate REPLAN gain in our three-model decomposition. Under a shared three-revision-call cap, execution feedback adds 1.05–1.49 pp on initially invalid workflows for those models. We report model-level gains and realized inference budgets (Section 4; Appendix J.2).

5. Does efficiency survive recovery? Amortization and limitations (hPy3, HufH). In the main comparison, STEP-BY-STEP uses 23.2× as many LLM calls and 3.64× as many completion tokens. GLOBAL reaches completion-token break-even after 0.94–3.49 executions for generation alone, or 5–8 executions including three full REPLAN rounds. We clarify how unhandled failures stop execution (Appendices F, H).

6. Does training generalize? Separate SFT/RL and unseen templates (hPy3). We separately report average accuracy gains of 10.29 pp (SFT) and 11.93 pp (RL), replacing the earlier best-of-two summary. On SOPs without matching training workflow templates, gains remain 5–19 pp (Section 4; Appendix J.3).

Overall, these additions directly address the major concerns and reinforce our central result: GLOBAL can become substantially more accurate while preserving inference-efficiency advantages on structured enterprise workflows. We sincerely thank the reviewers for their valuable feedback and respectfully invite the AC to consider the additional evidence and revisions in the final assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.