acceptodds
Under review as a conference paper at ICLR 2027

Proactive Agentic Orchestration

Abstract

Agentic orchestration delegates tasks to sub-agents with independent contexts, providing a key path to extending the effective execution horizon of large language model agents. However, existing orchestration methods face two limitations: in execution, current methods fix sub-agents' goals and harnesses at creation time, limiting their adaptability to dynamic real-world environments; in optimization, sparse and delayed feedback constrains effective learning of the orchestration policy. To address these limitations, we introduce POrchestra, an on-the-fly proactive agentic orchestration framework. It allows sub-agents to proactively communicate with the orchestrator at any logical point to request harness adjustments while maintaining continuity of local execution. This mechanism not only enables sub-agents to adapt seamlessly to dynamic environments but also naturally inspires On-Policy Orchestration Distillation (OPOD) based on hindsight feedback. OPOD attributes sub-agents' proactive revision requests back to preceding orchestration decisions to derive targeted improvement signals, and leverages these signals as process supervision alongside the final reward to optimize the orchestration policy. We evaluate POrchestra on three long-horizon agent benchmarks spanning deep research, dynamic digital assistance, and software engineering. POrchestra improves task success by an average of 9.1% relative to the strongest baseline; combined with OPOD, it improves small-orchestrator success by an average of 10.0% relative to existing training baselines. POrchestra also achieves a more favorable cost–performance trade-off. Our code is publicly available at https://anonymous.4open.science/r/POrchestra-23F3.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, agentic orchestration, long-horizon tasks, adaptive agent harnesses, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T8ssOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PORCHESTRA lets a sub-agent pause mid-task, ask the orchestrator to change its goal, tools or budget, and carry on with its context intact. OPOD then uses these requests as hindsight hints to train a small orchestrator. Across GAIA, GAIA2 and a 100-issue SWE-bench subset, with three executor models, it baets the best baseline by 7 to 11 points on GAIA2, by less on GAIA, and by one issue on SWE-bench, while costing less.

Critical strengths

On GAIA2, the one benchmark where things change mid-task, the gain is large and consistent across all three executors. A same-model comparison shows it comes from the communication mechanism itself. It also cuts cost per task from 1.21 to 0.72 relative to AOrchestra.

Critical weaknesses

OPOD isn't established. Its edge over a simpler baseline is a few tasks, its settings were tuned on the test set, and it closely follows self-distillation work it barely cites. Outside GAIA2 the gains are within noise. Inside GAIA2, the prompt tells sub-agents to report every notification, which effectively acts as a trigger, and this is never controlled for.

Questions for authors

How many runs underlie each number? How did the baselines receive GAIA2 notifications, and what would a simple notification trigger score? For future work, OPOD should be compared with GRPO at matched compute.

Reviewer_tNNbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes POrchestra, which allows sub-agents to request changes to their tasks, tools, context, or budgets during execution. It also introduces OPOD to turn this feedback into supervision for training the orchestrator. Experiments on GAIA, GAIA2, and SWE-Bench show improved task success, with additional ablations and cost analysis.

Critical strengths
  1. The paper studies a practical problem in agent orchestration. Letting sub-agents request adjustments while keeping their execution history is intuitive and useful for tasks with changing requirements.

  2. The experiments cover three benchmarks and multiple execution models. The comparisons with polling and error-triggered intervention, together with the component ablations, provide useful evidence that proactive feedback helps.

Critical weaknesses
  1. The evaluation protocol makes some gains hard to judge. Appendix E.3 says that only the top two methods in each task category are evaluated twice, while others are evaluated once. Without consistent repeated runs and uncertainty estimates, it is unclear how stable the rankings are. The SWE-Bench improvement over the strongest baseline is also only one additional solved issue out of 100 for each backbone.

  2. The OPOD description is not fully consistent with the implementation. Section 3.3 describes current-policy trajectories and a same-parameter self-teacher, while Algorithm 2 and Appendix E use stored SFT trajectories and a frozen SFT teacher. Student actions are sampled during training, but this distinction matters for understanding what is actually on-policy.

Questions for authors
  1. Can the authors report repeated runs under the same protocol for the main methods, with uncertainty estimates? Do the gains in Table 2 remain consistent?

  2. Can the authors clarify which parts of OPOD are on-policy and align Section 3.3 with Algorithm 2? Are execution trajectories or feedback hints ever refreshed during training?

Reviewer_JcnMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper points out that multi-agent systems now mainly have two kinds of problems: 1. The sub-agent's goal context is fixed from the beginning; 2. When training the main agent, training is basically done on the success or failure of the final task, without judging whether there are wrongly assigned tasks in the middle. For the above two problems the paper's main approach is: when a sub-agent gets stuck it can pause and reply to the main agent with current status/evidence and other information, and then use this kind of information to in turn train the main Agent. The effects the paper mainly claims are that the sub-agent can adjust more timely to avoid task failure and compared with directly judging task success or failure this kind of training method is more fine-grained and the effect is better. The paper tested on three benchmarks, and at the same time on GAIA and GAIA2 used the trained Qwen3.5-9B as the main agent, and compared with several common other training methods.

Critical strengths

The overall structure description is relatively clear.

Critical weaknesses
  1. Lack of innovativeness, the paper's main work is designing a mechanism in which a sub-agent encountering a problem proactively reports to the main agent and adjusts the task, this kind of mechanism is already very common in current mainstream harnesses such as claude code codex cursor;
  2. The significance of the improvement from main agent training is not high and is not sufficient to prove the effectiveness of the design;
  3. The final comparison experiments are also very thin and did not compare current mainstream harnesses such as claude code codex cursor opencode etc. models also have no comparison with current sota models. The experiments also only ran pass@1 the final results differ by only one case and almost have no significance.
Questions for authors

Can mainstream harnesses and the method proposed by the paper be used, use sota models, to test on more complex long-horizon benchmarks?

Comments on the paper

8 comments

We thank the authors for the extensive additional experiments; however, the novelty concern remains, since the harness revision is still delivered as an observation appended to the sub-agent's trajectory and is functionally close to mechanisms already available in mainstream harnesses; the contribution of OPOD's process supervision over its ablation is not shown to be significant; and most new comparisons (SWE-Bench, AutomationBench, DeepSWE) fall within run-to-run variance, with Claude Code missing, harness configurations unspecified, and no harness comparison on GAIA/GAIA2, so I maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer Follow-up Comments (Part 1/2)

We thank the reviewer for the follow-up comments and the opportunity to provide further clarification.

The novelty concern remains, since the harness revision is still delivered as an observation appended to the sub-agent's trajectory and is functionally close to mechanisms already available in mainstream harnesses; the contribution of OPOD's process supervision over its ablation is not shown to be significant;

We thank the reviewer for their comments. We understand that the novelty concerns primarily involve two aspects: (1) the Harness Revision mechanism in PORCHESTRA and (2) the statistical significance of OPOD's process supervision.

(1) PORCHESTRA's Harness Revision mechanism is fundamentally different from existing approaches.

We would like to clarify a key technical misunderstanding: PORCHESTRA's Harness Revision is not implemented by appending observations to the Sub-agent's trajectory, but instead directly modifies the underlying Harness configuration. Specifically, Task and Context Revision directly replace the task description and context; Budget Revision modifies the actual execution budget; and Tool Revision adjusts the set of available tools, thereby changing the Sub-agent's actual action space.

Observations merely notify the Sub-agent of configuration changes that have already taken effect; they are not the mechanism through which these changes are implemented. To the best of our knowledge, existing mainstream agent harnesses do not provide an equivalent mechanism that allows Sub-agents to proactively trigger modifications to the actual Harness configuration while preserving their execution history. Our ablation results demonstrate the performance benefits of this capability, as well as the necessity of the complete Communication Semantics (Status, Request, Evidence) protocol in supporting this mechanism.

(2) OPOD's process supervision provides statistically significant improvements over its ablations.

To directly address the reviewer's concern that "OPOD's process supervision over its ablation is not shown to be significant," we conducted additional one-sided Welch's t-tests comparing the full OPOD against two ablated variants. The results are summarized below:

BenchmarkOPOD > w/o RWR (OPD only)OPOD > w/o OPD (RWR only)
GAIA0.030350.03892
GAIA20.034100.04570

All four comparisons yield p-values below 0.05, demonstrating statistically significant improvements of the full OPOD over both ablated variants. In particular, the significant improvements over w/o OPD (RWR only) provide direct empirical evidence for the independent contribution of process supervision.

(3) PORCHESTRA introduces technical contributions at the conceptual, protocol, and learning levels.

PORCHESTRA's core contributions span three complementary dimensions:

  • Conceptual contribution: Extending proactive communication beyond information synchronization to Sub-agent-initiated dynamic Harness Evolution;
  • Protocol contribution: Systematically defining a proactive agent orchestration protocol, with ablation studies demonstrating both its effectiveness and the necessity of its protocol components;
  • Learning contribution: Introducing OPOD, which transforms Sub-agent orchestration feedback into process supervision, with additional statistical significance tests supporting its effectiveness.

As explicitly stated in the ICLR 2027 Reviewer Guidelines [11], submissions should be evaluated based on whether they contribute "new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)" to the research community.

We believe that PORCHESTRA's combined contributions at the conceptual, protocol, and learning levels constitute a meaningful research advance, supported by clear technical distinctions, empirical evidence, and practical value. We respectfully encourage the reviewer to reconsider the novelty assessment in light of these clarifications and additional results.

[11] ICLR 2027. Reviewer Guidelines, Section "Reviewing a submission: step-by-step."

AuthorsOpenReview ↗

Response to Reviewer Follow-up Comments (Part 2/2)

most new comparisons (SWE-Bench, AutomationBench, DeepSWE) fall within run-to-run variance, with Claude Code missing, harness configurations unspecified, and no harness comparison on GAIA/GAIA2

(1) PORCHESTRA's performance gains exceed the strongest baseline's run-to-run variability, rather than reflecting marginal numerical differences.

To address the reviewer's concern about run-to-run variance, we clarify that PORCHESTRA's mean performance exceeds not only the strongest baseline's mean, but also its mean plus one standard deviation in the relevant comparisons:

μ_PORCHESTRA > μ_BestBaseline + σ_BestBaseline

Here, μ and σ denote the mean and standard deviation across independent runs. Notably, this criterion has also been explicitly used to highlight performance advantages in an accepted ICLR paper [12].

Beyond reporting means, standard deviations, and Pass@3 across multiple runs, we consistently achieve gains exceeding this criterion on multiple benchmarks. To the best of our knowledge, our evaluation provides comparatively comprehensive and statistically robust evidence relative to existing agent orchestration studies, official model technical reports, and prior work on the same benchmarks. We believe these results provide strong empirical support for PORCHESTRA's effectiveness and robustness.

(2) Additional comparisons with Claude Code, Codex, and OpenCode on GAIA and GAIA2.

To address the missing Claude Code baseline and the lack of mainstream harness comparisons on GAIA/GAIA2, we extend our previous evaluations on SWE-bench, AutomationBench, and DeepSWE with new results for OpenCode, Claude Code, and Codex on GAIA and GAIA2:

BenchmarkOpenCodeClaude CodeCodexPORCHESTRA
GAIA70.03% ± 1.6073.74% ± 4.4771.92% ± 2.2979.60% ± 4.04
GAIA244.53% ± 5.1248.44% ± 5.6349.22% ± 2.7155.21% ± 5.08

PORCHESTRA exceeds the strongest baseline's mean plus one standard deviation on both benchmarks. These results further demonstrate that PORCHESTRA's performance advantages persist in direct comparisons with mainstream agent harnesses.

(3) The mainstream harness configurations

We use the official Codex CLI v0.167.0, OpenCode v1.19.5, and Claude Code v2.1.311, invoking them through codex exec, opencode run, and claude --print, respectively. Task descriptions and associated attachments or repository information are provided as user inputs.

For action spaces, GAIA, GAIA2, and AutomationBench expose shared environment actions to all baselines through MCP tools, while SWE-bench and DeepSWE provide shell and file-operation tools in isolated containers. In GAIA2, dynamic notifications are delivered through tool responses or subsequent user messages, with native sessions resumed for multi-turn interactions.

Each harness retains its native execution loop, session history, and context compaction mechanisms. The adaptation layer handles only environment integration, execution-budget enforcement, and evaluation. Benchmark-specific integrations, prompts, tool permissions, step limits, and timeout settings are documented in https://anonymous.4open.science/r/POrchestra-23F3/harness_adaptations.md.

[12] Uendes et al. Start Smart: Leveraging Gradients for Enhancing Mask-based XAI Methods. ICLR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.