acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Orchestration Monoculture: Training Generalizable LLM Agents via Multi-Orchestration Reinforcement Learning

Abstract

Reinforcement learning (RL) has become the dominant paradigm for training LLM-based agents on tool-use tasks, yet virtually all existing work trains under a single, fixed orchestration such as REACT throughout the entire RL process. This orchestration monoculture limits both generalization and data utilization efficiency: agents overfit to one reasoning pattern and become brittle on out-of-distribution queries that demand different problem-decomposition strategies. We argue that the root cause runs deeper than a static choice: the training value of an orchestration drifts as the policy's capability grows, so early leaders may cap the long-run ceiling or even collapse while complex scaffolds only pay off once the policy can exploit them. We therefore treat orchestration as a self-evolving curriculum: a fixed pool of structurally diverse orchestrations whose training-time allocation is routed online by the policy's own reward stream. Instantiated as Multi-Orchestration Reinforcement Learning (MORL) with Discounted-UCB routing, the curriculum reweights rather than rewrites, adapting to capability drift while keeping the rollout distribution stable. On seven Search QA benchmarks, MORL outperforms a rollout-matched single-orchestration baseline by 5.3 exact-match points on average (43.3% vs. 38.0% with Qwen2.5-3B-Ins), with notable gains on OOD multi-hop sets (+7.8 on Bamboogle, +5.7 on MuSiQue), demonstrating strong task generalizability and robust orchestration adaptability across model sizes and domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agent, RL

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_oWBDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper is trying to address a problem that in agentic RL, training is usually only under one orchestration (orchestration monoculture as called by the authors) which can overfit to one pattern and limits the generalization capability of the models. They propose an algorithm called MORL which pools 5 types of orchestrations where each orchestration gets its own GRPO group and a variant D-MORL which adds discounted-UCB routing on top of MORL and selects top-3 orchestrations per query. They evaluate the method on Search QA, ALFWorld and WebShop with Qwen 2.5 models of various sizes and shows advantage over existing single-orchestration RL baselines.

Critical strengths
  1. It is a simple and cheap method that addresses an important problem, whether we should fix the orchestration while doing agentic RL.
  2. Empirical results show that RL with multiple orchestrations lead to better results on standard benchmarks on QA, embodies planning and web navigation.
  3. The paper offers interesting insights of the failure modes in mono-orchestration RL training. e.g. the tool atrophy observed in REACT-only training in Fig.3.
Critical weaknesses
  1. Main result tables lack paired significant tests while comparing different methods.
  2. To justify the advantage of routing a comparison to a random choice of 3 out of 5 orchestration baseline is needed.
Questions for authors
  1. Can you provide statistical test results for the main tables? Or is it the same as table 6?
  2. Can you shed light on when MORL would benefit from D-UCB routing based on the experiment results?
  3. Which orchestrations make up the ALFWorld and WebShop pools?
2 comments
AuthorsOpenReview ↗

Response to Reviewer oWBD (1/2)

Thank you for recognizing the simple, inexpensive approach to the important question of whether orchestration should remain fixed during agentic RL, the multi-orchestration gains in QA, embodied planning, and web navigation, and the analysis of ReAct tool-use atrophy. We address statistical evidence, random routing and its applicability, and the orchestration pools below.

W1 & Q1: Statistical evidence for the main results

Thank you for your comment. Table 6 reports repeated evaluations corresponding to the main Search QA results. Appendix E.2 evaluates the trained checkpoints for each method in three independent runs, showing that the main-table gains remain stable and statistically significant.

W2: Random 3-of-5

Thank you for requesting this control. We add random 3-of-5 to Table 8 in Appendix E.6.1. For each training query, it samples three orchestrations uniformly without replacement from ReAct, Self-Ask, Step-Back, Verify, and Breadth, with N=4 trajectories per selected orchestration. The comparison uses Qwen2.5-3B-Instruct on NQ/HotpotQA and matches the total rollout budget B in Table 1. Random 3-of-5, standard UCB, sliding-window UCB, and D-UCB all use K=3 and N=4, with the other training and evaluation settings following Table 1.

Response Table 5. Routing-strategy comparison on Search QA (Qwen2.5-3B, EM %). Settings and total rollout budget follow main-text Table 1, with K=3 and N=4. Random 3-of-5 samples uniformly without replacement; sliding-window UCB uses a 50-step window.

SchedulerNQ*TriviaQA†PopQA†HotpotQA*2Wiki†MuSiQue†Bamboogle†Avg
Random 3-of-548.063.251.044.639.820.838.643.7
UCB48.863.250.845.140.021.738.043.9
Sliding-window UCB49.064.051.045.639.822.938.844.4
D-UCB49.363.851.245.640.423.539.044.7

The four routing strategies attain seven-benchmark mean EMs of 43.7, 43.9, 44.4, and 44.7, respectively. D-UCB exceeds random selection by 1.0 point and sliding-window UCB by 0.3 points. Alongside this modest end-to-end lead, its rollout shares have properties suited to a non-stationary training curriculum. Policy–orchestration compatibility changes during training. Random selection does not use these changes; full-history UCB may retain too much stale feedback, while sliding-window UCB removes old observations at window boundaries. D-UCB smoothly downweights old feedback through exponential discounting and maintains exploration.

To better understand the effects of different routing strategies, we add rollout share evolution curves to the D-UCB ablations in Appendix E.6.1 (Figure 8). D-UCB combines responsiveness to capability changes, continued exploration of temporarily weaker orchestrations, and smoother rollout share updates. At 3B, D-UCB initially favors ReAct and broadens coverage as the other reward curves catch up; at 1.5B, it progressively reduces the persistently less effective ReAct's rollout share while retaining exploration. Relative to random selection (Figure 8(c, i) versus 8(f, l)), it responds to training stage and orchestration effectiveness; relative to full-history UCB (Figure 8(d, j) versus 8(f, l)), it discounts stale feedback more promptly; relative to sliding-window UCB (Figure 8(e, k) versus 8(f, l)), its later rollout shares fluctuate less. These rollout share dynamics further explain the suitability of exponential forgetting for this training curriculum.

AuthorsOpenReview ↗

Response to Reviewer oWBD (2/2)

Q2: When D-UCB is useful

D-UCB is particularly useful when policy–orchestration compatibility changes substantially during training and some orchestrations temporarily degrade or become less productive. Sections 5.2 and 5.4 and Appendix E.5 analyze two representative cases: ReAct degradation on small models, and more complex orchestrations catching up with an initially stronger ReAct as policy capabilities improve.

The first case is orchestration degradation. The early-winner phenomenon in Section 5.2 (original manuscript lines 400–405 and Figure 3) shows ReAct starting with the highest reward, but its search use atrophies within about 30 steps under single-orchestration training and other orchestrations subsequently overtake it. Initial rankings should therefore not fix allocation. Section 5.4, lines 471–473, and Appendix E.5 explain how D-UCB downweights degradation through declining discounted reward estimates while preserving recovery through exploration. In the newly added 1.5B rollout share comparison, Figure 8l reduces the later ReAct share to about 12% while retaining exploration; full-history UCB instead produces late zero-share windows in Figure 8j.

The second case is changing orchestration value as capabilities improve. Section 5.4, lines 462–465, and Figure 5 show different growth rates and reward ceilings: ReAct is more effective initially, but complex orchestrations become more useful as the shared policy improves, warranting a shift of budget toward them. In the newly added 3B comparison in Figure 8f, the ReAct share falls from about 29% in the first 50 steps to about 20% in steps 300–400 as coverage of the other orchestrations broadens. Full-history UCB retains its early preference until about step 200 (Figure 8d). D-UCB discounts both rewards and effective visit counts, reducing stale influence while restoring exploration opportunities for orchestrations not visited recently.

This adaptive filtering also reduces dependence on orchestration generation: every pool member need not be highly valuable at every training stage. Less productive orchestrations are downweighted while effective ones continue supplying training signals, improving tolerance to uneven pool quality and temporary degradation while preserving opportunities for weaker orchestrations to become useful again.

Q3: The ALFWorld and WebShop orchestration pools

The tables below follow the Name–Reasoning Structure–Inductive Bias / Primary Use Case format of the Search QA orchestration table in Appendix B.1 and list the ALFWorld and WebShop pools separately. The implementations will be included in the code release after the paper is accepted.

Response Table 6. ALFWorld five-orchestration overview

NameReasoning StructureInductive Bias / Primary Use Case
ReAct (react)Observe → reason → act (loop)Choose admissible actions from the current observation
Plan-then-Execute (plan)Form a complete plan → execute step by stepOrganize long-horizon actions around a global plan
Reflection (reflection)Review feedback → revise approach → actAdapt to environment feedback and avoid repeated mistakes
Subgoal Tracking (subgoal)Decompose goal → track subgoals → actAdvance subgoals and monitor task progress
Systematic Exploration (exploration)Search likely locations systematically → locate target → executeOrganize search when the target location is unknown

Response Table 7. WebShop five-orchestration overview

NameReasoning StructureInductive Bias / Primary Use Case
ReAct (react)Observe → reason → act (loop)Choose search, inspection, or purchase from the current page
Criteria-first (criteria)Specify attributes/constraints → search → inspectOrganize decisions around the requested product criteria
Compare-before-buy (compare)Retrieve candidates → compare products → buyCompare how candidates satisfy the target criteria
Query Refinement (refine)Search → inspect feedback → refine queryIteratively improve search terms from the results
Plan-then-Shop (plan)Plan shopping → search → inspect → selectFollow a planned shopping process
Reviewer_HjHUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of using a fixed orchestration during reinforcement learning for LLM agents and proposes Multi-Orchestration Reinforcement Learning (MORL). The method jointly trains a shared policy under multiple reasoning scaffolds, computes group-wise GRPO advantages separately for each orchestration, and aggregates their gradients for policy optimization. The authors further introduce Discounted-UCB (D-UCB) to adaptively allocate rollout budgets based on the reward dynamics of different orchestrations.

Experiments on Search QA, ALFWorld, and WebShop show that MORL consistently outperforms single-orchestration training across different model sizes and tasks, while improving generalization to unseen orchestrations. D-MORL achieves additional improvements under matched rollout budgets.

Critical strengths
  • The paper investigates how orchestration choices affect agent learning and generalization during RL training. The observed differences in training dynamics across orchestrations, including tool-use degradation under single-orchestration training, provide meaningful empirical motivation for training with diverse reasoning scaffolds.
  • The experiments cover multiple task domains and model scales, with rollout-budget-matched comparisons, cross-orchestration generalization tests, and several ablation studies. The results consistently support the effectiveness of multi-orchestration training, particularly for OOD generalization.
Critical weaknesses
  • The core of MORL consists of sampling trajectories under multiple existing reasoning scaffolds, applying standard group-wise GRPO advantage normalization, and jointly optimizing a shared policy. D-MORL further applies the established Discounted-UCB algorithm for rollout budget allocation. The methodological contribution largely amounts to applying existing training diversification and adaptive scheduling techniques at the orchestration level, without introducing a substantially new algorithmic mechanism.
  • The paper presents the self-evolving curriculum as a central contribution, yet D-MORL does not outperform uniform MORL under a matched training-data budget. Its additional gains under matched rollout budgets partly result from processing more training queries. Moreover, comparisons against simpler alternatives are missing, e.g., random Top-K or fixed Top-K scheduling. Therefore, it remains unclear whether reward-driven adaptive scheduling provides meaningful benefits beyond simpler budget allocation strategies. In particular, an orchestration's current reward does not necessarily reflect its contribution to subsequent policy improvement.
Questions for authors
  • Under matched rollout and training-data budgets, does D-MORL outperform simpler alternatives such as random Top-K or fixed Top-K scheduling? This comparison would help clarify the independent contribution of adaptive scheduling.
  • D-UCB uses the average rollout reward of each orchestration as its scheduling signal. Have the authors investigated how well this metric correlates with the orchestration's contribution to subsequent policy improvement?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HjHU (1/3)

Thank you for recognizing the empirical motivation from orchestration-dependent training dynamics and tool-use degradation, and the value of evaluation across task domains and model scales, budget-matched comparisons, and cross-orchestration generalization. We clarify the contribution and jointly address budgets, simpler routing controls, and the scheduling signal.

W1: Methodological contribution and research question

Thank you for discussing the contribution. The problem we identify is that agentic RL typically selects one orchestration before training and keeps it fixed while the policy's capabilities continue to change. Yet an orchestration's value depends on the policy's current capabilities and the task structure: strong initial performance does not establish its long-term training value. Section 5.2 illustrates this early-winner trap. ReAct starts with the highest reward on small models but can suffer tool-use atrophy under single-orchestration training, while initially weaker, more complex orchestrations overtake it as capabilities improve.

Existing offline workflow search primarily optimizes orchestrations against a frozen policy. This leaves a training-time gap: how should experience from different orchestrations be organized, and how should the curriculum adapt when the policy itself is changing?

We address this gap through the self-evolving orchestration curriculum, instantiated as MORL / D-MORL. A shared policy jointly learns under structurally diverse orchestrations. Each orchestration forms an independent GRPO group with within-group advantage normalization, allowing complementary success and failure patterns to contribute to the shared update. D-UCB allocates rollout budget using the policy's continually updated feedback, with discounting and exploration tracking changing policy–orchestration compatibility. Policy updates reshape orchestration effectiveness; the curriculum adapts in turn and shapes subsequent learning, creating a two-way adaptation process during training.

We hope the contribution can also be assessed through the research question and the innovation in the curriculum framework and perspective, which are orthogonal to mechanism innovations in policy optimizers such as GRPO. GRPO addresses how relative advantages from given trajectories are used to optimize a policy; our work addresses which orchestrations should generate training experience and how limited rollout compute should be allocated as policy capabilities change. MORL/D-MORL instantiate this training framework with existing optimization and scheduling tools, making orchestration a dimension of the training distribution and online curriculum so that the curriculum and shared policy adapt to each other during training. This is the contribution we aim to emphasize.

AuthorsOpenReview ↗

Response to Reviewer HjHU (2/3)

W2 & Q1: Budget matching and routing controls

There appears to be a potential misunderstanding here. The matched-data comparison in Appendix E.1 and the matched-rollout comparison in Table 1 address questions at two different levels. The matched-data comparison demonstrates D-MORL's computational efficiency in reaching near-full-MORL performance with less compute, while the matched-rollout comparison demonstrates its performance benefit under a fixed total compute budget. First, the matched-data setting fixes training samples and rollout group size. Full MORL generates five groups per sample, while D-MORL selects three, so full MORL uses more rollout compute and obtains more groups of training signal from the same data. D-MORL uses approximately 3/5 of full MORL's rollout compute per training step while reaching 42.7/48.8 mean EM at 3B/7B, close to full MORL's 43.3/49.5 and above the strongest single-orchestration RL results of 37.8/42.0 in Table 5. This demonstrates near-full-MORL performance with less compute and complements Table 1 from the perspective of data use and computational efficiency.

Second, under the same total rollout budget in Table 1, processing more training queries is an efficiency benefit enabled by adaptive allocation. As capabilities change, D-MORL moves budget from degraded or temporarily mismatched orchestrations toward those better suited to the current policy. Reducing less productive sampling frees rollouts for additional queries. The resulting 3B/7B mean EMs are 44.7/50.4, compared with uniform MORL's 43.3/49.5. The curriculum therefore adjusts both which orchestrations supply current training experience and how many queries a fixed compute budget can cover. Together, the matched-training-data and matched-rollout comparisons demonstrate two complementary advantages of D-MORL: computational efficiency in approaching full MORL's performance with fewer rollouts, and improved performance when the total rollout budget is fixed.

To further compare routing rules, we add random 3-of-5, standard UCB, and sliding-window UCB alongside D-UCB in Appendix E.6.1 (Table 8). The generalization results for Fix-3 were already reported in Section 5.3 (Figure 4) of the original submission. The four routing strategies use Qwen2.5-3B-Instruct, NQ/HotpotQA data, the same five-orchestration pool and total rollout budget as Table 1, with K=3 and N=4.

Response Table 4. Routing-strategy comparison on Search QA (Qwen2.5-3B, EM %). Settings and total rollout budget follow main-text Table 1, with K=3 and N=4. Random 3-of-5 samples uniformly without replacement; sliding-window UCB uses a 50-step window.

SchedulerNQ*TriviaQA†PopQA†HotpotQA*2Wiki†MuSiQue†Bamboogle†Avg
Random 3-of-548.063.251.044.639.820.838.643.7
UCB48.863.250.845.140.021.738.043.9
Sliding-window UCB49.064.051.045.639.822.938.844.4
D-UCB49.363.851.245.640.423.539.044.7

The four three-orchestration schedulers use the same pool, K=3, N=4, and total rollout budget, so query exposure is also matched among them.

Their seven-benchmark mean EMs are 43.7, 43.9, 44.4, and 44.7, respectively. D-UCB exceeds random selection, standard UCB, and sliding-window UCB by 1.0, 0.8, and 0.3 percentage points. Alongside this modest end-to-end lead, its rollout shares combine responsiveness, continued exploration, and smooth adjustment. The newly added Figure 8 shows these allocation changes at 3B and 1.5B. At 3B, D-UCB initially favors ReAct, then broadens coverage as the other orchestrations catch up: ReAct's share falls from about 29% in the first 50 steps to about 20% in steps 300–400 (Figure 8f). At 1.5B, it reduces the degraded ReAct's later share to about 12% while retaining exploration (Figure 8l). Unlike random selection (Figure 8(c, i) versus 8(f, l)), this allocation responds to training stage and orchestration effectiveness.

Full-history UCB retains its early 3B ReAct preference until about step 200 and produces late zero-share windows for 1.5B ReAct (Figure 8(d, j) versus 8(f, l)), reflecting slower adaptation to stale feedback. Sliding-window UCB emphasizes recent feedback but exhibits larger later fluctuations (Figure 8(e, k) versus 8(f, l)). D-UCB gradually discounts stale feedback while maintaining exploration, enabling more timely redistribution without abrupt changes at window boundaries. These allocation patterns help explain why exponential forgetting suits a curriculum in which policy–orchestration compatibility changes during training.

AuthorsOpenReview ↗

Response to Reviewer HjHU (3/3)

W2 & Q2: Scheduling signal and subsequent learning gains

Regarding the scheduling signal and subsequent learning gains, we agree that an orchestration's current reward does not necessarily reflect its contribution to subsequent policy improvement. This is also a reason for our use of D-UCB. We also wish to clarify that “the average rollout reward” in Q2 describes only one component of the D-UCB selection score: the score additionally includes an exploration term based on discounted visit counts, so interpreting the scheduling signal solely as current mean reward is incomplete. Section 5.2 of the original manuscript (lines 400–405) already identifies early-reward rankings as unreliable proxies for long-term value: ReAct starts highest but suffers tool-use atrophy under single-orchestration training, while initially weaker orchestrations overtake it. Figure 3 additionally shows ReAct-only search calls falling to near zero within about 30 steps.

The D-UCB design considers this risk: Section 3.5 (lines 250–265) combines discounted reward estimates with an exploration term based on discounted visit counts. Exponential discounting gradually reduces stale feedback, allowing allocation weights to respond promptly to changes in policy–orchestration compatibility. Historical information fades gradually rather than being abruptly discarded at a window boundary, keeping the mean estimate relatively stable and supporting smoother weight adjustments. Section 5.4 (lines 471–473) and Appendix E.5 explicitly preserve recovery opportunities when downweighting a degraded orchestration. Newly added Figure 8 illustrates the advantage over full-history UCB: UCB retains the early 3B ReAct preference longer (8d versus 8f) and produces late zero-share windows for 1.5B ReAct (8j versus 8l); D-UCB reallocates more promptly while retaining recent exploration. The scheduling signal thus combines a discounted estimate of current compatibility with exploration, rather than a simple historical reward average. For the subsequent contribution raised in Q2, exploration gives initially weaker orchestrations opportunities to be sampled again at later stages. If an orchestration becomes better suited to the model's current capabilities as the shared policy improves, newly observed rewards can raise its recent compatibility estimate and lead D-UCB to increase its subsequent rollout allocation. For example, in the 3B case of Figure 8, other orchestrations initially trail ReAct but later catch up in reward, and D-UCB correspondingly broadens their coverage. This example illustrates how D-UCB's design links sampling weights to subsequent policy improvement: as the shared policy improves, continued exploration reveals changes in orchestration effectiveness, and updated reward estimates allow rollout allocation to adapt accordingly. Continued exploration, reassessment, and feedback updates keep orchestration selection aligned with evolving capabilities without requiring an accurate early prediction of each orchestration's long-term contribution to subsequent policy improvement.

Dear Reviewer,

Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.

Thank you for your time and effort!

Reviewer_AcMUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a research gap that current LLM agent RL only uses a single, fixed orchestration. It proposes a Multi-Orchestration Reinforcement Learning framework to do RL training on structurally diverse orchestrations. It also uses Discounted-UCB routing to reweight roll outs efforts on different orchestration. The paper conducts experiments on Search QA benchmarks and show extra performance gain of this MORL training over traditional single orchestration training method.

Critical strengths
  1. The paper finds the drawbacks of current single orchestration RL training method and Innovatively propose an MORL training method that training on diverse orchestrations to solve the drawbacks of current single orchestration RL training method.
  2. The paper finds that dynamically allocates roll out efforts to different orchestrations can have extra gain of performance and design a D-UCB routing to implement it.
Critical weaknesses
  1. The detailed implementation of how an orchestration is represented during the training process is missing from the paper. It only states that it is encoded as a triple of SystemPrompt, FormatSpec, InteractionPattern. What they really looks like is not provided. The function of InteractionPattern is also not well represented in the training process as only format is counted in the reward but not InteractionPattern according to the reward formula (3) in the paper

  2. There are still some experiments need to be done to show the soundness of the advantage gain. First, the experiments are only done on simple agent tasks such as search QA with very limited tool access. The more advanced tasks such as coding which requires more tool accesses and much longer roll out trajectories. I think an experiment on a more complicated task like coding is necessary to show the advantage of the method. Moreover, different orchestrations may require different budget of tool calling and interaction steps. The experiment in the paper sets a fixed budget for all the orchestrations, which may limited the performance of some orchestrations. I want an ablation study to show how the budget of tool calling and interaction steps affect the result.

Questions for authors
  1. How different orchestrations are represented in the format of SystemPrompt, FormatSpec, InteractionPattern? Can you give a detailed example? What have been done other than changing the system prompt?
6 comments
AuthorsOpenReview ↗

Response to Reviewer AcMU (1/2)

Thank you for recognizing the limitations of single fixed-orchestration training, the value of jointly training under diverse orchestrations, and the additional gains from dynamically allocating rollouts. We address implementation details and task/budget evaluation below.

W1 & Q1: The orchestration triple, implementation, and rewards

We apologize for the confusion. Appendix B.1 (Orchestration Pool) of the original submission already describes the implementation logic and reasoning structures of the Search QA orchestrations. For example, ReAct follows a thought–action–observation loop, Self-Ask decomposes the question before retrieval, and Verify checks evidence consistency after retrieval. In our implementation, InteractionPattern is an abstract logical process realized through loops/workflows and stage-specific prompts/context, using information such as retrieved evidence and verification feedback to guide subsequent reasoning and actions. The complete orchestration prompts are too long to include in full in the appendix; the source code will be released upon acceptance.

The SystemPrompt, FormatSpec, InteractionPattern triple is our logical abstraction informed by prior work on agent orchestration and multi-module reinforcement learning, such as AOrchestra and AgentFlow. It summarizes how a workflow or loop organizes problem solving, rather than prescribing a particular encoding. For example, a Breadth multi-query loop can be organized through code: k sub-agents propose a total of n independent, non-overlapping queries from different subquestions or retrieval facets. The main agent collects and deduplicates them, calls search sequentially, and combines the results to extract and connect multi-hop facts. If the evidence supports an answer, it generates one; otherwise, unresolved facts become context for the next round of queries targeting those gaps. Role instructions correspond to SystemPrompt; the message structures for queries, search results, and unresolved facts to FormatSpec; and query generation, retrieval aggregation, and the transfer of unresolved facts to the next round to InteractionPattern. An agent workflow/loop/multi-agent task requires an organized execution process: SystemPrompt specifies roles, goals, and tool and task constraints; FormatSpec specifies the structure of inter-stage messages, tool calls, and returned information; InteractionPattern describes control flow and information flow: how stages are ordered or repeated, how their outputs are passed on, and how that information influences subsequent decisions.

The two reward signals in Eq. (3) serve different purposes. The small format reward R_format helps the model align with the interface format more quickly, so that queries, actions, and messages can be parsed by the environment or subsequent stages. This is consistent with the accuracy/format reward distinction in DeepSeek-R1 [1]. The effect of InteractionPattern on task solving is already reflected in R_correct: within a retrieve–verify–continue-search-or-answer loop, the model's query choices, evidence use, and decisions to search again or answer affect the final outcome evaluated by R_correct. GRPO compares rewards within each orchestration's trajectory group and encourages the model-generated decisions in relatively higher-reward trajectories. InteractionPattern thus receives correctness feedback through the solution trajectories it organizes; a separate reward for the prescribed workflow is unnecessary.

AuthorsOpenReview ↗

Response to Reviewer AcMU (2/2)

W2: Task complexity and interaction budgets

Thank you for the suggestion. Our evaluation extends beyond Search QA. In the embodied household environment in Table 2, picking up and placing objects alone yield over 1,000 type-level tool–object–receptacle combinations. Agents must explore, try actions, and adapt to feedback through interactions that can span tens of steps. The shopping environment includes approximately 1.18 million real products and 842,849 distinct options, requiring retrieval, comparison, and matching across pages. These settings test long-horizon interaction at a scale suitable for multi-turn agent RL training of our models; our choice of these task domains is therefore consistent with GiGPO [2], SPA-RL [3], and SkillRL (2026) [4].

We study two connected questions: how training a shared policy under structurally diverse orchestrations supports transfer across tasks and orchestrations, and how a limited rollout-compute budget should be allocated as policy capabilities and orchestration effectiveness change during training, so that the curriculum remains adapted to the current policy. The three evaluated domains cover multi-hop retrieval, embodied planning with action preconditions, and interactive decisions over large candidate spaces. These long-horizon interaction tasks let us observe the training and transfer benefits of orchestration diversity and dynamic sampling across settings. Coding is a valuable extension, but existing coding benchmarks typically emphasize functional correctness and passing tests, with less focus on orchestration structure and its role in training. Our current evaluation examines the core research questions through training results across task domains, leave-one-orchestration-out transfer, and dynamic-sampling analysis. We therefore leave orchestration design and evaluation for coding to future work.

To further analyze how interaction-step budgets affect different orchestrations, we compare three single-orchestration RL checkpoints in each of Search QA and ALFWorld. The shared headers O₁/O₂/O₃ denote ReAct/Self-Ask/Verify for Search QA and ReAct/Reflexion/Plan-and-Exec for ALFWorld. Search QA uses Qwen2.5-3B, search-call caps of 2/4/8, retrieval top-3, and equal-weight mean EM over seven benchmarks; ALFWorld uses Qwen2.5-1.5B, environment-action caps of 10/30/50, and success rate (SR).

Response Table 3. Performance of three orchestrations under different action caps (%).

TaskCapO₁O₂O₃
Search QA220.325.628.1
Search QA4 (default)37.238.139.9
Search QA837.238.639.7
ALFWorld1067.268.869.3
ALFWorld30 (default)77.481.082.8
ALFWorld5077.680.283.0

All three orchestrations lose performance under tight caps: 2 rather than 4 searches lowers Search QA EM by 11.8–16.9 percentage points, and 10 rather than 30 steps lowers ALFWorld SR by 10.2–13.5 points. Changes for 4→8 searches and 30→50 actions are small, with no persistent need for a larger cap in one orchestration. Sufficient retrieval and environment interaction are therefore prerequisites for benefiting from orchestration structure: tight caps can truncate workflows before enough evidence is collected or the required action sequence is completed, and decomposition, reflection, or planning cannot readily replace missing external feedback. During training, terminal failure at the cap implicitly favors completion within the available steps; relaxing the evaluation cap can instead prolong trajectories through repeated retrieval or unproductive retries. Extra steps are useful when the policy produces new, effective queries or actions.

References

[1] Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., …, Zhang, Z. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081), 633–638. https://doi.org/10.1038/s41586-025-09422-z

[2] Feng, L., Xue, Z., Liu, T., & An, B. (2025). Group-in-group policy optimization for LLM agent training [Preprint]. arXiv. https://arxiv.org/abs/2505.10978

[3] Wang, H., Leong, C. T., Wang, J., Wang, J., & Li, W. (2025). SPA-RL: Reinforcing LLM agents via stepwise progress attribution [Preprint]. arXiv. https://arxiv.org/abs/2505.20732

[4] Xia, P., Chen, J., Wang, H., Liu, J., Zeng, K., Wang, Y., Han, S., Zhou, Y., Zhao, X., Chen, H., Zheng, Z., Xie, C., & Yao, H. (2026). SkillRL: Evolving agents via recursive skill-augmented reinforcement learning [Preprint]. arXiv. https://arxiv.org/abs/2602.08234

Thank you authors for providing this extra experiment to show the reson for selection of current default action caps. However, my question is more about setting different action budget for different orchestrations during one trainning process. Your current experiment results also show different performance trends for different orchestrations corresponding to the action budget (some keep goes up as action budget goes up, some goes up and down instead), which makes me thinking that a uniformed fixed action budget for all the orchestraions may not be appropriate. Can you add an addtional experiments to show if result will change by setting different action budget for different orchestrations during one trainning process or explain why you choose current uniformed fixed action budget for all the orchestraions.

Reviewer_RrhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Orchestration Reinforcement Learning (MORL), a framework designed to improve the generalization of LLM-based agents by training them with multiple orchestration strategies instead of relying on a single fixed orchestration paradigm.

Critical strengths

Relevant and meaningful research problem: The paper identifies an important limitation in existing RL-based agent training as most approaches rely on a fixed orchestration strategy throughout training. Investigating whether orchestration diversity can improve agent generalization is a meaningful research direction.

Reasonable motivation for adaptive orchestration selection:The observation that different orchestration strategies may have different levels of effectiveness at different training stages is intuitive. Treating orchestration selection as an adaptive curriculum provides a reasonable motivation for the proposed framework.

Evaluation across multiple benchmarks:The reported experiments cover seven Search QA benchmarks, including OOD multi-hop reasoning tasks. The reported improvements on Bamboogle and MuSiQue suggest that the proposed approach may offer benefits beyond the training distribution.

Critical weaknesses

Limited comparison with other routing algorithms: Table 7 studies the effects of Top-K, the discount factor, and the exploration coefficient, but does not compare Discounted-UCB with other routing algorithms, such as standard UCB or other non-stationary bandit methods. Although D-MORL outperforms uniform MORL, it is still unclear whether the improvement comes from Discounted-UCB specifically or whether other adaptive routing methods could achieve similar results.

Unclear contribution of each orchestration: Table 9 compares single-orchestration training with MORL, but does not show how much each orchestration contributes to joint training. It is unclear whether all five orchestrations are useful or whether most of the gains come from only a few of them. A leave-one-orchestration-out ablation would help clarify the contribution of each orchestration to the final performance.

Limited evaluation of generalization across orchestrations: Figure 4 and Appendix E.10 evaluate MORL trained with three orchestrations on two other orchestrations that were not used during training. However, only one combination of training orchestrations is tested. It would be useful to exclude each orchestration from training in turn and then evaluate the resulting model using that orchestration.

Questions for authors

Please address the concerns in weakness section.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Rrhn (1/3)

W1: Comparison with other routing strategies

Thank you for the suggestion. Without claiming a unique advantage for D-UCB, our insight is that policy–orchestration compatibility drifts during RL: dynamic rollout allocation should respond promptly to this drift while adjusting smoothly to avoid frequent curriculum oscillations.

In this setting, standard UCB's full-history averages may lag behind capability changes, while sliding-window UCB emphasizes recent feedback but removes old observations at window boundaries. D-UCB smoothly downweights old feedback through exponential discounting while retaining exploration pressure. It is thus a natural way to balance historical information, responsiveness, and weight stability.

We add the routing-strategy comparison in revised Appendix E.6.1. The table below compares end-to-end Search QA performance using Qwen2.5-3B-Instruct, NQ/HotpotQA training data, and the same five-orchestration pool. Training, evaluation, and the total rollout budget follow main-text Table 1.

Response Table 1. Routing-strategy comparison on Search QA (Qwen2.5-3B, EM %). Settings and total rollout budget follow the main results table, with K=3 and N=4. Random 3-of-5 samples uniformly without replacement; sliding-window UCB uses a 50-step window.

SchedulerNQ*TriviaQA†PopQA†HotpotQA*2Wiki†MuSiQue†Bamboogle†Avg
Random 3-of-548.063.251.044.639.820.838.643.7
UCB48.863.250.845.140.021.738.043.9
Sliding-window UCB49.064.051.045.639.822.938.844.4
D-UCB49.363.851.245.640.423.539.044.7

Sliding-window UCB and D-UCB modestly outperform random 3-of-5 and standard UCB on average. This pattern supports the importance of tracking policy–orchestration compatibility promptly: random selection cannot distinguish stage-dependent effectiveness, while full-history estimates can retain early advantages and delay reallocation. Recent windows and exponential forgetting reduce the influence of stale feedback. D-UCB has a further modest lead in mean performance, making exponential forgetting a suitable choice for this curriculum.

To better understand the effects of different routing strategies, we add rollout share evolution curves to the D-UCB ablations in Appendix E.6.1 (Figure 8). Figure 8 further visualizes the difference between the two recent-feedback strategies: D-UCB adjusts coverage as capabilities change, retains exploration of temporarily weaker members, and changes rollout shares more smoothly than sliding-window UCB. In this comparison, D-UCB combines responsiveness to capability changes, continued exploration of temporarily weaker workflows, and smoother rollout share updates. At 3B, D-UCB initially favors ReAct and broadens coverage as the other reward curves catch up; at 1.5B, it progressively reduces the persistently less effective ReAct's rollout share while retaining exploration. Relative to random selection (Figure 8(c, i) versus 8(f, l)), it responds to training stage and orchestration effectiveness; relative to full-history UCB (Figure 8(d, j) versus 8(f, l)), it discounts stale feedback more promptly; relative to sliding-window UCB (Figure 8(e, k) versus 8(f, l)), its later rollout shares fluctuate less. These rollout share patterns explain the suitability of exponential forgetting for this training curriculum.

AuthorsOpenReview ↗

Response to Reviewer Rrhn (2/3)

W2: Joint-training value and the roles of individual orchestrations

Thank you for raising this question. Beyond examining each orchestration's individual contribution to joint training, the more important purpose of original Table 9 (Table 10 in the revision) is to test the value of multi-orchestration training itself: whether training a shared policy under different orchestrations on the same tasks enables knowledge and capability transfer across orchestrations and exceeds the performance attained by single-orchestration RL in our experimental setting. The single-versus-joint-training ablation is designed to address this question, rather than merely assign a fixed contribution percentage to each of the five orchestrations.

Our contribution is to treat orchestration as a dimension of the RL training distribution and curriculum, and to introduce a framework for jointly optimizing a shared policy across orchestrations. The contribution is not the provision of these five particular orchestrations, nor does the framework assume that every pool member always provides a sufficiently substantial gain. In our view, requiring the orchestration-generation stage to guarantee such gains from every member throughout RL would impose a stronger assumption, substantially increase the demands on orchestration generation, and restrict the framework's generality for automatically generated pools.

This also motivates D-UCB: feedback that changes during training allocates more trial budget to orchestrations that better match the policy's current capabilities, supporting further capability improvement while reducing persistent budget consumption by potentially weaker members and retaining exploration as the policy evolves.

We are also happy to provide additional analysis: revised Appendix E.10.2 (Table 12) reports the leave-one-orchestration-out experiments, detailed in the next response. These results help quantify the positive contributions of the selected orchestrations in this implementation. We respectfully suggest that focusing on the contributions of individual pool members does not fully address the more fundamental question of whether joint training enables cross-orchestration capability transfer.

However, because an orchestration must match the policy's current capabilities, its role can vary considerably across tasks, model scales, and training stages, making a universally applicable set of fixed contribution percentages difficult to establish. In fact, our analysis in Section 5.4, Figure 5, and Appendix E.5 already discusses both regimes raised by the reviewer. For Qwen2.5-3B, the five orchestrations attain broadly similar reward levels, and their cumulative selection counts grow relatively evenly, with modest differences across stages. This illustrates a regime in which multiple orchestrations supply useful training signals together. For Qwen2.5-1.5B, ReAct exhibits tool-use degradation: its reward remains substantially below those of the other orchestrations, and its cumulative selection count grows much more slowly later in training. Meanwhile, rewards and selection counts continue to grow for the remaining orchestrations, which supply most of the effective training signal in the later stages. The analysis of small-model ReAct tool atrophy in Section 5.2 and Figure 3 is consistent with this behavior.

Thus, whether all five orchestrations continue to supply useful signals or the main later-stage gains concentrate in a subset, D-UCB can adapt allocation to current policy–orchestration compatibility. It maintains relatively balanced coverage in the former regime and directs more trials toward orchestrations that remain productive in the latter, while preserving exploration of temporarily weaker members. This allows the framework to accommodate changing effectiveness within the pool and continue benefiting from cross-orchestration knowledge and capability transfer.

AuthorsOpenReview ↗

Response to Reviewer Rrhn (3/3)

W3: Five-fold leave-one-orchestration-out generalization

Thank you for the suggestion. The original Fix-3 experiment in Figure 4 / Appendix E.10 tests one training-orchestration combination. Revised Appendix E.10.2 adds five-fold leave-one-orchestration-out results on Search QA, excluding each member in turn and evaluating with that held-out member.

Starting from the same Qwen2.5-3B checkpoint, we train on NQ/HotpotQA while excluding each of ReAct, Self-Ask, Step-Back, Verify, and Breadth in turn. Each fold uses the other four orchestrations with 20 training trajectories per query. Evaluation reports equal-weight mean EM over seven Search QA benchmarks, separating the ID mean of seen orchestrations from the held-out orchestration’s OOD performance. The search-call cap is 4 with retrieval top-3; OOD gain is the improvement over the corresponding single-orchestration RL baseline in percentage points.

Response Table 2. Five-fold leave-one-orchestration-out generalization on Search QA (Qwen2.5-3B, EM %).

Held-out orchestrationID mean EMOOD EMSingle EMOOD gain (pp)
ReAct44.740.537.2+3.3
Self-Ask44.540.938.1+2.8
Step-Back43.941.439.2+2.2
Verify44.340.939.9+1.0
Breadth43.941.038.9+2.1

The results cover all five single-member exclusions. Joint training outperforms the corresponding single-orchestration RL baseline under every held-out orchestration, indicating that task capabilities learned during training can support workflows absent from the training pool. This transfer may arise from shared abilities such as query decomposition, evidence retrieval and integration, and answer verification: complementary training signals encourage the shared policy to learn task structure, allowing learned capabilities to be reused across workflows. This is consistent with the discussion of implicit regularization through orchestration diversity in Appendix F.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their constructive feedback. We are encouraged that orchestration diversity is viewed as a “meaningful research direction” (Rrhn) and our approach as “simple and cheap” (oWBD). We also appreciate the recognition of adaptive rollout allocation (AcMU), cross-domain benefits (HjHU, oWBD), OOD generalization (Rrhn, HjHU), and the tool-use atrophy analysis (HjHU, oWBD).

The main concerns relate to contribution and routing controls. In response, we have added:

  1. A routing-strategy comparison in Appendix E.6.1, including random 3-of-5, standard UCB, sliding-window UCB, and D-UCB (Rrhn, HjHU, oWBD).
  2. Five-fold leave-one-orchestration-out evaluations in Appendix E.10.2, covering each held-out orchestration (Rrhn).
  3. An interaction-budget comparison in Appendix E.12.1 across search-call and action caps (AcMU).

We have also clarified the orchestration triples and pool implementations (AcMU, oWBD), the two budget-matching protocols (HjHU), and related issues. Detailed responses and results follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.