- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper is trying to address a problem that in agentic RL, training is usually only under one orchestration (orchestration monoculture as called by the authors) which can overfit to one pattern and limits the generalization capability of the models. They propose an algorithm called MORL which pools 5 types of orchestrations where each orchestration gets its own GRPO group and a variant D-MORL which adds discounted-UCB routing on top of MORL and selects top-3 orchestrations per query. They evaluate the method on Search QA, ALFWorld and WebShop with Qwen 2.5 models of various sizes and shows advantage over existing single-orchestration RL baselines.
Critical strengths
- It is a simple and cheap method that addresses an important problem, whether we should fix the orchestration while doing agentic RL.
- Empirical results show that RL with multiple orchestrations lead to better results on standard benchmarks on QA, embodies planning and web navigation.
- The paper offers interesting insights of the failure modes in mono-orchestration RL training. e.g. the tool atrophy observed in REACT-only training in Fig.3.
Critical weaknesses
- Main result tables lack paired significant tests while comparing different methods.
- To justify the advantage of routing a comparison to a random choice of 3 out of 5 orchestration baseline is needed.
Questions for authors
- Can you provide statistical test results for the main tables? Or is it the same as table 6?
- Can you shed light on when MORL would benefit from D-UCB routing based on the experiment results?
- Which orchestrations make up the ALFWorld and WebShop pools?
2 comments
Response to Reviewer oWBD (2/2)
Q2: When D-UCB is useful
D-UCB is particularly useful when policy–orchestration compatibility changes substantially during training and some orchestrations temporarily degrade or become less productive. Sections 5.2 and 5.4 and Appendix E.5 analyze two representative cases: ReAct degradation on small models, and more complex orchestrations catching up with an initially stronger ReAct as policy capabilities improve.
The first case is orchestration degradation. The early-winner phenomenon in Section 5.2 (original manuscript lines 400–405 and Figure 3) shows ReAct starting with the highest reward, but its search use atrophies within about 30 steps under single-orchestration training and other orchestrations subsequently overtake it. Initial rankings should therefore not fix allocation. Section 5.4, lines 471–473, and Appendix E.5 explain how D-UCB downweights degradation through declining discounted reward estimates while preserving recovery through exploration. In the newly added 1.5B rollout share comparison, Figure 8l reduces the later ReAct share to about 12% while retaining exploration; full-history UCB instead produces late zero-share windows in Figure 8j.
The second case is changing orchestration value as capabilities improve. Section 5.4, lines 462–465, and Figure 5 show different growth rates and reward ceilings: ReAct is more effective initially, but complex orchestrations become more useful as the shared policy improves, warranting a shift of budget toward them. In the newly added 3B comparison in Figure 8f, the ReAct share falls from about 29% in the first 50 steps to about 20% in steps 300–400 as coverage of the other orchestrations broadens. Full-history UCB retains its early preference until about step 200 (Figure 8d). D-UCB discounts both rewards and effective visit counts, reducing stale influence while restoring exploration opportunities for orchestrations not visited recently.
This adaptive filtering also reduces dependence on orchestration generation: every pool member need not be highly valuable at every training stage. Less productive orchestrations are downweighted while effective ones continue supplying training signals, improving tolerance to uneven pool quality and temporary degradation while preserving opportunities for weaker orchestrations to become useful again.
Q3: The ALFWorld and WebShop orchestration pools
The tables below follow the Name–Reasoning Structure–Inductive Bias / Primary Use Case format of the Search QA orchestration table in Appendix B.1 and list the ALFWorld and WebShop pools separately. The implementations will be included in the code release after the paper is accepted.
Response Table 6. ALFWorld five-orchestration overview
| Name | Reasoning Structure | Inductive Bias / Primary Use Case |
|---|---|---|
| ReAct (react) | Observe → reason → act (loop) | Choose admissible actions from the current observation |
| Plan-then-Execute (plan) | Form a complete plan → execute step by step | Organize long-horizon actions around a global plan |
| Reflection (reflection) | Review feedback → revise approach → act | Adapt to environment feedback and avoid repeated mistakes |
| Subgoal Tracking (subgoal) | Decompose goal → track subgoals → act | Advance subgoals and monitor task progress |
| Systematic Exploration (exploration) | Search likely locations systematically → locate target → execute | Organize search when the target location is unknown |
Response Table 7. WebShop five-orchestration overview
| Name | Reasoning Structure | Inductive Bias / Primary Use Case |
|---|---|---|
| ReAct (react) | Observe → reason → act (loop) | Choose search, inspection, or purchase from the current page |
| Criteria-first (criteria) | Specify attributes/constraints → search → inspect | Organize decisions around the requested product criteria |
| Compare-before-buy (compare) | Retrieve candidates → compare products → buy | Compare how candidates satisfy the target criteria |
| Query Refinement (refine) | Search → inspect feedback → refine query | Iteratively improve search terms from the results |
| Plan-then-Shop (plan) | Plan shopping → search → inspect → select | Follow a planned shopping process |
Response to Reviewer oWBD (1/2)
Thank you for recognizing the simple, inexpensive approach to the important question of whether orchestration should remain fixed during agentic RL, the multi-orchestration gains in QA, embodied planning, and web navigation, and the analysis of ReAct tool-use atrophy. We address statistical evidence, random routing and its applicability, and the orchestration pools below.
W1 & Q1: Statistical evidence for the main results
Thank you for your comment. Table 6 reports repeated evaluations corresponding to the main Search QA results. Appendix E.2 evaluates the trained checkpoints for each method in three independent runs, showing that the main-table gains remain stable and statistically significant.
W2: Random 3-of-5
Thank you for requesting this control. We add random 3-of-5 to Table 8 in Appendix E.6.1. For each training query, it samples three orchestrations uniformly without replacement from ReAct, Self-Ask, Step-Back, Verify, and Breadth, with N=4 trajectories per selected orchestration. The comparison uses Qwen2.5-3B-Instruct on NQ/HotpotQA and matches the total rollout budget B in Table 1. Random 3-of-5, standard UCB, sliding-window UCB, and D-UCB all use K=3 and N=4, with the other training and evaluation settings following Table 1.
Response Table 5. Routing-strategy comparison on Search QA (Qwen2.5-3B, EM %). Settings and total rollout budget follow main-text Table 1, with K=3 and N=4. Random 3-of-5 samples uniformly without replacement; sliding-window UCB uses a 50-step window.
The four routing strategies attain seven-benchmark mean EMs of 43.7, 43.9, 44.4, and 44.7, respectively. D-UCB exceeds random selection by 1.0 point and sliding-window UCB by 0.3 points. Alongside this modest end-to-end lead, its rollout shares have properties suited to a non-stationary training curriculum. Policy–orchestration compatibility changes during training. Random selection does not use these changes; full-history UCB may retain too much stale feedback, while sliding-window UCB removes old observations at window boundaries. D-UCB smoothly downweights old feedback through exponential discounting and maintains exploration.
To better understand the effects of different routing strategies, we add rollout share evolution curves to the D-UCB ablations in Appendix E.6.1 (Figure 8). D-UCB combines responsiveness to capability changes, continued exploration of temporarily weaker orchestrations, and smoother rollout share updates. At 3B, D-UCB initially favors ReAct and broadens coverage as the other reward curves catch up; at 1.5B, it progressively reduces the persistently less effective ReAct's rollout share while retaining exploration. Relative to random selection (Figure 8(c, i) versus 8(f, l)), it responds to training stage and orchestration effectiveness; relative to full-history UCB (Figure 8(d, j) versus 8(f, l)), it discounts stale feedback more promptly; relative to sliding-window UCB (Figure 8(e, k) versus 8(f, l)), its later rollout shares fluctuate less. These rollout share dynamics further explain the suitability of exponential forgetting for this training curriculum.