- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AgentReroute, a framework that trains a separate recovery controller while keeping the task-solving LLM frozen. The controller learns when to continue or roll back an agent trajectory and provides corrective feedback to guide regeneration. The method is trained using task-level rewards and evaluated on WebShop, Game24, and four question-answering datasets. The results show improvements over several baselines, with additional ablations and transfer experiments.
Critical strengths
- The method provides a practical way to learn rollback decisions and corrective feedback jointly, without requiring step-level recovery labels. Keeping the main generator frozen also makes the approach easy to understand and potentially reusable.
- The experiments cover several different task settings. The ablations provide evidence for the contributions of reinforcement learning and corrective feedback, while the recovery behavior and inference-cost analyses offer useful insights beyond final task performance.
Critical weaknesses
- The training signal is relatively coarse. All Rerouter responses in an episode share the same terminal advantage, including decisions made before a later rollback. This may introduce noisy credit assignment, but the paper does not investigate how sensitive the method is to this design.
- Some evaluation details need clarification. The paper does not report variability across independent training runs, and it is unclear whether the MuSiQue training examples are disjoint from the evaluation examples. In addition, cross-generator transfer is tested only on Game24 with two generators, so the evidence for broader transferability remains limited.
Questions for authors
- Can the authors report variability across independent Rerouter training runs and provide the key GRPO hyperparameters needed to reproduce the results?
- Why is the same terminal advantage assigned to all Rerouter responses in an episode? Have the authors tested alternative credit-assignment strategies?
2 comments
Response to D7nU (2/2): W2-2–W2-3; Q1-2
W2-2 & Q1-2: Key GRPO hyperparameters and strictly disjoint MuSiQue train/eval splits.
Tables C–E provide the complete GRPO hyperparameters and confirm that our 1,000 MuSiQue training questions (from the official train split) and 500 evaluation questions (from the official dev split) are strictly disjoint.
First, following ReSearch [5], our QA Rerouter is trained exclusively on 1,000 questions sampled from the official MuSiQue train set and evaluated on a fixed 500-question subset of the official dev set, with zero overlap between training and evaluation questions (Table C). Furthermore, HotpotQA, 2WikiMultihopQA, and Bamboogle are never seen during training and serve as pure zero-shot out-of-distribution evaluations (as also commended by Reviewer YcER). Therefore, our evaluation protocol enforces strict train-test isolation with zero data leakage, supporting the interpretation that the gains reflect a generalizable trajectory-recovery policy rather than dataset memorization.
Second, Tables D–E report the complete shared parameters, task-specific GRPO hyperparameters, and checkpoint-selection criteria.
| MuSiQue Split Role | Official Data Source | Number of Questions | Overlap with Evaluation Set |
|---|---|---|---|
| Rerouter GRPO Training | Official MuSiQue train split | 1,000 | 0 (Strictly disjoint) |
| Benchmark Evaluation | Official MuSiQue dev split fixed subset | 500 | - |
Table C: Strictly disjoint training and evaluation splits on MuSiQue.
| Shared Parameter | Setting |
|---|---|
| Advantage Estimation | GRPO, , shared across Rerouter decisions within an episode |
| Optimizer & LR Schedule | AdamW (, weight decay 0.01); constant LR ; warmup ratio 0 |
| Clipping & Regularization | PPO clip [0.2, 0.2]; grad clip 1.0; entropy coeff 0; Actor low_var_kl coeff 0.001 (use_kl_in_reward=false) |
| Batching, Sampling & Loss | PPO micro-batch/GPU 1; PPO epochs 1; sampling temp=1.0, top_p=1.0, top_k=-1; seed=42; seq-mean-token-mean |
Table D: Shared GRPO optimization hyperparameters.
| Task-Specific Parameter | Qwen / Game24 | Llama / Game24 | Qwen / WebShop | Llama / WebShop | Qwen / QA |
|---|---|---|---|---|---|
| Training Problems | 250 | 250 | 750 | 750 | 1,000 |
| Tasks per Batch / Mini-batch | 9 / 9 | 9 / 9 | 5 / 5 | 5 / 5 | 8 / 8 |
| GRPO Group Size () | 6 | 6 | 4 | 4 | 4 |
| Rollout Episodes per Batch | 54 | 54 | 20 | 20 | 32 |
| Max Prompt / Response Tokens | 4,096 / 8,000 | 4,096 / 8,000 | 8,192 / 32,768 | 8,192 / 32,768 | 8,192 / 16,384 |
| Max Assistant Turns | 15 | 15 | 31 | 31 | 17 |
| Total Training Steps | 289 | 289 | 450 | 450 | 125 |
| Terminal Outcome Reward | Binary SR | Binary SR | Raw Score | Raw Score | Token F1 |
| Checkpoint Selection Criterion | Top Val SR | Top Val SR | Top Val Exact SR | Top Val Exact SR | Top Val Token F1 |
Table E: Task-specific GRPO configurations and checkpoint selection.
W2-3: Expanded cross-Generator transfer across model scales (32B), WebShop, and QA.
Frozen Rerouters transfer zero-shot to Qwen3-32B, across Generators on WebShop, and from Qwen to Llama on MuSiQue QA with substantial gains.
As shown in Table F, a frozen 4B Rerouter boosts Qwen3-32B on Game24 by +52.4% (26.4% 78.8%), transfers from Llama-3.2-3B to Qwen3-8B on WebShop to reach 36.4% SR (approaching native 37.6%), and transfers from Qwen3-8B to Llama-3.2-3B on MuSiQue QA, raising EM 7× (2.00% 14.00%) and F1 from 10.97% to 18.43%:
| Task & Target Generator | Source of Frozen 4B Rerouter | Unadapted / Base SR (or EM/F1) | Transferred Rerouter SR (or EM/F1) ↑ |
|---|---|---|---|
| Game24 Qwen3-32B (N=500) | Qwen3-8B-trained | 26.4% (Standalone) | 78.8% (+52.4%) |
| WebShop Qwen3-8B (N=500) | Llama-3.2-3B-trained | 29.4% (Untrained 4B) | 36.4% (+7.0%, vs. 37.6% native) |
| WebShop Llama-3.2-3B (N=500) | Qwen3-8B-trained | 12.2% (Untrained 4B) | 12.4% (+0.2%) |
| MuSiQue QA Llama-3.2-3B (N=50) | Qwen3-8B-trained | 2.00% / 10.97% (Standalone) | 14.00% / 18.43% (+12.00% EM) |
Table F: Expanded zero-shot cross-Generator transfer across model scales and tasks.
Refs. [1] DeepSeek-AI: DeepSeek-R1 (2025); [2] Jin: Search-R1 (2025); [3] Feng: ReTool (2025); [4] Yan: Memory-R1 (2025); [5] Chen: ReSearch (2025).
Response to D7nU (1/2): W1; W2-1; Q1-1; Q2
Thank you for recognizing our modular design that keeps the Generator frozen, our outcome-based RL formulation requiring no step-level annotations, and our token/latency efficiency analyses.
W1-1 & Q2-1: Why all Rerouter responses in an episode share the terminal advantage.
Sharing the terminal advantage across an episode is not noisy credit assignment, but an integral part of our label-free trajectory-level formulation that optimizes sequential recovery decisions as a coupled whole.
First, decisions made before a later rollback are causally coupled to final task success rather than independent noise. In AgentReroute (Equations 3 and 8), each Rerouter decision zt is evaluated under its exact generation context hil. Even when an earlier
Continueor partialRollbackis followed by a deeper rollback later, that earlier decision directly determines which intermediate branch is explored, populates the recovery record ρt with pruned negative evidence, and shapes the remaining action budget and recoverable set Kt that enable the subsequent rollback to succeed.Second, episode-shared outcome advantage is an inseparable core mechanism of AgentReroute that eliminates handcrafted step-level recovery labels. Following the standard multi-turn GRPO paradigm in DeepSeek-R1 [1], Search-R1 [2], ReTool [3], and Memory-R1 [4], optimizing the complete decision chain via group-relative terminal outcomes (A^i=Ri−G1∑Rj) reduces the risk of reward hacking from heuristic intermediate penalties (e.g., penalizing every pre-rollback
Continuewould cause the Rerouter to over-trigger rollbacks on every step) while preserving the label-free practicality commended in both your review and Reviewer YcER's.W1-2 & Q2-2: Sensitivity analysis via time-decayed credit assignment.
AgentReroute is highly robust to credit-assignment weighting: introducing time-decayed credit assignment (0.9T−t) maintains WebShop performance and further improves Game24 SR to
81.4%.To directly test sensitivity to intra-episode credit assignment, we weight the policy loss of the t-th Rerouter response in an episode of length T by wt∝0.9T−t (normalized to mean
1within each trajectory so earlier decisions prior to a later rollback receive decayed credit). Across 500 test items per task (Table A), time-decayed weighting performs comparably to the default shared advantage on WebShop (37.0%vs.37.6%) and further improves Game24 SR by+3.6%(reaching81.4%), indicating that AgentReroute is robust to this credit-assignment change while remaining orthogonally compatible with fine-grained temporal weighting:Table A: Comparison between default episode-shared advantage and time-decayed credit assignment.
W2-1 & Q1-1: Variability across independent Rerouter training runs.
AgentReroute remains highly stable across 3 independent GRPO training seeds (
73.2% ± 4.10%SR on 500 Game24 items) and inference seeds (80.0% ± 1.73%), consistently outperforming GA-Rollback (59.6%) on every run.Table B: Variability across independent training runs and inference seeds on Game24.
Refs. [1] DeepSeek-AI: DeepSeek-R1 (2025); [2] Jin: Search-R1 (2025); [3] Feng: ReTool (2025); [4] Yan: Memory-R1 (2025); [5] Chen: ReSearch (2025).