acceptodds
Under review as a conference paper at ICLR 2027

AgentReroute: Training LLMs to Correct Agent Trajectories with Reinforcement Learning

Abstract

Once a large language model agent makes an erroneous decision early in a task, the resulting error can accumulate along its autoregressive trajectory and ultimately lead to task failure. Dominant self-correction methods rely on reflection and feedback to improve outputs, but the model's self-evaluation is often influenced by its original erroneous judgments, making it difficult to correct accumulated errors. To this end, external assistance methods introduce dedicated modules for diagnosis and feedback to correct accumulated errors. However, these methods often rely on heuristic rules or prompt-based judgments, making it difficult to achieve both accurate rollback and effective correction. To address this issue, we propose AgentReroute, an externally assisted reinforcement learning framework that uses task outcomes as reward signals to jointly optimize rollback control and corrective feedback. Specifically, AgentReroute freezes the action generator, Generator, and trains only an independent recovery controller, Rerouter. Given the task instruction, the active trajectory, and the recovery record, Rerouter decides whether to continue execution or select a rollback point and generate corrective feedback, thereby guiding Generator to regenerate subsequent actions. Extensive experiments on four question-answering tasks, WebShop, and the Game24 interactive environment show that AgentReroute significantly improves task success rates while keeping Generator frozen, and that it can be combined with Reflexion for further gains. Further analysis shows that the learned recovery policy reduces repetitive behavior after rollback and maintains stable gains when reused across question-answering datasets and across Generators, demonstrating strong generalization and cross-model transferability. Our code and dataset are anonymously available at https://anonymous.4open.science/r/AgentReroute-DCCA/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Sequential Decision Making, Error Recovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D7nUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AgentReroute, a framework that trains a separate recovery controller while keeping the task-solving LLM frozen. The controller learns when to continue or roll back an agent trajectory and provides corrective feedback to guide regeneration. The method is trained using task-level rewards and evaluated on WebShop, Game24, and four question-answering datasets. The results show improvements over several baselines, with additional ablations and transfer experiments.

Critical strengths
  1. The method provides a practical way to learn rollback decisions and corrective feedback jointly, without requiring step-level recovery labels. Keeping the main generator frozen also makes the approach easy to understand and potentially reusable.
  2. The experiments cover several different task settings. The ablations provide evidence for the contributions of reinforcement learning and corrective feedback, while the recovery behavior and inference-cost analyses offer useful insights beyond final task performance.
Critical weaknesses
  1. The training signal is relatively coarse. All Rerouter responses in an episode share the same terminal advantage, including decisions made before a later rollback. This may introduce noisy credit assignment, but the paper does not investigate how sensitive the method is to this design.
  2. Some evaluation details need clarification. The paper does not report variability across independent training runs, and it is unclear whether the MuSiQue training examples are disjoint from the evaluation examples. In addition, cross-generator transfer is tested only on Game24 with two generators, so the evidence for broader transferability remains limited.
Questions for authors
  1. Can the authors report variability across independent Rerouter training runs and provide the key GRPO hyperparameters needed to reproduce the results?
  2. Why is the same terminal advantage assigned to all Rerouter responses in an episode? Have the authors tested alternative credit-assignment strategies?
2 comments
AuthorsOpenReview ↗

Response to D7nU (1/2): W1; W2-1; Q1-1; Q2

Thank you for recognizing our modular design that keeps the Generator frozen, our outcome-based RL formulation requiring no step-level annotations, and our token/latency efficiency analyses.

W1 & Q2. The training signal is relatively coarse. All Rerouter responses in an episode share the same terminal advantage, including decisions made before a later rollback. Why is the same terminal advantage assigned to all Rerouter responses in an episode? Have the authors tested alternative credit-assignment strategies?

W1-1 & Q2-1: Why all Rerouter responses in an episode share the terminal advantage.

Sharing the terminal advantage across an episode is not noisy credit assignment, but an integral part of our label-free trajectory-level formulation that optimizes sequential recovery decisions as a coupled whole.

First, decisions made before a later rollback are causally coupled to final task success rather than independent noise. In AgentReroute (Equations 3 and 8), each Rerouter decision ztz_t is evaluated under its exact generation context hilh_{il}. Even when an earlier Continue or partial Rollback is followed by a deeper rollback later, that earlier decision directly determines which intermediate branch is explored, populates the recovery record ρt\rho_t with pruned negative evidence, and shapes the remaining action budget and recoverable set Kt\mathcal{K}_t that enable the subsequent rollback to succeed.

Second, episode-shared outcome advantage is an inseparable core mechanism of AgentReroute that eliminates handcrafted step-level recovery labels. Following the standard multi-turn GRPO paradigm in DeepSeek-R1 [1], Search-R1 [2], ReTool [3], and Memory-R1 [4], optimizing the complete decision chain via group-relative terminal outcomes (A^i=Ri−1G∑Rj\hat{A}_i = R_i - \frac{1}{G}\sum R_j) reduces the risk of reward hacking from heuristic intermediate penalties (e.g., penalizing every pre-rollback Continue would cause the Rerouter to over-trigger rollbacks on every step) while preserving the label-free practicality commended in both your review and Reviewer YcER's.

W1-2 & Q2-2: Sensitivity analysis via time-decayed credit assignment.

AgentReroute is highly robust to credit-assignment weighting: introducing time-decayed credit assignment (0.9T−t0.9^{T-t}) maintains WebShop performance and further improves Game24 SR to 81.4%.

To directly test sensitivity to intra-episode credit assignment, we weight the policy loss of the tt-th Rerouter response in an episode of length TT by wt∝0.9T−tw_t \propto 0.9^{T-t} (normalized to mean 1 within each trajectory so earlier decisions prior to a later rollback receive decayed credit). Across 500 test items per task (Table A), time-decayed weighting performs comparably to the default shared advantage on WebShop (37.0% vs. 37.6%) and further improves Game24 SR by +3.6% (reaching 81.4%), indicating that AgentReroute is robust to this credit-assignment change while remaining orthogonally compatible with fine-grained temporal weighting:

Task (Qwen3-8B Generator, N=500)Shared Terminal Advantage (Default)Time-Decayed Credit Assignment (0.9T−t0.9^{T-t})
Game24 (SR ↑)77.8%81.4% (+3.6%)
WebShop (SR ↑)37.6%37.0% (-0.6%)

Table A: Comparison between default episode-shared advantage and time-decayed credit assignment.

W2 & Q1. Some evaluation details need clarification. The paper does not report variability across independent training runs, and it is unclear whether the MuSiQue training examples are disjoint from the evaluation examples. In addition, cross-generator transfer is tested only on Game24 with two generators... Can the authors report variability across independent Rerouter training runs and provide the key GRPO hyperparameters?

W2-1 & Q1-1: Variability across independent Rerouter training runs.

AgentReroute remains highly stable across 3 independent GRPO training seeds (73.2% ± 4.10% SR on 500 Game24 items) and inference seeds (80.0% ± 1.73%), consistently outperforming GA-Rollback (59.6%) on every run.

Evaluation DimensionSeed 42Seed 43Seed 44Mean ± SD
Independent GRPO Training Runs (Test N=500)69.0%77.2%73.4%73.2% ± 4.10%
Fixed Model Across Generator Inference Runs (Test N=100)79.0%79.0%82.0%80.0% ± 1.73%

Table B: Variability across independent training runs and inference seeds on Game24.

Refs. [1] DeepSeek-AI: DeepSeek-R1 (2025); [2] Jin: Search-R1 (2025); [3] Feng: ReTool (2025); [4] Yan: Memory-R1 (2025); [5] Chen: ReSearch (2025).

AuthorsOpenReview ↗

Response to D7nU (2/2): W2-2–W2-3; Q1-2

W2-2 & Q1-2: Key GRPO hyperparameters and strictly disjoint MuSiQue train/eval splits.

Tables C–E provide the complete GRPO hyperparameters and confirm that our 1,000 MuSiQue training questions (from the official train split) and 500 evaluation questions (from the official dev split) are strictly disjoint.

First, following ReSearch [5], our QA Rerouter is trained exclusively on 1,000 questions sampled from the official MuSiQue train set and evaluated on a fixed 500-question subset of the official dev set, with zero overlap between training and evaluation questions (Table C). Furthermore, HotpotQA, 2WikiMultihopQA, and Bamboogle are never seen during training and serve as pure zero-shot out-of-distribution evaluations (as also commended by Reviewer YcER). Therefore, our evaluation protocol enforces strict train-test isolation with zero data leakage, supporting the interpretation that the gains reflect a generalizable trajectory-recovery policy rather than dataset memorization.

Second, Tables D–E report the complete shared parameters, task-specific GRPO hyperparameters, and checkpoint-selection criteria.

MuSiQue Split RoleOfficial Data SourceNumber of QuestionsOverlap with Evaluation Set
Rerouter GRPO TrainingOfficial MuSiQue train split1,0000 (Strictly disjoint)
Benchmark EvaluationOfficial MuSiQue dev split fixed subset500-

Table C: Strictly disjoint training and evaluation splits on MuSiQue.

Shared ParameterSetting
Advantage EstimationGRPO, A=R−mean(group R)A = R - \text{mean}(\text{group } R), shared across Rerouter decisions within an episode
Optimizer & LR ScheduleAdamW (β=(0.9,0.999)\beta=(0.9, 0.999), weight decay 0.01); constant LR 1×10−61\times 10^{-6}; warmup ratio 0
Clipping & RegularizationPPO clip [0.2, 0.2]; grad clip 1.0; entropy coeff 0; Actor low_var_kl coeff 0.001 (use_kl_in_reward=false)
Batching, Sampling & LossPPO micro-batch/GPU 1; PPO epochs 1; sampling temp=1.0, top_p=1.0, top_k=-1; seed=42; seq-mean-token-mean

Table D: Shared GRPO optimization hyperparameters.

Task-Specific ParameterQwen / Game24Llama / Game24Qwen / WebShopLlama / WebShopQwen / QA
Training Problems2502507507501,000
Tasks per Batch / Mini-batch9 / 99 / 95 / 55 / 58 / 8
GRPO Group Size (GG)66444
Rollout Episodes per Batch5454202032
Max Prompt / Response Tokens4,096 / 8,0004,096 / 8,0008,192 / 32,7688,192 / 32,7688,192 / 16,384
Max Assistant Turns1515313117
Total Training Steps289289450450125
Terminal Outcome RewardBinary SRBinary SRRaw Score ∈[0,1]\in [0,1]Raw Score ∈[0,1]\in [0,1]Token F1
Checkpoint Selection CriterionTop Val SRTop Val SRTop Val Exact SRTop Val Exact SRTop Val Token F1

Table E: Task-specific GRPO configurations and checkpoint selection.

W2-3: Expanded cross-Generator transfer across model scales (32B), WebShop, and QA.

Frozen Rerouters transfer zero-shot to Qwen3-32B, across Generators on WebShop, and from Qwen to Llama on MuSiQue QA with substantial gains.

As shown in Table F, a frozen 4B Rerouter boosts Qwen3-32B on Game24 by +52.4% (26.4% →\rightarrow 78.8%), transfers from Llama-3.2-3B to Qwen3-8B on WebShop to reach 36.4% SR (approaching native 37.6%), and transfers from Qwen3-8B to Llama-3.2-3B on MuSiQue QA, raising EM 7× (2.00% →\rightarrow 14.00%) and F1 from 10.97% to 18.43%:

Task & Target GeneratorSource of Frozen 4B RerouterUnadapted / Base SR (or EM/F1)Transferred Rerouter SR (or EM/F1) ↑
Game24 →\rightarrow Qwen3-32B (N=500)Qwen3-8B-trained26.4% (Standalone)78.8% (+52.4%)
WebShop →\rightarrow Qwen3-8B (N=500)Llama-3.2-3B-trained29.4% (Untrained 4B)36.4% (+7.0%, vs. 37.6% native)
WebShop →\rightarrow Llama-3.2-3B (N=500)Qwen3-8B-trained12.2% (Untrained 4B)12.4% (+0.2%)
MuSiQue QA →\rightarrow Llama-3.2-3B (N=50)Qwen3-8B-trained2.00% / 10.97% (Standalone)14.00% / 18.43% (+12.00% EM)

Table F: Expanded zero-shot cross-Generator transfer across model scales and tasks.

Refs. [1] DeepSeek-AI: DeepSeek-R1 (2025); [2] Jin: Search-R1 (2025); [3] Feng: ReTool (2025); [4] Yan: Memory-R1 (2025); [5] Chen: ReSearch (2025).

Reviewer_NeriOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies recovery in rollback-capable agent task settings: when the agent should roll back during execution, and how far. The authors introduce a Rerouter that operates independently of the agent loop, train it with reinforcement learning, and conduct basic validation experiments on multi-hop question answering, a simple game (Game24), and WebShop, with frozen 3B/8B generators. The learned Rerouter selects a rollback point and emits natural-language corrective feedback that conditions the first regenerated action.

Critical strengths

The problem decomposition is clean — recovery control is separated from action generation, with the Generator frozen throughout — and the feedback-removal ablation gives a clear mechanistic demonstration that rollback without corrective guidance drives the agent back into repeated states.

Critical weaknesses
  1. Limited novelty. The problem of when and how far to roll back follows existing rollback-recovery work, and the method adapts existing RL technique to it.

  2. Idealized settings. The settings do not reflect real long-horizon agentic execution, and offer no handling of the case where rollback is impossible (irreversible actions) — only an Ethics Statement acknowledgment of the gap. Moreover, rollbacks are executed by hand-written, rule-based code, markedly weaker than comparable recoverable-execution work (e.g., AgentRewind).

  3. Practical viability is not established. The Rerouter is invoked after every environment observation, adding a considerable runtime burden (74.58s vs. 4.09s end-to-end on Game24; Table 3), and effectiveness is validated only with 3B/8B generators — not the robust, large-parameter models practical agents would use. A fundamental issue is also left unexamined: capable models can already revert or redo actions within their own context; whether this would clash or duplicate with an independent external Rerouter is not analyzed.

Questions for authors
  1. For larger, robust generators that can themselves revert or redo actions in-context, would the model's own recovery behavior clash or duplicate with the independent Rerouter? (Reversion within a single attempt, distinct from the between-attempt reflection combined in Table 1.)

  2. How does the method behave when actions are irreversible or environment state cannot be reconstructed?

  3. Would the gains persist with a larger, more robust generator?

11 comments
AuthorsOpenReview ↗

Response to Neri (1/4): W1

Thank you for recognizing our decoupling of recovery control from action generation, our dual-view conditioning on active trajectories and pruned history, and our finding that rollback without corrective feedback leads to repeated state visitation.

W1. Limited novelty. The problem of when and how far to roll back follows existing rollback-recovery work, and the method adapts existing RL technique to it.

This characterization appears to conflate the rollback problem setup with the recovery policy learning mechanism: AgentReroute is the first framework to jointly learn when to roll back, how far to roll back, and what corrective feedback to provide purely from terminal task outcomes with a frozen Generator.

First, as Reviewers YcER and D7nU both highlighted in their Critical Strengths, eliminating step-level recovery labels by jointly optimizing rollback target selection and corrective feedback over dual-view history (τt\tau_t and pruned record ρt\rho_t) fundamentally distinguishes our approach from prior heuristic, prompted, or step-supervised rollback methods [1–4] (Table A).

Second, manuscript ablations (w/o RL, w/o Feedback) and new controlled GRPO variants (Feedback-only GRPO and Rollback-1 + Feedback GRPO) indicate that jointly learning multi-step rollback positions and feedback via RL is important. On Qwen/Game24 (500 items, Table A) and MuSiQue QA (100 items), removing RL drops Game24 SR from 77.8% to 15.6% (17.00% to 14.00% EM on QA); training feedback without rollback (Feedback-only GRPO) reaches only 37.6% SR (76.48% state recurrence vs. 28.57% for AgentReroute; 13.00% EM on QA); and restricting GRPO to single-step rollback (Rollback-1 + Feedback GRPO) reaches 57.6% SR. Therefore, neither prompted heuristics nor partial RL variants can replace our joint policy optimization:

MethodSupervision / Policy AcquisitionRollback Target SelectionCorrective SignalGenerator Frozen?Game24 SR
GA-Rollback [1]Prompted heuristics (no training)Prompted error index (span-capped)Prompted explanation & adviceYes59.6%
WebRollback [2]Zero-shot prompt or expert SFTPrompted/SFT state selectorCritique module evaluationShared SFT-
BacktrackAgent [3]Step-level SFT + step rewardFixed to immediate last stepReflector directly rewrites actionNo (co-trained)-
AgentRewind [4]Agent self-invokes rewind toolAgent selects saved checkpointAccumulated rewind memoryYes (no training)-
w/o RL TrainingZero-shot 4B promptPrompted over Kt\mathcal{K}_tPrompted feedbackYes15.6%
Feedback-only (GRPO)Terminal outcome GRPONone (no rollback)Learned next-step feedbackYes37.6%
Rollback-1 + Feedback (GRPO)Terminal outcome GRPOFixed to immediate last stepLearned next-step feedbackYes57.6%
AgentReroute (Ours)Terminal outcome GRPO (no step labels)Learned over recoverable set Kt\mathcal{K}_tJointly optimized natural language feedbackYes77.8%

Table A: Mechanistic and empirical comparison against prior rollback methods and controlled GRPO variants.

Refs. [1] Li: GA-Rollback (EMNLP'25); [2] Zhang: WebRollback (EACL'26); [3] Wu: BacktrackAgent (EMNLP'25); [4] Zhuang: AgentRewind (2026); [5] Shinn: Reflexion (NeurIPS'23); [6] Yang: DART (2026); [7] Dong: DeltaBox (2026); [8] Huang: Self-Correction (ICLR'24).

AuthorsOpenReview ↗

Response to Neri (2/4): W2-1

W2 & Q2. Idealized settings. The settings do not reflect real long-horizon agentic execution, and offer no handling of the case where rollback is impossible (irreversible actions)... Moreover, rollbacks are executed by hand-written, rule-based code, markedly weaker than comparable recoverable-execution work (e.g., AgentRewind). How does the method behave when actions are irreversible or environment state cannot be reconstructed?

W2-1: Local recovery on corrupted intermediate states vs. non-disruption on normal states.

AgentReroute operates on recoverable sub-horizons within long-horizon execution, rescuing 66.0% of corrupted intermediate states (vs. 4.0% for GA-Rollback) with 0% false-rollback disruption on normal states.

First, as Reviewers YcER and D7nU recognized, our evaluation spans web interaction (WebShop), multi-hop retrieval (4 QA benchmarks), and compositional planning (Game24)—the standard agent recovery benchmarks [1,2,5]. These benchmarks capture realistic multi-step agentic challenges: WebShop involves multi-turn catalog search, attribute verification, and dynamic DOM state transitions over 1.18M real-world products; the four multi-hop QA benchmarks require open-domain noisy passage retrieval and multi-step evidence chaining; and Game24 enforces strict combinatorial state dependencies where a single early error invalidates all downstream branches.

Second, we conduct controlled tests on 50 corrupted intermediate states and 19 normal intermediate states on Qwen3-8B / Game24 (capped at 30 actions and 35,000 tokens). Table B shows that on corrupted intermediate states, direct continuation scores 0%, full restart and GA-Rollback score only 4.0%, whereas AgentReroute rescues 66.0% (33/50); on the 19 normal intermediate states, AgentReroute achieves 94.7% (18/19, with 0 false-rollback disruptions), indicating that AgentReroute does not rely on idealized global resets, but instead pinpoints and repairs mid-trajectory errors locally within ongoing long-horizon execution without disturbing valid reasoning progress:

Intermediate State ConditionDirect ContinuationFull RestartGA-RollbackAgentReroute
Corrupted Intermediate States (N=50)0.0% (0/50)4.0% (2/50)4.0% (2/50)66.0% (33/50)
Normal Intermediate States (N=19)89.5% (17/19)--94.7% (18/19, 0 disruptions)

Table B: Controlled recovery on corrupted vs. normal intermediate states.

Refs. [1] Li: GA-Rollback (EMNLP'25); [2] Zhang: WebRollback (EACL'26); [3] Wu: BacktrackAgent (EMNLP'25); [4] Zhuang: AgentRewind (2026); [5] Shinn: Reflexion (NeurIPS'23); [6] Yang: DART (2026); [7] Dong: DeltaBox (2026); [8] Huang: Self-Correction (ICLR'24).

AuthorsOpenReview ↗

Response to Neri (3/4): W2-2–W2-3; Q2

W2-2 & Q2: Handling irreversible actions via Pending gates and restricted recoverable sets Kt\mathcal{K}_t.

The recoverable position set Kt\mathcal{K}_t in Eq. (4) explicitly models partial recoverability constraints; facing irreversible actions, AgentReroute operates safely via pre-commit Pending gates and restricted local recovery windows.

Following tool-agent safety protocols in DART [6] and AgentRewind [4], irreversible actions (e.g., click[Buy Now] in WebShop) can be staged as a Pending state for Rerouter inspection before execution, while nodes prior to an already-executed irreversible action are excluded from Kt\mathcal{K}_t. As shown in Table C: (1) adding a pre-purchase Pending check on WebShop (100 items) preserves the full 37.0% SR while safely intercepting erroneous checkouts; and (2) restricting Kt\mathcal{K}_t on the 50 corrupted Game24 states to the last action of the initial prefix or later steps (masking earlier nodes) still recovers 40.0% of trajectories (vs. 0% for continuation):

Irreversibility Mechanism & TaskSample SizeBaseline / UnrestrictedConstrained / Pending AgentReroute
WebShop Pre-commit Pending Gate10037.0% (Direct commit)37.0% (Safe pre-commit verification)
Game24 Restricted Kt\mathcal{K}_t (Partial lock)500.0% (Continue) / 66.0% (Full Kt\mathcal{K}_t)40.0% (Post-anchor local rollback only)

Table C: Performance under pre-commit Pending verification and restricted recoverable sets Kt\mathcal{K}_t.

W2-3: Orthogonal division of labor between our learned recovery policy and runtime snapshot executors.

This critique conflates the upper-level recovery policy (learning when/where to roll back and what feedback to give) with the lower-level state restoration executor (saving and restoring environment snapshots): AgentReroute and systems like AgentRewind [4] or DeltaBox [7] are orthogonal and complementary.

Our manuscript explicitly cites and discusses AgentRewind [4] and DeltaBox [7]: such systems provide low-level checkpointing and state restoration infrastructure, but rely on unadapted prompts or the base agent itself to decide when to rewind. Conversely, AgentReroute trains the upper-level recovery policy via RL and calls the underlying executor through a modular interface (target k, retained prefix) -> restored state (Table D). In deployment, our learned Rerouter can directly invoke AgentRewind or DeltaBox as its backend environment-restoration engine:

System LayerModule InputModule Output & Core Responsibility
Rerouter: Upper-Level Recovery Policy (Ours)Task instruction, active trajectory τt\tau_t, pruned history ρt\rho_t, recoverable set Kt\mathcal{K}_tOutputs Continue or (rollback target k, feedback f_t); learns when/where to recover and how to guide subsequent generation via GRPO
Low-Level State Restoration Executor (Rule replay / AgentRewind [4] / DeltaBox [7])Target index k∈Ktk \in \mathcal{K}_t, retained prefix τt<k\tau_t^{<k}, environment state logRestores environment state prior to action kk (recomputing numbers in Game24, restoring page in WebShop, or truncating query prefix in QA) and cleans active context
Generator: Task Execution Layer (Frozen)Restored prefix τt<k\tau_t^{<k}, current observation, corrective feedback ftf_tGenerates alternative action from the clean restored state and feedback without updating main model weights

Table D: Orthogonal division of labor between the learned Rerouter policy and the low-level state executor.

Refs. [1] Li: GA-Rollback (EMNLP'25); [2] Zhang: WebRollback (EACL'26); [3] Wu: BacktrackAgent (EMNLP'25); [4] Zhuang: AgentRewind (2026); [5] Shinn: Reflexion (NeurIPS'23); [6] Yang: DART (2026); [7] Dong: DeltaBox (2026); [8] Huang: Self-Correction (ICLR'24).

Reviewer_YcEROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentReroute, an externally assisted trajectory-correction framework. It freezes the task-performing Generator and trains a separate 4B Rerouter with GRPO. After each interaction, the Rerouter decides whether to continue or roll back to an earlier action while providing corrective feedback. Experiments on WebShop, Game24, and four multi-hop QA datasets show substantial improvements, compatibility with Reflexion, and some cross-dataset and cross-Generator transfer.

Critical strengths
  • Jointly learns rollback-point selection and corrective feedback from task outcomes.

  • Can improve existing agents while keeping the Generator frozen.

  • Evaluates across web interaction, arithmetic reasoning, and multi-hop QA.

  • Produces large and consistent gains on Game24 with two Generators.

  • Ablations and behavioral analysis support the importance of RL and corrective feedback.

Critical weaknesses
  • The experiments do not isolate rollback from the contribution of an additional 4B model that can directly reason about the task and suggest the next action.

  • AgentReroute receives task-specific outcome training, whereas several baselines appear prompt-based or receive no comparable adaptation.

  • No multi-seed results, confidence intervals, or significance tests are reported.

  • Inference is slow: on Llama/Game24, AgentReroute requires 74.58 seconds per task versus 28.94 for GA-Rollback.

Questions for authors
  • Can the authors compare against a feedback-only controller using the same 4B model and a similar inference budget but without rollback?

  • How often does the Rerouter’s Game24 feedback directly provide the correct next operation? Could its own solving ability explain much of the gain?

  • Do all methods receive comparable interaction, retry, and computational budgets? Were the QA baselines given any task-specific training?

  • Are the results stable across training and inference seeds? Can confidence intervals or paired significance tests be provided?

  • Please provide the complete GRPO hyperparameters, group size, training steps, and checkpoint-selection procedure.

  • Was cross-Generator transfer also tested on WebShop or QA?

6 comments
AuthorsOpenReview ↗

Response to YcER (1/4): W1; Q1–Q2

Thank you for recognizing our decoupling of recovery control from action generation, outcome-driven GRPO training without step-level labels, and zero-shot cross-dataset/cross-generator transfer.

W1, Q1 & Q2. The experiments do not isolate rollback from the contribution of an additional 4B model that can directly reason about the task and suggest the next action.

W1-1 & Q1: Compare against a feedback-only controller using the same 4B model.

AgentReroute's gains come from coupling state rollback with feedback, not from adding a 4B model alone.

First, prior rollback studies show that when early erroneous actions remain in the context, external feedback alone cannot prevent the generator from repeating faulty history [1,2]. As Reviewers Neri and D7nU also noted, jointly learning rollback control and corrective feedback is key to avoiding repeated failures.

Second, both manuscript Table 1 and a new GRPO-trained Feedback-only control indicate that a 4B feedback controller without rollback fails to resolve state stagnation. In Table 1, w/o RL Training uses the exact same 4B model and prompt interface yet achieves only 15.6% success rate (SR) on Qwen/Game24 (vs. 77.8% for AgentReroute). We further train a controlled Feedback-only (GRPO) baseline from the same Qwen3-4B-Instruct-2507 under identical GRPO settings (Qwen3-8B frozen), which retains the full trajectory and gives next-step feedback without rolling back. Across 500 Game24 tasks, Feedback-only (GRPO) reaches only 37.6% SR with a 76.48% state-recurrence rate (Table A):

Method (Qwen3-8B / Game24, N=500)Success Rate (SR) ↑Repeated States / New ActionsState Recurrence Rate ↓
w/o RL Training15.6%2,568 / 4,33759.21%
Feedback-only (GRPO, no rollback)37.6%3,119 / 4,07876.48%
AgentReroute (Full)77.8%911 / 3,18928.57%

Table A: Comparison against Feedback-only (GRPO) and state-recurrence statistics.

W1-2 & Q2: Next-operation suggestions in Game24 feedback and the 4B model's own solving ability.

The 4B model's own solving ability (8.0% SR) cannot explain the gain; its feedback provides single-step local guidance that works only after rollback restores a solvable state.

First, consistent with prior work on lightweight RL critics and retrospective controllers [3,4], an auxiliary controller excels at local error redirection rather than standalone long-horizon solving—a modular separation also highlighted by Reviewer Neri.

Second, auditing 100 rollback feedback messages alongside a Rollback-1 + Feedback (GRPO) ablation indicates that next-step suggestions fail without multi-step rollback. In Table B, while 92.0% of feedback messages suggest a next operation and 69.0% are valid and preserve solvability after AgentReroute restores the chosen checkpoint, the standalone 4B model (Act-only) solves only 8.0% of problems (Table C). Moreover, restricting a GRPO-trained Rerouter to only undo the last action (Rollback-1 + Feedback) drops SR from 77.8% to 57.6%. Once early steps corrupt the remaining numbers into a dead end, no single-step suggestion can reach 24 unless multi-step rollback first restores a valid historical state:

Feedback Audit Metric (N=100)CountRatio
Explicitly suggests next operation92 / 10092.0%
Unique executable operation identifiable91 / 10091.0%
Suggested operation is legally valid78 / 9185.7%
Remains solvable for 24 after valid operation69 / 7888.5%
Valid and preserves solvability (overall)69 / 10069.0%

Table B: Audit of 100 Game24 rollback feedback messages.

Controller Setting (Game24)Eval SizeSuccess Rate (SR) ↑
Standalone 4B Solver (Act-only, no 8B Generator)1008.0%
Feedback-only (GRPO, no rollback)50037.6%
Rollback-1 + Feedback (GRPO, undo last step only)50057.6%
AgentReroute (Learned multi-step rollback + Feedback)50077.8%

Table C: Standalone 4B solving ability vs. constrained and full rollback controllers.

Refs. [1] Li: GA-Rollback (EMNLP'25); [2] Zhang: WebRollback (EACL'26); [3] Yao: Retroformer (ICLR'24); [4] Xie: CTRL (ICML'25); [5] Yan: Memory-R1 (2025); [6] Shang: AgentSquare (ICLR'25); [7] Chen: HarnessForge (2026); [8] Zhang: ACE (2026); [9] Su: MNL (2026); [10] Yao: Tree-of-Thoughts (NeurIPS'23); [11] Li: Search-o1 (2025).

AuthorsOpenReview ↗

Response to YcER (2/4): W2; Q3-1–Q3-2

W2 & Q3. AgentReroute receives task-specific outcome training, whereas several baselines appear prompt-based or receive no comparable adaptation. Do all methods receive comparable interaction, retry, and computational budgets? Were the QA baselines given any task-specific training?

W2-1 & Q3-1: Baseline task adaptation and task-level RL controls on QA and Game24.

All methods keep the Generator frozen, strong baselines already receive task-specific adaptation, and AgentReroute still outperforms Feedback-only (GRPO) trained with identical task-level RL.

First, following established paradigms that train lightweight auxiliary modules while freezing the main policy [3,5], keeping the Generator frozen ensures fair base execution capacity—a practical advantage commended by Reviewer D7nU.

Second, strong baselines across Game24, WebShop, and QA already receive task-specific development search or trajectory adaptation. Following standard protocols [6–9], AgentSquare evaluates 30 candidate module combinations on task development data (MuSiQue for QA) before freezing the best combination, and ACE iteratively updates a playbook using task trajectories and feedback for up to 3 reflection rounds, within an 80K-token cap, and freezes it for evaluation (Table E).

Finally, we add a parameter-trained Feedback-only (GRPO) control on MuSiQue QA using the exact same 4B backbone and GRPO setup. Across 100 shared MuSiQue test items (Table D), AgentReroute leads Feedback-only (GRPO) by +4.00 EM and +8.14 F1 points, indicating that task-level RL on a 4B feedback controller alone cannot replace our joint rollback-and-feedback recovery policy:

QA Method (Qwen3-8B / MuSiQue, N=100)Exact Match (EM) ↑Token F1 ↑
w/o RL Training14.00%21.29%
Feedback-only (GRPO)13.00%19.00%
AgentReroute (Full)17.00%27.14%

Table D: Matched task-level GRPO control on MuSiQue QA.

MethodTasks CoveredWeights Updated?Task-Specific Adaptation Mechanism
Act-only / ReAct / Naive RAGGame24, WebShop, QANoTask-specific action schemas, environment interfaces, and task prompts
Iter-RetGen / IRCoT / Search-o1QANoTask-tuned iterative retrieval-reasoning pipelines with frozen generator
Reflexion / GA-RollbackGame24, WebShopNoCross-episode reflection memory or prompted error localization and rollback
AgentSquare [6]Game24, WebShop, QANoEvaluates 30 candidate module combinations on task dev set and freezes the winner
ACE [8]Game24, WebShop, QANoIteratively updates a playbook (80K-token cap) over task trajectories and feedback, then freezes it for evaluation
Feedback-only (GRPO)Game24, QA4B Controller onlyOutcome-driven GRPO training for next-step feedback; Generator frozen; no rollback
AgentRerouteGame24, WebShop, QA4B Rerouter onlyOutcome-driven GRPO jointly learning rollback decision, target, and feedback; Generator frozen

Table E: Baseline adaptation protocols and parameter update settings.

W2-2 & Q3-2: Comparable interaction, retry, and token budgets.

Under strictly matched cumulative action budgets and total token budgets, AgentReroute outperforms all baselines by a wide margin.

In manuscript Table 3, AgentReroute already uses fewer total tokens (26.75k) than GA-Rollback (34.66k). On Llama-3.2-3B / Game24 (100 items), we count cumulative consumption across all models, failures, reflections, and retries, allowing baselines to repeatedly retry until exhausting 30 Generator actions or 35,000 input+output tokens. Table F indicates that AgentReroute wins by a large margin under both matched-budget conditions:

Method (Llama-3.2-3B / Game24, N=100)SR @ 30 Generator Actions ↑SR @ 35,000 Total Tokens ↑
Act-only (with multi-episode retries)1.0%1.0%
Reflexion1.0%1.0%
GA-Rollback11.0%9.0%
AgentReroute76.0%43.0%

Table F: Success rates under matched cumulative action and token budgets.

Refs. [1] Li: GA-Rollback (EMNLP'25); [2] Zhang: WebRollback (EACL'26); [3] Yao: Retroformer (ICLR'24); [4] Xie: CTRL (ICML'25); [5] Yan: Memory-R1 (2025); [6] Shang: AgentSquare (ICLR'25); [7] Chen: HarnessForge (2026); [8] Zhang: ACE (2026); [9] Su: MNL (2026); [10] Yao: Tree-of-Thoughts (NeurIPS'23); [11] Li: Search-o1 (2025).

AuthorsOpenReview ↗

Response to YcER (3/4): W3–W4; Q3-3; Q4

W3 & Q4. No multi-seed results, confidence intervals, or significance tests are reported. Are the results stable across training and inference seeds?

W3-1 & Q4-1: Stability across training and inference seeds.

AgentReroute is highly stable across both independent GRPO training seeds (73.2% ± 4.10%) and Generator inference seeds (80.0% ± 1.73%).

Table G reports three independent GRPO training runs (seeds 42, 43, 44, each initialized from the raw 4B checkpoint and evaluated on the 500 Game24 test items) and three inference seeds regenerating the Generator's initial actions (100 items):

Stability Evaluation SettingSeed 42Seed 43Seed 44Mean ± SD
Independent GRPO Training Seeds (Test N=500)69.0%77.2%73.4%73.2% ± 4.10%
Independent Generator Inference Seeds (Test N=100)79.0%79.0%82.0%80.0% ± 1.73%

Table G: Training and inference seed stability on Game24.

W3-2 & Q4-2: Confidence intervals and paired significance tests.

All four main comparisons against GA-Rollback in Table 1 are statistically significant (95% paired CIs above zero, exact McNemar p<0.001p < 0.001).

Generator / Task (N=500 each)Δ SR vs. GA-Rollback (% pts)95% Paired CI (% pts)Exact McNemar pp-value
Qwen3-8B / Game24+18.2[+13.8, +22.6]9.12×10−159.12 \times 10^{-15}
Llama-3.2-3B / Game24+46.2[+41.6, +50.8]2.80×10−602.80 \times 10^{-60}
Qwen3-8B / WebShop+7.2[+3.6, +10.8]1.30×10−41.30 \times 10^{-4}
Llama-3.2-3B / WebShop+8.8[+5.2, +12.4]2.15×10−62.15 \times 10^{-6}

Table H: Paired confidence intervals and exact McNemar significance tests.

W4 & Q3-3. Inference is slow: on Llama/Game24, AgentReroute requires 74.58 seconds per task versus 28.94 for GA-Rollback.

Under an identical per-task wall-clock budget, AgentReroute still maintains a substantial lead in success rate.

The longer single-attempt latency arises because AgentReroute converts trajectories that otherwise fail rapidly (6.6% SR for GA-Rollback) into multi-step successful recoveries (52.8% SR)—a natural cost of precise trajectory restoration. Following test-time scaling evaluation standards [10,11], we allocate an identical 75-second wall-clock budget per task across all methods on Llama-3.2-3B / Game24 (100 items), allowing faster baselines to repeatedly restart or recover until time expires (Table I). The results show that even when faster baselines are granted multiple retries within the exact same 75-second limit, AgentReroute achieves nearly 5× the cumulative success rate of GA-Rollback (44.0% vs. 9.0%), indicating that its runtime investment translates directly into effective task completion rather than wasted overhead:

Method (Llama-3.2-3B / Game24, N=100)SR under Matched 75s Wall-Clock Budget ↑
Act-only (with repeated restarts)1.0%
Reflexion2.0%
GA-Rollback9.0%
AgentReroute44.0%

Table I: Cumulative success rate under a matched 75-second wall-clock budget per task.

Refs. [1] Li: GA-Rollback (EMNLP'25); [2] Zhang: WebRollback (EACL'26); [3] Yao: Retroformer (ICLR'24); [4] Xie: CTRL (ICML'25); [5] Yan: Memory-R1 (2025); [6] Shang: AgentSquare (ICLR'25); [7] Chen: HarnessForge (2026); [8] Zhang: ACE (2026); [9] Su: MNL (2026); [10] Yao: Tree-of-Thoughts (NeurIPS'23); [11] Li: Search-o1 (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.