acceptodds
Under review as a conference paper at ICLR 2027

Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agents

Abstract

Large language model agents increasingly operate in long-horizon environments where early mistakes can trigger cascading failures and trap agents in dead-end trajectories. Agents must therefore be able to proactively recover from such states and turn their exploratory experience into guidance for subsequent decisions. We introduce **ReTRAIL**, a framework that enables agents to rewind the environment to earlier states while preserving valid progress and carrying forward knowledge learned from abandoned exploration. The environment state moves backward while the agent's knowledge continues to move forward. ReTRAIL trains a shared policy with two complementary signals: State-Anchored GRPO compares alternative continuations from revisited states to improve execution, while a branch-advancement reward guides reflection through the progress of subsequent exploration. Experiments across four interactive benchmark MineSweeper, Sokoban, ALFWorld and ScienceWorld show that simply adding a rewind primitive provides limited benefit and can even degrade performance. In contrast, ReTRAIL enables agents to rewind from unproductive states and use insights from abandoned branches to guide subsequent exploration. These results demonstrate the value of coupling environment rewinding with experience reuse within an ongoing task for long-horizon task solving.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Reinforcement Learning, Rollback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o63WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLM agents can recover from mistakes within an ongoing task instead of restarting from the beginning. The authors propose ReTRAIL, which gives the agent an environment-level rewind action. The agent can restore an earlier state, write a reflection on what it learned from the abandoned branch, and continue from the restored state with that reflection. The agent is trained with RL using two signals. State-Anchored GRPO compares continuations from the same state to improve actions. A branch-advancement reward scores a reflection by how much more progress the resumed branch makes than the abandoned one. With Qwen3-4B on MineSweeper, Sokoban, ALFWorld, and ScienceWorld, ReTRAIL achieves the highest success when rewinds and retries are allowed.

Critical strengths
  • The idea of moving the environment backward while carrying knowledge forward is intuitive and well-motivated. It is also clearly distinguished from multi-episode adaptation, which restarts from the initial state.
  • The two learning signals fit the setting well. Continuations from the same state give a natural group for comparing actions, and scoring a reflection by the progress of the branch that follows gives reflection its own learning signal.
  • The paper provides ablations of rewind, reflection, and both learning objectives, along with analyses by difficulty, number of attempts, and task length.
Critical weaknesses
  • Baselines are not trained with rewind: ReTRAIL is the only method trained with the rewind interface. The RL baselines are trained without it and only given rewind at test time, so it is expected that they use it poorly. When State-Anchored GRPO is replaced with GiGPO inside the same rewind framework (Table 3), most of the gap closes.
  • Rewind changes what the benchmarks test: On MineSweeper, after the agent reveals a mine, it can restore an earlier state of the same board. Similarly, Sokoban is meant to test planning with moves that cannot be undone, and rewind removes that difficulty.
  • Limited practicality: The method assumes any earlier state can be restored for free, either from saved snapshots or by replaying actions in a deterministic environment. Replayed actions are also not counted in the budget. Most realistic settings the paper uses as motivation, such as web or software tasks with side effects, cannot be rewound this way. Training also relies on progress scores that are rarely available in practice, such as the shortest-solution distance found by search in Sokoban.
  • Limited evaluation: The tasks are small and short: 6×6 MineSweeper boards with three mines, Sokoban rooms with two or three boxes, and a 60-call budget. This makes the long-horizon framing a stretch. ALFWorld is evaluated only on the seen split, not on the unseen split that tests generalization, and all fine-tuned methods already exceed 89%. All results use a single 4B model, limiting generalization.
Questions for authors
  • Could the authors train the RL baselines with the same rewind interface so that the comparison isolates the proposed objectives?
  • On MineSweeper, how often do successful T@3 attempts depend on mine locations revealed by earlier failed attempts?
  • Do the RL baselines receive the same progress signals as ReTRAIL, such as the Sokoban progress score computed by search, or only the task outcome?
  • How much training compute does ReTRAIL use compared with the baselines?
5 comments
AuthorsOpenReview ↗

Response to Reviewer o63W (Part 1/3)

Thank you for recognizing that moving the environment backward while carrying knowledge forward is intuitive and clearly distinguished from multi-episode adaptation and the two learning signals fit the setting. We have marked all revisions in red in the updated PDF and respond to your concerns point by point below. Section, appendix, and table numbers refer to the revised PDF.

W1 & Q1: Training the baselines with the rewind interface

We thank the reviewer for this suggestion. We would first like to clarify that the GiGPO row of Table 3 is not a baseline trained with rewind. It is ReTRAIL with only State-Anchored GRPO replaced by GiGPO, and it keeps retrospective exploration and the branch-advancement reward for reflection. We agree that the baselines in Table 1 only receive rewind at test time, so we trained GRPO and GiGPO with the same rewind interface as ReTRAIL.

● Setup: During training, the baselines can rewind and write a reflection exactly as ReTRAIL does, under the same call ceiling and attempt limit.

MethodMineSweeper T@1MineSweeper T@3Sokoban T@1Sokoban T@3
GRPO51.660.868.982.8
GRPO, trained with the rewind interface52.968.571.284.6
GiGPO56.465.579.785.3
GiGPO, trained with the rewind interface57.173.980.687.2
ReTRAIL with the GiGPO objective (Table 3)57.690.184.290.3
ReTRAIL63.595.387.593.7

● Baselines trained with the interface: Training with the interface improves the baselines, for example GiGPO gains 8.4 points of T@3 on MineSweeper. They nevertheless remain 21.4 points below ReTRAIL on MineSweeper and 6.5 points below on Sokoban. Having access to rewind during training is not enough: the policy also has to learn what to carry over from an abandoned branch.

The gap between GiGPO trained with the interface and the Table 3 variant (73.9 versus 90.1 on MineSweeper) comes from learning reflection through branch advancement. State-Anchored GRPO adds a further 5.2 points of T@3 and 5.9 points of T@1.

Revision. We have added the interface-trained baselines to Appendix C.3 (Table 5), referenced from §5.2, and clarified in §6 what the GiGPO variant of Table 3 retains.

W2: Rewind changes what the benchmarks test

We thank the reviewer for this comment. We agree that rewind changes the task setting, and we therefore report both settings.

Under P@1, which keeps the original irreversible setting of each benchmark, ReTRAIL achieves the best result on MineSweeper (58.8), ALFWorld (92.7) and ScienceWorld (56.7), and stays within 2.3 points of GiGPO on Sokoban (76.6 versus 78.9, Table 1).

T@1 and T@3 measure success in a recoverable environment, which is the setting our work targets. A revealed mine or a deadlocked box becomes a failed branch that can be reverted. Whether the agent can recognize such a branch and make use of what it revealed is crucial for autonomous execution and full exploration. Rewind does not remove this difficulty by itself: with rewind enabled, GRPO drops from 54.6 to 51.6 on MineSweeper and from 70.3 to 68.9 on Sokoban under the same protocol (Table 1). We analyze how much revealed mine locations contribute in our response to Q2.

Revision. We have clarified in §5.1 what P@1 and T@1/T@3 each measure.

AuthorsOpenReview ↗

Response to Reviewer o63W (Part 2/3)

W3: Limited practicality We thank the reviewer for raising these points and address them in turn.

● Practicality: Environments for autonomous agent execution are increasingly restorable by design: virtual machines support snapshots [2], sandbox runtimes roll back an agent's workspace in milliseconds [3], and web agents are given explicit rollback operations [1]. In these environments, the side effects of an action, such as file edits, application state and browser sessions, stay inside the workspace and can be reverted. What is missing is a policy that uses this ability well. With rewind at test time, the trained baselines change by only −3.0 to +1.8 points, whereas ReTRAIL gains up to 10.9 points from proactive rewind and 36.5 points with three attempts (Table 1). Learning when to rewind and what to carry over is the gap that ReTRAIL fills. We acknowledge that some actions have effects outside the workspace that cannot be undone, such as completing a payment. In such cases, ReTRAIL simply acts without rewind, and its performance without rewind (P@1) is still competitive.

● Progress signals: In all four environments, the progress score is computed on the environment side from the task state, including the solver distance in Sokoban, and requires no human annotation. Such verifiable progress is increasingly available, through checkpoints and state-based tests in realistic benchmarks [4] and through learned progress models [5], and ReTRAIL can use any of these as its branch score. To test how much ReTRAIL depends on such scores, we conducted an experiment in which every method is trained with only the binary task outcome:

Method (binary outcome only)MineSweeper T@1MineSweeper T@3Sokoban T@1Sokoban T@3
GiGPO54.860.277.383.1
LaMer56.068.969.181.6
ReTRAIL60.289.183.989.6

Under this sparse signal, ReTRAIL still trains well. It loses 6.2 and 4.1 points of T@3 relative to Table 1, and remains 20.2 and 6.5 points above the stronger baseline.

Revision. We have added a discussion of restorable environments, this scope, and the availability of progress signals to Appendix B, the restoration cost to Appendix D.3, and the binary-outcome experiment to Appendix C.8 (Table 11).

[1] WebRollback: Enhancing Web Agents with Explicit Rollback Mechanisms. EACL 2026. [2] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024. [3] DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback. ACM ATC 2026. [4] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. ACL 2024. [5] Web-Shepherd: Advancing PRMs for Reinforcing Web Agents. NeurIPS 2025.

W4: Limited evaluation

We thank the reviewer for these comments and address them one by one.

● Task size and horizon: We agree that the default instances are relatively small. In figure 4(a) we scales their difficulty, and ReTRAIL keeps its advantage at every level. We additionally scaled the step budget on the hardest instances, using the same checkpoints:

T@3 (%)MineSweeper, 5 mines: 60 steps90 steps120 stepsSokoban, 4 boxes: 60 steps90 steps120 steps
GiGPO34.636.737.538.140.241.0
LaMer45.347.748.445.548.049.2
ReTRAIL63.971.575.855.263.368.0

With a larger budget, ReTRAIL improves by 11.9 and 12.8 points, against at most 3.7 for the baselines. Its benefit also grows with the horizon within the main benchmarks: the retry gain rises from 4.5 points on tasks with short reference solutions to 19.0 points on long ones (Appendix C.1).

ALFWorld unseen split: We additionally evaluated the same checkpoints on the unseen split (134 tasks) under the same protocol:

MethodP@1T@1T@3
GiGPO86.685.890.3
LaMer84.385.191.8
ReTRAIL88.190.396.3

All methods lose 2.2 to 5.4 points relative to the seen split, and ReTRAIL remains the best method in every column.

We trained GiGPO, LaMer and ReTRAIL on MineSweeper and Sokoban with Qwen2.5-7B and Llama-3.1-8B, which covers a larger model and a second model family:

ModelMethodMineSweeper T@1MineSweeper T@3Sokoban T@1Sokoban T@3
Qwen2.5-7BGiGPO48.056.353.159.8
Qwen2.5-7BLaMer48.862.144.958.2
Qwen2.5-7BReTRAIL55.188.761.772.3
Llama-3.1-8BGiGPO44.151.631.336.7
Llama-3.1-8BLaMer49.264.532.844.1
Llama-3.1-8BReTRAIL53.985.243.058.6

ReTRAIL still achieve the best method in every column, with T@3 margins of 12.5 to 26.6 points.

In summary, the advantage of ReTRAIL holds on harder instances, on the unseen split of ALFWorld, and across three base models from two families.

AuthorsOpenReview ↗

Response to Reviewer o63W (Part 3/3)

Q2: Dependence of successful T@3 attempts on revealed mine locations

About two thirds of the successful T@3 trials succeed within the first macro-attempt and never reveal a mine (T@1 is 63.5 and T@3 is 95.3). Among the remaining third, the reflection states the location of the revealed mine in 81% of the trials, which is 27% of all successful trials. To measure the actual dependence, we re-evaluated the same checkpoints with the coordinates of revealed mines removed from the reflection and from every retained observation:

MethodT@3T@3, mine locations masked
GiGPO65.563.9
LaMer70.866.0
ReTRAIL95.388.4

Masking costs ReTRAIL 6.9 points, so part of its retry gain does come from revealed mine locations. Most of the gain remains: with the locations masked, ReTRAIL is still 22.4 points above LaMer under the same masking.

Revision. We have added this analysis to Appendix C.9 (Table 12) and summarized it in §6.

Q3: Progress signals received by the RL baselines

For a fair comparison, all trained baselines receive the same environment feedback as ReTRAIL, namely the progress score of Table 14, including the search-based score on Sokoban. The only signal specific to ReTRAIL is the branch-advancement reward, which compares an abandoned branch with the branch resumed after a rewind and is therefore defined only under retrospective exploration.

Revision. We have listed the training signals of every method in Appendix D.2 (Table 15).

Q4: Training compute of ReTRAIL compared with the baselines

ReTRAIL uses no more training compute than the baselines. All methods are trained for 300 updates on 8 NVIDIA H200 GPUs, and the table reports the realized GPU-hours:

GPU-hours for 300 updatesMineSweeperSokobanALFWorldScienceWorld
GiGPO441482186359
LaMer480523203391
ReTRAIL416427174332

ReTRAIL samples 8 trials per task where the baselines sample 24 trajectories, and it uses 61% to 85% of their tokens. The full breakdown into model calls, tokens and evaluation cost is given in our response to Reviewer 6QEV (W1 & Q1).

Revision. We have added the full breakdown to Appendix D.3 (Tables 17 and 18).

Thank you again for the detailed and constructive feedback, which shaped most of the additional experiments in this revision. We hope these results and clarifications address your concerns, and we would be glad to discuss any remaining questions.

Reviewer_6QEVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReTRAIL, a framework that enables agents to rewind the environment to earlier states while preserving valid progress and carrying forward knowledge learned from abandoned exploration. It studies how language agents can recover from mistakes during long tasks while retaining useful progress and lessons from failed exploration. ReTRAIL combines environment rewinding with a shared execution and reflection policy trained using RL. Experiments with Qwen3-4B on four interactive benchmarks show higher success rates than the evaluated baselines under a shared test-time model-call budget.

Critical strengths

1, this paper address an useful problem, has a good motivation on the problem that recovering from a bad trajectory without restarting the entire task or forgetting what went wrong. Two factors here, restoring the environment and retaining the experience, both are clearly motivated. 2, The improvements seem to be significant on several benchmarks. 3, The ablations are reasonably informative, and suggest that the proposed methods contribute meaningfully to the observed gains.

Critical weaknesses

1, The cost comparisons are limited. A shared model-call ceiling does not control token usage or the environment work required for restoration. Actual resource usage would help assess the efficiency claims.

Questions for authors

Could you report actual training and evaluation calls, tokens, and wall-clock time, including environment restoration?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6QEV

Thank you for recognizing the motivation of recovering from a bad trajectory without restarting the task or forgetting what went wrong, the improvements across benchmarks, and the informativeness of our ablations. We have marked all revisions in red in the updated PDF and respond to your concern below. Section, appendix, and table numbers refer to the revised PDF.

W1 & Q1: Actual training and evaluation cost, including environment restoration

We appreciate the reviewer's suggestion. We agree that a shared model-call ceiling controls neither token usage nor the work of restoring the environment. We therefore report the realized cost of ReTRAIL and of the two strong baselines, GiGPO and LaMer.

● Training cost: The table reports the realized cost per task and optimizer update, averaged over the 300 updates. ReTRAIL samples 8 trials per task and the baselines sample 24 trajectories. Calls include execution and reflection, and all runs use 8 NVIDIA H200 GPUs.

EnvironmentMethodModel calls per taskTokens per taskGPU-hours for 300 updates
MineSweeperGiGPO4201.30M441
MineSweeperLaMer4441.43M480
MineSweeperReTRAIL3520.87M416
SokobanGiGPO4201.32M482
SokobanLaMer4441.46M523
SokobanReTRAIL3360.96M427
ALFWorldGiGPO3000.27M186
ALFWorldLaMer3170.32M203
ALFWorldReTRAIL1920.23M174
ScienceWorldGiGPO4560.87M359
ScienceWorldLaMer4800.96M391
ScienceWorldReTRAIL3600.72M332

The realized calls are well below the ceilings of Table 16 (1,440 for the baselines and 480 for ReTRAIL). ReTRAIL uses 61% to 85% of the tokens of the baselines and no more GPU-hours on any environment.

● Evaluation cost: The table reports the average cost of one test task under T@3, over all test instances including failed trials:

EnvironmentMethodModel callsTokensWall-clock
MineSweeperGiGPO55.9146k140 s
MineSweeperLaMer56.4151k139 s
MineSweeperReTRAIL43.5108k109 s
SokobanGiGPO51.6155k129 s
SokobanLaMer54.1164k130 s
SokobanReTRAIL40.8116k102 s
ALFWorldGiGPO26.034k69 s
ALFWorldLaMer32.436k68 s
ALFWorldReTRAIL21.729k59 s
ScienceWorldGiGPO48.497k131 s
ScienceWorldLaMer47.6100k129 s
ScienceWorldReTRAIL43.587k117 s

ReTRAIL needs 9% to 33% fewer model calls per test task than the baselines, as it resumes from a checkpoint instead of restarting from the initial state.

● Environment restoration: MineSweeper and Sokoban restore snapshots. ALFWorld and ScienceWorld reset the task and replay the retained actions, which requires no model call. A replayed action takes 0.18 s and 0.13 s, so replay adds about 0.4 s and 1.4 s to a test task, less than 2% of its wall-clock. The wall-clock numbers above include restoration.

Revision. We have added both tables and the restoration cost to Appendix D.3 (Tables 17 and 18), and §5.1 now points to the realized cost.

Thank you again for the constructive suggestion, which led us to report realized cost alongside the call ceilings. We hope these numbers address your concern, and we welcome further discussion.

AuthorsOpenReview ↗

Anticipating Your Response

Dear Reviewer 6QEV, thank you again for your valuable suggestions. As the discussion period ends in one day, we would like to check whether our response has addressed your concerns. We are happy to discuss any further questions and would appreciate it if you could reconsider your evaluation.

Reviewer_DsXEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReTRAIL, which trains an LLM agent to learn to restore earlier environment states, while retaining reflections of the branches it has abandoned. The execution objective combines segment-endpoint progress, and return comparisons among continuations that share the environment state and history. The reflection objective rewards progress after recovery, relative to the abandoned branch. Experiments cover MineSweeper, Sokoban, ALFWorld and ScienceWorld with Qwen3-4B model. The results support improved recovery-enabled success.

Critical strengths
  1. Recovery gains persist under a shared inference-call ceiling.
  • Table 1 reports highest T@1, T@3 success on all four environments, where T@3 reaches 95.3% versus the strongest baseline's performance of 70.8% on MineSweeper.
  • Since both execution and reflection both count toward the call-ceiling, the gains are not from more model calls.
  1. The ablations are meaningful. Replacing the proposed objective with GiGPO lowers T@3 on both MineSweeper and Sokoban. Replacing progress-difference reflection rewards with post-recovery progress lowers the performance as well. The results support specific training and memory-design choices beyond exposing a rewind command.
Critical weaknesses
  1. The execution objective does not credit proactive rewind decisions.
  • Rewind terminates the execution return with zero transition reward. Hence, different proactive targets receive the same execution advantage regardless of subsequent success.
  • Post-recovery feedback trains the separate reflection output instead.
  • Hence I believe it is less clear whether the execution objective learns to rewind time by evaluating the resulting recovery.
  1. The claimed execution-efficiency is not established by the cost analysis.
  • Figure 3 compares action counts on each method's success, which can be different instance subsets.
  • This does not establish lower cost on attached tasks or account for failed trials.
  • Reset-and-replay incurs additional environment actions outside the model calls.
Questions for authors
  • Were the Table 2 variants retrained with the modified interfaces, or were the components removed only at evaluation?
  • What was the sampling configuration for GiGPO objective in Table 3?
4 comments
AuthorsOpenReview ↗

Response to Reviewer DsXE (Part 1/2)

Thank you for recognizing that the recovery gains persist under a shared inference-call ceiling and that the ablations support specific training and memory-design choices. We have marked all revisions in red in the updated PDF and respond to your concerns point by point below. Section, appendix, and table numbers refer to the revised PDF.

W1: Credit for proactive rewind decisions

We appreciate the reviewer's insightful observation. We agree that the execution return ends at the rewind request, so the progress of the resumed branch does not flow back into the advantage of the rewind decision. We would like to clarify that proactive rewind could be learned in an indirect way. ReTRAIL splits the credit: the execution objective learns whether and when to leave a branch implicitly, and the reflection objective learns from what happens after the rewind.

● Before the rewind: State-Anchored GRPO compares continuations that start from the same state, so the policy learns which states and actions lead to progress and which do not. The rewind request is taken by the same policy and draws on this learned sense of when a branch is unproductive.

● After the rewind: The resumed branch is compared with the abandoned one, and its progress rewards the reflection that accompanies every recovery. A rewind is therefore followed by a better-informed continuation.

This is reflected in the results. With proactive rewind enabled, ReTRAIL improves over its own forward execution on all four environments (T@1 − P@1 = +4.7, +10.9, +1.3 and +5.3), whereas GRPO loses 3.0, 1.4, 0.5 and 1.4 points (Table 1). We further analyzed how the trained policy uses proactive rewind at test time. Each proactive rewind involves two states: the state at which it was requested and the earlier state it returns to. We restored both and rolled out the policy three times from each under the same remaining budget: from the request state, the policy continues without rewind; from the rewind target, it continues without the reflection, so that the comparison isolates the choice of state. We average the final progress over the three rollouts and report the rewind win rate, i.e., the share of rewinds for which the rewind target has the higher average progress:

EnvironmentProactive rewinds per trialRewind win rate
MineSweeper1.4269.3%
Sokoban1.3872.1%
ALFWorld0.9862.2%
ScienceWorld1.5567.7%

The rewind target yields higher average progress in 62.2% to 72.1% of the cases. The policy thus tends to request a rewind where returning to the earlier state pays off.

We agree that studying how to provide a direct reward for the rewind decision is a promising future direction.

Revision. We have stated this limitation in the Limitations (Appendix A), added a discussion of how proactive rewind is learned to Appendix B, and added the analysis above to Appendix C.4 (Table 6).

W2: Execution efficiency and the cost analysis

We thank the reviewer for pointing this out. We agree that the original Figure 3(b) compared each method on its own successful trials, which are different instance subsets and leave out failed trials. We have therefore repeated the comparison over all test instances.

We count every action step of a trial, including all macro-attempts, steps on abandoned branches, and invalid actions, and the table reports the average over the full test set, so both methods are measured on the same instances and failed trials are included:

EnvironmentMethodT@3 successActions per trial, all trials
MineSweeperLaMer70.854.6
MineSweeperReTRAIL95.341.6
SokobanLaMer83.351.3
SokobanReTRAIL93.739.2
ALFWorldLaMer95.730.1
ALFWorldReTRAIL98.520.6
ScienceWorldLaMer60.246.5
ScienceWorldReTRAIL71.241.2

Over all trials, ReTRAIL takes fewer actions than LaMer on all four environments, for example 41.6 versus 54.6 on MineSweeper. After a failure, LaMer restarts from the initial state, whereas ReTRAIL resumes from a checkpoint and keeps the valid part of its progress.

● Reset-and-replay: MineSweeper and Sokoban restore snapshots and replay nothing. On ALFWorld and ScienceWorld, replayed actions involve no model call and are not included in the counts above. Their cost in time is small: a replayed action takes 0.18 s and 0.13 s, so replay adds about 0.4 s and 1.4 s to a trial, less than 2% of its wall-clock. The reset itself is the same operation that LaMer performs at the start of every episode.

Revision. We have replaced Figure 3(b) with the all-trial comparison and revised §5.2 accordingly, added this table to Appendix D.4 (Table 20), and reported the replay cost in Appendix D.3.

AuthorsOpenReview ↗

Response to Reviewer DsXE (Part 2/2)

Q1: Were the Table 2 variants retrained with the modified interfaces?

Yes. Each variant is trained from the same Qwen3-4B checkpoint with the modified interface, under the settings of Table 16, and evaluated with the same interface. We note that "w/o reflection" replaces the distilled reflection with the raw abandoned suffix, so it does not remove branch memory altogether.

Revision. We have stated this in the ablation analysis of §6.

Q2: Sampling configuration of the GiGPO objective in Table 3

It is identical to ReTRAIL. The variant only replaces State-Anchored GRPO with GiGPO as the execution objective; the retrospective-exploration rollout, the reflection objective and its weight are unchanged. Concretely, it samples 8 trials per task and update, each with at most 3 macro-attempts and a 60-call ceiling covering execution and reflection, for 300 updates with the optimizer settings of Table 16.

Revision. We have clarified in §6 what this variant retains and how it differs from the GiGPO baselines.

Thank you again for the careful reading of our credit assignment and cost analysis, which sharpened both. We hope the added analyses address your concerns, and we would be glad to discuss further.

AuthorsOpenReview ↗

Anticipating Your Response

Dear Reviewer DsXE, thank you again for your valuable feedback. As the discussion period ends in one day, we would like to check whether our response has addressed your concerns. We are happy to discuss any further questions and would appreciate it if you could reconsider your evaluation.

Thank you for the detailed response and revisions. The cost analysis the two of my questions have been adequately resolved.

However, my main concern regarding weakness 1 still partially remains. The authors confirm the execution return ends at the rewind request, which is learned indirectly. As I see this as one of the core contributions of this work (please correct me if I misunderstood), I decide to retain my score as borderline. That being said, I am open to seeing how other reviewers assess the work after the revision.

Comments on the paper

1 comment

Dear reviewers,

Thank you for your work reviewing this paper!

The authors have submitted their responses. If you haven't already, please acknowledge them in a comment and adjust your reviews as needed. The discussion period finishes on November 18.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.