acceptodds
Under review as a conference paper at ICLR 2027

REPO: Replay-Enabled Policy Optimization for Terminal Agents via Trajectory-Reconstructed Environments

15 views

Abstract

Realistic, executable environments are scarce for terminal-agent post-training, while recorded trajectories are plentiful but remain fixed demonstrations. We introduce REPO (Replay-Enabled Policy Optimization), which reconstructs an executable workspace from each trajectory by deterministic replay and agentic completion, then adds per-turn container snapshots for counterfactual rollouts. The recovered workspace supports intent recovery, new single- and cross-workspace tasks, and multi-round sessions, each paired with an executable verifier. It expands each workspace along breadth (cross-project dependencies) and depth (persistent user follow-ups). We further introduce PAPO, which anchors sibling rollouts at verifier-diagnosed pivotal turns, and CBAO, which branches at high-entropy turns to obtain local credit signals. On public trajectories, REPO yields k task-sufficient environments. SFT warm-start followed by PAPO+CBAO improves Qwen3.5-27B by 11.9 points on Terminal-Bench 2.1 and by 18.3 points in EvoCode-Bench v2 MT@4.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SFpHOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for terminal-agent post-training, which replays recorded trajectories to generate executable workspaces via per-turn container snapshots for counterfactual rollouts. The method generates a environments from different public trajectories, uses Qwen3.7-Max to produce solution rollouts, which are then used to fine-tune Qwen3.5-27B. The paper further proposed two modifications of GRPO for policy optimization: pivotal-anchored policy optimization (PAPO) and contrastive branch advantage optimization (CBAO). The model is evaluated on Terminal-Bench 2.1 and EvoCode-Bench v2, where it improves upon the base model.

I have flagged the paper because I think it violates the guidelines since it does not contain the required AI use statement while showing clear signs of AI usage. As indicated by the low confidence, I was assigned this submission outside my expertise, so I may not be able to properly assess the paper, but I'd recommend to reject the paper in its current form.

Critical strengths
  • Improves upon the base model
Critical weaknesses
  • My biggest concern is that the submission does not contain the required AI use statement, which violates the guidelines. Especially, since the gray text and captions in Figures 5 and 6 suggest that the plots were generated with generative AI and the shown numbers are not the real numbers.
  • The method requires a better model to generate a dataset, which is then used to fine-tune a weaker model, which may not be available. In general, the pipeline seems complex (5 stages) and there is no information about the additional computational effort. This raises the question why not just use a better model directly, e.g. Qwen3.7-Max? Does the relatively small improvement justify the additional effort? Biggest performance improvement comes from SFT while improvements from CBAO and PAPO are much smaller.
  • Some parts are hard to follow due to the (dense) writing style, for example the whole Appendix A is hard to follow containing sentences with language that is hard to comprehend like "The framework therefore benefits from stronger source agents while retaining an explicit information ceiling imposed by the traces".

Minor:

  • All papers are Arxiv preprints although some were accepted at conferences
  • There is already a paper called "RePO: Replay-Enhanced Policy Optimization" (https://arxiv.org/abs/2506.09340), so it may be better to use a different name/acronym to avoid confusion
  • ToC for Appendix: A after J, which probably only needs another compile
  • Also no reproducibility and ethics statement
Questions for authors
  • Can you please elaborate why the proposed method should be used instead of simply using a better model? The overall approach seems quite complex and I am unsure whether the relatively small improvements justify the additional effort.
  • How large is the additional computational effort of the method compared to the baselines?
  • Could you please clarify the actual contributions and position the paper more clearly towards related work?
Reviewer_Y8b1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces REPO, which reconstructs an executable workspace from each trajectory by deterministic replay and agentic completion, then adds per-turn container snapshots for counterfactual rollouts. Replay restores the earliest observed version of accessed files and holds out the agent's later edits, while an agentic completion step supplies missing project context without solving the target task. The recovered workspace supports intent recovery, new single- and cross-workspace tasks, and multi-round sessions, each paired with an executable verifier. It expands each workspace along breadth, meaning cross-project dependencies, and depth, meaning persistent user follow-ups. They further introduce PAPO, which anchors sibling rollouts at verifier-diagnosed pivotal turns, and CBAO, which branches at high-entropy turns to obtain local credit signals. They train Qwen3.5-27B on the reconstructed environments and evaluate it on Terminal-Bench and EvoCode-Bench v2. Controlled ablations aim to isolate the effects of reconstruction, task breadth, session depth, and local credit assignment.

Critical strengths
  • The reconstruction recipe is careful and reusable. It holds out post-edit file versions and limits the completion agent to filling in context without solving the task. A sufficiency judge filters workspaces, and verifiers must pass a red check on the unmodified workspace. Together, these give a credible way to turn existing logs into verifiable environments.

  • The appendix prompts are detailed enough to reproduce the pipeline, and the matched-record result that new environments help more than extra queries or extra solutions is a practical finding.

  • Per-turn snapshots fit sparse terminal rewards well. Restarting siblings from a restored mid-episode state gives a concrete counterfactual unit for turn-level advantages.

Critical weaknesses
  • The paper says its own central results are not real. Section 4.2 states that "the numerical values in the REPO rows are provisional placeholders for the first complete training run". The Table 2 caption says to "replace the displayed pilot values with the final repeated-run estimates", and Table 3 calls its entries "pilot values to be replaced". Table 7 says "Values are provisional placeholders", and Table 16 says the pilot values should be replaced "before submission". Figures 5 and 6 are labeled illustrative and "must be replaced by the final census".

  • PAPO and CBAO are presented as the paper's methodological contribution, yet they have no valid empirical support. The abstract still reports gains of 11.9 and 18.3 points as if they were measured.

  • Inside the submission, Table 19 reports 58.4 for a plain Single-WS plus Cross-WS SFT mix, which is higher than the final RL model. Table 20 reports 18.4 to 21.0 MT@4 for SFT variants, while Table 1 lists the SFT warm-start at 16.8. Table 23 also keeps a column named Terminal-Universe-27B. Read together, either the SFT baseline is understated, or Stage-2 adds nothing over a stronger SFT mix. Either way, the claim that local credit assignment drives the gain does not hold.

  • Some method claims are also weak. PAPO puts a teacher-written correction hint into the history of sibling rollouts, but this hint is absent at test time, so the updated tokens come from a privileged context.

Questions for authors

Clarifications that could change my assessment:

  • Has any full Stage-2 run finished? Please report the final numbers with variance over the six Terminal-Bench runs, along with RL steps, GPU hours, and rollout tokens per update for GRPO, PAPO, CBAO, and the joint model.

  • Which checkpoint initializes Stage 2? Why does REPO-SFT score 53.8 on Terminal-Bench 2.1, when the Table 19 SFT mix scores 58.4 and the public SFT-only model on the same 32.0k corpus scores 58.1?

  • How does PAPO perform without the teacher hint in the sibling context? How often does the teacher pick a pivot that sibling outcomes confirm, and what does the teacher call cost per failed rollout?

Reviewer_7c6POpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes REPO, a framework that reconstructs executable environments from recorded terminal-agent trajectories. It uses replay and agentic completion to recover workspaces, then generates additional tasks within and across workspaces and through multi-round interactions. The authors also introduce PAPO and CBAO, which use intermediate snapshots to train on alternative continuations. REPO is evaluated on Terminal-Bench and EvoCode-Bench v2.

Critical strengths
  • The one strength of REPO that I want to point out is that it provides a practical way to reuse recorded trajectories as executable training environments. This is useful since the same workspace can support new attempts and additional tasks without manually constructing each environment.
  • The proposed use of intermediate snapshots is meaningful for understanding where an agent fails. Comparing different continuations from the same point could provide better supervision than assigning the same reward to the whole trajectory. Furthermore, the multi-round setting considers whether the agent preserves earlier requirements while making new changes.
Critical weaknesses
  • The biggest concern I have is whether the reported experiments are complete. Section 4.2 explicitly calls the REPO results “provisional placeholders,” while Tables 2 and 3 state that the pilot values need replacing. However, the abstract presents the improvements as established results. This makes it difficult to understand which claims are supported by completed experiments. The authors need to clarify the status of these results and provide the measured outcomes, including uncertainty across runs.

  • The comparison against GRPO does not sufficiently isolate the contribution of the proposed methods. PAPO receives a correction hint from a stronger teacher, which itself could improve the continuation. Furthermore, although Section 4.3 claims matched rollout budgets, it does not give the total generated tokens or teacher-call costs for each method. Without these controls, it is hard to understand whether the reported gains come from better credit assignment, additional guidance, or differences in sampling cost. Comparing against random branch selection would also help establish whether the proposed selection rules are necessary.

  • The claim of exact state restoration needs more technical support. Section 3.1 defines the environment state to include files, the working directory, and process state, but the snapshot mechanism is described only as an incremental container commit. The paper does not explain how running processes, in-memory state, and mounted files are restored, or report a test of restoration correctness. This is important because the comparison between alternative continuations depends on them starting from the same execution state.

Questions for authors
  • Which results in Tables 1–3 come from completed experiments? Are the placeholder statements outdated editing notes, or are the corresponding runs still incomplete? Please clarify this for the reported improvements in the abstract as well.

  • How much of PAPO’s reported improvement remains when the teacher correction hint is removed? Alternatively, does PAPO still outperform GRPO when both receive the same teacher guidance?

  • What exactly is matched in the rollout-budget comparison? Please report total generated tokens, teacher calls, and execution costs for GRPO, PAPO, CBAO, and their combination.

  • Do verifier-selected pivots and entropy-selected branches outperform randomly selected turns under the same budget? How do the authors check whether the teacher-selected pivot is actually the earliest error?

  • What state does each snapshot preserve, and how is it restored? Can the authors demonstrate that replaying the same actions from a restored snapshot reproduces the original execution outcomes, including tasks with persistent processes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.