acceptodds
Under review as a conference paper at ICLR 2027

Retrospective Progress-Aware Self-Refinement for LLM Agent Training

Abstract

Long-horizon LLM-based agents receive rich environmental observations during interaction, yet outcome rewards provide limited explicit supervision about how individual actions advance task completion. We investigate whether agents can turn this interaction evidence into useful training signals through retrospective progress assessment. A WebShop pilot shows that direct progress prompting reduces task success, whereas hindsight-annotated demonstrations improve it. We introduce RePro, Retrospective Progress-Aware Training, with a forward-then-reflect rollout: the agent estimates progress while acting, then reassesses each step using the completed trajectory and outcome. After warmup with externally generated demonstrations, policy optimization combines self-generated progress differences, online-retrospective alignment, and format rewards with environment feedback, requiring neither a separate process reward model nor ongoing teacher annotation. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to success rate gains.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM/AI agents, agent training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yMtQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RePro trains an LLM agent to estimate progress during execution and reassess it after observing a complete trajectory and outcome. External-model-labelled successful trajectories initialise this behaviour; subsequent RL uses the agent's own retrospective estimates for progress-difference rewards, online-retrospective agreement and formatting objectives, alongside environment reward and GiGPO advantages. The evaluation covers WebShop, ALFWorld and Sokoban with several Qwen models. Reported WebShop average success gains over the progress-prompted baseline are 8.98, 11.57 and 5.82 percentage points. Warm-up controls and three-seed WebShop runs for the 1.5B and 3B models provide supporting evidence.

Critical strengths
  1. The paper tests a concrete and consequential distinction: eliciting progress estimates is not enough, whereas teaching retrospective assessment and incorporating it into training can improve task success. The warm-up-plus-GiGPO control in Table 3 helps distinguish the full method from simply adding supervised demonstrations, and the 7B result without warm-up shows that the dependence on bootstrapping varies with model capability.

  2. The empirical support extends across three environments and multiple model sizes. In particular, Figure 3 includes independent training seeds for two WebShop models, and Appendix A reports a fixed-tail three-seed comparison of 76.22 ± 1.20 versus 65.78 ± 8.68 for the 1.5B model. This is more persuasive than the best-checkpoint results alone.

Critical weaknesses
  1. The progress diagnostics do not isolate improved online progress judgement. Table 4 and Figure 4 compare the baseline's online estimates with RePro's retrospective estimates on different trajectories. The latter explicitly receive the final outcome, so stronger separation of successful and failed episodes can arise from privileged outcome information. The paper acknowledges that these are retrospective signals, but the comparison cannot independently validate accurate intermediate credit or the transfer to online estimates. This limits the mechanistic interpretation of the success gains.

  2. Main-table model selection and averaging are outcome-dependent. Appendix C.2 defines Best by the highest evaluation success rate and chooses the Avg window by moving backwards when performance falls. Section 5.1 refers to test samples, whereas the multi-seed discussion uses validation terminology. Without a clear separation of validation-based selection from a held-out final test, the main numerical gains may include selection optimism. The fixed-tail multi-seed evidence partly addresses robustness but is not the reporting protocol used throughout Tables 1–3.

Questions for authors
  1. Clarification and targeted analysis: can you compare RePro's online and retrospective estimates, and the baseline estimates, on the same held-out trajectories or shared prefixes? Even a small existing-trace analysis would help distinguish hindsight outcome separation from better online state assessment.

  2. Clarification: which split determines checkpoints and the convergence windows, and which split is used only for final evaluation? Please provide fixed-final or validation-selected test results from existing checkpoints for the main comparison, with training-seed uncertainty where available.

  3. Clarification: Equation 4 defines progress before action a_t, but Appendix B's example and terminal boundary target refer to task completion at the last step. Which observation does each percentage describe in the implementation, and how is the final action aligned with its progress difference?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yMtQ

We thank the reviewer for the feedback and address each point below.

W1: The progress diagnostics do not isolate improved online assessment or validate intermediate credit accuracy.

Please see the General Response for the added online comparison and matched-prefix analysis. RePro’s online estimates show higher IntermDisc and larger success–failure gaps across all four temporal metrics on each policy’s own rollouts. On matched prefixes, RePro shows higher temporal R2R^2 and lower plateau rates under identical action–observation histories.

Our claim concerns the usefulness of retrospective progress as auxiliary supervision, supported by improved agent performance. Independently validating intermediate credit accuracy versus ground truth is beyond the scope of this study.

W2: Outcome-dependent checkpoint window selection and unclear validation–test separation. The fixed-tail multi-seed evidence partly addresses robustness.

Thank you for recognizing Figure 3. The original convergence-window average was intended to reduce evaluation fluctuations, but its rule makes selection outcome-dependent. Following verl-agent’s setup, training and evaluation tasks were separate, but "validation" and "test" referred to the same pool. We therefore report fixed-window results below.

For WebShop, Last 1 uses step 120 and Last 5 averages the five evaluation checkpoints from steps 100 to 120, consistent with Figure 3. For ALFWorld and Sokoban, we also use the final checkpoint and the final five evaluation checkpoints. Baseline denotes GiGPO + Meta Prompt.

Δ\Delta denotes RePro Last 5 minus RePro Avg in Table 1.

TaskModelMetricBaseline Last 1RePro Last 1Baseline Last 5RePro Last 5Δ\Delta
WebShopQwen2.5-1.5BSR69.1480.0864.8476.80+0.67
Score87.4490.4484.0188.22+0.66
WebShopQwen2.5-3BSR63.6777.7366.1779.84+0.11
Score83.1791.6782.9491.05+0.30
WebShopQwen2.5-7BSR78.9180.8674.1481.41+0.94
Score85.9389.8784.7790.04+0.91
ALFWorldQwen2.5-1.5BSR87.1192.9785.3993.83−2.19
ALFWorldQwen2.5-7BSR92.1998.0592.7398.83+1.29
SokobanQwen3-4BSR78.1370.3177.6680.00−1.64

Three-seed results are reported as mean ± standard deviation. For Last 5, we first average within each run, then compute statistics across seeds. Δ\Delta compares the three-seed mean with Avg in Table 1.

ModelMetricBaseline Last 1RePro Last 1Baseline Last 5RePro Last 5Δ\Delta
Qwen2.5-1.5BSR70.44 ± 7.0577.21 ± 5.3065.78 ± 8.6876.22 ± 1.20+0.09
Score85.58 ± 2.9987.43 ± 6.8882.03 ± 7.1587.84 ± 2.00+0.28
Qwen2.5-3BSR65.49 ± 1.6379.17 ± 1.2667.45 ± 1.1478.96 ± 1.21−0.77
Score81.12 ± 2.9590.79 ± 0.7982.72 ± 0.9090.60 ± 0.47−0.15
  • RePro achieves higher Last 5 success rates in all six main-table settings and both multi-seed comparisons. RePro's Last 1 is also higher on WebShop and ALFWorld, whereas Sokoban exhibits a performance drop but only for final step (Last 5 SR: 80.47、85.16、87.50、76.56、70.31).
  • RePro’s Last 5 results remain close to the original Table 1 averages: success rates differ by at most 2.19 percentage points across the 6 settings, and WebShop scores by less than 1 point. The gains over the baseline persist under these fixed-window evaluations.
AuthorsOpenReview ↗

Response to Reviewer yMtQ (Part 2)

Q1: Can you compare RePro’s online and retrospective estimates with baseline estimates on identical trajectories or shared prefixes? Even a small existing-trace analysis would help.

Thank you for this advice.Please see the General Response for the added online comparison and shared-prefix analysis using existing traces.

Q2: Which splits are used for checkpoint/window selection and final evaluation? Please provide fixed-final or validation-selected results with training-seed uncertainty where available.

Please see our response to W2 for split clarification, fixed-window results, and three-seed means and standard deviations.

Q3: Clarification: Equation 4 defines progress before action a_t, but Appendix B's example and terminal boundary target refer to task completion at the last step. Which observation does each percentage describe in the implementation, and how is the final action aligned with its progress difference?

Both the online estimate p~t\tilde p_t and retrospective estimate ptp_t describe the state after observing oto_t and before taking ata_t. Retrospective progress uses the same step indexing. For t<Tt<T, action ata_t receives shaping reward β(pt+1−pt)\beta(p_{t+1}-p_t). The final action receives the environment outcome reward but no difference-based shaping (rp(T)=0r_p(T)=0 in Eq. (10)).

In Appendix B, your estimate=90% is the original online estimate before the final action. The instruction "the last step should be near 100" guides retrospective reassessment, while the boundary reward softly encourages the final online estimate toward 1 on successful trajectories. These are soft targets and do not force predictions to equal 100%.

Thank you for the clarifications. The fixed-window results and three-seed comparisons substantially reduce my concern about outcome-dependent reporting, and the action/progress indexing is now clearer. The online and matched-prefix analyses are useful, but the small matched subset and temporal metrics may not independently establish progress accuracy. Perhaps retain this qualification. The evidence supports the method’s practical usefulness, and I remain at weak acceptance.

Thank you for reviewing the additional results and clarifications. We are glad they helped address your concerns about reporting and indexing. We will retain the qualification regarding progress accuracy.

Reviewer_eszkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RePro, a two-stage framework that trains LLM agents using retrospective progress assessments. After supervised warm-up with externally annotated demonstrations, the agent reassesses completed trajectories and uses these estimates for reward shaping and online–retrospective alignment. Experiments on WebShop, ALFWorld, and Sokoban demonstrate improvements over the evaluated RL baselines.

Critical strengths
  1. The idea of converting completed interaction trajectories into self-generated progress supervision is useful for agent training with sparse outcome rewards. It avoids a separate process reward model and ongoing teacher annotation after warm-up.

  2. The evaluation covers three environments and multiple model sizes. WebShop results show substantial improvements, with additional evidence across model sizes.

  3. The warmup and training ablations provide useful evidence beyond the main comparisons. The overall workflow is easy to understand, and the qualitative cases acknowledge that progress assessment can remain confidently incorrect.

Critical weaknesses
  1. The technical novelty looks somewhat limited. RePro combines retrospective reflection, progress-difference rewards, and GiGPO, but lacks a key point. Discussions of related progress-supervision methods (see some references below) would better establish its contribution.
  • Wang et al., SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution. arXiv'2025.
  • Chai et al., Progra: Progress-Aware Reinforcement Learning for Multi-Turn Function Calling. ACL Findings'2026.
  1. The experiments mainly compare against generic RL baselines and internal variants. Direct comparisons with progress-related RL methods are missing.

  2. The ablations do not isolate the main reward components. Table 3 is informative about warm-up, but does not separately remove progress shaping and correlation rewards. Their individual contributions and potential interactions remain unclear. Table 4 compares the baseline’s online estimates with RePro’s retrospective estimates on different trajectories. Since RePro observes the final outcome, stronger outcome discrimination does not establish a more accurate online progress assessment.

  3. The overall paper writing needs improvement, especially in logical coherence and human readability. Some implementation details are not clear, which weakens clarity and reproducibility.

Questions for authors
  1. Could you compare both methods’ online estimates on identical held-out trajectories, and separately compare their retrospective estimates?

  2. Could you provide component ablations for progress shaping, pointwise alignment, and correlation rewards? What prevents agreement rewards from favoring mutually consistent but inaccurate estimates?

  3. Eq. (10) makes the undiscounted shaping reward telescope to (\beta(p_T-p_1)). How much benefit comes from its intermediate timing under discounted step advantages, versus simply adding this endpoint reward?

  4. Please clarify progress indexing around the terminal action, the reward scale, and whether retrospective-generation tokens receive any training loss during RePro-PO. Also, please report the warmup dataset size and how often retrospective sequences require truncation or padding.

4 comments
AuthorsOpenReview ↗

Response to Reviewer eszk

We thank the reviewer for the feedback and address each point below.

W1: Novelty: Comparison with related progress-supervision methods.

The agent assesses stepwise progress from completed rollouts to construct shaping rewards and alignment targets for online estimates, without a separately trained progress estimator. Progress assessment is both the content of retrospective reflection and the means by which reflection informs learning. It is not a second component added to an independent reflection-training objective.

The differences from the two papers the reviewer mentioned are:

  • SPA-RL trains a separate progress estimator to assign stepwise contributions whose sum predicts the terminal reward. With a different motivation, RePro instead uses the acting agent’s own retrospective assessments.
  • Progra learns a structured online reasoning pattern: summarizing interaction history and planning subsequent actions. The new reasoning pattern is gained from a GPT-generated, verified demonstrations, followed by task-reward optimization. Differently, without this verified strong teacher's supervision, RePro uses self-generated retrospective progress assessments as auxiliary training signals.

W2: Comparisons with progress-related RL methods are missing.

We compare RePro with published results from recent progress-related RL methods using the same backbone model sizes and benchmarks, while noting differences in methods and setups. RePro’s average results are better than or comparable to the reported results.

ModelMethodWebShop SRWebShop ScoreALFWorld SR
Qwen2.5-1.5BBEACON [1]75.686.191.4
GiGPO + ProGPO [2]72.4 ± 0.685.6 ± 0.591.4 ± 1.6
GiGPO + ProAS [3]66.7 ± 3.287.5 ± 1.791.9 ± 0.5
RePro (Avg)76.1387.5696.02
RePro (Best)81.6491.7299.22
RePro (3 seeds in Fig. 3)76.22 ± 1.2087.84 ± 2.00—
Qwen2.5-7BBEACON [1]79.787.794.5
GiGPO + ProGPO [2]76.2 ± 1.185.9 ± 0.292.5 ± 0.9
GiGPO + ProAS [3]70.1 ± 2.389.1 ± 2.794.3 ± 2.4
RePro (Avg)80.4789.1397.54
RePro (Best)84.3891.18100.00

BEACON [1] uses predefined, task/domain-specific detection rules to construct rewards. ProGPO [2] uses first-visit observation coverage within a trajectory as a progress. ProAS [3] identifies valid transitions to previously unvisited states in a reference graph constructed from successful trajectories. RePro instead uses the acting agent’s own retrospective assessments.

[1] Milestone-Guided Policy Learning for Long-Horizon Language Agents.

[2] Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks.

[3] ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning.

Our existing controls complement these comparisons: L1, L2, and the variants in Table 3 evaluate online progress signals, while Warmup + GiGPO evaluates SFT initialization without retrospective-progress supervision during RL.

W3: Reward-component contributions and interactions are unclear. Table 4 does not isolate improvements in online assessment.

  1. Additional component ablations for Table 3. We ablate the reward components discussed here and in Question 2, training each variant for 120 steps.

    MetricSRScore
    RePro81.6491.72
    w/o shaping78.5291.46
    w/o correlation79.3091.98
    w/o alignment77.3490.89
    w/o shaping, correlation, alignment, format51.1763.02

    RePro uses these auxiliary rewards in two ways: shaping modifies stepwise rewards, while alignment and correlation encourage agreement between online and retrospective progress estimates. Removing each of these three components individually lowers success rate, while jointly removing all leads to substantial late-training degradation.

  2. Additional analysis for Table 4. Please see the General Response for online progress comparisons and analysis under matched action–observation prefixes.

W4: The overall paper writing needs improvement.

We will revise the paper to improve readability and incorporate the implementation clarifications provided in our responses.

AuthorsOpenReview ↗

Response to Reviewer eszk (Part 2)

Q1: Can both methods’ online and retrospective estimates be compared under identical histories?

Please see the General Response for the online comparison and for comparison on shared prefixes. The baseline does not generate retrospective estimates, so a retrospective comparison between the two methods is not included.

Q2: Can you ablate the reward components, and what prevents consistent but inaccurate estimates?

Please see W3 for the component ablations. Agreement rewards are accompanied by format and boundary constraints that anchor progress to the task’s start and terminal outcome. Empirically, the outcome separation and temporal dynamics reported in Table 4 and our General Response provide evidence against both estimates collapsing together.

Q3: How much benefit comes from intermediate shaping versus an endpoint reward?

The undiscounted shaping rewards sum to β(pT−p1)\beta(p_T-p_1). Under GiGPO’s discounted step-level returns ( γ=0.95\gamma=0.95), however, intermediate reward timing affects step-level advantages, so moving this total to the terminal step produces different training. Therefore, the contribution of intermediate reward timing has not been separately quantified.

Q4: Clarify terminal-step indexing, reward scales, retrospective-token loss, warmup data size, and truncation/padding frequency.

  1. Progress indexing. Both online and retrospective estimates are indexed to the state after observing oto_t and before taking ata_t. For t<Tt<T, action ata_t receives shaping reward β(pt+1−pt)\beta(p_{t+1}-p_t). The final action receives the environment outcome reward but no difference-based shaping (rp(T)=0r_p(T)=0, Eq. (10)). The terminal boundary constraint softly encourages high progress on successful trajectories without overwriting the predicted value.

  2. Reward scale and training loss. Progress percentages are normalized to [0,1][0,1]. WebShop and ALFWorld provide an environment reward of 10 for success and 0 otherwise. Sokoban uses a completion bonus of +10, box-on/off-target rewards of +1/−1, and a per-step penalty of −0.1. Other setups are listed in the appendix. Retrospective-generation tokens receive no direct training loss during RePro-PO. Their outputs are converted into rewards used to optimize forward-response tokens, including reasoning, online progress estimates, and actions.

  3. Warmup dataset size. The WebShop warmup pools contain 52793, 77861, and 40293 trajectory-level examples for the 1.5B, 3B, and 7B models, respectively. We use an 80/20 training–validation split.

  4. Retrospective sequence truncation and padding. Across all saved WebShop evaluation checkpoints, truncation and padding are infrequent. Excess progress values are truncated while shorter sequences are padded with their last value.

    ModelTruncatedPadded
    Qwen2.5-1.5B1.78%2.10%
    Qwen2.5-3B0.94%0.13%
    Qwen2.5-7B0.05%0.01%

Thanks for the detailed responses and additional experiments. I still think the novelty of introducing progress supervision is not as sufficient as I expected, since the differences compared to existing work are narrow. The experimental results largely addressed my concerns about how the performance gains are achieved. Considering the overall work now, I'd like to increase my score to 3.

Thank you very much for reviewing our responses. We are glad they addressed some of your concerns. We sincerely appreciate your time and thoughtful feedback.

Reviewer_bmx5OpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces RePro, a training framework that turns retrospective progress assessments into supervision for LLM agents. The agent estimates progress while acting, then reassesses intermediate steps using the completed trajectory and final outcome. Following supervised warmup with externally annotated demonstrations, reinforcement learning combines environment rewards with retrospective progress shaping and online–retrospective alignment. Experiments on WebShop, ALFWorld, and Sokoban with multiple Qwen models report improved task success. The evaluation includes a prompting pilot study, training ablations, and analyses of the generated progress estimates.

Critical strengths
  1. A practical approach to generating intermediate training signals. RePro uses the acting model's retrospective assessments to supplement environment rewards, without a separate process reward model or continued teacher annotation after warmup. This offers a useful way to extract additional supervision from completed interactions.

  2. Performance gains across multiple tasks and model scales. RePro improves task success over the progress-prompted RL baseline across all three benchmarks. The WebShop results also include three-seed runs for the 1.5B and 3B models. These results provide substantive evidence that the training procedure is useful beyond a single model–environment setting.

Critical weaknesses
  1. The incremental contribution of progress-aware reflection over existing self-reflection training is unclear.

Early Experience [1] already uses self-reflection on actions and their consequences to construct training supervision. RePro closely couples retrospective reflection with progress estimation for reward construction and online alignment, but the basis for quantifying progress and its additional training value are not sufficiently explained. The experiments do not separate the effects of reflection itself from those of progress modeling, leaving the substantive contribution beyond existing reflection-based training unclear.

  1. The progress-quality analysis does not distinguish improved assessment ability from the effects of additional information.

Section 6.2 compares the baseline's online estimates with RePro's retrospective estimates generated with knowledge of the final outcome, and the two are evaluated on different trajectories. Greater score separation between successful and failed trajectories could therefore reflect better progress assessment, but also hindsight information and differences between the trajectories. This limits the analysis's support for improved progress awareness.

Reference

[1] Kai Zhang et al. Agent Learning via Early Experience. arXiv:2510.08558, 2025. https://arxiv.org/abs/2510.08558

Questions for authors
  1. Compared with the self-reflection training in other related works, what independent value does RePro gain by combining retrospective reflection with progress estimation? What useful training information does progress modeling add beyond reflection on task states and action outcomes?

  2. Table 4 compares the baseline's online estimates, based on the history available at each step, with RePro's retrospective estimates, which use the full trajectory and final outcome. The two methods are also evaluated on different trajectories. Is there evidence that RePro improves online progress assessment when both methods are given the same interaction history, without access to future observations or outcomes?

  3. How do the authors view RePro's applicability to longer tasks with less clearly defined stages that require exploration, backtracking, or replanning? What are the main conditions under which retrospective progress would continue to provide useful supervision in these settings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer bmx5

We thank the reviewer for the feedback and address each point below.

W1: The contribution of progress-aware reflection over existing self-reflection training (Early Experience [1]). Separating reflection from progress modeling.

In RePro, progress assessment is both the content of retrospective reflection and the means by which reflection informs learning. It is not a second component added to an independent reflection-training objective.

  1. Differences from existing self-reflection training. A common use of reflection is an attempt–reflect–retry loop, where reflection guides another execution. uses the agent’s own retrospective progress assessments to construct stepwise rewards and align online estimates reasoning. Here, retrospection describes how the assessment is performed, while progress describes what is assessed.

    Specifically, Early Experience [1] compares agent and expert actions, explains why the expert action is preferable, and uses these explanations and expert actions as supervised targets. RePro instead assesses task completion at successive states using the completed episode and converts these assessments into auxiliary rewards.

  2. Progress quantification and training role. Progress percentages are model-generated estimates of task completion based on the finished trajectory. Differences between successive estimates provide shaping rewards, while retrospective estimates serve as references for online–retrospective agreement rewards.

  3. What the existing experiments separate. What the existing experiments separate. Warmup + GiGPO receives the same SFT initialization for retrospective reflection without retrospective-progress reward during RL. L1 assesses online progress estimates without hindsight. Neither matches RePro’s gains, indicating that reflection initialization or online progress assessment alone is insufficient to reproduce the benefits of RePro’s retrospective-progress training.

W2: Section 6.2 compares the baseline's online estimates with RePro's retrospective estimates on different trajectories.

We add online comparisons and a shared-prefix analysis to address the differences in available information and trajectories. Please see the General Response.

Questions 1: Comparison with the self-reflection training.

The progress assessment is both what retrospective reflection produces and how it informs learning, rather than an additional component attached to an independent reflection-training objective. Please see W1 for its role, differences from prior work, and supporting ablations.

Question 2: Whether RePro improves online assessment on the same history.

Please see our General Response for the added online comparison and matched-prefix analysis. The added results show that RePro’s online estimates better distinguish successful from failed trajectories across IntermDisc and all four metrics. The matched-prefix analysis further shows improved temporal dynamics under identical action–observation histories.

AuthorsOpenReview ↗

Response to Reviewer bmx5 (Part 2)

Question 3: Applying RePro to longer tasks involving exploration, backtracking, or replanning, what conditions are required?

  1. RePro is designed to self-assess progress without predefined task stages or stepwise ground-truth labels. It uses the completed trajectory and outcome to estimate task progress at each step, allowing plateaus during unproductive interactions and downward revisions when setbacks occur. Applying RePro to longer tasks requires no additional assumptions about task structure.

  2. Our current results provide relevant evidence. RePro’s 7B online estimates decline more frequently on failed trajectories than on successful ones, with a larger gap in retrospective estimates (see the General Response). Each row shows the action, a brief action description, retrospective progress, and an excerpt from the rationale from <think> output.

Example 1: Decline and recovery
Task: Purchase orange, XX-large men's pants under $50.

StepProgressActionAction descriptionModel-generated rationale
10%search[men's pants long sleeve XX-large orange price lower than 50.00]Search requirements—
220%click[b09q67h373]Open the 1st candidate—
340%click[xx-large]Select XX-large—
440%click[xx-large]Repeat the selection"possibly redundant, but maybe confirms size, still 40%."
540%click[d-navy]Select d-navy rather than the requested orange"wrong color, not helpful"
640%click[d-navy]Repeat d-navy selection—
730%click[back to search]Return to search"Back to search — resets progress, maybe 30% (some items seen)."
850%search[men's pants color orange size xx-large color orange price lower than 50.00]Search again with a revised query"New search with more details — better targeting"
970%click[b099231v35]Open another candidate—
1080%click[orange]Select orange"correct color"
1190%click[xx-large]Select XX-large"correct size"
12100%click[buy now]Purchase"purchase completed"

Example 2: Exploration across candidate products
Task: Purchase a black wall-mounted mount under $40.

StepProgressActionAction descriptionModel-generated rationale
15%search[wall-mounted mounts black]Search requirements—
220%click[b09pfwvxvs]Open the 1st candidate"viewing a product but not yet confirmed it's black or ready to buy."
320%click[back to search]Return to search"still 20% because we didn't complete anything new."
420%search[wall-mounted mounts black]Repeat the search—
530%click[b09qyyn4s4]Open the 2nd candidate"we're looking at another product."
630%click[back to search]Return to search—
730%search[wall-mounted mounts black]Repeat the search—
850%click[b0924pklq4]Open the 3rd candidate"we're on a product page that likely matches."
980%click[black]Select black"select the correct color variant"
10100%click[buy now]Purchase"purchase completed"

Thank you for the clarifications and additional analyses. I understand how RePro differs from Early Experience and appreciate the new online results. However, I am still leaning toward keeping my current score.

For Q3, the examples show that progress can plateau, decline, and recover. My concern is how overall progress can be meaningfully assessed in longer tasks with many interdependent subgoals. In a SWE task, an agent may modify several modules, undo changes, or discover new dependencies. Reflection can explain why an unsuccessful attempt or a rollback was useful. But how should these judgments translate into changes in overall completion? Allowing progress to decline does not resolve how different local changes should be evaluated on the same scale.

This is why I raised W1. Early Experience uses reflection to explain why an expert action is preferable to an alternative, without estimating overall completion. When progress is hard to quantify in a complex task, RePro may still rely on reflection to explain what an action achieved, what went wrong, or why a rollback was useful. Although the training mechanisms differ, this role of reflection does not seem substantially different from prior work. What additional value does the overall progress estimate provide beyond that reflective understanding?

I would also appreciate the authors’ thoughts on when this representation is useful and where it may fall short. The SWE example illustrates this concern; I am not asking for additional experiments.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: Online progress assessment and matched-history analysis

We thank the reviewers for their feedback. A shared concern is whether RePro improves online progress estimation. We extend Section 6.2 with RePro’s online estimates and a matched-prefix analysis to distinguish improvements in online assessment from the effects of hindsight.

1. Improved progress estimation comparison. We extend the comparison to baseline online, RePro online, and RePro retrospective progress and recompute all metrics over the final 5 evaluation checkpoints. RePro’s online and retrospective estimates share the same trajectories, while the baseline is evaluated on its own rollouts.

MetricOutcome1.5B Baseline1.5B RePro online1.5B RePro retro3B Baseline3B RePro online3B RePro retro7B Baseline7B RePro online7B RePro retro
IntermDisc—−0.076.2110.93−0.455.4425.81−1.183.5011.99
Temporal R2R^2Success98.2791.2494.830.8798.4197.3149.3388.9195.22
Failure98.6384.5389.360.3795.9759.9147.0768.7168.17
Mono.Success99.9797.9898.82100.0099.99100.0099.9999.9899.96
Failure100.0093.3197.60100.0099.6090.99100.0098.1993.64
DeclineSuccess0.117.884.750.000.100.000.100.200.00
Failure0.0022.7710.120.001.9036.600.0024.5330.55
PlateauSuccess0.3412.163.5799.160.000.94100.000.000.10
Failure0.2717.1315.9199.692.287.07100.003.086.23
  • Outcome separation. RePro’s online IntermDisc exceeds the baseline’s on their respective rollouts, while retrospective assessment further increases IntermDisc on the same RePro trajectories, supporting the use of hindsight to construct training signals.

  • Repro's online estimates exhibit meaningful structure compared with baselines. Compared with the baseline, the success–failure gaps are larger across all four progress dynamic metrics for all three model sizes. For 3B and 7B, RePro also substantially reduces stagnation, lowering plateau rates.

2. Shared trace analysis. We futher matched saved WebShop evaluation traces, retained the longest identical action–observation prefix, and required at least three valid steps.

Table 1: Frequency

ModelMatched prefixMatched trajectoryCheckpoint distribution
1.5B7 (0.09%)1 (0.01%)10–35
3B4 (0.08%)0 (0.00%)5–30
7B69 (0.85%)36 (0.45%)0–45: 18; 50–95: 40; 100–105: 11

Table 2: Analysis

MetricBaseline onlineRePro online
Temporal R2R^2 (×100)65.1896.11
Plateau (%)98.463.08

Table 3: Case study.

(1) A men’s polo in military green, size 3x-large, priced below $60. Qwen2.5-7B, Checkpoint 100.

StepBaseline onlineRePro onlineAction
10%0%search[...]
220%30%click[b01mg1ltms]
320%60%click[3x-large]
420%70%click[military green]
520%80%click[buy now]

(2) A women’s top in b21-a, size medium, priced below $30. Qwen2.5-1.5B, Checkpoint 140.

StepBaseline onlineRePro onlineAction
10%0%search[...]
220%20%click[b09npml43m]
340%40%click[medium]
460%70%click[b21-a]
580%70%click[medium] — repeated selection
690%70%click[buy now]
  • As shown in Table 1, identical action–observation prefixes are rare and occur mainly at early and intermediate training checkpoints.
  • Given the limited matches for 1.5B and 3B, Table 2 reports results for the 7B matched-prefix subset using the metrics defined in Table 4. RePro exhibits higher temporal R2R^2 and a lower plateau rate, indicating reduced stagnation in online progress estimates under matched environment histories.
  • The 3B and 7B examples primarily illustrate reduced stagnation in online progress estimates. In the 1.5B example, the baseline’s estimates keep increasing, whereas RePro’s estimates are more conservative: the baseline’s estimate rises from 80% to 90% after a repeated size selection, while RePro’s remains at 70%.

Table 4 originally characterizes the baseline’s online and RePro’s retrospective progress signals on their respective rollouts. It examines the signals used to improve the policy, consistent with our pilot-study motivation that hindsight can supplement online assessment, rather than isolating estimator quality on fixed trajectories. We have updated the Section 6.2 and Table 4, with cases in Appendix F.

AuthorsOpenReview ↗

Rebuttal Summary

We thank all three reviewers and the AC for their time, careful consideration, and contributions to the review process. We especially thank the reviewers for their active participation in the discussion. We are encouraged by the reviewers’ recognition of the strengths of our work. Their feedback helped us improve the paper through the following additions and clarifications:

  • Online progress comparisons and shared-prefix analysis. (All reviewers)
  • RePro’s contribution and distinctions from related methods. (bmx5 and eszk)
  • Reward-component ablations. (Reviewer eszk)
  • Outcome-independent evaluation windows and three-seed statistics. (Reviewer yMtQ)
  • Case studies demonstrating exploration and backtracking. (Reviewer bmx5)
  • Implementation details. (Reviewer eszk and yMtQ)

All three reviewers responded to our rebuttal, acknowledged that the clarifications and additional results addressed some important concerns, and now give the paper a score of 3. Lastly, we thank the reviewers and AC for their time and effort again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.