- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RePro trains an LLM agent to estimate progress during execution and reassess it after observing a complete trajectory and outcome. External-model-labelled successful trajectories initialise this behaviour; subsequent RL uses the agent's own retrospective estimates for progress-difference rewards, online-retrospective agreement and formatting objectives, alongside environment reward and GiGPO advantages. The evaluation covers WebShop, ALFWorld and Sokoban with several Qwen models. Reported WebShop average success gains over the progress-prompted baseline are 8.98, 11.57 and 5.82 percentage points. Warm-up controls and three-seed WebShop runs for the 1.5B and 3B models provide supporting evidence.
Critical strengths
-
The paper tests a concrete and consequential distinction: eliciting progress estimates is not enough, whereas teaching retrospective assessment and incorporating it into training can improve task success. The warm-up-plus-GiGPO control in Table 3 helps distinguish the full method from simply adding supervised demonstrations, and the 7B result without warm-up shows that the dependence on bootstrapping varies with model capability.
-
The empirical support extends across three environments and multiple model sizes. In particular, Figure 3 includes independent training seeds for two WebShop models, and Appendix A reports a fixed-tail three-seed comparison of 76.22 ± 1.20 versus 65.78 ± 8.68 for the 1.5B model. This is more persuasive than the best-checkpoint results alone.
Critical weaknesses
-
The progress diagnostics do not isolate improved online progress judgement. Table 4 and Figure 4 compare the baseline's online estimates with RePro's retrospective estimates on different trajectories. The latter explicitly receive the final outcome, so stronger separation of successful and failed episodes can arise from privileged outcome information. The paper acknowledges that these are retrospective signals, but the comparison cannot independently validate accurate intermediate credit or the transfer to online estimates. This limits the mechanistic interpretation of the success gains.
-
Main-table model selection and averaging are outcome-dependent. Appendix C.2 defines Best by the highest evaluation success rate and chooses the Avg window by moving backwards when performance falls. Section 5.1 refers to test samples, whereas the multi-seed discussion uses validation terminology. Without a clear separation of validation-based selection from a held-out final test, the main numerical gains may include selection optimism. The fixed-tail multi-seed evidence partly addresses robustness but is not the reporting protocol used throughout Tables 1–3.
Questions for authors
-
Clarification and targeted analysis: can you compare RePro's online and retrospective estimates, and the baseline estimates, on the same held-out trajectories or shared prefixes? Even a small existing-trace analysis would help distinguish hindsight outcome separation from better online state assessment.
-
Clarification: which split determines checkpoints and the convergence windows, and which split is used only for final evaluation? Please provide fixed-final or validation-selected test results from existing checkpoints for the main comparison, with training-seed uncertainty where available.
-
Clarification: Equation 4 defines progress before action a_t, but Appendix B's example and terminal boundary target refer to task completion at the last step. Which observation does each percentage describe in the implementation, and how is the final action aligned with its progress difference?
4 comments
Response to Reviewer yMtQ (Part 2)
Q1: Can you compare RePro’s online and retrospective estimates with baseline estimates on identical trajectories or shared prefixes? Even a small existing-trace analysis would help.
Thank you for this advice.Please see the General Response for the added online comparison and shared-prefix analysis using existing traces.
Q2: Which splits are used for checkpoint/window selection and final evaluation? Please provide fixed-final or validation-selected results with training-seed uncertainty where available.
Please see our response to W2 for split clarification, fixed-window results, and three-seed means and standard deviations.
Q3: Clarification: Equation 4 defines progress before action a_t, but Appendix B's example and terminal boundary target refer to task completion at the last step. Which observation does each percentage describe in the implementation, and how is the final action aligned with its progress difference?
Both the online estimate and retrospective estimate describe the state after observing and before taking . Retrospective progress uses the same step indexing. For , action receives shaping reward . The final action receives the environment outcome reward but no difference-based shaping ( in Eq. (10)).
In Appendix B, your estimate=90% is the original online estimate before the final action. The instruction "the last step should be near 100" guides retrospective reassessment, while the boundary reward softly encourages the final online estimate toward 1 on successful trajectories. These are soft targets and do not force predictions to equal 100%.
Thank you for the clarifications. The fixed-window results and three-seed comparisons substantially reduce my concern about outcome-dependent reporting, and the action/progress indexing is now clearer. The online and matched-prefix analyses are useful, but the small matched subset and temporal metrics may not independently establish progress accuracy. Perhaps retain this qualification. The evidence supports the method’s practical usefulness, and I remain at weak acceptance.
Thank you for reviewing the additional results and clarifications. We are glad they helped address your concerns about reporting and indexing. We will retain the qualification regarding progress accuracy.
Response to Reviewer yMtQ
We thank the reviewer for the feedback and address each point below.
Please see the General Response for the added online comparison and matched-prefix analysis. RePro’s online estimates show higher IntermDisc and larger success–failure gaps across all four temporal metrics on each policy’s own rollouts. On matched prefixes, RePro shows higher temporal R2 and lower plateau rates under identical action–observation histories.
Our claim concerns the usefulness of retrospective progress as auxiliary supervision, supported by improved agent performance. Independently validating intermediate credit accuracy versus ground truth is beyond the scope of this study.
Thank you for recognizing Figure 3. The original convergence-window average was intended to reduce evaluation fluctuations, but its rule makes selection outcome-dependent. Following verl-agent’s setup, training and evaluation tasks were separate, but "validation" and "test" referred to the same pool. We therefore report fixed-window results below.
For WebShop, Last 1 uses step 120 and Last 5 averages the five evaluation checkpoints from steps 100 to 120, consistent with Figure 3. For ALFWorld and Sokoban, we also use the final checkpoint and the final five evaluation checkpoints. Baseline denotes GiGPO + Meta Prompt.
Δ denotes RePro Last 5 minus RePro Avg in Table 1.
Three-seed results are reported as mean ± standard deviation. For Last 5, we first average within each run, then compute statistics across seeds. Δ compares the three-seed mean with Avg in Table 1.