- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper treats different rollouts by the same agent as a set of mutually instructive experiences. It uses progress differences among failed trajectories to determine the learning frontier, then selects local demonstrations from successful or failed attempts that cross the next frontier, driving multi-round self-evolution through more targeted SFT.
Critical strengths
-
Where to learn and how far to learn are distinct questions. Rather than merely changing the optimization objective, the paper focuses on training-data construction, which is practically relevant for interactive agent self-training.
-
It requires neither an additional trained PRM nor a stronger teacher providing revised trajectories each round. The complexity lies mainly in offline state matching and segment selection.
-
The experiments compare full successful trajectories, next-action supervision, full continuation, successful-donor-only selection, and removal of position weighting.
Critical weaknesses
W1. The relative frontier may not reflect actual capability bottlenecks. The method requires matching each reference state in order without skipping, so an early mismatch prevents later, closer-to-success states from counting as progress. The frontier thus measures ordered path consistency with one reference rather than distance to task completion. This affects the core learning signal, not just future extensions.
Please compare
- single vs. multiple references
- strict vs. skip-allowed matching
- ordered-path vs. task-relevant state progress
and report sensitivity to reference choice.
W2. The Agent-R baseline omits its core mechanisms (Appendix A.2). On ALFWorld, Agent-R is implemented by collecting successful MCTS paths as fine-tuning data. However, Agent-R's key design involves the actor selecting revision points, constructing revised trajectories that incorporate failure experience, adding revision signals, and learning to recover from errors. Training only on successful MCTS paths DOES NOE adequately represent Agent-R.
Please implement the full method or clarify this deviation when interpreting the comparison.
W3. The closest strong baselines are missing (§5.1). The main experiments lack of
- GiGPO
- SALT
- full UI-Voyager-style fork correction
- milestone-based RL methods.
The paper should include at least one credible modern step-level RL baseline and one faithful matched-state correction baseline. The next-action ablation is not equivalent to full UI-Voyager, as it uses the paper's own frontier starting points rather than UI-Voyager's fork detection and transition alignment. The current results thus show gains over these specific trajectory-level schemes, but do not establish that the local learning mechanism outperforms the most relevant prior methods.
W4. The gain from failed donors conflates coverage and segment quality. Relative to successful-donor-only selection, the full method improves by 3.36 points on ALFWorld and 3.60 points on WebShop. This shows that allowing failed donors makes the full scheme more effective, but not that segments from failed donors are of higher quality than those from successful donors. Including failed donors may also
- increase learner coverage
- the number of usable samples
- path diversity
- the proportion of short segments.
Please compare the two settings with matched coverage, sample count, and segment length.
W5. What is the precise definition of state matching? When a donor segment is replayed from the learner's state, what fraction of replays reach the target milestone?
W6. The reported statistics appear inconsistent with the stated evaluation protocol. Table A.3 specifies four decoding seeds over 134 ALFWorld tasks, but ReSTEM’s overall result of 96.64 ± 0.53 in Table 1 cannot arise from four per-seed success rates under either the conventional sample or population standard deviation. Instead, it matches the sample statistics of two runs with 129 and 130 successes.
Please clarify the actual number of evaluation seeds, provide per-seed results, and reconcile the tables. Moreover, decoding-only repetitions do not quantify training variability, so independent training runs are needed to substantiate robustness.
Questions for authors
See Weakness
2 comments
Response to Reviewer 2Wp1 - 2
2Wp1-W3 Missing Step-Level RL and Full UI-Voyager-Style Baselines
Reply to 2Wp1-W3
Our main comparisons focus on SFT baselines to evaluate how the choice of training content affects self-evolution. RL also changes credit assignment and policy optimization. Following your suggestion, we evaluated GiGPO, which uses its own rollouts and environment feedback without external training data or process supervision.
We adapted GiGPO from full-parameter training to LoRA, using the same Qwen2.5-7B-Instruct base model and rollout budgets as our method, with 4,800 on ALFWorld and 8,000 on WebShop. The ALFWorld budget is one quarter of GiGPO's published 19,200 rollouts. We retained its native prompt and interaction history.
The original learning rate of may not be optimal for LoRA. After an initial run at this rate, we tested , , and on ALFWorld and selected the best-performing rate, , which we also used on WebShop. Our method achieved higher success on both benchmarks under these settings.
Table 2Wp1-W3. Success rates under matched training rollout budgets.
| Method | ALFWorld | WebShop |
|---|---|---|
| GiGPO (LoRA) | 66.60% ± 2.47 | 7.15% ± 1.08 |
| Our method | 97.76% ± 1.61 | 71.00% ± 0.52 |
We have not yet included a full UI-Voyager comparison. Its original screenshot-based matching requires adaptation for our text environments, including fork detection and transition alignment. Our next-action ablation holds the PRFL frontier fixed and compares learning only the donor’s next action with learning the segment up to the next milestone. This tests supervision scope and does not reproduce the full UI-Voyager pipeline.
2Wp1-W4 What adding failed donors establishes
Reply to 2Wp1-W4
We do not claim that failed donors provide higher-quality segments than successful donors in general. We include them to give each learner more routes to the same milestone. The benefits you mentioned, including a wider choice of paths and the possibility of covering more learners, are precisely why we expand the donor pool.
Following your suggestion, we compared training coverage across all rounds (Table 2Wp1-W4). Relative to using successful donors alone, coverage was slightly lower on ALFWorld (89.2% vs. 90.2%) and higher on WebShop (76.6% vs. 69.9%). The higher coverage on WebShop illustrates one intended benefit of including failed donors: a larger share of eligible learners contributed training examples.
Table 2Wp1-W4. Training coverage in the donor-source ablation. The numerator counts failed learners used for training, and the denominator counts eligible failed learners in the corresponding run.
| Environment | Successful donors only | Our method |
|---|---|---|
| ALFWorld | 716/794 (90.2%) | 696/780 (89.2%) |
| WebShop | 1,306/1,869 (69.9%) | 1,316/1,718 (76.6%) |
2Wp1-W5 Exact state matching and replay from the learner’s state
Reply to 2Wp1-W5
1. Replay success. Donor segments reached the next milestone in nearly every case when replayed from the learner’s state. On ALFWorld, 780/780 selected segments (100%) reached their target in post-hoc replay. On WebShop, 1,449/1,459 replays (99.31%) reached their target among attempts that began executing donor actions from the restored learner state. We count a replay as successful if the state after the final donor action matches the target milestone or, for a terminal target, if the environment reports success.
2. State matching.
- ALFWorld. Two states match when their complete simulator fact sets are identical after we normalize names and sort the facts, preserving object identities and argument order.
- WebShop. Page type, product ID, selected options, reward, and any recorded cart, purchase, or terminal fields must agree. If the next reference action clicks a product on a search page, the product must also be clickable in the matched state. In R1, its ID must also appear in the recorded observation.
2Wp1-W6 ReST-EM results on ALFWorld
Reply to 2Wp1-W6
Our review identified an inconsistency between the ReST-EM statistics in Table 1 and the evaluation protocol reported in Table A.3. Across two additional runs on ALFWorld, ReST-EM achieved 94.22% mean success (sample SD 1.85 percentage points). We will report this result in Table 1 and correct the corresponding protocol description in Table A.3.
Response to Reviewer 2Wp1 - 1
We thank the reviewer for recognizing the practical value of our training-data construction approach and the distinction between where and how far to learn. We also appreciate your recognition of the design without an additional trained PRM or stronger teacher, and of the ablation comparisons. Thank you for your time, effort, and valuable questions. Below, we address your concerns and present additional experiments and analyses.
2Wp1-W1 Frontier definition and reference choice
Reply to 2Wp1-W1
The relative frontier is the last reference state a failed rollout matches in order. It marks the start of that rollout’s learning interval. We use the shortest successful rollout as our default reference, although a single reference can underestimate progress along another valid path, as discussed in §6. Longer successful rollouts can include detours. If a location visited on a detour becomes a milestone, the learner may be trained to repeat that detour.
We evaluated the multi-reference variant on ALFWorld and WebShop. The other two proposals, which are allowing reference states to be skipped and measuring progress through task-relevant states, typically require more detailed prior knowledge of the dataset and task to determine which states can be skipped or which changes represent meaningful progress. They therefore introduce more environment-specific rules and manual intervention. Our method aims to provide a general framework across environments, within which these proposals can serve as environment-specific extensions and refinements.
Using multiple references gave some failed rollouts a later learning start, but final success rates did not improve in either environment after retraining (Table 2Wp1-W1). Each successful rollout provided its own milestone sequence. For each failed rollout, we chose the reference whose last match came latest in the failed rollout itself. We kept the prefix through that match and used the next milestone on the chosen reference as the target. Final success rates were close on WebShop (70.90% with one reference and 69.60% with multiple references). On ALFWorld, the single-reference run scored 8.21 percentage points higher (98.51% versus 90.30%). In some cases, the new target was simply a location visited during search. Even when the target stayed the same, the selected segment could include more detours.
Table 2Wp1-W1. Effect of reference choice on learning intervals and final success rates
2Wp1-W2 Scope of the Agent-R implementation
Reply to 2Wp1-W2
We agree that Agent-R on ALFWorld is a search-only adaptation and does not represent the full method. Agent-R’s revision procedure, tested on WebShop, does not transfer directly to our ALFWorld setup because actions in the failed prefix can change the object states needed to execute the successful continuation. In both environments, we excluded ShareGPT conversational data to keep supervision derived from each method’s own rollouts. However, our main conclusions remain supported. On WebShop, Agent-R retains MCTS search, policy-selected revision points, revision signals, and the concatenation of failed and successful trajectories. PRFL still outperforms it, achieving 71.00% versus 58.70% success. Our ALFWorld conclusion also does not depend on Agent-R: the strongest baseline there is ReST-EM at 96.64%, compared with 97.76% for PRFL.
We omitted revision trajectories on ALFWorld for environment compatibility. Intermediate states cannot be saved directly and must be restored by replaying actions from the initial state. Agent-R’s revision trajectories concatenate a failed branch’s prefix with a successful branch’s continuation. In ALFWorld, actions such as picking up, placing, or heating objects persistently change the environment, so observations in the concatenated trajectory can become inconsistent with the actual state. To avoid training on such inconsistencies, we used only successful paths actually executed during search.
We will label this result as an adaptation in Table 1, document the mechanisms retained and omitted in each environment in Appendix A.2, and polish the experimental analysis.