- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VEL trains agents to distill experience using environment-verifiable progress differences between consecutive attempts, so that reflection after failure actually helps the next retry rather than merely producing plausible-sounding summaries. Across four task types, cumulative success over the first three attempts improves by 49.3% on average relative to the first attempt, and the method shows preliminary cross-environment transfer.
Critical strengths
-
For long-horizon interactive tasks, distilling useful information from failures may be more valuable than simply raising single-attempt success rates.
-
In environments with computable progress, this reduces noise from subjective evaluation.
-
Table 3 examines the reward and grouping design, and Table 6 examines the cross-episode information carrier.
Critical weaknesses
W1. Insufficient comparison with the closest baseline (Table 1). All 63 baseline success rates in Table 1 match LaMer's Table 1 exactly, yet the paper claims all methods are evaluated under its own cross-episode, experience-only protocol.
Please mark which results are cited versus re-run, and explain how LaMer's GiGPO optimizer is reconciled with VEL's GRPO.
More importantly, LaMer's reflection-only configuration, closest to VEL's experience-only setting, is not compared. Its reported p@3 (80.5 / 56.4 / 92.8) exceeds the LaMer numbers here (74.4 / 51.6 / 89.1) and VEL on Sokoban (55.5).
Please compare against it under the same split, optimizer, test budget, and checkpoint selection.
W2. Limited novelty relative to prior work. The novelty of VEL is not sufficiently established.
- SCoRe [1] already uses the reward difference between consecutive attempts to encourage improvement, although its shaping bonus is applied to the second response rather than an intermediate experience summary.
- Moreover, LaMer [2] already trains cross-episode self-reflection using subsequent-episode rewards and evaluates a reflection-only memory configuration.
Thus, neither improvement-based reward shaping nor the cross-episode reflection framework is new in itself. The paper should explicitly position these components against the existing mechanisms and provide controlled evidence that their combination offers a substantive advance, rather than primarily benefiting from additional dense progress supervision.
[1] Training Language Models to Self-Correct via Reinforcement Learning. ICLR, 2025.
[2] Meta-RL Induces Exploration in Language Agents. ICLR, 2026.
W3. EGAR does not causally attribute progress to experience quality (Eq. 7). Even with a fixed task and policy, the progress difference also reflects action sampling, environment stochasticity, and regression to the mean. Moreover, given , acts as a constant baseline that leaves the expected policy gradient unchanged. EGAR's gains may thus come from altered baselines, variance, and advantage ranking rather than causal attribution.
Please fix the previous trajectory and actor, and compare real, empty, shuffled, and other-task experience over multiple rollouts.
W4. Weak evidence for generalization and recursive improvement (Table 2). Transfer from MineSweeper to ALFWorld is a valuable experiment, but one model outperforming another does not show a stronger experience-transfer mechanism. The gap may reflect general execution ability, prompt following, or training recipe. Table 2 compares only training methods and cannot isolate the contribution of cross-task experience transfer.
Please compare the same VEL model with
- no prior-task experience
- correct same-category experience
- shuffled experience
- unrelated-category experience
- fixed generic strategy text
W5. The method relies on hand-constructed verifiers. All four benchmarks have well-defined progress structures, but open-ended real-world tasks may lack computable shortest-solution distances, full environment states, reliable intermediate milestones, or continuous completion scores.
For example, BFS-based progress in Sokoban is reasonable for small boards but does not scale to large, open-ended planning problems. The paper thus addresses how to learn from experience given a reliable progress signal, but not how to obtain such a signal automatically in general environments.
Please discuss this limitation and how progress signals could be constructed beyond these benchmarks.
This work reads more as an effective reward/advantage recipe than as convincing evidence of the claimed independent, attributable, and recursively transferable experience-learning capability.
Questions for authors
See Weakness