- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents World State Generator (WSG), a fine-tuned replanning component built upon the existing State-Centric Decision Process (SDP) framework. The authors construct synthetic environments across seven domains and collect failure–repair trajectories through agent interactions. Approximately 239K repair examples are retained from 226K episodes and used to fine-tune two language models with supervised learning.
WSG is activated when an intermediate target state fails, generating a revised sequence of states based on environmental feedback. Evaluations on seven agent benchmarks show improvements over the prompted SDP baseline, with additional analyses of recovery rates and execution efficiency.
Critical strengths
-
Practical synthetic data construction. The paper presents a systematic pipeline for collecting failure–repair examples from executable synthetic environments across multiple domains. This provides a potentially useful source of supervision for training agent recovery behavior.
-
Consistent empirical improvements. WSG improves the reported metrics across seven benchmarks and two open-weight backbones. The comparison with prompted SDP is reasonably controlled, as the underlying runtime and other agent components remain unchanged. The recovery analysis provides some supporting evidence for improved replanning behavior.
Critical weaknesses
- Limited methodological novelty and insufficient experimental evidence supporting the core design.
The proposed method relies heavily on the existing SDP framework, while the actual training procedure is standard supervised fine-tuning with LoRA. The primary contribution therefore lies in the construction and use of failure–repair supervision. However, the paper lacks important controlled comparisons, including successful-trajectory SFT, training without refutation information, and different levels of repair verification.
Although Section 5.1 discusses training on initial plans, no quantitative comparison is provided. Consequently, it remains unclear whether the improvements specifically result from the proposed failure-conditioned supervision or from more general adaptation to the SDP planning format. This limits the strength of the paper's central methodological claim.
- Insufficiently justified verification claims and unclear presentation of the proposed approach.
The paper repeatedly emphasizes verified failure–repair pairs, yet the filtering criterion only requires at least one state in a repaired continuation to be certified. Only 48.9% of retained repairs come from episodes that eventually reach the goal, and intermediate states are validated by an LLM in most domains. Since the entire repaired continuation is used for training, the actual reliability of the supervision remains insufficiently established.
The presentation further weakens the clarity of the contribution. Figure 1 is particularly difficult to follow: its decorative visual style, numerous small labels, icons, and dense connections make the overall pipeline hard to understand. As the main methodological illustration, it fails to clearly distinguish the existing SDP components from the proposed contributions. A simpler, more structured diagram would substantially improve readability.
The writing also relies on repetitive and metaphorical descriptions rather than consistently providing precise technical explanations. Important distinctions, such as programmatic verification versus LLM-based state validation, are not communicated clearly enough. Overall, the paper requires considerable effort to identify what is technically new and how the proposed training mechanism differs from the underlying framework.
Several reporting inconsistencies further reduce clarity and confidence in the experimental presentation. The main text reports 238,563 retained repair examples, whereas Appendix Table 7 refers to 207,660. The reported average repair length also differs (5.5 versus 4.9), and the DeepPlanning evaluation set contains 150 tasks in Table 8 but only 120 in the breakdown of Table 18. Additionally, some claims about end-to-end success improvements are imprecise because several benchmarks report partial task scores rather than strict success rates.
Questions for authors
Q1. Can the authors provide controlled ablations comparing the proposed repair SFT against successful-trajectory SFT and repair training without the refutation signal? These comparisons are important for establishing the necessity of the proposed supervision.
Q2. How reliable are the intermediate-state certifications produced by the LLM validator? What percentage of retained repair sequences are fully verified, and how does training exclusively on goal-reaching repairs affect performance?
Q3. Could the authors clarify the inconsistencies in dataset size, average repair length, and DeepPlanning evaluation sample count? Are these statistics derived from different filtering stages or experimental configurations?
Q4. Can the authors more explicitly distinguish the contributions of WSG from the existing SDP framework and explain which components are newly introduced? A clearer methodological diagram and a more precise description of the verification process would also help assess the contribution.