- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WEDGE, a framework that uses policy-specific navigation failures to guide training environment generation. An LLM identifies weakness features from successful and failed episodes, which are statistically validated and used to generate targeted environments. The generated environments are filtered based on rollout success rates before policy adaptation. Experiments on DD-PPO, Uni-NaVid, and Gemini show consistent improvements on ProcTHOR and cross-domain transfer to HM3D. The problem is interesting, and the results are promising. However, the methodological novelty and experimental validation have some limitations.
Critical strengths
- The paper addresses an interesting problem of generating training environments based on policy-specific failures. 2. The identified weakness features are interpretable and provide measurable descriptions of navigation failures. 3. The method shows consistent improvements across three agents with different adaptation mechanisms. 4. The cross-domain improvements on HM3D demonstrate promising generalization beyond the training simulator.
Critical weaknesses
- The methodological novelty is limited. Weakness discovery largely uses predefined simulator primitives, while environment selection and generation rely on statistical filtering, fixed heuristics, and existing procedural operators. The paper also lacks a comparison with simpler feature-selection methods without an LLM. 2. The claim of policy-specific environment generation is not sufficiently validated. The paper does not examine whether training environments targeting one agent's weakness are less effective for other agents. 3. The benefit of weakness-conditioned generation is not fully isolated from difficulty-based selection. A baseline using randomly generated environments with the same rollout-based calibration is missing. 4. The statistical reliability of the results is unclear, as confidence intervals and multiple-seed evaluations are not reported. Some improvements are relatively small, particularly for Gemini on ProcTHOR.
Questions for authors
- How does LLM-based weakness discovery compare with directly ranking predefined simulator features? 2. How does WEDGE compare with random environment generation using the same rollout-based calibration? 3. Can the authors evaluate cross-policy weakness conditioning to better support the claim of policy-specific training? 4. How sensitive are the results to the fixed rollout selection threshold of 2–3 successes out of five trials?