acceptodds
Withdrawn from ICLR 2027

Fail2Scene: Turning Navigation Failures into Training Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Improving embodied navigation requires training experience that targets the failure conditions behind incorrect navigation decisions. However, reproducing the objects and room types of a failed episode can omit the relationships among distracting cues, alternative routes, and the goal. We introduce Fail2Scene, a failure-driven curriculum framework that turns failed navigation decisions into new training environments. Its central idea is to preserve the visual and spatial relationships among competing choices while varying the surrounding layout. First, evidence-grounded failure attribution compares chosen and alternative views alongside the navigator's stated rationale to propose and verify explanations for failed decisions. Cross-scene consolidation identifies recurring failure conditions, which guide specifications of room connectivity and relevant visual cues. Second, mechanism-conditioned curriculum construction generates new layouts from these specifications. It then retains navigable scenes that reproduce the diagnosed failure conditions. Rendered trajectories provide candidate-selection examples for reinforcement learning, while repeated diagnosis updates subsequent scene construction as the navigator's failure patterns evolve. Trained in AI2-THOR and evaluated in Habitat, Fail2Scene achieves 70.50% success and 47.62% SPL on GOAT-Bench tasks in held-out buildings, improving success by 9.35 percentage points over a random curriculum with the same scene budget.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, Vision-Language Navigation, Failure Attribution, VLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3fedOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Fail2Scene, a failure-driven curriculum learning framework for embodied navigation. The framework optimizes navigation policies via reinforcement learning by diagnosing the causes of navigation failures, identifying recurring failure patterns, and generating targeted training environments.

Critical strengths
  1. The paper addresses a critical challenge in embodied navigation: how to leverage an agent’s own failure experiences to construct more informative training environments, rather than simply collecting more trajectories or generating scenes randomly.

  2. On the unseen GOAT-Bench environment, Fail2Scene achieves a Success Rate (SR) of 70.50% and a Success weighted by Path Length (SPL) of 47.62%—outperforming the "Random Curriculum" baseline by 9.35 and 4.46 percentage points, respectively. Performance improvements on A-EQA tasks further demonstrate the method's versatility across various embodied tasks.

  3. The authors analyze several key components of the framework, including the identification and validation of failure mechanisms, as well as the adaptive curriculum update process. These experiments provide robust evidence for the effectiveness of the overall framework.

Critical weaknesses
  1. The proposed framework primarily integrates existing techniques—such as Large Language Model (LLM)-based failure attribution, scene generation, curriculum learning, and the GRPO algorithm. While the overall pipeline is effective, the technical innovation beyond the combination of these existing components is relatively limited.

  2. Although the paper compares Fail2Scene against baselines like "Random Curriculum," "Failure Replay," and SAGE, it lacks comparisons with other highly relevant adaptive curriculum learning and environment generation methods, such as "Prioritized Level Replay" or "regret-based environment design." Including these comparisons would better highlight the advantages of the strategy of "generating environments based on failure mechanisms."

  3. While the paper confirms that the generated scenarios do indeed contain the diagnosed failure conditions, it remains unclear whether the resulting training samples consistently and effectively represent these conditions. Furthermore, differences in scenario complexity and trajectory distribution could also be contributing factors to the observed performance improvements.

  4. The experimental section reports results from only a single independent training run, with no estimates of variance provided. Although the appendix includes experiments with real-world robots, these are limited to qualitative demonstrations and lack systematic, quantitative comparisons.

Questions for authors

see weaknesses

Reviewer_tMNpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to construct training environments from failed navigation decisions to improve a VLM’s candidate-selection ability during navigation. Specifically, the authors propose the Fail2Scene framework, which first collects failed navigation decisions and uses a large language model to analyze the failure reasons behind incorrect choices, consolidating them into reusable failure mechanisms and corresponding scene specifications. The authors then use a scene generator to construct new environments that reproduce the relevant decision conditions and filter the generated scenes through navigability and mechanism-presence checks. On this basis, they convert visual observations along navigation trajectories into Frontier selection and Memory selection tasks and use GRPO to fine-tune the VLM through reinforcement learning. On held-out GOAT-Bench buildings, Fail2Scene achieves 70.50% SR and 47.62% SPL, improving over the Random curriculum by 9.35% and 4.46%, respectively, under the same scene budget.

Critical strengths
  1. The authors construct training scenes based on specific failure reasons, such as the relationships between incorrectly selected candidates and the goal, providing a meaningful perspective on how to better use the experience accumulated during navigation.
  2. The experimental evaluation is reasonably comprehensive and compares navigation performance using the same model and the same number of training examples.
Critical weaknesses
  1. The authors emphasize preserving the failure conditions of the original failed decisions in newly constructed scenes. However, the current experiments do not isolate this contribution. Content-only removes both competing-route relations and the corresponding filtering, making it difficult to determine which component is more important.

  2. Although the paper identifies the Qwen3-VL-2B backbone and the Frontier/Memory selection interface, the source of the initial policy π₀ and the implementation of components such as frontier generation and memory construction remain unclear. These omissions limit reproducibility and make implementation differences across methods difficult to assess.

  3. The authors use failure cases from discovery buildings to guide training-scene generation and optimize the model based on its performance in these buildings. These buildings are subsequently included in the full-set evaluation. Consequently, the full-set evaluation includes scenes already used for adaptation, and its results cannot be regarded as a fully independent test of generalization. Although the paper separately reports independent held-out results, the data access and adaptation conditions across methods in the full-set comparison remain insufficiently specified.

  4. Section 3.2 uses geodesic distance to identify decisions in failed trajectories where the policy did not select the candidate closest to the goal. However, the navigator must act on the information available at the time, without knowing the true goal location. A choice that appears worse in hindsight may still be a reasonable exploratory action under incomplete information. The paper does not sufficiently distinguish such exploration from avoidable mistakes, leaving the validity of the resulting failure mechanisms insufficiently established.

Questions for authors
  1. Content-only removes both competing-route relations and the corresponding filtering. Could the authors report SR/SPL when each component is removed separately, while keeping all other settings identical?

  2. Which checkpoint is used to initialize π₀, and has it undergone navigation-specific training?

  3. In the full-set comparison, what discovery data and feedback does each baseline use, including failure traces, goal locations, geodesic distances, and navigation scores for model selection? For external baselines with different adaptation conditions, could the authors additionally report results on the same held-out buildings to reduce the influence of differences in evaluation scenes?

Reviewer_i8i7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds a navigation curriculum from failures. It looks within failed subtasks for decisions where a closer candidate view was available. An examiner uses the selected view, alternatives, and the navigator’s rationale to propose an explanation and a scene specification; a verifier checks them against the recorded evidence. Explanations that recur across scenes are consolidated into failure conditions used to generate AI2-THOR environments. The authors then use A* trajectories in these scenes to create candidate-selection examples and train Qwen3-VL-2B with GRPO. On held-out GOAT-Bench buildings, Fail2Scene achieves 70.50% SR and 47.62% SPL, compared with 61.15% and 43.16% for the same-budget Random curriculum. The paper also reports results on Snapshot and A-EQA, along with several ablations.

Critical strengths

The paper presents a substantial closed-loop system spanning failure-trajectory collection, explanation generation and verification, cross-scene consolidation, environment generation, training-input construction, policy optimization, and held-out evaluation. The main text and appendices describe the examiner/verifier interfaces, scene filters, training-example construction, GRPO rewards, hyperparameters, and prompts in detail. The experiments also include comparisons with Random, Content-only, No consolidation, No verification, and Frozen mechanisms.These results provide useful evidence that policy failures can guide the allocation of a synthetic training budget.

Critical weaknesses
  1. The scope and training exposure of recurring failure conditions need clearer reporting. The paper defines mechanisms as recurring, observable failure conditions grounded in recorded evidence. The verifier prompt also states that the navigator's rationale supports what it stated, not an unobserved internal cause. Figure 6 reports mechanism coverage for 107 of 140 unique input decisions; Appendix I.1 clarifies that decisions assigned to multiple mechanisms are counted once. Because these decisions were selected from failed subtasks using a geodesic-gap filter, this statistic describes coverage within the selected attribution set, not prevalence across policy decisions more generally. Figure 5(c) reports condition presence and input exposure at the scene level, leaving unclear how many candidate-selection examples actually consumed by GRPO instantiate each condition. Please report input-level counts or proportions from the training logs if available; otherwise, state this limitation.
  2. The mechanism-specific source of the performance gains remains unclear. The held-out results and ablations provide evidence that the full Fail2Scene pipeline is useful, while the current experiments do not isolate how much of the gain comes from training on the inferred recurring conditions. Appendix Table 13 reports round-5 medians of 4 vs. 2 rooms and 12.50 m vs. 9.36 m planned path length for Random and Fail2Scene, respectively. Appendix Table 12 reports round-3 success rates of 52.52% vs. 66.19% and abort_parse rates of 17.27% vs. 5.04%. These results point to plausible contributions from scene difficulty and output or termination reliability. Appendix Table 5 reports one independent training run, so seed-level variability is not measured. A focused analysis using existing data, or more cautious wording about mechanism-specific gains, would help distinguish these explanations.
Questions for authors
  1. Could the authors briefly describe how the verifier judges whether a proposed recurring condition is supported by the recorded evidence, and whether any consistency checks were performed? This would help readers assess how much confidence to place in the attributions used to guide scene generation. 2.Could the authors report, if logged, how many examiner-generated explanations were rejected by the verifier and the main reasons for rejection?
  2. Can the authors report the count and proportion of candidate-selection examples actually consumed by GRPO that instantiate each condition? Figure 5(c) reports scene-level input exposure; if input-level counts are unavailable, please clarify this limitation.
  3. Appendix Table 5 reports one independent training run. Can the authors provide seed-level variability for the main SR/SPL comparison, or, if additional runs are unavailable, qualify the result as a single-run estimate?
  4. Using existing data, can the authors provide one focused control or stratified analysis addressing a major alternative contributor, such as scene/path difficulty or parse/termination differences? If such an analysis is unavailable, please qualify the attribution of gains to mechanism-conditioned training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.