Fail2Scene: Turning Navigation Failures into Training Environments

Improving embodied navigation requires training experience that targets the failure conditions behind incorrect navigation decisions. However, reproducing the objects and room types of a failed episode can omit the relationships among distracting cues, alternative routes, and the goal. We introduce Fail2Scene, a failure-driven curriculum framework that turns failed navigation decisions into new training environments. Its central idea is to preserve the visual and spatial relationships among competing choices while varying the surrounding layout. First, evidence-grounded failure attribution compares chosen and alternative views alongside the navigator's stated rationale to propose and verify explanations for failed decisions. Cross-scene consolidation identifies recurring failure conditions, which guide specifications of room connectivity and relevant visual cues. Second, mechanism-conditioned curriculum construction generates new layouts from these specifications. It then retains navigable scenes that reproduce the diagnosed failure conditions. Rendered trajectories provide candidate-selection examples for reinforcement learning, while repeated diagnosis updates subsequent scene construction as the navigator's failure patterns evolve. Trained in AI2-THOR and evaluated in Habitat, Fail2Scene achieves 70.50% success and 47.62% SPL on GOAT-Bench tasks in held-out buildings, improving success by 9.35 percentage points over a random curriculum with the same scene budget.