From Planning Failures to Grounded Actions through Synthetic Training on Ambiguous Embodied Instructions

Embodied AI studies agents that interpret human instructions and translate them into grounded action sequences for interaction with physical or simulated environments. A major challenge is that human instructions are often ambiguous, containing vague references, missing attributes, pronouns, and distracting conversational context. Small Language Models (SLMs) are attractive for efficient local planning, but their performance drops substantially under such ambiguity, while inference time prompting provides only limited improvement. To solve this issue, we introduce FAST-GEN, a failure driven synthetic training data generation framework that converts ambiguity induced planning failures into targeted supervision for textual SLM planners. Unlike prior approaches that primarily address ambiguity at inference time or train on clean demonstrations, FAST-GEN explicitly uses planner failures to generate corrective, ambiguity focused training data. FAST-GEN generates grounded ambiguous planning examples, detects and categorizes failed plans using grounding checks and a critic, and feeds the identified failure type back into the pipeline to generate corrected trajectories that are retained as supervised finetuning data. On REI-Bench, FAST-GEN improves 1B to 4B SLMs by roughly 35 to 62 absolute percentage points over inference time prompting baselines and also outperforms proprietary LLMs like GPT-5. Direct FAST-GEN training on AmbiK improves commonsense and safety success by 13.2 and 7.7 points over zero shot transfer. In out of distribution evaluation, FAST-GEN improves CLARA macro-F1 by 6.14 points and AmbiK safety success by 5.8 points, boosts DialFRED execution success by 21.5 to 29.0 points, and reduces hazardous task risk on SafeAgentBench by up to 7.7 points. Overall, FAST-GEN provides a scalable recipe for training ambiguity robust SLM planners in embodied domain.