acceptodds
Under review as a conference paper at ICLR 2027

From Planning Failures to Grounded Actions through Synthetic Training on Ambiguous Embodied Instructions

Abstract

Embodied AI studies agents that interpret human instructions and translate them into grounded action sequences for interaction with physical or simulated environments. A major challenge is that human instructions are often ambiguous, containing vague references, missing attributes, pronouns, and distracting conversational context. Small Language Models (SLMs) are attractive for efficient local planning, but their performance drops substantially under such ambiguity, while inference time prompting provides only limited improvement. To solve this issue, we introduce FAST-GEN, a failure driven synthetic training data generation framework that converts ambiguity induced planning failures into targeted supervision for textual SLM planners. Unlike prior approaches that primarily address ambiguity at inference time or train on clean demonstrations, FAST-GEN explicitly uses planner failures to generate corrective, ambiguity focused training data. FAST-GEN generates grounded ambiguous planning examples, detects and categorizes failed plans using grounding checks and a critic, and feeds the identified failure type back into the pipeline to generate corrected trajectories that are retained as supervised finetuning data. On REI-Bench, FAST-GEN improves 1B to 4B SLMs by roughly 35 to 62 absolute percentage points over inference time prompting baselines and also outperforms proprietary LLMs like GPT-5. Direct FAST-GEN training on AmbiK improves commonsense and safety success by 13.2 and 7.7 points over zero shot transfer. In out of distribution evaluation, FAST-GEN improves CLARA macro-F1 by 6.14 points and AmbiK safety success by 5.8 points, boosts DialFRED execution success by 21.5 to 29.0 points, and reduces hazardous task risk on SafeAgentBench by up to 7.7 points. Overall, FAST-GEN provides a scalable recipe for training ambiguity robust SLM planners in embodied domain.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Task Planning, Small Language Models, Ambiguous Instructions, Failure-Driven Data Generation, Synthetic Training Data, Supervised Fine-Tuning, Language Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S7DqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper mainly propose FAST-GEN. It use failure recovery and analysis to generate data for fine-tuning small language model planners. The fine-tuned planner gets stronger ability in interpreting ambiguous instructions. The method is tested on on REI-Bench, AmbiK, CLARA, DialFRED, and SafeAgentBench.

Critical strengths
  • The paper's focusing on an interesting question, discussing how improving SLM planning under ambiguous instructions.
  • The evaluation is complete and has a wide scope. The result covers multiple benchmarks, including both in-domain and out-of-domain situations.
  • The ablation study on component is well done, and synthetic-data controls ensures isolated effects comparison of the methods.
Critical weaknesses
  1. No real-robot validation. I consider this a fatal gap in supporting the claimed real-world applicability, as robustness to perception errors and physical execution uncertainty remains untested.

  2. Insufficient comparison with related methods. After reading the paper, I think PRISM is the only approach that includes data-generation + fine-tuning, and it performs far worse than basic template-only SFT. This comparison provides unconvincing evidence of contribution of FAST-GEN over existing training or data generation methods.

  3. The major improvement seems comes from SFT. In the results shown by the author, simple template-based SFT already provides substantial improvements. The additional gains from proposed components is relatively modest. This makes the added value on the main benchmark unclear.

  4. Unclear component contributions to OOD generalization. In-distribution gains are expected after task-specific training. More importantly, the method should demonstrate that its proposed components contribute to generalized ambiguity instructions. However, component ablations on OOD execution benchmarks is very limited.

Questions for authors
  1. Can the authors provide real-robot evaluation to support the claimed practical applicability under perception and execution uncertainty?

  2. Can the authors compare against additional failure-driven data generation and training methods? Why PRISM performs even worse than simply template-only SFT?

  3. Can the authors directly measure the fine-tuned students’ target-object and destination resolution accuracy? This can isolate the proposed components' effects in improving ambiguity understanding? Is it better than simply template-based SFT?

  4. Can the authors extend the simple SFT and no-Critic controls to OOD execution benchmarks such as DialFRED to evaluate the generalization of ambiguity understanding?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer S7Dq-Part 1

We thank the reviewer for the careful and constructive review. We answer each point below.

W1/Q1. No real-robot validation. I consider this a fatal gap in supporting the claimed real-world applicability, as robustness to perception errors and physical execution uncertainty remains untested.

Ans: We thank the reviewer for raising this important point, and we agree that real-robot validation would strengthen the work. Unfortunately, our lab does not have access to a physical robot platform, so due to this resource constraint we are not able to provide real-robot experiments. We kindly ask the reviewer's understanding on this.

What we can provideDetails
Simulated executionPlans are executed in the AI2-THOR simulator on DialFRED, and success is decided by ALFRED's own goal checker, not by plan matching (see W4 / Q4).
ResultExecution success improves from 3.5 (base) to 25.0 (ours) with Llama-3.2-3B.
Scope of our claimOur contribution is at the language-planning level (resolving ambiguous instructions into correct plans). Perception and low-level control are not addressed.

We will state the absence of real-robot evaluation clearly as a limitation and revise the wording on real-world applicability accordingly. We consider real-robot evaluation valuable future work.

W2/Q2. Insufficient comparison with related methods. After reading the paper, I think PRISM is the only approach that includes data-generation + fine-tuning, and it performs far worse than basic template-only SFT. This comparison provides unconvincing evidence of contribution of FAST-GEN over existing training or data generation methods. An s: Thank you. We agree that template-only SFT is the stronger baseline, and we will present it as the main SFT baseline instead of PRISM.

Why PRISM is low: PRISM generates data for unambiguous ALFRED-style instructions. Models trained on it write plans that skip steps REI-Bench requires (for example the find step before pick_up).

PRISM-trained model on REI-BenchSRFailures that are "action omission"
Llama-3.2-3B25.778% (564 / 722)
Qwen3-4B28.980% (556 / 691)

So PRISM's low score is mostly a plan-convention mismatch, not evidence about ambiguity. For a fair comparison of data-generation choices under the same budget, format and training recipe, we use controlled variants of our own pipeline:

Training data (same 1,800 budget)REI-Bench SRAmbiK SR (OOD)DialFRED Exec SR (OOD)
PRISM (AI2-THOR port)25.723.3not run
Template-only SFT91.537.717.2
Unambiguous instructions only92.557.223.6
Ours (full)96.064.125.0

We are not aware of another failure-driven data-generation method for ambiguous embodied instructions that we could run in this period; we will add any the reviewer suggests.

W3. The major improvement seems comes from SFT. In the results shown by the author, simple template-based SFT already provides substantial improvements. The additional gains from proposed components is relatively modest. This makes the added value on the main benchmark unclear.

Ans: Thank you, this is correct for the overall REI-Bench number, and we will say so in the paper. The added value appears in two places:

WhereBaseTemplate-only SFTOurs (full)
REI-Bench, overall41.291.596.0
REI-Bench, implicit instructions only38.080.695.7
AmbiK (OOD)63.237.764.1
DialFRED Exec SR (OOD, goal + QA)3.517.225.0
  • On implicit instructions, where ambiguity matters, the gain over template-only is +15.1.
  • Out of distribution, template-only SFT falls well below the base model on AmbiK (37.7 vs 63.2), while our data keeps base-level performance.

W4/Q4. Unclear component contributions to OOD generalization. In-distribution gains are expected after task-specific training. More importantly, the method should demonstrate that its proposed components contribute to generalized ambiguity instructions. However, component ablations on OOD execution benchmarks is very limited.

Ans: Thank you for this suggestion. We ran it for this response: each ablation model was executed in AI2-THOR on the 424 DialFRED validation episodes (success decided by ALFRED's goal checker). AmbiK is added as a second OOD benchmark.

Training dataDialFRED Exec SR, goal onlyDialFRED Exec SR, goal + QAAmbiK SR
Base (no fine-tuning)2.13.563.2
Template-only SFT12.717.237.7
Unambiguous instructions only19.420.557.2
Ambiguous data, no critic20.221.661.7
Ours (full)22.225.064.1

Thank you for the response and additional experiments.

  1. Narrowing the claims acknowledges the lack of real-robot validation but does not resolve it. Given the substantial sim-to-real gap in robotics, real-world applicability remains unsupported.
  2. No additional external method is compared. Since PRISM’s failures largely stem from mismatched action conventions, a comparison under aligned conventions is necessary. The current baseline comparison remains inadequate.
  3. Q3 remains unanswered: target-object and destination resolution accuracy are not separately evaluated. The reported overall SR does not establish whether the gains come from improved ambiguity understanding.
  4. The additional OOD ablations address Q4, and I appreciate this evidence of component effectiveness.

I understand the constraints on real-robot evaluation, and reframing the paper as a language-oriented study would be reasonable. However, weaknesses 2 and 3 remain critical: even when assessed purely as an LLM study, the experimental evidence is insufficient to establish the method’s advantage over existing approaches or attribute its gains to improved ambiguity understanding.

In conclusion, substantial concerns remain, and I maintain my original rating.

Reviewer_jritOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FAST-GEN, a failure-driven synthetic data generation framework for improving small language models (SLMs) on embodied task planning with ambiguous instructions. The framework first categorizes planning failures into eight error types, then generates grounded scenarios in AI2-THOR and uses a resolver, planner, grounding checker, and failure-aware critic to construct supervised fine-tuning data. Experiments on Llama-3.2 and Qwen3 models show substantial improvements on REI-Bench over prompting baselines and larger LLMs.

Critical strengths

FAST-GEN achieves approximately 94–96% average success on REI-Bench across the evaluated SLMs, substantially outperforming their base models and inference-time prompting baselines.

Critical weaknesses

The paper argues converting observed planning failures into targeted training supervision as a contribution, but Stage 1's failures are only used to define eight error categories, rather than directly informing individual training examples. In addition, such failure modes come from the target benchmark (i.e., the failure modes are collected from the benchmark where the model is going to be evaluated). This raises concerns that the proposed supervision is tailored to failure patterns observed on the target benchmark, rather than providing a generalizable failure-driven learning mechanism. For this, the paper provides additionaly analyses on different benchmarks such as AmbiK, CLARA, etc., but their results are mixed (e.g., Figure Table 1 vs. Figure 4, Figure 5, ...).

Questions for authors
  • Were the REI-Bench instances used to derive the failure taxonomy disjoint from those used for the final evaluation in Table 1? If not, how can the authors distinguish the benefits of failure-driven supervision from adaptation to the target benchmark's failure patterns?
  • Could the authors generate the synthetic dataset without REI-Bench and provide the analysis of how its performance differs from the one with REI-Bench? This can be useful to see whether the proposed method is dependent on the failure modes obtained from the target benchmark.
  • How are ambiguous instructions identified in Stage 1? Are ambiguity categories manually annotated or automatically determined? How are ambiguity-induced failures distinguished from ordinary planning errors?
  • How were the eight failure categories selected, and do they sufficiently characterize planning failures on unseen tasks or other benchmarks?
  • Can the authors compare FAST-GEN with ambiguity-augmented synthetic SFT using the same scenario generation, data budget, and verification procedures, but without failure-driven correction? This would help establish whether the proposed failure-driven mechanism contributes beyond conventional synthetic supervision.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer jrit-Part 1

We thank the reviewer for the careful and constructive review. We answer each point below.

W1. The paper argues converting observed planning failures into targeted training supervision as a contribution, but Stage 1's failures are only used to define eight error categories, rather than directly informing individual training examples. In addition, such failure modes come from the target benchmark (i.e., the failure modes are collected from the benchmark where the model is going to be evaluated). This raises concerns that the proposed supervision is tailored to failure patterns observed on the target benchmark, rather than providing a generalizable failure-driven learning mechanism. For this, the paper provides additionaly analyses on different benchmarks such as AmbiK, CLARA, etc., but their results are mixed (e.g., Figure Table 1 vs. Figure 4, Figure 5, ...).

Ans: Thank you, this is a fair reading. Our clarifications:

ConcernClarification
Failures only define categoriesCorrect. Failures are used at design time (the taxonomy and the critic's checks), not per training example. We will state this clearly in Sec. 3.
Tailored to the target benchmarkOnly the category definitions come from REI-Bench. The generator never reads a REI-Bench item; scenarios are built from AI2-THOR templates.
Is the mechanism generalizable?Yes. The mechanism is "analyse failures → define categories → critic checks them", and it is not tied to REI-Bench. We applied the same pipeline to AmbiK, a different task (kitchen plans with preferences, common sense and safety), by changing only the category definitions.
Mixed OOD resultsThe OOD benchmarks test different tasks, so the gain depends on how far the task is from the training task.

Same pipeline applied to AmbiK:

AmbiK categoryBaseZero-shot transfer from REI-Bench dataPipeline applied to AmbiK
Preferences34.535.038.2
Common sense40.635.748.9
Safety34.640.448.1

So the weak zero-shot numbers on AmbiK reflect the task distance, not a limit of the failure-driven mechanism: when the pipeline is given the new task's failure categories, the ambiguous categories improve.

Q1. Were the REI-Bench instances used to derive the failure taxonomy disjoint from those used for the final evaluation in Table 1? If not, how can the authors distinguish the benefits of failure-driven supervision from adaptation to the target benchmark's failure patterns?

Ans: Thank you for this question. They were not disjoint, and we will say so in the paper: the taxonomy was derived from base-model failures on REI-Bench. What crosses from the benchmark into training is limited to the 8 category definitions.

What could carry benchmark informationUsed in data generation?
REI-Bench instructions, dialogues, plansNo
REI-Bench scenes / objectsNo (sampled from the AI2-THOR skill set)
8 failure-category definitions (critic prompt)Yes

To separate the two effects, we removed the benchmark-derived part (the categories) and re-trained:

Qwen3-4BREI-Bench SR
Base (no fine-tuning)32.72
Our pipeline, critic without categories (no REI-Bench failure information)93.31
Our pipeline, critic with 8 categories (full)95.99

Without any failure information from REI-Bench the model still gains about 60 points over the base model. The categories add a further 2.7 points. So the main benefit comes from the synthetic ambiguous data, and only a small part can be attributed to the benchmark-derived categories. We also report train/test separation (embedding and n-gram analysis) in the appendix.

AuthorsOpenReview ↗

Reply to Reviewer jrit-Part 2

Q2. Could the authors generate the synthetic dataset without REI-Bench and provide the analysis of how its performance differs from the one with REI-Bench? This can be useful to see whether the proposed method is dependent on the failure modes obtained from the target benchmark.

Ans: We thank the reviewer for this suggestion. FAST-GEN generates synthetic data independently using AI2-THOR, without using REI-Bench examples or failure trajectories. Only the eight failure-category definitions are transferred to the Critic.

1. Removing REI-Bench-derived failure categories (Appendix B.5, Figure 15)

MethodREI-Bench Success Rate
FAST-GEN (Full)95.99%
FAST-GEN (without error categories)93.31%
Difference−2.68 pp

The small performance drop shows that FAST-GEN remains effective without the REI-Bench-derived error taxonomy, although these categories provide additional benefits.

2. Independent generation on AmbiK (Appendix H, Figure 21)

We also instantiate FAST-GEN directly on AmbiK, demonstrating its adaptability beyond REI-Bench.

AmbiK CategoryREI-Bench-trainedAmbiK-trained
Preference35.0%38.2%
Commonsense35.7%48.9%
Safety40.4%48.1%

Results for Llama-3.2-3B-Instruct.

These experiments suggest that FAST-GEN is not solely dependent on REI-Bench-derived failure modes. We will clarify this distinction and highlight these results in the revised manuscript.

Q3. How are ambiguous instructions identified in Stage 1? Are ambiguity categories manually annotated or automatically determined? How are ambiguity-induced failures distinguished from ordinary planning errors?

Ans: Initially humans make the error categories by reading the trajectories and then LLM as a judge is used to scale up in 972 test samples.Here, human-LLM agreement score is 97 percent.

StepHow it is determined
Ambiguity type of a benchmark itemGiven by REI-Bench: each item belongs to one of 9 conditions (explicit / mixed / implicit × standard / noised / short).
Ambiguity type of a generated exampleBy construction: the Augmentor is asked to write the instruction for a specific condition.
Failure categoryAn LLM judge sees the instruction, target object, gold plan and generated plan, and assigns exactly one of the 8 categories.

The 8 categories separate the two kinds of error:

GroupCategoriesShare of base failures (Llama-3.2-3B / 1B)
Ambiguity-relatedpronoun failure, attributive misidentification, noise distraction, context loss8.7% / 20.3%
Planning-relatedaction omission, action ordering, hallucinated action, wrong destination91.3% / 79.7%

Q4. How were the eight failure categories selected, and do they sufficiently characterize planning failures on unseen tasks or other benchmarks?

Ans: Thank you for this question. The categories were selected from an error analysis of base-model failures (Llama-3.2-1B and 3B) on REI-Bench. They are generic: four describe reference errors and four describe plan errors. To check that they hold beyond REI-Bench, we labelled base-model failures on three other benchmarks with an LLM judge that could also answer "other" (none of the eight) or "no plan produced":

Benchmark (not used for the taxonomy)ModelFailures checkedCovered by the 8 categories"Other"No plan produced
DialFREDLlama-3.2-3B20099.5%0%0.5%
Qwen3-4B20087.0%0%13.0%
TEAChLlama-3.2-3B200100%0%0%
Qwen3-4B20095.0%0%5.0%
SafeAgentBench (safe tasks)Llama-3.2-3B190100%0%0%
Qwen3-4B15099.3%0%0.7%

In 5 of 6 cases the eight categories cover at least 95% of failures, and the judge never needed "other"; the remaining cases are outputs with no plan at all. We do not claim the taxonomy is complete: it does not describe failures that need world knowledge or user preferences (AmbiK common sense). We will add this table and state the limitation.

AuthorsOpenReview ↗

Reply to Reviewer jrit-Part 3

Q5. Can the authors compare FAST-GEN with ambiguity-augmented synthetic SFT using the same scenario generation, data budget, and verification procedures, but without failure-driven correction? This would help establish whether the proposed failure-driven mechanism contributes beyond conventional synthetic supervision.

Ans: We thank the reviewer for this suggestion. We have already conducted a closely related ablation in Appendix B.6 (Figure 16), where we remove the Critic responsible for failure-driven correction while keeping the training budget (1,800 examples), backbone, and training hyperparameters fixed.

1. Effect of Failure-Driven Correction (Appendix B.6, Figure 16)

EvaluationFull FAST-GENWithout CriticImprovement
REI-Bench (Average SR)96.0%94.7%+1.3 pp
AmbiK (Preference)35.0%29.6%+5.4 pp
AmbiK (Safety)40.4%28.8%+11.6 pp

Results for Llama-3.2-3B-Instruct.

2. Comparison with Conventional Synthetic Supervision (Appendix B.7, Figure 17)

Training MethodREI-Bench Average SR
FAST-GEN (Full)96.0%
Unambiguous-only SFT92.5%
Template-only SFT91.5%

All settings use the same 1,800-example training budget.

These results show that failure-driven correction contributes beyond simply generating synthetic demonstrations, particularly for out-of-domain generalization, where removing the Critic reduces AmbiK Safety performance by 11.6 percentage points.

While the component-removal ablation provides the closest comparison to the requested setting, the unambiguous-only and template-only baselines also change other aspects of data generation. We will clarify this distinction and highlight the Critic ablation more prominently in the revised manuscript.

Reviewer_jritOpenReview ↗

Official Review of Submission27309 by Reviewer jrit

I thank the authors for their detailed response. I still have several concerns remaining, especially related to generalizability beyond the failure-driven design of the synthetic data for the downstream tasks, and thus, I'd like to keep my original rating.

  • For the answer to W1, the authors mention that the generator never reads a REI-Bench item, but this doesn't mean that the failure modes are not designed so.
  • For the answers to Q1 and Q2, thank you for your additional analysis. The authors mention that FAST-GEN generates synthetic data independently using AI2-THOR, without using REI-Bench examples or failure trajectories, but its detail is missing. How is the pipeline without the categories trained? What are the difference between the training data (observation-action pairs, language instructions, prompts, etc.) used with and without the categories? Only the presence of the categories in the prompts? If not, what are they?
  • For the answers to Q3 and Q4, if the eight failure categories are manually defined, there should be clear and objective criteria for selecting and distinguishing them, but these criteria remain insufficiently explained. Although the authors provide a table showing the coverage of the eight categories across different benchmarks in response to Q4, it is still unclear what criteria are used to assign failures to these categories and determine whether the taxonomy adequately covers the observed failure modes.
  • For the answer to Q5, thank you for your additional analysis. I believe these experimental results of the failure-driven component should be in the main paper somewhere as this is closely related to the core of the failure-driven characteristics of the proposed pipeline.
Reviewer_6nK9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FAST-GEN generates verified synthetic training data to enhance the performance of small language model planners when processing ambiguous instructions. The system employs grounding checks and a failure-category-guided critic to provide corrective supervision during the fine-tuning process. Reported results demonstrate significant in-domain improvements and selective transfer across benchmarks related to ambiguity, execution, and safety. Fine-tuned models reach 92.6 to 98.1% success on REI-Bench, outperforming prompting baselines and large prompted models including GPT-5, with smaller gains on four out-of-domain benchmarks. The pipeline is carefully engineered, but the “failure-driven” contribution is not isolated experimentally, and the in-domain gains are hard to separate from close alignment between the generator and the benchmark.

Critical strengths
  1. Grounded, verifiable data without manual annotation: Gold plans are created directly from AI2-THOR task specs, and each candidate plan is checked for accuracy in a consistent way. Every training example has a clear reference, so the process is reproducible and easy to build on.

  2. Significant and consistent improvements within the domain: These gains appear across three different backbones and all nine REI-Bench conditions (for example, Llama-3.2-3B’s mean success rises from 41.2% to 96.0%). The improvements are not limited to a single model family or type of ambiguity.

  3. Helpful negative findings about prompting: Table 1 shows that CoT, action-aware prompting, and ToCC lead to mixed or negative results for planners between 1B and 4B, while ICL is the only strategy that consistently helps. This is useful information for anyone developing on-device planners.

  4. Practical ablation studies and wide-ranging evaluation: The per-class budget and class-coverage ablations offer clear advice on data needs, and the out-of-domain tests cover four benchmarks with different types of tasks.

Critical weaknesses
  1. The methodological novelty is incremental. Synthetic data generation, critique, correction, and supervised fine-tuning are established techniques. Also the contribution relies on demonstrating the specific value of the ambiguity-focused failure taxonomy.
  2. The failure-driven framing requires extra clarification. The pipeline uses fixed failure-category definitions instead of directly incorporating failures or adapting generation to the student’s current weaknesses. As a result, the extent of failure-driven adaptation seems unclear.
  3. The use of the correction pipeline is not sufficiently established by the available evidence. Given that grounded gold plans are available during generation, a critical baseline would be to generate ambiguous instructions and train directly on those plans. Comparisons against prompting primarily demonstrate the benefit of specialized training but do not isolate the proposed mechanism. The referenced appendix ablations should address this concern.
  4. Generalization performance is inconsistent. Certain OoD improvements are modest, and the reported AmbiK commonsense regression requires further discussion. More rigorous evaluation using independently collected human instructions would better establish robustness beyond synthetic language patterns.
  5. The efficiency motivation requires quantitative measurements. Small parameter counts alone do not demonstrate practical deployment advantages. What is latency, memory usage, and an accuracy? efficiency comparison with moderately larger local models would strengthen the argument.

(Minor issue:) 1.No sufficient motivation for using SLM,

  1. Structured reasoning. A comparison with semantic-role-based reasoning such as CoT-TL, or Lang2LTL would complement the graph-based baselines.

  2. Context loss and noise distraction are almost absent from base failures (0.13%), which suggests REI-Bench dialogues don’t stress memory.

Questions for authors
  1. Given A.2, B.7, and Appendix G, it appears that the majority of the REI-Bench improvement is attributable to procedural supervision. If this is the case, the abstract and introduction should be revised to reflect this finding.
  2. Hyperparameter selection: Was the grid search described in Appendix F conducted on the REI-Bench test set? If yes it is recommended to perform hyperparameter selection on a held-out validation split. Additionally,also clarify why the optimal Llama grid configuration differs from the main result despite using the same settings.
  3. Please clarify why the value 95.99% appears as the main result for Llama, the Qwen B.5 result, and the Qwen-4B M.2 result, whereas the main Qwen result is reported as 96.4%.
  4. SafeAgentBench outcome analysis: What proportion of fine-tuned “safe” outcomes result from plans failing due to vocabulary or format issues, as opposed to refusals or intentionally safe plans?
  5. DialFRED dialogue use: Why does the inclusion of human QA reduce symbolic success for both fine-tuned models? Does this show that the models are not utilizing clarification dialogue?
  6. AmbiK common sense: Why do all SFT variants perform worse than the base model on common sense, and should not this be part of the main text?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 6nK9-Part 1

We thank the reviewer for the careful and constructive review. We answer each point below.

W1. The methodological novelty is incremental. Synthetic data generation, critique, correction, and supervised fine-tuning are established techniques. Also the contribution relies on demonstrating the specific value of the ambiguity-focused failure taxonomy.

Ans: We thank the reviewer and agree the individual components are established. The novelty is in what the data contains. Prior data-generation methods for embodied planning produce clear, unambiguous instructions, and models trained on them do not handle ambiguous instructions. To our knowledge, FAST-GEN is the first data-generation pipeline that creates ambiguous instructions paired with the correct plan and the reasoning that resolves the ambiguity.

Comparison with PRISM + AI2-THOR, same 1,800 training examples:

ModelTraining dataREI-Bench SR (in-domain)AmbiK SR (OOD)CLARA macro-F1 (OOD)
Llama-3.2-3BPRISM + AI2-THOR25.723.324.0
FAST-GEN (ours)96.064.124.4
Qwen3-4BPRISM + AI2-THOR28.942.741.2
FAST-GEN (ours)96.058.343.5

FAST-GEN is higher on all three benchmarks for both models; the margin is large in-domain and on AmbiK, and small on CLARA. We will state the contribution in these terms in the revision.

W2. The failure-driven framing requires extra clarification. The pipeline uses fixed failure-category definitions instead of directly incorporating failures or adapting generation to the student’s current weaknesses. As a result, the extent of failure-driven adaptation seems unclear.

Ans: We thank the reviewer for pointing this out. "Failure-driven" refers to how the pipeline is designed, not to online adaptation:

Where failures are usedHow
TaxonomyThe 8 error categories come from an error analysis of base-model failures.
CriticEach generated example is checked against these categories and rejected if it shows one.
Student adaptationNot done: the categories are fixed and generation does not adapt to the current student's weaknesses.

The category-based critic improves Qwen3-4B by 2.7 points over a critic without categories (95.99 vs 93.31). We will state the above explicitly in Sec. 3, use "failure-informed" wherever the text suggests adaptation, and list online failure harvesting as future work.

W3. The use of the correction pipeline is not sufficiently established by the available evidence. Given that grounded gold plans are available during generation, a critical baseline would be to generate ambiguous instructions and train directly on those plans. Comparisons against prompting primarily demonstrate the benefit of specialized training but do not isolate the proposed mechanism. The referenced appendix ablations should address this concern.

Ans: Thank you, for raising this issue. All rows train Llama-3.2-3B on 1,800 examples with the same recipe.

Training dataOverall
Template-only: plain instruction → gold plan91.46
Unambiguous instructions through the full pipeline92.49
Ambiguous instruction → gold plan (no resolver reasoning)93.21
Full pipeline95.99 (5 seeds: 95.25 ± 1.00)

W4. Generalization performance is inconsistent. Certain OoD improvements are modest, and the reported AmbiK commonsense regression requires further discussion. More rigorous evaluation using independently collected human instructions would better establish robustness beyond synthetic language patterns.

Ans: Thank you for this point. The OOD benchmarks test different tasks, so the size of the gain depends on how hard the task is and how far it is from what our data teaches (resolving references from dialogue and writing a grounded plan). No OOD example is seen in training.

OOD benchmarkWhat it requiresDistance from our training taskBase → + Ours
DialFRED (human instructions + human QA, execution SR)Plan from human dialogue, executed in simulatorClose: same task, human language3.5 → 25.0 (Llama), 3.3 → 32.3 (Qwen)
CLARA (macro-F1)Classify a command as clear / ambiguous / infeasibleMedium: different output, no plan18.29 → 24.43 (Llama)
AmbiK (overall SR)Kitchen tasks needing preferences, common sense, safety knowledgeFar: needs world knowledge not in the dialogue63.2 → 64.1 (Llama)

The gain is largest where the task is closest to ours and smallest where the task needs knowledge our data does not contain (AmbiK common sense, see Q6). DialFRED also answers the request for independently collected human instructions. We will add this table and discussion to the main text.

AuthorsOpenReview ↗

Reply to Reviewer 6nK9-Part 2

W5. The efficiency motivation requires quantitative measurements. Small parameter counts alone do not demonstrate practical deployment advantages. What is latency, memory usage, and an accuracy? efficiency comparison with moderately larger local models would strengthen the argument.

Ans: We agree and have measured it. Setting: one A100, bf16, batch size 1, greedy decoding, about 100 output tokens per plan (reasoning + plan), 54 REI-Bench prompts.

Fine-tuned modelParamsPeak GPU memoryLatency per plan, mean (p95)REI-Bench SR
Qwen3-0.6B0.6B1.2 GB2.95 s (3.56)95.58
Llama-3.2-1B1.2B2.3 GB1.34 s (1.65)93.93
Qwen3-1.7B1.7B3.3 GB2.84 s (3.30)96.81
Llama-3.2-3B3.2B6.1 GB2.31 s (2.64)95.99
Qwen3-4B4.0B7.6 GB3.85 s (4.37)95.99
Qwen3-8B8.2B15.3 GB4.06 s (4.77)96.30

Accuracy is flat from 0.6B to 8B after fine-tuning, while memory grows 13× and latency up to 3×. The adapters are merged, so fine-tuning adds no inference cost over the base model.

MW1. No sufficient motivation for using SLM.

Ans: Two reasons: on-robot inference without network access or data leaving the device, and the observation that scale alone does not solve the task under the same zero-shot prompt.

ModelREI-Bench SR
Qwen3-235B (zero-shot)43.9
GPT-OSS-120B (zero-shot)50.2
Gemma-4-31B (zero-shot)57.1
Llama-3.2-1B + ours93.9
Llama-3.2-3B + ours96.0

MW2. Structured reasoning. A comparison with semantic-role-based reasoning such as CoT-TL, or Lang2LTL would complement the graph-based baselines.

Ans: Thank you for the suggestion. We ran both on all 972 items with the same base models.

MethodLlama-3.2-3BQwen3-4B
Base (zero-shot)41.1532.72
Lang2LTL54.5358.74
Lang2LTL + LLM referring-expression grounding with dialogue (our variant)82.5186.21
CoT-TL77.3792.49
CoT-TL w/o semantic roles65.6491.15
Ours (SFT)95.9995.99

Both are NL→LTL translators, so we ported them to emit REI-Bench plans. Substitutions we will disclose: a rule-based semantic-role tagger, a plan well-formedness check in place of the model checker, and an open embedding model with a prompted translator for Lang2LTL. Lang2LTL is strong on explicit references (82–92) and weak on mixed/implicit (about 40).

MW3. Context loss and noise distraction are almost absent from base failures (0.13%), which suggests REI-Bench dialogues don’t stress memory.

Ans: We agree. REI-Bench dialogues are short and do not stress memory: base failures are dominated by action omission and ordering (84.6% for Llama-3.2-3B), and noised context does not lower base accuracy for the 3B and 4B models. We will state this as a limitation of the benchmark and avoid memory-related claims. DialFRED (31 scene objects per prompt, human dialogue) is our longer-context test.

Q1. Given A.2, B.7, and Appendix G, it appears that the majority of the REI-Bench improvement is attributable to procedural supervision. If this is the case, the abstract and introduction should be revised to reflect this finding. Ans: Yes, and we will revise the abstract and introduction accordingly.

Llama-3.2-3BOverallImplicit (std / noised / short)
Base41.1528.7 / 48.1 / 37.0
Template-only SFT (procedure only)91.4671.3 / 83.3 / 87.0
FAST-GEN SFT95.9996.3 / 96.3 / 94.4

About 50 of the 55 points come from learning the grounded plan procedure. The ambiguity-targeted data contributes the rest, concentrated on implicit references (+7 to +25).

Q2. Hyperparameter selection: Was the grid search described in Appendix F conducted on the REI-Bench test set? If yes it is recommended to perform hyperparameter selection on a held-out validation split. Additionally,also clarify why the optimal Llama grid configuration differs from the main result despite using the same settings. Ans: Thank you for raising this. We took the best configuration from the grid and rerun training for the main table. The main-table values are therefore separate runs from the grid cells; the small gaps are within the statistical range of our 5-seed runs.

ModelGrid cellMain table (re-run)5-seed mean ± std
Llama-3.2-3B95.6895.9995.25 ± 1.00
Qwen3-4B96.5096.495.64 ± 0.74
AuthorsOpenReview ↗

Reply to Reviewer 6nK9-Part 3

Q3. Please clarify why the value 95.99% appears as the main result for Llama, the Qwen B.5 result, and the Qwen-4B M.2 result, whereas the main Qwen result is reported as 96.4%.

Ans: We apologise for the confusion. These values are single-seed runs reported in different places: 96.4% is one Qwen3-4B training run, and 95.99% (933/972) is another; the Llama run happens to reach the same 933/972. For statistical reliability we also trained each model with 5 seeds:

ModelSingle-seed values in the paper5-seed mean ± stdLowest seedBest prompting baseline (ICL)
Llama-3.2-3B95.9995.25 ± 1.0093.5244.4
Qwen3-4B96.4 (main), 95.99 (B.5, M.2)95.64 ± 0.7494.3461.0

The difference between 96.4 and 95.99 is within seed variation, and even the lowest seed is more than 30 points above the best prompting baseline. In the revision we will report the 5-seed mean ± std consistently in all tables.

Q4. SafeAgentBench outcome analysis: What proportion of fine-tuned “safe” outcomes result from plans failing due to vocabulary or format issues, as opposed to refusals or intentionally safe plans?

Ans: Thank you for this question. We separated the outcomes. On the detailed hazardous tasks, most non-completed outcomes of the fine-tuned models are well-formed plans, not format failures:

Fine-tuned model (detailed hazardous, 300)Well-formed planFormat / vocabulary issueRefusal
Llama-3.2-3B + ours67.6%24.5%7.9%
Qwen3-4B + ours80.2%19.8%0.0%

To check that the lower risk is not caused by format failures, we recomputed the risk rate on well-formed plans only. The reduction remains:

Risk rate on well-formed plans only (↓)Detailed: base → oursAbstract: base → ours
Llama-3.2-3B17.7 → 8.79.5 → 4.4
Qwen3-4B27.8 → 24.318.4 → 17.4

Q5. DialFRED dialogue use: Why does the inclusion of human QA reduce symbolic success for both fine-tuned models? Does this show that the models are not utilizing clarification dialogue?

Ans: Thank you for the question. The models do read the dialogue: adding QA changes the plan in 62–67% of episodes. The QA answers only say where objects are (e.g., "the alarm clock is on the dresser"), so they have two opposite effects:

Effect of adding QA (424 episodes)Llama-3.2-3BQwen3-4B
Helps: execution success gained / lost33 / 2142 / 31
Hurts: symbolic match gained / lost37 / 4333 / 52
  • Helps: the location answers let the model find the right object, so execution success rises (+2.8 and +2.6 points).
  • Hurts: the model sometimes takes a location named in an answer as the target or destination, or drops a step, so the plan no longer matches the gold plan.

So the models use the clarification dialogue, but not reliably. We will add this analysis and list it as a limitation.

Q6. AmbiK common sense: Why do all SFT variants perform worse than the base model on common sense, and should not this be part of the main text?

Ans: The reviewer is right, and we will move this to the main text and the limitations.

AmbiK common sense (SR)Base+ Ours
Llama-3.2-1B23.815.9
Llama-3.2-3B40.635.7
Qwen3-4B45.325.7

Our reading: these items need world knowledge or a clarification question, not reference resolution from the dialogue. Our data contains neither, and it trains the model to commit to a plan from the stated context. Adding common-sense and ask-for-help cases to the generator is the natural next step.

Thanks for the response and the additional experiments. The multi seed exp results and answers for w2, q1, 6 are helpful. Still I see some issues. Q2 is not actually answered clearly: the hyperparams and the n=200 seems to have been chosen on the REI-Bench set. The gain of the full pipeline over ambiguous -> gold plan is small and the best prompting baselines such as CoT-TL at 92.5 on Qwen3-4B is now very close to your SFT. That really does significantly reduce the contribution as presented in the paper. The on-device motivation would also need to be backed up by real measurements on real world/robot rather than just on an A100. I maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.