- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper mainly propose FAST-GEN. It use failure recovery and analysis to generate data for fine-tuning small language model planners. The fine-tuned planner gets stronger ability in interpreting ambiguous instructions. The method is tested on on REI-Bench, AmbiK, CLARA, DialFRED, and SafeAgentBench.
Critical strengths
- The paper's focusing on an interesting question, discussing how improving SLM planning under ambiguous instructions.
- The evaluation is complete and has a wide scope. The result covers multiple benchmarks, including both in-domain and out-of-domain situations.
- The ablation study on component is well done, and synthetic-data controls ensures isolated effects comparison of the methods.
Critical weaknesses
-
No real-robot validation. I consider this a fatal gap in supporting the claimed real-world applicability, as robustness to perception errors and physical execution uncertainty remains untested.
-
Insufficient comparison with related methods. After reading the paper, I think PRISM is the only approach that includes data-generation + fine-tuning, and it performs far worse than basic template-only SFT. This comparison provides unconvincing evidence of contribution of FAST-GEN over existing training or data generation methods.
-
The major improvement seems comes from SFT. In the results shown by the author, simple template-based SFT already provides substantial improvements. The additional gains from proposed components is relatively modest. This makes the added value on the main benchmark unclear.
-
Unclear component contributions to OOD generalization. In-distribution gains are expected after task-specific training. More importantly, the method should demonstrate that its proposed components contribute to generalized ambiguity instructions. However, component ablations on OOD execution benchmarks is very limited.
Questions for authors
-
Can the authors provide real-robot evaluation to support the claimed practical applicability under perception and execution uncertainty?
-
Can the authors compare against additional failure-driven data generation and training methods? Why PRISM performs even worse than simply template-only SFT?
-
Can the authors directly measure the fine-tuned students’ target-object and destination resolution accuracy? This can isolate the proposed components' effects in improving ambiguity understanding? Is it better than simply template-based SFT?
-
Can the authors extend the simple SFT and no-Critic controls to OOD execution benchmarks such as DialFRED to evaluate the generalization of ambiguity understanding?
2 comments
Thank you for the response and additional experiments.
- Narrowing the claims acknowledges the lack of real-robot validation but does not resolve it. Given the substantial sim-to-real gap in robotics, real-world applicability remains unsupported.
- No additional external method is compared. Since PRISM’s failures largely stem from mismatched action conventions, a comparison under aligned conventions is necessary. The current baseline comparison remains inadequate.
- Q3 remains unanswered: target-object and destination resolution accuracy are not separately evaluated. The reported overall SR does not establish whether the gains come from improved ambiguity understanding.
- The additional OOD ablations address Q4, and I appreciate this evidence of component effectiveness.
I understand the constraints on real-robot evaluation, and reframing the paper as a language-oriented study would be reasonable. However, weaknesses 2 and 3 remain critical: even when assessed purely as an LLM study, the experimental evidence is insufficient to establish the method’s advantage over existing approaches or attribute its gains to improved ambiguity understanding.
In conclusion, substantial concerns remain, and I maintain my original rating.
Reply to Reviewer S7Dq-Part 1
We thank the reviewer for the careful and constructive review. We answer each point below.
W1/Q1. No real-robot validation. I consider this a fatal gap in supporting the claimed real-world applicability, as robustness to perception errors and physical execution uncertainty remains untested.
Ans: We thank the reviewer for raising this important point, and we agree that real-robot validation would strengthen the work. Unfortunately, our lab does not have access to a physical robot platform, so due to this resource constraint we are not able to provide real-robot experiments. We kindly ask the reviewer's understanding on this.
We will state the absence of real-robot evaluation clearly as a limitation and revise the wording on real-world applicability accordingly. We consider real-robot evaluation valuable future work.
W2/Q2. Insufficient comparison with related methods. After reading the paper, I think PRISM is the only approach that includes data-generation + fine-tuning, and it performs far worse than basic template-only SFT. This comparison provides unconvincing evidence of contribution of FAST-GEN over existing training or data generation methods. An s: Thank you. We agree that template-only SFT is the stronger baseline, and we will present it as the main SFT baseline instead of PRISM.
Why PRISM is low: PRISM generates data for unambiguous ALFRED-style instructions. Models trained on it write plans that skip steps REI-Bench requires (for example the
findstep beforepick_up).So PRISM's low score is mostly a plan-convention mismatch, not evidence about ambiguity. For a fair comparison of data-generation choices under the same budget, format and training recipe, we use controlled variants of our own pipeline:
We are not aware of another failure-driven data-generation method for ambiguous embodied instructions that we could run in this period; we will add any the reviewer suggests.
W3. The major improvement seems comes from SFT. In the results shown by the author, simple template-based SFT already provides substantial improvements. The additional gains from proposed components is relatively modest. This makes the added value on the main benchmark unclear.
Ans: Thank you, this is correct for the overall REI-Bench number, and we will say so in the paper. The added value appears in two places:
W4/Q4. Unclear component contributions to OOD generalization. In-distribution gains are expected after task-specific training. More importantly, the method should demonstrate that its proposed components contribute to generalized ambiguity instructions. However, component ablations on OOD execution benchmarks is very limited.
Ans: Thank you for this suggestion. We ran it for this response: each ablation model was executed in AI2-THOR on the 424 DialFRED validation episodes (success decided by ALFRED's goal checker). AmbiK is added as a second OOD benchmark.