- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AGR, an ICL approach for embodied task planning. It aims to retrieve examples that share relevant action types and action sequences with the query task. Specifically, the method extracts action-set and action-sequence labels from annotated plans and employs supervised contrastive learning to train two instruction embedding projectors. Experiments on ALFRED, EB-ALFRED, and WAH-NL demonstrate improvements over existing retrieval-based planning methods across multiple backbones.
Critical strengths
-
The paper identifies a niche limitation of similarity-based ICL methods in embodied task planning. The proposed approach is also relatively lightweight and does not require fine-tuning the planning LLM.
-
AGR achieves improvements over existing retrieval strategies on multiple embodied task planning benchmarks and across different planning LLMs.
Critical weaknesses
-
The overall methodology largely follows a widely used pipeline using contrastive learning in ICL. Similar learning-based retrieval approaches like EPR, UDR, and CEIL have been extensively explored. The proposed pipeline combines LLM-generated data augmentation, supervised contrastive learning, and similarity-based retrieval, but the methodological and novelty contribution beyond these existing techniques appears relatively limited.
-
The comparisons do not isolate the benefits of the proposed design from those of additional task-specific supervision and synthetic data. AGR leverages ground-truth action labels and LLM-generated augmented instructions to train its retriever, while the main retrieval baselines do not receive equivalent supervised representation learning. Comparisons against simpler supervised retrievers using the same augmented training data are missing.
-
The training objective treats instructions as positive pairs only when they share exactly the same action set or action sequence. Consequently, instructions with substantial overlap in their required actions but different action compositions are treated as negatives. My concern is whether the learned embeddings capture compositional action similarity or primarily distinguish action patterns observed during training.
-
The paper does not clearly establish how AGR performs when the query requires a novel action combination with no exact match among the available demonstrations. Such cases are important for evaluating whether the method can generalize beyond previously observed action patterns.
-
The necessity of maintaining two separate representation spaces is not sufficiently established. Although Table 4 shows that combining action-set and action-sequence representations generally performs well, the improvements over individual projectors are often modest, and the combined model does not consistently outperform both alternatives. A clearer analysis of when each representation is beneficial, and why their combination is necessary, would strengthen the methodological justification.
Questions for authors
Please refer to the weakness section.
Further, the evaluation primarily uses LLaMA-2, GPT-3.5, and GPT-4o. Would the improvements persist with more recent and capable instruction-following or reasoning models?
7 comments
Response to Reviewer Ly69 (2/6) — [W1] Comparison with Existing Retrieval Methods
W1: Comparison with Existing Retrieval Methods
As the reviewer mentioned, AGR builds on established techniques, including contrastive learning, linear projectors, and LLM-based augmentation. However, the main contribution of AGR lies in introducing action-level relevance as a novel retrieval criterion for embodied task planning, rather than developing new learning algorithms.
Existing learning-based methods train a retriever model using various criteria. EPR evaluates the usefulness of each demonstration based on the probability that an LLM assigns to the correct output when the demonstration is included in the prompt, and uses this information to train a retriever. UDR leverages LLM feedback to learn relative rankings among demonstrations, while CEIL jointly considers the relevance of demonstrations to the query and the diversity among demonstrations to optimize the selection of in-context examples. Unlike these approaches, AGR explicitly considers action-level relevance for in-context example retrieval by learning two separate instruction representation spaces that capture required action types and their patterns to complete the task. This approach allows AGR to retrieve action-relevant demonstrations using only the query instruction.
To evaluate the effectiveness of our retrieval strategy against existing approaches, we compared AGR with EPR, UDR, and CEIL. Since these baselines require annotated instruction-plan pairs for training, we trained them using only the 100 original demonstrations, as our augmentation generates alternative instructions without corresponding task plans. For retrieval, all methods used the same pool of 100 original demonstrations. With both LLaMA2 and GPT-4o, AGR achieved higher SR and GC than all baselines on ALFRED-seen, ALFRED-unseen, and EB-ALFRED. These results suggest that action-level relevance provides a more effective criterion for retrieving in-context examples in embodied task planning than the existing retrieval strategies.
To further isolate the effect of our retrieval strategy, we also compared AGR with alternative supervised retrievers using the same augmented instructions; please see our response to W2 for these results.
We will clarify AGR’s contribution in Section 1 and report these comparisons in the appendix.
| ALFRED‑seen | ALFRED‑unseen | EB‑ALFRED | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| LLM | Method | SR | GC | HLP | SR | GC | HLP | SR | GC | HLP |
| LLaMA2 | EPR | 20.12 | 33.62 | 26.95 | 19.24 | 32.41 | 28.99 | 20.67 | 33.42 | 25.67 |
| UDR | 13.90 | 30.30 | 16.59 | 11.94 | 30.47 | 17.78 | 15.33 | 35.24 | 17.33 | |
| CEIL | 21.22 | 34.99 | 33.05 | 23.14 | 35.99 | 33.62 | 19.00 | 28.48 | 25.67 | |
| AGR | 23.78 | 37.60 | 35.73 | 26.19 | 40.00 | 44.09 | 23.67 | 40.83 | 30.33 | |
| GPT‑4o | EPR | 39.39 | 48.98 | 61.07 | 36.05 | 47.36 | 68.33 | 35.00 | 41.74 | 45.67 |
| UDR | 39.51 | 49.31 | 61.83 | 37.76 | 49.76 | 67.11 | 36.00 | 47.07 | 44.00 | |
| CEIL | 38.66 | 48.41 | 56.59 | 36.91 | 48.35 | 62.73 | 34.67 | 39.92 | 43.00 | |
| AGR | 40.37 | 49.50 | 61.83 | 38.25 | 50.28 | 68.94 | 37.67 | 48.89 | 46.33 |
Response to Reviewer Ly69 (3/6) — [W2] Retrieval Baselines Based on Augmented Dataset
W2: Retrieval Baselines Based on Augmented Dataset
Our action-preserving augmentation generates alternative instructions by modifying the linguistic expressions and the referenced objects or receptacles while retaining the action-set and action-sequence labels of the source demonstration. Since the modified objects and receptacles may be inconsistent with the original task plans, we use augmented instructions only for learning projectors, not for in-context example retrieval.
LLM-Planner and FLARE do not involve retriever training, so they cannot use augmented instructions as additional training data. Meanwhile, EPR, UDR, and CEIL require valid instruction-plan pairs to train their retriever, making it difficult to utilize our augmented instructions without corresponding valid task plans.
Therefore, to isolate the contribution of AGR's retrieval design from that of action-label supervision and augmented data, we trained alternative retrievers using our augmented dataset and compared their performance with AGR. Specifically, we employed linear classifiers that share the single-layer structure of AGR's projectors and LLM classifiers using Qwen2.5-7B fine-tuned with LoRA. For both approaches, we trained two separate classifiers to predict action sets and action sequences, respectively. At inference time, we obtained the predicted probability distributions over action sets and sequences for the query. For each training demonstration, we averaged the predicted probabilities of the query corresponding to the demonstration's action-set and action-sequence labels to compute a retrieval score. We then retrieved the nine demonstrations with the highest scores, using linguistic similarity to resolve ties. For the LLM classifiers, we applied softmax to the generation probabilities of candidate action labels given the query instruction.
With the same augmented training data and action labels, AGR achieved higher SR and GC than the linear classifiers on all three benchmarks with both LLaMA2 and GPT-4o. Compared with the fine-tuned Qwen2.5-7B classifiers, AGR achieved higher SR and GC on ALFRED-seen and ALFRED-unseen, while the classifiers achieved higher scores on EB-ALFRED with both planners. This advantage on EB-ALFRED came with a trade-off in the number of trainable parameters: fine-tuning Qwen2.5-7B with LoRA required updating 40.4M parameters, compared with 0.2M for AGR. Furthermore, Qwen2.5-7B requires more computation during inference than AGR, which relies on a frozen BERT encoder and lightweight projectors. The comparison with the linear classifiers also shows that the choice of retrieval method affects planning performance even when the training data and supervision are fixed. In addition, Table 5 of our manuscript presents an ablation study evaluating the effects of action-preserving augmentation.
We will report the supervised retriever comparisons in the appendix.
| ALFRED‑seen | ALFRED‑unseen | EB‑ALFRED | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| LLM | Predictor | SR | GC | HLP | SR | GC | HLP | SR | GC | HLP |
| LLaMA2 | Linear | 21.71 | 35.89 | 31.34 | 21.80 | 36.75 | 34.10 | 21.67 | 37.84 | 23.67 |
| Qwen2.5‑7B (FT) | 23.05 | 37.08 | 35.61 | 25.82 | 39.72 | 43.73 | 24.33 | 40.96 | 31.00 | |
| AGR | 23.78 | 37.60 | 35.73 | 26.19 | 40.00 | 44.09 | 23.67 | 40.83 | 30.33 | |
| GPT‑4o | Linear | 39.15 | 48.93 | 58.90 | 35.69 | 47.97 | 65.16 | 35.67 | 44.60 | 43.67 |
| Qwen2.5‑7B (FT) | 40.24 | 49.27 | 61.22 | 38.00 | 50.09 | 68.57 | 39.33 | 50.85 | 48.00 | |
| AGR | 40.37 | 49.50 | 61.83 | 38.25 | 50.28 | 68.94 | 37.67 | 48.89 | 46.33 |
Response to Reviewer Ly69 (1/6)
Thank you for your detailed comments on our work. We hope our responses clarify our work and address your concerns.
Before addressing the reviewer's comments, we would like to clarify that GPT-3.5 (gpt-3.5-turbo-instruct), which was used in our original experiments, was retired from the OpenAI API on September 28, 2026. Therefore, we used LLaMA2 and GPT-4o for the additional task planning experiments in this rebuttal.