acceptodds
Under review as a conference paper at ICLR 2027

Action-Guided In-Context Example Retrieval for Embodied Task Planning

Abstract

Given only a small set of annotated instruction-plan demonstrations, large language models (LLMs) have shown promising performance in embodied task planning through in-context learning. However, existing approaches primarily retrieve in-context examples based on linguistic or visual similarity, so the retrieved examples may fail to include the actions required by the query task and thus provide limited guidance for generating an appropriate task plan. To address this limitation, we propose an action-guided retrieval approach that retrieves examples relevant to the actions implied by the query to better guide task plan generation. To estimate this relevance, we perform action-guided representation learning based on action sets and action sequences derived from annotated demonstrations, yielding instruction embeddings that capture the action types and their action patterns necessary to complete the task. We further introduce an action-preserving augmentation strategy that diversifies training instructions while preserving their underlying action information to alleviate the scarcity of annotated demonstrations. Experiments on embodied task planning benchmarks show that the proposed method outperforms existing retrieval strategies when integrated with multiple LLMs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Task Planning, In-Context Example Retrieval, In-Context Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ly69OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AGR, an ICL approach for embodied task planning. It aims to retrieve examples that share relevant action types and action sequences with the query task. Specifically, the method extracts action-set and action-sequence labels from annotated plans and employs supervised contrastive learning to train two instruction embedding projectors. Experiments on ALFRED, EB-ALFRED, and WAH-NL demonstrate improvements over existing retrieval-based planning methods across multiple backbones.

Critical strengths
  • The paper identifies a niche limitation of similarity-based ICL methods in embodied task planning. The proposed approach is also relatively lightweight and does not require fine-tuning the planning LLM.

  • AGR achieves improvements over existing retrieval strategies on multiple embodied task planning benchmarks and across different planning LLMs.

Critical weaknesses
  • The overall methodology largely follows a widely used pipeline using contrastive learning in ICL. Similar learning-based retrieval approaches like EPR, UDR, and CEIL have been extensively explored. The proposed pipeline combines LLM-generated data augmentation, supervised contrastive learning, and similarity-based retrieval, but the methodological and novelty contribution beyond these existing techniques appears relatively limited.

  • The comparisons do not isolate the benefits of the proposed design from those of additional task-specific supervision and synthetic data. AGR leverages ground-truth action labels and LLM-generated augmented instructions to train its retriever, while the main retrieval baselines do not receive equivalent supervised representation learning. Comparisons against simpler supervised retrievers using the same augmented training data are missing.

  • The training objective treats instructions as positive pairs only when they share exactly the same action set or action sequence. Consequently, instructions with substantial overlap in their required actions but different action compositions are treated as negatives. My concern is whether the learned embeddings capture compositional action similarity or primarily distinguish action patterns observed during training.

  • The paper does not clearly establish how AGR performs when the query requires a novel action combination with no exact match among the available demonstrations. Such cases are important for evaluating whether the method can generalize beyond previously observed action patterns.

  • The necessity of maintaining two separate representation spaces is not sufficiently established. Although Table 4 shows that combining action-set and action-sequence representations generally performs well, the improvements over individual projectors are often modest, and the combined model does not consistently outperform both alternatives. A clearer analysis of when each representation is beneficial, and why their combination is necessary, would strengthen the methodological justification.

Questions for authors

Please refer to the weakness section.

Further, the evaluation primarily uses LLaMA-2, GPT-3.5, and GPT-4o. Would the improvements persist with more recent and capable instruction-following or reasoning models?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Ly69 (1/6)

Thank you for your detailed comments on our work. We hope our responses clarify our work and address your concerns.

Before addressing the reviewer's comments, we would like to clarify that GPT-3.5 (gpt-3.5-turbo-instruct), which was used in our original experiments, was retired from the OpenAI API on September 28, 2026. Therefore, we used LLaMA2 and GPT-4o for the additional task planning experiments in this rebuttal.

AuthorsOpenReview ↗

Response to Reviewer Ly69 (2/6) — [W1] Comparison with Existing Retrieval Methods

W1: Comparison with Existing Retrieval Methods

As the reviewer mentioned, AGR builds on established techniques, including contrastive learning, linear projectors, and LLM-based augmentation. However, the main contribution of AGR lies in introducing action-level relevance as a novel retrieval criterion for embodied task planning, rather than developing new learning algorithms.

Existing learning-based methods train a retriever model using various criteria. EPR evaluates the usefulness of each demonstration based on the probability that an LLM assigns to the correct output when the demonstration is included in the prompt, and uses this information to train a retriever. UDR leverages LLM feedback to learn relative rankings among demonstrations, while CEIL jointly considers the relevance of demonstrations to the query and the diversity among demonstrations to optimize the selection of in-context examples. Unlike these approaches, AGR explicitly considers action-level relevance for in-context example retrieval by learning two separate instruction representation spaces that capture required action types and their patterns to complete the task. This approach allows AGR to retrieve action-relevant demonstrations using only the query instruction.

To evaluate the effectiveness of our retrieval strategy against existing approaches, we compared AGR with EPR, UDR, and CEIL. Since these baselines require annotated instruction-plan pairs for training, we trained them using only the 100 original demonstrations, as our augmentation generates alternative instructions without corresponding task plans. For retrieval, all methods used the same pool of 100 original demonstrations. With both LLaMA2 and GPT-4o, AGR achieved higher SR and GC than all baselines on ALFRED-seen, ALFRED-unseen, and EB-ALFRED. These results suggest that action-level relevance provides a more effective criterion for retrieving in-context examples in embodied task planning than the existing retrieval strategies.

To further isolate the effect of our retrieval strategy, we also compared AGR with alternative supervised retrievers using the same augmented instructions; please see our response to W2 for these results.

We will clarify AGR’s contribution in Section 1 and report these comparisons in the appendix.


ALFRED‑seenALFRED‑unseenEB‑ALFRED
LLMMethodSRGCHLPSRGCHLPSRGCHLP
LLaMA2EPR20.1233.6226.9519.2432.4128.9920.6733.4225.67
UDR13.9030.3016.5911.9430.4717.7815.3335.2417.33
CEIL21.2234.9933.0523.1435.9933.6219.0028.4825.67
AGR23.7837.6035.7326.1940.0044.0923.6740.8330.33
GPT‑4oEPR39.3948.9861.0736.0547.3668.3335.0041.7445.67
UDR39.5149.3161.8337.7649.7667.1136.0047.0744.00
CEIL38.6648.4156.5936.9148.3562.7334.6739.9243.00
AGR40.3749.5061.8338.2550.2868.9437.6748.8946.33
AuthorsOpenReview ↗

Response to Reviewer Ly69 (3/6) — [W2] Retrieval Baselines Based on Augmented Dataset

W2: Retrieval Baselines Based on Augmented Dataset

Our action-preserving augmentation generates alternative instructions by modifying the linguistic expressions and the referenced objects or receptacles while retaining the action-set and action-sequence labels of the source demonstration. Since the modified objects and receptacles may be inconsistent with the original task plans, we use augmented instructions only for learning projectors, not for in-context example retrieval.

LLM-Planner and FLARE do not involve retriever training, so they cannot use augmented instructions as additional training data. Meanwhile, EPR, UDR, and CEIL require valid instruction-plan pairs to train their retriever, making it difficult to utilize our augmented instructions without corresponding valid task plans.

Therefore, to isolate the contribution of AGR's retrieval design from that of action-label supervision and augmented data, we trained alternative retrievers using our augmented dataset and compared their performance with AGR. Specifically, we employed linear classifiers that share the single-layer structure of AGR's projectors and LLM classifiers using Qwen2.5-7B fine-tuned with LoRA. For both approaches, we trained two separate classifiers to predict action sets and action sequences, respectively. At inference time, we obtained the predicted probability distributions over action sets and sequences for the query. For each training demonstration, we averaged the predicted probabilities of the query corresponding to the demonstration's action-set and action-sequence labels to compute a retrieval score. We then retrieved the nine demonstrations with the highest scores, using linguistic similarity to resolve ties. For the LLM classifiers, we applied softmax to the generation probabilities of candidate action labels given the query instruction.

With the same augmented training data and action labels, AGR achieved higher SR and GC than the linear classifiers on all three benchmarks with both LLaMA2 and GPT-4o. Compared with the fine-tuned Qwen2.5-7B classifiers, AGR achieved higher SR and GC on ALFRED-seen and ALFRED-unseen, while the classifiers achieved higher scores on EB-ALFRED with both planners. This advantage on EB-ALFRED came with a trade-off in the number of trainable parameters: fine-tuning Qwen2.5-7B with LoRA required updating 40.4M parameters, compared with 0.2M for AGR. Furthermore, Qwen2.5-7B requires more computation during inference than AGR, which relies on a frozen BERT encoder and lightweight projectors. The comparison with the linear classifiers also shows that the choice of retrieval method affects planning performance even when the training data and supervision are fixed. In addition, Table 5 of our manuscript presents an ablation study evaluating the effects of action-preserving augmentation.

We will report the supervised retriever comparisons in the appendix.


ALFRED‑seenALFRED‑unseenEB‑ALFRED
LLMPredictorSRGCHLPSRGCHLPSRGCHLP
LLaMA2Linear21.7135.8931.3421.8036.7534.1021.6737.8423.67
Qwen2.5‑7B (FT)23.0537.0835.6125.8239.7243.7324.3340.9631.00
AGR23.7837.6035.7326.1940.0044.0923.6740.8330.33
GPT‑4oLinear39.1548.9358.9035.6947.9765.1635.6744.6043.67
Qwen2.5‑7B (FT)40.2449.2761.2238.0050.0968.5739.3350.8548.00
AGR40.3749.5061.8338.2550.2868.9437.6748.8946.33
Reviewer_tey7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Action-Guided Retrieval (AGR), a method for choosing in-context examples for LLM-based embodied task planning when only a few demonstrations are available. Existing methods retrieve examples by instruction or visual similarity. AGR instead trains two linear projectors on a frozen text encoder with supervised contrastive learning, so that instructions with the same action set, or the same action sequence, from their ground-truth plans end up close together. The scarce training instructions are expanded with LLM paraphrases that keep the plan's actions unchanged. At test time, the 9 nearest demonstrations under the combined similarity are given to the planner. Using 100 demonstrations, AGR improves over LLM-Planner and FLARE on ALFRED (validation and test), EB-ALFRED and WAH-NL with LLaMA-2, GPT-3.5 and GPT-4o.

Critical strengths
  1. A simple, cheap and broadly effective fix to a real failure mode.

The motivation analysis (Fig. 1) shows that planning success rises with the number of retrieved examples whose actions match the query. AGR roughly doubles the number of matched examples compared with FLARE: about 6.4 vs 2.7 out of 9 share the action set (Fig. D1). The method is only a linear projector on a frozen encoder, adds negligible cost at inference, and can be dropped into any ICL planner.

  1. The gains are consistent across settings.

Improvements hold across three planners, three benchmarks, two simulators (AI2-THOR and VirtualHome), and both seen and unseen environments. They also hold on the official ALFRED test leaderboard, and when replanning is disabled for all methods (App. C), which removes an advantage the baselines had. Higher exact-match plan accuracy (HLP) in every Table 2 setting suggests the gains really come from better plans.

Critical weaknesses
  1. The method largely reduces to predicting task type, and the obvious simpler alternatives are not compared.

In ALFRED, action sets and sequences are almost fully determined by the seven fixed task templates (pick & place, heat, cool, clean, look-at, two objects, movable receptacle). The action-guided embedding is therefore close to a task-type classifier trained on instructions. Instruction words such as "microwave" or "rinse" already reveal the type in most cases.

The technical parts are standard: supervised contrastive learning on label-defined positives, a linear projector, and LLM paraphrase augmentation. Retrievers trained with labels or outputs already exist for in-context example selection (e.g. Liu et al. 2025, which the paper cites; EPR, Rubin et al. 2022).

Missing baselines: retrieval based on action sets predicted by an LLM or a classifier, including the skill-based selection of An et al. (2023), which is cited but not compared; a keyword or task-type heuristic; an oracle that retrieves by ground-truth action labels, which would show how much of the achievable gain AGR captures.

Without these, it is unclear whether the learned representation adds anything beyond "retrieve demonstrations of the same task type".

  1. The evidence is statistically weak where it matters most, and the motivating analysis is correlational.

With the strongest planner (GPT-4o, Table 2), gains over FLARE are small: +2.1 / +3.4 SR on seen / unseen, and a tie on EB-ALFRED (37.67 for both).

On the test set, AGR uses GPT-4o while FLARE uses GPT-4, so the comparison is not like-for-like, and the results are mixed.

On WAH-NL, the SR gains of 2–4 points correspond to only 4–8 of 195 instructions.

The component ablation (Table 4) shows differences of under 1 point, and the combined projector is not consistently best: g_set alone has higher unseen SR.

Fig. 1 is likely confounded by task type: types that are common in the 100-shot pool get more matched examples and may also be easier. Breaking the results down per task type would separate the two effects.

Together these make it hard to judge how robust the improvements are, especially as stronger LLMs narrow the gap.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer tey7 (1/7)

We appreciate your helpful comments. We hope our responses clarify our work and address your concerns.

Before addressing the reviewer's comments, we would like to clarify that GPT-3.5 (gpt-3.5-turbo-instruct), which was used in our original experiments, was retired from the OpenAI API on September 28, 2026. Therefore, we used LLaMA2 and GPT-4o for the additional task planning experiments in this rebuttal.

AuthorsOpenReview ↗

Response to Reviewer tey7 (2/7) — [W1] Task Types, Action Labels, and Retrieval Setup

W1: Task Types, Action Labels, and Retrieval Baselines

To address these concerns, we 1) clarify the differences between task types and action labels, and 2) present additional experiments comparing AGR with alternative retrieval methods, including classification-based approaches and existing in-context example retrieval methods.

1) Difference Between Task Types and Action Labels

Although ALFRED defines several task types, they cannot fully determine the action sets and sequences required for task completion. Our empirical analysis reveals two important distinctions between task types and action labels. First, demonstrations within the same task type can require different actions. In the pool of 100 training demonstrations used for retrieval, six of the seven task types contained multiple action sets, and every task type contained between two and five distinct action sequences. Second, the same action sequence can appear across different task types. For example, "Two Objects" and "Movable Receptacle" tasks can both require the same action sequence (Pick, Put, Pick, Put). For these reasons, retrieval strategies based solely on task types may struggle to provide LLMs with demonstrations that accurately reflect the actions required to accomplish the given task. Furthermore, task-type labels may not always be available in real-world scenarios, and assigning such labels to each demonstration can require considerable annotation effort. These challenges may limit the practical applicability of retrieval methods that rely on task-type information.

2) Comparison with Alternative Retrieval Methods

2-1) Comparison with classification-based retrieval

[Experimental settings]

  • Retrieval Information: We compared task types and action labels (action sets and sequences) as retrieval criteria.

  • Classification Models: We compared AGR's projectors with linear classifiers, zero-shot Qwen2.5-7B, and Qwen2.5-7B fine-tuned using LoRA (Qwen2.5-7B FT). For each classification approach, we used one model to predict task types and two separate models to predict action sets and action sequences from query instructions.

  • Training Data: The linear classifiers and Qwen2.5-7B FT were trained using the 100 original demonstrations and 2,000 augmented demonstrations, which were used to train AGR's projectors.

[Retrieval score]

  • Task Type Classifier: We first predicted the task type of the query and then retrieved the nine demonstrations with the highest linguistic similarity to the query within the predicted task type. The training pool contains at least ten demonstrations for each task type.

  • Action Label Classifiers: Since most action labels are associated with fewer than nine training demonstrations, we used predicted probabilities from classifiers to assign a retrieval score to each training demonstration. For each query, we obtained softmax probabilities over possible action sets and action sequences from two separate classifiers, respectively. Then, for each training demonstration, we identified its action set and action sequence labels and obtained the probabilities assigned to these labels. We combined these two probabilities with equal weights of 0.5 each to compute the demonstration's retrieval score. Finally, we retrieved the nine demonstrations with the highest scores, using linguistic similarity to resolve ties. For the LLM classifiers, we applied softmax to the generation probabilities of candidate action labels given the query instruction.

  • Oracle Retrieval: We assumed access to the ground-truth task type and action labels of each query. For task types, we retrieved the nine demonstrations with the highest linguistic similarity to the query within its ground-truth task type. For action labels, we computed the Jaccard similarity between the query and each training demonstration using their action sets, and the normalized longest common subsequence similarity using their action sequences. We then combined these two similarities with equal weights of 0.5 each and retrieved the nine demonstrations with the highest scores.

AuthorsOpenReview ↗

Response to Reviewer tey7 (3/7) — [W1] Analysis of Classifier and Oracle Results

[Experimental Results]


  • Task Types vs. Action Labels: Across ALFRED-seen, ALFRED-unseen, and EB-ALFRED, retrieval using action labels achieved higher SR, GC, and HLP in most cases than retrieval using task types with linear classifiers, Qwen2.5-7B (zero-shot), Qwen2.5-7B FT, and oracle retrieval. However, the performance gaps were relatively small with Qwen2.5-7B FT and oracle retrieval using GPT-4o. Two factors may explain these smaller gaps.

    First, we found that although retrieval based on action labels consistently identified more demonstrations with action sets and sequences matching those of the query, the gap in the number of action-matched demonstrations between retrieval based on task types and action labels decreased with Qwen2.5-7B FT and oracle retrieval compared with linear classifiers. This may be because accurate task-type prediction allows linguistic similarity to retrieve demonstrations with matching actions from the predicted task type, potentially reducing the differences in task planning performance. Second, stronger LLM-based planners such as GPT-4o may generate accurate task plans by inferring the required actions even when retrieved demonstrations provide limited action-level guidance. As a result, the additional action-matched demonstrations retrieved using action-label information may yield relatively small improvements in task planning performance.

    Although task-type retrieval achieved competitive task planning performance, task-type labels may not always be available in real-world scenarios and may require additional annotation effort. Moreover, retrieval using action labels achieved better performance in most settings, supporting the use of action information derived from demonstration plans, as in AGR.


  • AGR vs. Linear Classifiers: AGR consistently achieved higher SR and GC than the linear classifiers across all evaluated settings, even though both used the same training data and a single linear layer. While the linear classifiers were trained using cross-entropy loss to predict action labels, AGR employed supervised contrastive learning to learn action-guided representations. These results support the effectiveness of learning representations tailored to demonstration retrieval rather than directly predicting action labels.

  • AGR vs. LLM Classifiers: AGR consistently outperformed zero-shot Qwen2.5-7B classifiers, while Qwen2.5-7B FT achieved higher SR and GC on EB-ALFRED. However, AGR achieved competitive performance with only 0.2M trainable parameters for its projectors, compared with 40.4M for LoRA fine-tuning. Furthermore, LLM classifiers require inference with a 7B parameter model for each query instruction, highlighting the trade-off between task planning performance and computational efficiency.

  • AGR vs. Oracle: Compared with oracle retrieval using ground-truth action sets and sequences, AGR retrieved an average of 1.10 fewer demonstrations with matching action sets and 1.94 fewer with matching action sequences among nine retrieved demonstrations. In terms of task planning performance, AGR achieved SR 1.33% to 2.93% lower than oracle retrieval across all evaluated settings. Although AGR achieved higher task planning performance compared to most alternative retrieval methods and existing baselines, the remaining gap to oracle suggests that there is still room to improve AGR by retrieving more demonstrations with matching actions.
Reviewer_TCRFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method to retrieve samples relevant to the current instruction in terms of actions and action sets, and uses the retrieved demonstrations to augment the instruction. The authors empirically demonstrate the effectiveness of augmenting instructions with relevant demonstrations.

Critical strengths
  1. Overall, I think the proposed method is intuitively reasonable, and selecting demonstrations that are more relevant to the current instruction based on action relevance is also sensible.

  2. The experiments demonstrate the effectiveness of the proposed method.

Critical weaknesses
  1. The proposed method is too incremental, even though it is effective. Augmenting the current instruction with more relevant demonstrations can lead to performance improvements, which is not surprising.

  2. Incorporating more demonstrations into the LLM input also increases token consumption. I think the authors should be cautious about providing too many demonstrations to the LLM, especially considering the high cost of LLM inference.

Questions for authors

Could the authors also consider evaluating the computational cost and token consumption when incorporating more demonstrations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TCRF (1/4)

Thank you for your helpful comments. We hope our responses help to clarify our work and answer all the concerns.

Before addressing the reviewer's comments, we would like to clarify that GPT-3.5 (gpt-3.5-turbo-instruct), which was used in our original experiments, was retired from the OpenAI API on September 28, 2026. Therefore, we used LLaMA2 and GPT-4o for the additional task planning experiments in this rebuttal.

AuthorsOpenReview ↗

Response to Reviewer TCRF (2/4) — [W1] Novelty of AGR

W1: Novelty of AGR

As the reviewer noted, providing demonstrations that are more relevant to the query can help LLMs perform the given task more effectively. However, the key challenge lies in how to identify such relevant demonstrations from the training pool.

Existing ICL-based approaches for embodied task planning primarily retrieve demonstrations based on linguistic similarity between instructions or visual similarity between the agents' observations. However, these retrieval strategies do not explicitly consider action-level relevance, making it difficult to provide LLMs with useful guidance on the actions required to accomplish the query task.

To address this limitation, AGR introduces action-level relevance as a novel retrieval criterion for embodied task planning, considering both the action types and their patterns required for task completion. Since the ground-truth action information of the query is unavailable at inference time, AGR learns instruction representations that capture action-set and action-sequence relevance using the annotated task plans paired with instructions in the training pool. These learned representations enable AGR to retrieve action-relevant demonstrations from the training pool using only the query instruction.

In summary, the contribution of AGR lies not in showing that more relevant demonstrations improve task planning performance, but in proposing an action-guided retrieval approach that identifies relevant demonstrations based on the actions required to accomplish the query task.

We will clarify AGR’s contribution in the introduction.

AuthorsOpenReview ↗

Response to Reviewer TCRF (3/4) — [W2] Performance with Different Numbers of In-Context Examples

W2: Performance with Different Numbers of In-Context Examples

In our experiments, we followed the evaluation protocol of LLM-Planner and FLARE by providing nine demonstrations as in-context examples to the LLMs. Thus, AGR does not incur additional token consumption by providing more in-context examples than the baselines.

To further address the concern about LLM inference cost, we conducted experiments with different numbers of in-context examples (K = 3, 5, 7, 9) on the ALFRED validation set (both seen and unseen environments) and EB-ALFRED. The results show that AGR outperforms the baselines in most cases across different numbers of in-context examples, including settings with fewer than nine examples. More importantly, we found that AGR achieves competitive performance with only five or seven in-context examples, demonstrating its potential for more cost-efficient task planning.

We will report task planning results for different numbers of demonstrations in the appendix.


ALFRED-seenALFRED-unseenEB-ALFRED
LLMKMethodSRGCHLPSRGCHLPSRGCHLP
LLaMA23LLM-Planner3.5413.286.344.0215.076.343.3310.922.67
FLARE17.9331.2924.7621.9235.4229.7214.6726.7917.00
AGR21.4634.7132.4424.3637.8838.7318.0032.6421.33
5LLM-Planner5.6115.557.684.0216.398.054.0011.443.67
FLARE23.0536.3228.7823.3936.6032.5216.3331.2121.00
AGR24.1538.5035.4928.2640.9444.3424.6739.6627.67
7LLM-Planner7.0717.028.904.2616.796.214.0012.222.33
FLARE24.6337.1331.8323.2336.2636.0517.6733.0322.33
AGR23.5437.4134.6326.9240.1444.9522.3338.1028.33
9LLM-Planner6.1016.268.664.3814.869.743.0013.653.67
FLARE24.8837.0833.5422.9036.3235.9321.6735.8923.67
AGR23.7837.6035.7326.1940.0044.0923.6740.8330.33
ALFRED-seenALFRED-unseenEB-ALFRED
LLMKMethodSRGCHLPSRGCHLPSRGCHLP
GPT-4o3LLM-Planner23.7833.9536.2223.6336.6542.1412.3321.8525.33
FLARE30.9840.9751.8331.1842.2660.0529.6737.4539.00
AGR36.4645.9955.2436.3047.7464.4331.3341.2241.00
5LLM-Planner25.4936.7042.8029.6042.9251.7714.6726.1427.67
FLARE34.8844.5754.8833.2544.2063.7033.0041.3542.67
AGR38.9048.7058.4136.5449.1065.7735.0046.0344.00
7LLM-Planner29.1538.9347.5632.6446.8955.4215.0025.6231.33
FLARE36.9546.2357.6834.1045.0967.2437.3347.9845.67
AGR40.1249.5560.8537.8850.2867.7237.6749.8046.00
9LLM-Planner31.2241.3550.8536.4249.8659.3216.6728.0928.67
FLARE38.2947.8959.7634.8446.1367.3637.6747.3345.00
AGR40.3749.5061.8338.2550.2868.9437.6748.8946.33
AuthorsOpenReview ↗

Response to Reviewer TCRF (4/4) — [Q1] Token Consumption and Inference Cost

Q1: Token Consumption and Inference Cost

We analyzed how the number of in-context examples affects token consumption and computational cost by comparing AGR with FLARE, a strong baseline for ICL-based embodied task planning. This analysis was conducted on the ALFRED validation set (seen environments) using LLaMA2 and GPT-4o with K = 3, 5, 7, 9, and 11. For task plan generation, both methods use a single LLM call with the same prompt format, differing only in the retrieved in-context examples. We measured the average input and output tokens and evaluated inference cost using latency for LLaMA2 and API costs for GPT-4o. The API costs of the GPT-4o were calculated using the official OpenAI pricing of $2.50 per million input tokens and $10.00 per million output tokens. For the latency analysis of LLaMA2, we used a system equipped with an AMD EPYC 9555 64-Core CPU, 2.1 TiB RAM, and an NVIDIA B300 SXM6 GPU with 288 GB of memory.

The results show that both AGR and FLARE exhibit higher inference latency and API costs as K increases. This increase is primarily driven by the additional input tokens from more in-context examples. Under the same K, we observed that AGR consumes slightly fewer input tokens than FLARE, resulting in lower inference costs. This suggests that AGR tends to retrieve shorter in-context examples than FLARE.

The results provided in W2 and our cost analysis suggest that AGR can achieve competitive embodied task planning performance with K = 5 or 7 while reducing inference latency and API costs compared with the K = 9 setting used in our original experiments.

We will report token usage, latency, and API cost in the appendix.


Input and output tokens and inference latency are averaged per query, while GPT-4o API costs are reported in USD per 100 queries. Token counts are measured using each LLM's tokenizer

LLaMA2GPT-4o
KMethodInput tokensOutput tokensLatency (s)Input tokensOutput tokensInput cost ($)Output cost ($)
3FLARE1083.542.00.59905.428.20.2260.028
AGR1001.638.60.54837.627.40.2090.027
5FLARE1473.340.60.631234.028.20.3080.028
AGR1349.538.90.581131.927.70.2830.028
7FLARE1875.440.20.671574.328.30.3940.028
AGR1717.238.80.621442.627.70.3610.028
9FLARE2293.139.90.721927.328.50.4820.028
AGR2084.638.40.651752.627.90.4380.028
11FLARE2721.740.60.762289.728.40.5720.028
AGR2456.338.50.682067.028.00.5170.028

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers and Manuscript Revisions

We thank all reviewers for carefully reading our paper and providing constructive feedback. In response to this valuable feedback, we have clarified the contribution of AGR and strengthened the evaluation of the proposed method through additional experiments and analyses.

We have incorporated these clarifications, additional experimental results, and discussions into the revised manuscript. For ease of reference, we have highlighted changes in the main text in yellow and marked new or revised appendix sections by highlighting their headings in yellow. Detailed point-by-point responses are provided under each review.

We hope these revisions and responses address the reviewers' concerns and further clarify the contributions of AGR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.