acceptodds
Under review as a conference paper at ICLR 2027

Task-specific Context Optimization for Training-free Robot Planning

Abstract

Frontier vision-language models (VLMs) can serve as off-the-shelf high-level planners in hierarchical robot policies, using multimodal reasoning to infer subtasks for low-level execution. However, their out-of-the-box use leaves ambiguity in offline subtask annotation (for low-level policy fine-tuning) and online subtask planning, resulting in inconsistent guidance that we find degrade long-horizon task performance. We introduce TAsk-Specific Context Optimization (TASCO), a framework that adapts frozen frontier VLMs through task-specific context to produce consistent offline annotations and align causal online predictions with them. Specifically, TASCO queries a separate VLM, termed the context optimizer, to induce this context from a reference demonstration, providing a common basis for consistent task decomposition. Next, for temporally consistent online planning, the context optimizer iteratively refines this context by diagnosing mismatches between offline annotations and online predictions, accepting updates only when their consistency improves. Across multiple frontier VLMs and low-level policies, TASCO substantially improves complex, long-horizon task performance on simulated and real-robot benchmarks, while naive deployment yields surprisingly poor results. These findings establish TASCO as a model-agnostic framework for harnessing the growing reasoning capabilities of frontier VLMs by aligning high-level planning with low-level execution. Project page is available at https://iclr-tasco.github.io/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Hierarchical Robot Planning, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RZabOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TASCO, a framework that optimizes task-specific context for frozen VLM robot planners. It first derives context from a reference demonstration to produce consistent subtask annotations for low-level policy training. It then refines this context to align online planning with the offline annotations, accepting updates that improve agreement on replayed demonstrations. Experiments across multiple VLM planners, low-level policies, and simulated and real-robot tasks show improved long-horizon execution performance.

Critical strengths

The paper has a clear motivation and addresses an important and timely problem. In long-horizon robot tasks, mismatches between the environment state and high-level subtask instructions can undermine execution, making their alignment a valuable research direction.

The evaluation covers multiple high-level VLMs, low-level policies, and both simulated and real-robot tasks. This breadth supports the method’s applicability across different hierarchical robot systems.

The ablations help isolate the benefit of context refinement by keeping the offline annotations and trained low-level policy fixed. They also demonstrate the importance of evaluating proposed context updates before accepting them.

Critical weaknesses
  1. There is no negative pair in the trajectory from the offline data, as they are all successes. In this case, what assumption do you have, or how can you have a formal way to verify that the subtask is reasonable?

  2. Context refinement uses recorded expert trajectories, while robot execution can deviate from them. How robust is the online planner to execution failures and observations absent from the demonstrations?

  3. Generalization beyond the demonstrated tasks remains unclear. Can the induced subtask definitions and contexts transfer to new task compositions in which similar actions require different prerequisites or completion criteria?

Questions for authors
  1. Agreement with VLM-generated offline annotations does not establish their correctness. How sensitive is TASCO to incorrect subtask labels or transition criteria, and can refinement reinforce these errors?

  2. I understand this work was probably done concurrently with GPT-6 Astra, which can directly control the robot for long-horizon tasks. But given this, which part of the algorithm is/isn’t still needed?

4 comments

Dear reviewer RZab,
We sincerely appreciate your efforts and insightful comments to improve the manuscript.


[W1,Q1] Validity of offline annotations.

We thank the reviewer for the insightful question. Our assumption is that successful demonstrations define a valid subtask sequence, and TASCO aims to decompose them consistently rather than to find a uniquely correct decomposition. Although the demonstrations contain only successes, refinement obtains negative signals from online predictions that diverge from the offline annotations. Formally, we consider a decomposition reasonable if its subtask boundaries can be identified consistently from available observations, which the offline–online agreement in Eq. (4) directly measures. We further verify the annotations externally: relative to simulator-defined subtask boundaries, they show a narrower offset distribution than Baseline (Figure 6), and they improve a separately fine-tuned planner, Qwen3-VL-8B, by 6.9–10.9%p on RMBench (Section 4.3).

Regarding incorrect subtask labels, a misplaced subtask boundary makes the per-timestep labels incorrect, and such errors are harmful when inconsistent: randomly shifting training transitions by up to one planning interval reduces success from 64.0% to 48.9% (Figure 2(a)), which TASCO mitigates (Figure 6). In contrast, refinement turns a consistent bias into consistent guidance: TASCO's offline transitions occur slightly later than the simulator boundaries, and aligning online transitions with them improves success from 44.2% to 51.8%. Refinement thus reinforces consistency with the annotations the low-level policy was trained on, while never modifying the annotations or the label set.


[W2] Robustness to execution deviations.

For execution failures, the task-specific context specifies the conditions under which each subtask is semantically achieved (e.g., confirming that an object was released into its bin; Figure 3), so the planner does not advance to the next subtask when execution fails to meet them. This behavior appears frequently in TASCO's successful rollouts on our project page (https://iclr-tasco.github.io/): when the low-level policy repeatedly fails to lift an object due to its limited precision, the planner keeps the subtask for that object until it is picked up.

For observations absent from the demonstrations, we evaluate two real-world tasks under visual domain shifts (changed backgrounds, swapped appearances of the drawer and board, and added distractors), keeping all checkpoints and contexts fixed (setups and results are provided in Figure 9 and Table 11 of the revised manuscript). As shown below, Refined-TASCO achieves the highest score under every condition and retains 89.3% of its standard-condition performance, indicating that such shifts do not substantially degrade TASCO.

\begin{array}{ll|cccc|cccc|c} \hline \rlap{\text{Method}} & & \rlap{~~~~~\text{Object Retrieval}} & & & & \rlap{\text{Layout Reconstruction}} & & & & \text{Avg.} \newline \text{High-level Planner} & \text{Subtask Annotations} & \text{Std.} & \text{Bg.} & \text{Int.} & \text{Dist.} & \text{Std.} & \text{Bg.} & \text{Int.} & \text{Dist.} & \text{(w/o Std.)} \newline \hline \text{None} & \text{N/A} & \text{25.0} & \text{16.7} & \text{16.7} & \text{8.3} & \text{35.4} & \text{31.3} & \text{25.0} & \text{20.8} & \text{19.8} \newline \text{Qwen3-VL-8B} & \text{Baseline} & \text{41.7} & \text{33.3} & \text{41.7} & \text{16.7} & \text{43.8} & \text{39.6} & \text{39.6} & \text{18.8} & \text{31.6} \newline \text{Gemini 3.7 Flash} & \text{Baseline} & \text{25.0} & \text{25.0} & \text{16.7} & \text{16.7} & \text{43.8} & \text{35.4} & \text{37.5} & \text{29.2} & \text{26.8} \newline \text{Gemini 3.7 Flash + TASCO} & \text{TASCO} & \text{58.3} & \text{50.0} & \text{58.3} & \text{41.7} & \text{45.8} & \text{45.8} & \text{35.4} & \text{39.6} & \text{45.1} \newline \text{Gemini 3.7 Flash + Refined-TASCO} & \text{TASCO} & \textbf{91.7} & \textbf{83.3} & \textbf{83.3} & \textbf{75.0} & \textbf{77.0} & \textbf{72.9} & \textbf{66.7} & \textbf{70.8} & \textbf{75.3} \newline \hline \end{array}

[W3] Generalization to new task compositions.

To evaluate whether induced contexts transfer to new task compositions, we additionally construct a composite real-world task that performs Object Retrieval followed by Layout Reconstruction, without collecting its demonstrations. Both tasks involve placing objects, but under different prerequisites (identifying items in the drawers vs. observing the initial layout) and completion criteria. We reuse the low-level policy co-trained on the original four real-world tasks and provide the planner with the composite instruction and the two task contexts, concatenated and labeled by task, without further optimization. The planner executes the entire sequence, including the transition between tasks, without human intervention. Over 12 trials, we measure progress on both tasks using their original partial-credit criteria.

As shown below, all baselines degrade sharply on the composite task (8.3–16.7%), whereas Refined-TASCO achieves 65.6%, outperforming the Gemini baseline by 57.3%p and induction-only TASCO by 32.3%p. This suggests that task-specific contexts can be reused to compose new long-horizon tasks, with each stage following its own prerequisites and completion criteria, without additional demonstrations. We have added these results to Table 14 in the revised manuscript.


\begin{array}{ll|cc|c} \hline \text{High-level Planner} & \text{Subtask Annotations} & \text{Object Retrieval (OR)} & \text{Layout Reconstruction (LR)} & \text{OR + LR Composite} \newline \hline \text{None} & \text{N/A} & \text{25.0} & \text{35.4} & \text{12.5} \newline \text{Qwen3-VL-8B} & \text{Baseline} & \text{41.7} & \text{43.8} & \text{16.7} \newline \text{Gemini 3.7 Flash} & \text{Baseline} & \text{25.0} & \text{43.8} & \text{8.3} \newline \text{Gemini 3.7 Flash + TASCO} & \text{TASCO} & \text{58.3} & \text{45.8} & \text{33.3} \newline \text{Gemini 3.7 Flash + Refined-TASCO} & \text{TASCO} & \textbf{91.7} & \textbf{77.0} & \textbf{65.6} \newline \hline \end{array}


[Q2] Role of TASCO with stronger frontier models.

Recent studies show that GPT-6 Astra can directly control robots, but also reveal its current limitations for low-level control. Each inference call takes over ten seconds, which is impractical for closed-loop control [1], and generating dexterous actions for high-DoF embodiments remains difficult: direct in-hand control struggles to coordinate finger contacts, and humanoid control relies on pretrained whole-body controllers [2]. We therefore expect hierarchical policies that pair frontier VLMs with learned low-level policies to remain practical, which is the setting TASCO targets.

Within this setting, context refinement remains necessary regardless of planner capability: offline annotation observes future frames while online planning cannot, so even GPT-6 Astra shows offline–online transition discrepancies (Figure 2(c)), which refinement reduces. Context induction may become less critical as planners annotate more consistently, but a shared annotation protocol is still needed, since subtask granularity and phrasing are not determined by the instruction alone (Figure 1(a)). Moreover, TASCO is model-agnostic, so stronger frontier VLMs such as GPT-6 Astra can directly serve as its planner or context optimizer.

[1] Yang et al., Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy, 2026.10.
[2] Galbot Team, Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies, 2026.09.


We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns beyond those already discussed, we would be grateful if you could share them so that we may address them more thoroughly during the remainder of the rebuttal period.

The authors’ response has largely addressed my concern. I am therefore raising my recommendation to Weak Accept.

Dear Reviewer RZab,

We sincerely appreciate your acknowledgement of our rebuttal and your updated recommendation. If you have any further comments or suggestions, please let us know, as we value your feedback in improving our work.

Best regards,
Authors

Reviewer_cTyfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Authors introduce TASCO, a method for task specific context optimization for VLMs serving as high level planners in robotics. TASCO is a dual stage framework that first generates temporally consistent subtask goals for training using a separate LLM call. It then refines this context further using a pseudo-online prompting strategy and a multi-seed matching mechanism to align online plans with offline trained subtasks for effective utilization of downstream VLAs as low level controllers.

Critical strengths
  • TASCO is an efficient solution towards context optimization to improve efficiency of downstream VLAs as with the refined context, the high level planner can provide subtasks that are aligned with the training distribution learnt by VLAs
  • Experiments on simulation and real-world tasks highlight the improvement first with just a consistent context across all frames, followed by the improvements with further refinement of the context for optimized online plans.
Critical weaknesses
  • It's not clear how the proposed planning mechanism generalizes to tasks outside the training distribution used to refine the context for online planning. While the refined context consistently improves success rates, the refinement occurs over several trials of the same task, making it unclear how to use such a system for a general robot task
  • The VLA policy trained on the subtask goals learns a language distribution that is consistent with the initial task context across all demos, but this also can restrict the diversity of the queries themselves, limiting the generalizability of the VLAs themselves to low level control.
Questions for authors
  • Can the authors please clarify how the proposed solution is suitable outside tasks from the training distribution, ie deploying it in tasks where demonstrations are not available?
3 comments

Dear reviewer cTyf,
We sincerely appreciate your efforts and insightful comments to improve the manuscript.


[W1,Q1] Out-of-distribution tasks.

We first clarify that context refinement does not require additional trials of the task; it replays three recorded training demonstrations offline, using only demonstrations already required for low-level policy training.

To evaluate TASCO beyond the training distribution, we additionally construct a composite real-world task that performs Object Retrieval followed by Layout Reconstruction, without collecting its demonstrations. We reuse the low-level policy co-trained on the original four real-world tasks and provide the planner with the composite instruction and the two task contexts, concatenated and labeled by task, without further optimization. The planner executes the entire sequence, including the transition between tasks, without human intervention. Over 12 trials, we measure progress on both tasks using their original partial-credit criteria.

As shown below, all baselines degrade sharply on the composite task (8.3–16.7%), whereas Refined-TASCO achieves 65.6%, outperforming the Gemini baseline by 57.3%p and induction-only TASCO by 32.3%p. This suggests that task-specific contexts can be reused to compose new long-horizon tasks without additional demonstrations. We have added these results to Table 14 in the revised manuscript.

\begin{array}{ll|cc|c} \hline \text{High-level Planner} & \text{Subtask Annotations} & \text{Object Retrieval (OR)} & \text{Layout Reconstruction (LR)} & \text{OR + LR Composite} \newline \hline \text{None} & \text{N/A} & \text{25.0} & \text{35.4} & \text{12.5} \newline \text{Qwen3-VL-8B} & \text{Baseline} & \text{41.7} & \text{43.8} & \text{16.7} \newline \text{Gemini 3.7 Flash} & \text{Baseline} & \text{25.0} & \text{43.8} & \text{8.3} \newline \text{Gemini 3.7 Flash + TASCO} & \text{TASCO} & \text{58.3} & \text{45.8} & \text{33.3} \newline \text{Gemini 3.7 Flash + Refined-TASCO} & \text{TASCO} & \textbf{91.7} & \textbf{77.0} & \textbf{65.6} \newline \hline \end{array}


[W2] Generalizability limits due to predefined subtasks.

We agree that training on canonical subtasks involves a trade-off between consistent supervision and the diversity of subtask language. To quantify both sides, we extend the instruction-only comparison in Table 6, where subtasks are generated freely from the task instruction, by applying an LLM-based paraphrasing filter to the planner's subtask outputs at test time, identically across all settings (we have added its details to Appendix D). As shown below (RMBench, Gemini 3.7 Flash with π0.5\pi_{0.5}), canonical subtasks improve success over instruction-only by 13.5%p, whereas paraphrasing reduces TASCO's success by only 1.7%p, so TASCO with paraphrased subtasks still outperforms instruction-only by 11.8%p. Thus, the consistency of canonical subtasks substantially improves execution at a minor cost in robustness to paraphrased subtasks. We have added these results to Table 6 in the revised manuscript.

\begin{array}{ll|cc} \hline \rlap{\text{Method}} & & \rlap{~~~~~~~~~\pi_{0.5}} & \newline \hline \text{High-level Planner} & \text{Subtask Annotations} & \text{Original} & \text{Paraphrased} \newline \hline & \text{Instruction-only} & \text{30.7} & \text{30.1 (-0.6)} \newline \text{Gemini 3.7 Flash} & \text{Baseline (canonical subtasks)} & \text{29.6} & \text{26.2 (-3.4)} \newline & \text{TASCO} & \textbf{44.2} & \textbf{42.5 (-1.7)} \newline \hline \end{array}


We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns beyond those already discussed, we would be grateful if you could share them so that we may address them more thoroughly during the remainder of the rebuttal period.

Reviewer_cTyfOpenReview ↗

Response to rebuttal

Thank you for the clarifications and additional experiments. My original question concerned tasks entirely outside the training distribution. The composite task still relies on demonstrations and contexts for each component task, so it remains unclear how much TASCO helps in such open-world settings, and I will maintain my score.

Dear Reviewer cTyf,

We appreciate your follow-up, but would like to respectfully clarify the scope of our work.

Our work targets user-defined, challenging tasks on which frontier VLMs, despite sufficient planning capability, fail to plan consistently due to the lack of task-specific context, and thus cannot be properly integrated with low-level policies. TASCO addresses this problem as a general framework that adapts such a VLM to any target task to be deployed (e.g., assembling parts according to predefined rules on an industrial assembly line), which we validate across 18 simulated and real-world tasks with two frontier VLM planners and two low-level policies.

This task-specific setting reflects how hierarchical policies are often deployed in practice: the low-level policy already requires demonstrations of the target task, and TASCO uses only these demonstrations, without additional data collection or planner training. For the high-level planner, a single reference demonstration suffices to induce a context for a new task, and the induced contexts can be reused for new compositions without their own demonstrations, as shown in our composite task. Planning on tasks without any demonstrations, for which the low-level policy itself could not be trained, therefore lies outside this setting rather than being a limitation specific to TASCO.

That said, we believe that a self-evolving planner that accumulates such task-specific experience to plan consistently on arbitrary tasks would be an interesting direction for future work.

Thank you for your valuable feedback.
Best regards,
Authors

Reviewer_PXGxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to improve hierarchical robot manipulation by using a frozen VLM as a high-level planner to guide low-level execution on long-horizon tasks. To resolve both semantic ambiguity in subtask annotation and temporal uncertainty during online transitions, the authors induce and iteratively refine a task-specific context from offline reference demonstrations to align online planning with offline annotations and improve overall task success rates.

Critical strengths

Strong empirical improvements: The framework demonstrates substantial gains in task success rates across both simulated and real-world benchmarks compared to out-of- the-box VLM baselines.

Critical weaknesses

Clarity of presentation and mathematical notation: The proposed method is currently difficult to follow. Although the paper provides several figures and mathematical equations, there is no unified diagram that clearly connects the visual pipeline to the formal notation. Additionally, several mathematical symbols are under-defined. For example, what is the exact representation of ℓ\ell (raw text or VLM embeddings)?

What is the structure or dimensionality of the textual memory mtm_t? Are the observation frames Ot\mathcal{O}_t single images in $\mathbb{R}^{C \times H \times W}$ or a sequence of frames, and how is the reference video Vref\mathcal{V}_{\text{ref}} formally represented?

The proposed method relies on offline annotations to guide online planning, which implicitly assumes that deployment conditions remain identical to the reference data. How does the method perform under a domain shift—for example, if the table color changes or the box colors are swapped?

Missing baselines: The current evaluation only compares against raw models without any adaptation. Please include additional baselines that perform similar task-adaptation or context-optimization strategies for a fairer comparison.

Computational overhead and latency: Based on the demo videos, the overall execution appears relatively slow. Could you please provide a detailed runtime analysis of your approach? Specifically, how much time is spent on offline context optimization versus online inference with the high-level planner?

Questions for authors
  1. What is the exact representation of ℓ\ell (raw text or VLM embeddings)? What is the structure or dimensionality of the textual memory mtm_t? Are the observation frames Ot\mathcal{O}_t single images in RC×H×W\mathbb{R}^{C \times H \times W} or a sequence of frames, and how is the reference video Vref\mathcal{V}_{\text{ref}} formally represented?

  2. How does the method perform under a domain shift—for example, if the table color changes or the box colors are swapped?

  3. Do you have additional baselines that perform similar task-adaptation or context- optimization strategies for a fairer comparison?

  4. Could you please provide a detailed runtime analysis of your approach? Specifically, how much time is spent on offline context optimization versus online inference with the high-level planner?

2 comments

Dear reviewer PXGx,
We sincerely appreciate your efforts and insightful comments to improve the manuscript.


[W1,Q1] Clarity of presentation and mathematical notation.

We thank the reviewer for pointing out the need for clearer definitions. We clarify the representations as follows:

  • ℓ\ell is the natural-language task instruction, provided as raw text (not a VLM embedding) and held fixed throughout an episode.
  • mtm_t is a variable-length, free-form text summary generated and updated by the high-level planner, rather than a fixed-dimensional vector. Following the textual memory in [1], it retains task-relevant past observations (e.g., "A metal ring was observed in the near compartment of the left drawer.").
  • Ot\mathbf{O}_t is a short sequence of recent multi-view RGB frames up to planning step tt, providing short-term visual context, while longer-range information is carried by mtm_t.
  • Vref\mathbf{V}^{\mathrm{ref}} is the video of one reference demonstration from the training set, provided to the context optimizer as timestamped RGB frames.

We have clarified these definitions in Section 3 of the revised manuscript, where we also define the task-specific context C\mathcal{C} as a structured text. To make the roles and temporal scope of the inputs explicit, we revised the notation as Ot→It−H:t\mathbf{O}_t \to \mathbf{I}_{t-H:t}, Vref→Dref\mathbf{V}^{\mathrm{ref}} \to \mathbf{D}^{\mathrm{ref}}, and mt−→mt−Hm_{t^-} \to m_{t-H}, where HH denotes the planning interval (consistent with Figure 2), and index demonstrations with subscripts (e.g., Di\mathbf{D}_i, yˉi,t\bar{y}_{i,t}). We have also updated Figure 3 to explicitly connect the pipeline components to the corresponding symbols and equations.

[1] Torne et al., MEM: Multi-Scale Embodied Memory for Vision Language Action Models, 2026.03.


[W2,Q2] Robustness to domain shift.

We would like to clarify that TASCO does not assume deployment conditions identical to the reference data. The reference demonstration and offline annotations are used only before deployment; at deployment, the planner predicts subtasks from its current observations and memory alone. Since refinement is driven by agreement on task progress, it naturally retains details tied to task semantics (e.g., how far a drawer should be opened to see its contents) rather than domain-specific attributes such as table or drawer color.

To evaluate robustness empirically, we conducted additional real-world experiments on Object Retrieval and Layout Reconstruction. Here, we changed only the evaluation environments while keeping all policy checkpoints and task-specific contexts fixed, and evaluated each condition over 12 trials with the same partial-credit metrics as Table 2 (setups and results are provided in Figure 9 and Table 11 of the revised manuscript):

  1. Background (table color): The tabletop was covered with a white tablecloth or a dark wood-patterned sheet, each for half of the trials.
  2. Interaction site (swapped appearance): The drawer was given a wood-patterned finish and the board a white finish, swapping their original appearances.
  3. Distractors: Task-irrelevant objects (e.g., plush toys, cups, and potted plants) were added to the workspace.

As shown below, Refined-TASCO achieves the highest score under every condition, averaging 75.3% vs. 45.1% for induction-only TASCO and 26.8% for the Gemini baseline, and retains the largest fraction of its standard-condition score (89.3% vs. 77.8% for the baseline). The flat policy without a high-level planner shows a similar drop (−10.4%p), suggesting that the remaining degradation stems mainly from the low-level policy rather than the context.

\begin{array}{ll|cccc|cccc|c} \hline \rlap{\text{Method}} & & \rlap{~~~~~\text{Object Retrieval}} & & & & \rlap{\text{Layout Reconstruction}} & & & & \text{Avg.} \newline \text{High-level Planner} & \text{Subtask Annotations} & \text{Std.} & \text{Bg.} & \text{Int.} & \text{Dist.} & \text{Std.} & \text{Bg.} & \text{Int.} & \text{Dist.} & \text{(w/o Std.)} \newline \hline \text{None} & \text{N/A} & \text{25.0} & \text{16.7} & \text{16.7} & \text{8.3} & \text{35.4} & \text{31.3} & \text{25.0} & \text{20.8} & \text{19.8} \newline \text{Qwen3-VL-8B} & \text{Baseline} & \text{41.7} & \text{33.3} & \text{41.7} & \text{16.7} & \text{43.8} & \text{39.6} & \text{39.6} & \text{18.8} & \text{31.6} \newline \text{Gemini 3.7 Flash} & \text{Baseline} & \text{25.0} & \text{25.0} & \text{16.7} & \text{16.7} & \text{43.8} & \text{35.4} & \text{37.5} & \text{29.2} & \text{26.8} \newline \text{Gemini 3.7 Flash + TASCO} & \text{TASCO} & \text{58.3} & \text{50.0} & \text{58.3} & \text{41.7} & \text{45.8} & \text{45.8} & \text{35.4} & \text{39.6} & \text{45.1} \newline \text{Gemini 3.7 Flash + Refined-TASCO} & \text{TASCO} & \textbf{91.7} & \textbf{83.3} & \textbf{83.3} & \textbf{75.0} & \textbf{77.0} & \textbf{72.9} & \textbf{66.7} & \textbf{70.8} & \textbf{75.3} \newline \hline \end{array}

[W3,Q3] Missing baselines.

We appreciate your suggestion to explore how additional baselines on i) task adaptation and ii) context optimization compare with TASCO. To measure this, we provide the following baselines on RMBench:

  • In-context learning [3,4]: following the few-shot prompting used for LLM-based planners, the planner receives three demonstrations annotated with ground-truth subtask labels as in-context examples, in place of an induced context, for both offline annotation and online planning. The low-level policy is trained on the resulting annotations.
  • OPRO [5]: in place of our refinement, it proposes new contexts from previously evaluated contexts and their scores, without observing the subtask predictions, and keeps the highest-scoring one. It starts from the same induced context and optimizes the same objective (i.e., agreement with the offline annotations on the same three demonstrations).
  • GEPA [6]: in the same setup, it reflects on the paired offline and online subtask sequences and their scores using its generic reflection prompt, and explores multiple candidates selected from the Pareto front of per-demonstration agreement.

As shown in the table below, TASCO outperforms in-context learning by 9.1%p, even though in-context learning uses three demonstrations with ground-truth subtask labels, whereas TASCO uses a single unlabeled reference demonstration. For context optimization, our refinement achieves the largest gain over the induced context (7.6%p), outperforming both OPRO and GEPA under the same objective, as it diagnoses what the planner could not observe without future frames rather than relying on scores or generic reflection. We have added these results to Table 12 in the revised manuscript.

\begin{array}{ll|c} \hline \text{High-level Planner} & \text{Subtask Annotations} & \text{RMBench Success Rate} \newline \hline \text{Gemini 3.7 Flash} & \text{Baseline} & \text{29.6} \newline \text{Gemini 3.7 Flash + In-context learning [3,4]} & \text{In-context learning [3,4]} & \text{35.1} \newline \text{Gemini 3.7 Flash + TASCO} & \text{TASCO} & \textbf{44.2} \newline \hline \text{Gemini 3.7 Flash + OPRO [5]} & \text{TASCO} & \text{50.5} \newline \text{Gemini 3.7 Flash + GEPA [6]} & \text{TASCO} & \text{43.7} \newline \text{Gemini 3.7 Flash + Refined-TASCO} & \text{TASCO} & \textbf{51.8} \newline \hline \end{array}

[3] Song et al., LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models, 2022.12.
[4] Singh et al., ProgPrompt: Generating Situated Robot Task Plans using Large Language Models, 2022.09.
[5] Yang et al., Large Language Models as Optimizers, 2023.09.
[6] Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, 2025.07.


[W4,Q4] Computational overhead and latency.

As requested, we provide a detailed runtime analysis on RMBench, comparing offline context optimization with online inference, with Gemini 3.7 Flash as the high-level planner and Claude Opus 5 as the context optimizer. The demo videos appear slow because the robot holds its pose while waiting for the API-based planner (an average of 7.1 s per call), which affects wall-clock time rather than task success and applies equally to the API-based Baseline. TASCO introduces no additional online model calls, and its context optimization runs once per task offline, without robot interaction. We have added this analysis as Table 13 in the revised manuscript.

\begin{array}{lllc} \hline \text{Stage} & \text{Frequency} & \text{Model calls} & \text{Latency} \newline \hline \text{Offline context induction} & \text{Once per task} & \text{1 optimizer call} & {\sim}\text{76 s} \newline \text{Offline context refinement} & \text{Once per task} & \text{2 optimizer calls + 3-demo replays} & {\sim}\text{20 min} \newline \text{Online inference} & \text{Once per planning step} & \text{1 planner call} & \text{7.1 s} \newline \hline \end{array}


We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns beyond those already discussed, we would be grateful if you could share them so that we may address them more thoroughly during the remainder of the rebuttal period.

Comments on the paper

1 comment
Area_Chair_1xpiOpenReview ↗

Reminder: Please Review Authors’ Rebuttal and Provide Feedback

Dear Reviewers,

Thank you for your reviews. If you have not already done so, please read the authors' rebuttal and share your feedback in the discussion thread at your earliest convenience. Your timely responses would be greatly appreciated.

Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.