- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces TASCO, a framework that optimizes task-specific context for frozen VLM robot planners. It first derives context from a reference demonstration to produce consistent subtask annotations for low-level policy training. It then refines this context to align online planning with the offline annotations, accepting updates that improve agreement on replayed demonstrations. Experiments across multiple VLM planners, low-level policies, and simulated and real-robot tasks show improved long-horizon execution performance.
Critical strengths
The paper has a clear motivation and addresses an important and timely problem. In long-horizon robot tasks, mismatches between the environment state and high-level subtask instructions can undermine execution, making their alignment a valuable research direction.
The evaluation covers multiple high-level VLMs, low-level policies, and both simulated and real-robot tasks. This breadth supports the method’s applicability across different hierarchical robot systems.
The ablations help isolate the benefit of context refinement by keeping the offline annotations and trained low-level policy fixed. They also demonstrate the importance of evaluating proposed context updates before accepting them.
Critical weaknesses
-
There is no negative pair in the trajectory from the offline data, as they are all successes. In this case, what assumption do you have, or how can you have a formal way to verify that the subtask is reasonable?
-
Context refinement uses recorded expert trajectories, while robot execution can deviate from them. How robust is the online planner to execution failures and observations absent from the demonstrations?
-
Generalization beyond the demonstrated tasks remains unclear. Can the induced subtask definitions and contexts transfer to new task compositions in which similar actions require different prerequisites or completion criteria?
Questions for authors
-
Agreement with VLM-generated offline annotations does not establish their correctness. How sensitive is TASCO to incorrect subtask labels or transition criteria, and can refinement reinforce these errors?
-
I understand this work was probably done concurrently with GPT-6 Astra, which can directly control the robot for long-horizon tasks. But given this, which part of the algorithm is/isn’t still needed?
4 comments
[W3] Generalization to new task compositions.
To evaluate whether induced contexts transfer to new task compositions, we additionally construct a composite real-world task that performs Object Retrieval followed by Layout Reconstruction, without collecting its demonstrations. Both tasks involve placing objects, but under different prerequisites (identifying items in the drawers vs. observing the initial layout) and completion criteria. We reuse the low-level policy co-trained on the original four real-world tasks and provide the planner with the composite instruction and the two task contexts, concatenated and labeled by task, without further optimization. The planner executes the entire sequence, including the transition between tasks, without human intervention. Over 12 trials, we measure progress on both tasks using their original partial-credit criteria.
As shown below, all baselines degrade sharply on the composite task (8.3–16.7%), whereas Refined-TASCO achieves 65.6%, outperforming the Gemini baseline by 57.3%p and induction-only TASCO by 32.3%p. This suggests that task-specific contexts can be reused to compose new long-horizon tasks, with each stage following its own prerequisites and completion criteria, without additional demonstrations. We have added these results to Table 14 in the revised manuscript.
\begin{array}{ll|cc|c} \hline \text{High-level Planner} & \text{Subtask Annotations} & \text{Object Retrieval (OR)} & \text{Layout Reconstruction (LR)} & \text{OR + LR Composite} \newline \hline \text{None} & \text{N/A} & \text{25.0} & \text{35.4} & \text{12.5} \newline \text{Qwen3-VL-8B} & \text{Baseline} & \text{41.7} & \text{43.8} & \text{16.7} \newline \text{Gemini 3.7 Flash} & \text{Baseline} & \text{25.0} & \text{43.8} & \text{8.3} \newline \text{Gemini 3.7 Flash + TASCO} & \text{TASCO} & \text{58.3} & \text{45.8} & \text{33.3} \newline \text{Gemini 3.7 Flash + Refined-TASCO} & \text{TASCO} & \textbf{91.7} & \textbf{77.0} & \textbf{65.6} \newline \hline \end{array}
[Q2] Role of TASCO with stronger frontier models.
Recent studies show that GPT-6 Astra can directly control robots, but also reveal its current limitations for low-level control. Each inference call takes over ten seconds, which is impractical for closed-loop control [1], and generating dexterous actions for high-DoF embodiments remains difficult: direct in-hand control struggles to coordinate finger contacts, and humanoid control relies on pretrained whole-body controllers [2]. We therefore expect hierarchical policies that pair frontier VLMs with learned low-level policies to remain practical, which is the setting TASCO targets.
Within this setting, context refinement remains necessary regardless of planner capability: offline annotation observes future frames while online planning cannot, so even GPT-6 Astra shows offline–online transition discrepancies (Figure 2(c)), which refinement reduces. Context induction may become less critical as planners annotate more consistently, but a shared annotation protocol is still needed, since subtask granularity and phrasing are not determined by the instruction alone (Figure 1(a)). Moreover, TASCO is model-agnostic, so stronger frontier VLMs such as GPT-6 Astra can directly serve as its planner or context optimizer.
[1] Yang et al., Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy, 2026.10.
[2] Galbot Team, Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies, 2026.09.
We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns beyond those already discussed, we would be grateful if you could share them so that we may address them more thoroughly during the remainder of the rebuttal period.
The authors’ response has largely addressed my concern. I am therefore raising my recommendation to Weak Accept.
Dear Reviewer RZab,
We sincerely appreciate your acknowledgement of our rebuttal and your updated recommendation. If you have any further comments or suggestions, please let us know, as we value your feedback in improving our work.
Best regards,
Authors
Dear reviewer RZab,
We sincerely appreciate your efforts and insightful comments to improve the manuscript.
[W1,Q1] Validity of offline annotations.
We thank the reviewer for the insightful question. Our assumption is that successful demonstrations define a valid subtask sequence, and TASCO aims to decompose them consistently rather than to find a uniquely correct decomposition. Although the demonstrations contain only successes, refinement obtains negative signals from online predictions that diverge from the offline annotations. Formally, we consider a decomposition reasonable if its subtask boundaries can be identified consistently from available observations, which the offline–online agreement in Eq. (4) directly measures. We further verify the annotations externally: relative to simulator-defined subtask boundaries, they show a narrower offset distribution than Baseline (Figure 6), and they improve a separately fine-tuned planner, Qwen3-VL-8B, by 6.9–10.9%p on RMBench (Section 4.3).
Regarding incorrect subtask labels, a misplaced subtask boundary makes the per-timestep labels incorrect, and such errors are harmful when inconsistent: randomly shifting training transitions by up to one planning interval reduces success from 64.0% to 48.9% (Figure 2(a)), which TASCO mitigates (Figure 6). In contrast, refinement turns a consistent bias into consistent guidance: TASCO's offline transitions occur slightly later than the simulator boundaries, and aligning online transitions with them improves success from 44.2% to 51.8%. Refinement thus reinforces consistency with the annotations the low-level policy was trained on, while never modifying the annotations or the label set.
[W2] Robustness to execution deviations.
For execution failures, the task-specific context specifies the conditions under which each subtask is semantically achieved (e.g., confirming that an object was released into its bin; Figure 3), so the planner does not advance to the next subtask when execution fails to meet them. This behavior appears frequently in TASCO's successful rollouts on our project page (https://iclr-tasco.github.io/): when the low-level policy repeatedly fails to lift an object due to its limited precision, the planner keeps the subtask for that object until it is picked up.
For observations absent from the demonstrations, we evaluate two real-world tasks under visual domain shifts (changed backgrounds, swapped appearances of the drawer and board, and added distractors), keeping all checkpoints and contexts fixed (setups and results are provided in Figure 9 and Table 11 of the revised manuscript). As shown below, Refined-TASCO achieves the highest score under every condition and retains 89.3% of its standard-condition performance, indicating that such shifts do not substantially degrade TASCO.
\begin{array}{ll|cccc|cccc|c} \hline \rlap{\text{Method}} & & \rlap{~~~~~\text{Object Retrieval}} & & & & \rlap{\text{Layout Reconstruction}} & & & & \text{Avg.} \newline \text{High-level Planner} & \text{Subtask Annotations} & \text{Std.} & \text{Bg.} & \text{Int.} & \text{Dist.} & \text{Std.} & \text{Bg.} & \text{Int.} & \text{Dist.} & \text{(w/o Std.)} \newline \hline \text{None} & \text{N/A} & \text{25.0} & \text{16.7} & \text{16.7} & \text{8.3} & \text{35.4} & \text{31.3} & \text{25.0} & \text{20.8} & \text{19.8} \newline \text{Qwen3-VL-8B} & \text{Baseline} & \text{41.7} & \text{33.3} & \text{41.7} & \text{16.7} & \text{43.8} & \text{39.6} & \text{39.6} & \text{18.8} & \text{31.6} \newline \text{Gemini 3.7 Flash} & \text{Baseline} & \text{25.0} & \text{25.0} & \text{16.7} & \text{16.7} & \text{43.8} & \text{35.4} & \text{37.5} & \text{29.2} & \text{26.8} \newline \text{Gemini 3.7 Flash + TASCO} & \text{TASCO} & \text{58.3} & \text{50.0} & \text{58.3} & \text{41.7} & \text{45.8} & \text{45.8} & \text{35.4} & \text{39.6} & \text{45.1} \newline \text{Gemini 3.7 Flash + Refined-TASCO} & \text{TASCO} & \textbf{91.7} & \textbf{83.3} & \textbf{83.3} & \textbf{75.0} & \textbf{77.0} & \textbf{72.9} & \textbf{66.7} & \textbf{70.8} & \textbf{75.3} \newline \hline \end{array}