acceptodds
Under review as a conference paper at ICLR 2027

TaskAnchor: Grounding Task State in Reactive VLAs for Long-Horizon Manipulation

Abstract

Reactive vision-language-action (VLA) policies suffer from task-state aliasing in long-horizon manipulation, where identical multimodal inputs call for distinct, context-dependent actions. Given that pretrained VLAs already possess rich control primitives to express diverse behaviors, we hypothesize that the execution bottleneck lies not in policy capacity, but in input ambiguity. In this paper, we propose TaskAnchor, a lightweight adapter that grounds task state by injecting execution context into the VLA’s native input space. During post-training, TaskAnchor learns to represent subtask stage as a milestone-supervised coordinate prepended to the language instruction, while incorporating fine-grained historical evidence via a residual update to the current visual tokens. This formulation avoids generating complex subtask instructions and leaves the backbone architecture unchanged. Across long-horizon benchmarks, TaskAnchor delivers substantial gains, achieving approximately the average success rate of the and X-VLA baselines on RMBench and more than doubling the task success rate of on RoboMemArena. Real-robot experiments further validate reliable multi-stage execution, with the same policy adapting its subsequent behaviors using earlier human interactions as in-context cues. Our project website is available at https://anonymous.4open.science/api/repo/taskanchor-80C3/file/index.html.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hrxgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed TASKANCHOR, a novel model structure to identify the task state problem that causing latency in the long-horizon manipulation tasks that requires the robot to (1) know the current subtask stage and (2) able to recover history-grounded task context. Contect-adaptive visual conditioning and Task-state coordinate prediction and conditioning jointly optimized the memory bank and trained the TASKANCHOR that is used for conditioning the ReactiveVLA. The paper is in good quality and the research problem is novel. Evaluation on RMBench and RoboMemArena demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper is researching task state aliasing problem, which is novel and important for current VLA paradigm for long-horizon tasks.
  2. The proposed TASKANCHOR is novel and useful to build a good memory bank to solve the "same observation, different intended action" problem.
  3. The experimental results on RMBench and RoboMemArena benchmark demonstrate the effectiveness of the proposed method. It improves long-horizon performance over prior methods.
  4. Ablation study verifies the effectiveness of the added feature of Coordinate and history observation.
  5. Only 2ms of additional latency is acceptable even though the TASKANCHOR brings additional model weight.
  6. Project page is stunning.
Critical weaknesses
  1. In long-horizon tasks, sometimes the robot will face failure and recovery. Say the robot was told to "move a red block onto a blue pad and then return it to its original slot", if the robot mistakenly returned the red block to the wrong slot, and retried to pick it up and place it to the original slot. In this failure recovery process, does the failure process counts history memory?
  2. What if a robot moves wrong at the very beginning of the manipulation so the accumulated memory is negative?
  3. Using TTT to compress sequence history is a controllable way to store history content. But will it bring forgetting as the task horizon increases? It will be good to show the relationship between the model performance and the task horizon.

Minor:

  1. Position of Table 3,4,5 are awkward.
Questions for authors

See weakness

Reviewer_h4dcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TaskAnchor addresses task-state aliasing in reactive vision-language-action policies, where similar observations can require different actions depending on the execution stage and prior interactions. It adds two signals to a pretrained VLA: a history-conditioned visual residual and a scalar task-state coordinate, predicted from milestone supervision and prepended to the task instruction. Its CA-TTT adapter constructs temporary fast weights from compressed visual history at each decision and discards them after use, while retaining an external history buffer. The authors evaluate the method with two VLA backbones on RMBench and RoboMemArena, and on three real-robot tasks.

Critical strengths

The paper gives a clear and useful account of two sources of ambiguity in long-horizon manipulation: identifying the current stage and recovering details from earlier interactions. The proposed conditioning signals map naturally onto these needs, and using the VLA's existing visual and language interfaces keeps the action-generation pathway intact.

The empirical results are promising across several settings. On RMBench, TaskAnchor substantially improves over its reactive backbones. On RoboMemArena, TaskAnchor-π0.5 reports 50.1% TSR and 64.7% CSR, compared with 21.5% and 38.7% for π0.5. The component ablation on RMBench also supports combining the coordinate and history-conditioned visual pathways: the full model reaches 0.70 average success, compared with 0.31 for coordinate-only and 0.20 for history-visual-only (Table 3). The real-robot results are encouraging, though limited to three tasks.

Critical weaknesses
  1. The contribution of CA-TTT is not isolated from the broader post-training recipe. The ablation shows that both conditioning signals are useful, but does not establish that CA-TTT is necessary compared with a simpler history encoder trained with the same milestone supervision. In addition, selected downstream VLA components are jointly fine-tuned after adapter warm-up (Appendix A.1), so the gains cannot be attributed solely to the proposed adapter. This makes the interpretation that TaskAnchor primarily unlocks capabilities already present in the pretrained policy less conclusive.

  2. The generalization evidence is limited to the evaluated task and object distributions. The two backbones and real-robot experiments demonstrate breadth across architectures and settings, but the paper does not report held-out object or scene evaluations. The extent to which the method transfers to unfamiliar objects and environments therefore remains unclear.

Questions for authors
  1. Could the authors clarify which downstream parameters are updated for each backbone, particularly π0.5, and how they view the contribution of CA-TTT relative to this downstream fine-tuning? The current ablations show that both conditioning pathways help, but do not isolate the specific benefit of CA-TTT over a simpler history encoder.

  2. What motivates representing task progress with a scalar coordinate rather than semantic subtask instructions, and how do the authors interpret the lower Sequence performance relative to PrediMem? Since the policy is trained with ground-truth coordinates but uses predictions at inference, it would also be useful to clarify how coordinate errors affect behavior on policy-generated rollouts.

  3. How should the generalization claims be scoped with respect to unseen objects and scenes? The paper evaluates two VLA backbones and includes real-robot tasks, but does not appear to use held-out object or scene splits; it would help to clarify what transfer these results establish and what remains untested.

Reviewer_YGzhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies task-state confusion in reactive vision-language-action models during long-horizon manipulation, where similar current observations and identical task instructions may require different actions depending on the execution stage or interaction history. The authors propose TaskAnchor, which incorporates historical information into the VLA’s native input interfaces through two complementary pathways. First, CA-TTT constructs temporary fast weights from historical visual features to produce a residual update to the current visual features. Second, the method predicts task-state coordinates supervised by semantic milestones and encodes them as a text prefix. The model is trained through joint post-training of the adapters and a subset of the VLA parameters, without requiring ground-truth stage annotations at inference time.

The authors evaluate TaskAnchor on RMBench, RoboMemArena, and three real-world robot tasks, demonstrating its effectiveness with both π₀.₅ and X-VLA backbones. The experiments show that the method significantly improves long-horizon execution on the evaluated tasks while maintaining low additional inference overhead.

Critical strengths
  1. The performance gains are substantial and span different backbones and multiple types of history-dependent tasks. On RMBench, the per-task training version of TaskAnchor-π₀.₅ improves the average success rate from 0.14 to 0.79, while the jointly trained version reaches 0.70; substantial improvements are also observed with X-VLA. On RoboMemArena, the average TSR/CSR reaches 50.1%/64.7%, compared with PrediMem’s 38.5%/55.2%, respectively. These results show that the gains extend beyond comparisons with relatively weak reactive baselines to comparisons with existing memory-based methods. Results from 25 trials on each of three real-world robot tasks provide further support for its effectiveness in practical execution.
  2. The dual-pathway interface follows a simple and effective design and the ablation results support the complementarity of the two pathways. The method preserves historical details through visual residuals and provides explicit stage information through a language prefix, without requiring a separate planner to generate subtask instructions. In Table 3, the full model achieves an average success rate of 0.70, substantially higher than 0.31 with coordinate-only conditioning and 0.20 with historical visual conditioning alone, indicating that the two forms of conditioning are complementary under the current setup. The approximately 27M additional parameters and 2 ms of additional latency per call reported in Table 5 also make the approach attractive for deployment.
Critical weaknesses
  1. The existing experiments do not sufficiently isolate the independent contributions of the proposed mechanisms. Table 3 validates the roles of the two output pathways, while Table 4 examines the effects of network depth and historical feature resolution. However, CA-TTT is not directly compared with simpler history fusion modules, such as standard cross-attention, under the same stage supervision, historical inputs, and training budget. It therefore remains unclear how much of the performance advantage comes from CA-TTT itself and how much comes from explicit stage supervision and dual-pathway conditioning. Furthermore, Appendix A.1 explicitly states that a subset of the downstream VLA parameters is jointly optimized after warm-up; for X-VLA, the trainable components include the action core and action head. Thus, these results support the claim that “task-state conditioning and joint post-training can improve long-horizon execution,” but are insufficient to establish the stronger interpretation that “the main bottleneck is input ambiguity rather than policy capability.” Figure 16 shows that the post-trained model indeed uses coordinate conditioning, but does not distinguish whether this response draws on existing capabilities or reflects a new mapping learned during post-training. This affects my assessment of the paper’s contribution at the mechanism level.
  2. The reliability of stage coordinate predictions and the discrepancy between training and inference have not been directly quantified. The paper does not report coordinate prediction errors, the distribution of errors near stage transitions, or the closed-loop performance gap between using ground-truth and predicted coordinates. The intervention experiments in Figure 16 show that coordinates can alter action selection, while also suggesting that incorrect stage predictions may directly lead to incorrect behavior. The existing experiments cannot establish how tolerant the model is to actual prediction errors, which also raises concerns about the reproducibility of the supervision scheme.
Questions for authors
  1. Are there results for replacing CA-TTT with standard cross-attention under the same historical inputs, stage annotations, and training configuration? Have the authors evaluated a version that freezes the entire VLA and trains only TaskAnchor? The former would help assess the necessity of CA-TTT, while the latter could support the claim of “activating existing long-horizon capabilities.”
  2. Please report the gap in closed-loop success rates between conditioning on predicted and ground-truth coordinates, along with prediction performance near stage transitions. How exactly are milestone boundaries obtained for each dataset? Have the authors analyzed the effects of coordinate perturbations or changes in annotation granularity?
  3. For the real-robot experiments, please provide the number of training demonstrations for each task, clarify whether TaskAnchor and X-VLA use the same data and matched training budgets, and specify the range of variation in object positions, object identities, and human cues across the 25 trials. This information is necessary to interpret the 25/25 success rate on all three tasks and understand the scope of these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.