acceptodds
Withdrawn from ICLR 2027

RecastVLA: From Past Interaction to Future Control with Adaptive Policy States

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sequential manipulation requires a robot to track what has already happened, even when the current scene no longer reveals it. Policies with explicit history representations make past interactions available as context for current decisions. We ask how action generation itself can form a persistent state for subsequent control. Building on action-side test-time training, RecastVLA maintains an adaptive policy state within a flow-matching vision-language-action policy. The state is represented by shared fast weights and remains fixed throughout action generation. Depth-specific interfaces read the same state, while features across depths and flow evaluations jointly define one update for the next policy call. Subsequent action losses train the initialization, interfaces, and update rule by differentiating through earlier state transitions. At deployment, updates use the policy's own action-generation features without expert action labels. Across LIBERO, RoboTwin, RoboDojo, and twelve real-robot tasks, RecastVLA improves mean success over a matched policy trained without test-time training, including 10.68 percentage points on RoboTwin Clean→Clean. In controlled RoboTwin comparisons, retaining state improves success, and the shared design exceeds independently trained layer-local TTT by 2.58 points.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action models, Robot learning, Test-time training, Fast weights, Adaptive policy states, Sequential manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_upfROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RecastVLA, a TTT framework for VLAs that uses a single shared state instead of layer-local states. RecastVLA improves over a matched baseline across diverse general manipulation benchmarks and real-robot tasks that require execution memory for successful completion.

Critical strengths
  • The paper demonstrates the effectiveness of the proposed TTT framework across diverse simulation benchmarks and real-robot tasks.

  • The experience transfer experiment with different state initializations (Figure 3, Table 2) is interesting and provides useful insight into how prior experience can affect control.

Critical weaknesses
  • RecastVLA’s main architectural contribution over RoboTTT is the use of a shared state instead of layer-local states. However, the motivation for this change is somewhat unclear, and the improvement from state sharing itself is relatively modest. The gains over the matched base seem to mostly come from the TTT architecture (Table 2, 3) instead of the shared state design.

  • The analysis of the proposed state-sharing design is conducted primarily on RoboTwin, which does not directly align with the paper’s motivation of execution memory. Additional comparisons on long-horizon or explicitly memory-dependent tasks would better demonstrate the benefits of RecastVLA.

Questions for authors

See weaknesses also.

  • How do the authors position the main novelty of RecastVLA relative to RoboTTT: the shared-state design, or the use of TTT for execution memory?
  • Why is using a single shared state preferable to layer-local states? Can the authors provide further comparisons between layer-local TTT and RecastVLA on long-horizon or memory-dependent benchmarks? (Table 3)
  • Can the authors further analyze how the degree of task similarity affects the effectiveness of experience transfer? (Table 2)
  • Further analysis of how the fast state changes throughout the rollout trajectory will help understanding the behavior of the TTT mechanism.
Reviewer_QCoGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents RecastVLA, a vision-language-action framework that incorporates persistent execution memory through shared adaptive policy states. Motivated by the difficulty of tracking task progress from current observations alone, the proposed method represents execution memory using the fast weights of a lightweight MLP shared across multiple depths of a flow-matching action expert.

The framework introduces depth-specific interfaces that allow different layers to access the shared state. Features collected across network depths and flow-matching evaluations are aggregated to perform a single gradient-based memory update after each policy call. The memory initialization, read/write interfaces, and update mechanism are jointly trained by differentiating subsequent action prediction losses through earlier state transitions. During deployment, the policy continuously updates its fast weights using internally generated features without requiring expert action supervision.

Experiments on LIBERO, RoboTwin, RoboDojo, and twelve real-world robotic manipulation tasks demonstrate consistent improvements over a matched baseline without TTT. Additional experiments investigate the effects of memory continuity, the relevance of prior experience, and shared versus layer-local memory organization.

Critical strengths
  1. The paper addresses an important challenge in sequential robotic manipulation: maintaining information about previous interactions when the current observation is insufficient to determine task progress. The proposed approach offers an interesting alternative to conventional history-conditioned policies by representing execution memory directly within the action expert through adaptive fast weights. The use of a shared state with depth-specific access interfaces also provides a reasonable mechanism for integrating information from different stages of action generation.

  2. The proposed memory update mechanism is technically well formulated. In particular, maintaining a fixed memory state throughout each action-generation process and performing a joint update at the policy-call boundary provides a clear separation between action generation and memory adaptation. Moreover, differentiating subsequent action prediction losses through previous fast-weight updates offers a principled way to learn memory representations that are useful for future control. This design enables test-time memory adaptation without requiring additional expert action labels.

  3. The experimental evaluation covers multiple simulation benchmarks and twelve real-world manipulation tasks, providing evidence for the effectiveness of the proposed framework across different settings. Beyond comparisons with the matched baseline, the paper includes several informative ablations. The Full versus Reset experiment demonstrates the benefit of maintaining state continuity across policy calls, while the Similar versus Other initialization experiment highlights the importance of the relevance of prior experience. The comparison against independently trained layer-local TTT is also valuable, as it directly examines the proposed shared-state organization under a consistent per-state memory architecture.

Critical weaknesses
  1. While the proposed framework is technically reasonable, its main methodological contributions appear somewhat incremental relative to existing TTT-based approaches. Specifically, RoboTTT has already explored fast-weight-based memory within robotic action experts, while uTTT has investigated globally shared fast-weight modules across network depths. RecastVLA builds upon these ideas by introducing a shared adaptive state into a flow-matching action expert, but the paper could more clearly articulate the methodological advances beyond their combination. More importantly, the experimental evidence supporting the advantages of shared memory remains relatively limited. As shown in Table 3, RecastVLA improves RoboTwin Clean→Clean success from 69.80% with layer-local TTT to 72.38%, corresponding to a 2.58-percentage-point improvement. Although achieving better performance with fewer persistent-state parameters is encouraging, this improvement is considerably smaller than the overall 10.68-point gain over the matched baseline. Furthermore, the layer-local comparison fixes the architecture of each individual memory module rather than matching total memory capacity or computational cost. The absence of results across multiple training seeds also makes it difficult to assess the robustness of the relatively modest improvement. A more comprehensive comparison with closely related TTT architectures, including capacity-matched baselines and additional component-wise ablations, would strengthen the paper's central methodological claim.

  2. Although the real-world evaluations demonstrate promising overall improvements, the gains are highly concentrated in a small subset of tasks. As reported in Table 8, the first three button-pressing tasks account for 26 of the 29 additional successful trials compared with π0.5. These tasks specifically require tracking previously executed button presses and therefore closely match the proposed motivation of execution memory. However, across the remaining nine tasks, the average success rate improves only marginally over π0.5, from 48.89% to 50.56%. While RecastVLA still demonstrates meaningful improvements over its matched baseline on these tasks, the concentration of gains relative to π0.5 raises questions about how broadly the proposed memory mechanism benefits more diverse manipulation settings. Additionally, the paper does not include a direct memory-continuity ablation in the real-robot experiments. Given that the strongest improvements occur on tasks requiring explicit progress tracking, evaluating the same trained policy with and without persistent memory would provide more direct evidence that these gains are attributable to the proposed execution-memory mechanism. Overall, the current experiments support the effectiveness of the framework, but the evidence for generalizable long-horizon memory benefits could be strengthened.

  3. The proposed method introduces a potentially important discrepancy between memory training and deployment. During training, the memory update mechanism is optimized using windows of only four consecutive demonstration observations, with write features constructed from action-generation computations conditioned on noisy expert actions. During deployment, however, the memory is continuously updated using features generated by the policy itself over potentially much longer execution trajectories. It remains unclear whether such short training windows are sufficient to learn stable and effective memory updates over extended interactions, particularly when the policy encounters accumulated prediction errors or previously unseen intermediate states. Another concern is that memory updates are performed before the generated action chunks are physically executed. Consequently, the update does not directly incorporate feedback about whether the intended actions were successfully completed. Although subsequent observations may allow the policy to correct its behavior, the robustness of the memory mechanism to unsuccessful or partially completed actions is not explicitly evaluated. Furthermore, the gradient-based updates introduce additional computational requirements during inference. The paper does not provide sufficient quantitative analysis of inference latency, computational overhead, or memory-update stability over long trajectories. These aspects are particularly relevant for evaluating the practicality of the proposed approach in real-time robotic control.

Questions for authors
  1. Could the authors further clarify the methodological distinctions between RecastVLA, RoboTTT, and uTTT? In particular, could the authors provide additional comparisons with a faithful RoboTTT implementation and layer-local TTT variants with matched total memory capacity or computational budgets? It would also be helpful to independently ablate the contributions of cross-depth shared reads and joint memory updates, ideally with results across multiple training seeds, to better understand which components are responsible for the observed improvements.

  2. Given that the real-world improvements are primarily concentrated in the button-pressing tasks, could the authors provide a Full versus Reset ablation on these tasks to more directly establish the contribution of persistent memory? Additionally, how does the proposed mechanism handle discrepancies between generated actions and actual execution outcomes? For example, if a robot attempts to press a button but fails to establish contact, can the policy correctly recognize the incomplete operation and repeat the action? Controlled experiments involving failed actions or external perturbations would help clarify the robustness of the learned execution memory.

  3. How sensitive is the proposed method to the length of the training sequence? Since the memory update mechanism is trained using only four consecutive observations, it would be valuable to investigate whether longer training windows or deployment-aligned temporal sampling improve long-horizon performance and stability. Furthermore, could the authors provide quantitative comparisons of inference latency, GPU memory overhead, and performance as a function of execution horizon? These results would help establish the computational practicality and scalability of the proposed gradient-based memory updates.

Reviewer_C3KYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RecastVLA introduces a shared fast-weight state into a flow-matching VLA policy to retain information across successive action-generation calls. Depth-specific interfaces read the same state, which remains fixed during action generation and receives one joint update afterward. The update rule is trained through subsequent action losses and requires no expert labels at deployment. Experiments on three simulation benchmarks and twelve real-robot tasks show improvements over a matched non-TTT baseline. State interventions support the usefulness of persistent adaptation, while a layer-local comparison examines the benefit of sharing memory across depths.

Critical strengths

Clear and coherent design. Holding the state fixed throughout action generation and aggregating writes at call boundaries provides a well-defined recurrent interface for iterative action policies.

Useful controlled comparisons. The matched baseline retains the same sequential objective and update budget. Reset and imported-state experiments further examine how accumulated history affects control.

Consistent empirical gains. RecastVLA improves over the matched baseline across all reported simulation settings and raises real-robot success from 30.83% to 54.58%. Improvements on button sequencing are particularly relevant to the execution-memory motivation.

Compact shared state. The shared design exceeds the layer-local control by 2.58 percentage points on RoboTwin while using one-eighth as many persistent-state values.

Critical weaknesses
  1. Training writes use noisy expert action chunks from adjacent frames, whose targets substantially overlap. This may encourage short-term action continuation rather than persistent event memory; deployment uses self-generated actions and different temporal spacing. (Sections 3.4 and Appendix A.1.)

  2. Action-side TTT and globally shared fast weights have precedents in RoboTTT and uTTT. The sharing-specific gain is demonstrated on one setting without multi-seed uncertainty or a comparison matching total state capacity.

  3. Execution-memory interpretation remains incomplete: Resetting state establishes that recurrence helps, but does not identify what it stores. Since updates precede execution, targeted tests are needed to distinguish remembered outcomes from intended actions or elapsed task phase.

Questions for authors

How does RecastVLA compare with a matched recurrent or explicit-history baseline? Can the policy correctly track progress after a failed button press, interrupted action, or inserted delay? Is there any significant improvement compare to RoboTTT, I feel it's quite similar to RoboTTT and RoboTTT have same effect?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.