- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an architecture, ACT3, leveraging a VLM and a world model as the foundational models. ACT3 keeps the semantic and dynamic stream independent, while the action expert cross-attends to both streams through layerwise cross-attention.
It uses pi05 and cosmos-predict video model as the backbone for the semantic and the dynamic stream and trained joint end-to-end for action generation and future visual prediction.
ACT3 reports results on LIBERO, LIBERO-PLUS, RoboCasa and fine-tuning on real-world robot data.
The central claims are
- How should the semantic and dynamic streams be updated ?
- Layerwise cross-attention is better suited for action-generation than final layer features
- Combining VLM with WM improves general manipulation performance.
Critical strengths
- ACT3 proposes a well motivated design question about how the semantic and dynamic stream interact and influence the action expert.
- Success rate improvement over baseline pi05, especially on RoboCasa and robust across shifts in LIBERO-Plus.
- Ablations are well motivated. Tab.3 Ablations are convincing about how the stream should interact. Table 7. Ablation aims to investigate which streams of the model should be kept frozen (authors might consider moving it to main paper). Sec 4.3 aims to analyze the how the action query attend to the three streams.
Critical weaknesses
-
This paper is missing analysis on how the dynamics and semantic streams affect the action generations. What are the effect of degraded dynamics generation ? What happens when two of the streams decide on conflicting paths ? e.g. ‘for put the vegetables in the bowl, VLM might have a bias towards carrots, and WM towards potato, how does ACT3 action stream handle such cases ? Controlled experiments are missing to distinguish the benefits of the dynamics stream vs just additional features.
-
This work fails to clearly isolate the source of performance gains. ACT3 relies on both a pretrained world-model and pi05. The work further claims generated future obs, trainable backbone, prediction supervision all contribute to improved performance. The real-world finetuning experiments compared with pi05 might not be fairest comparison. A baseline with both a VLA + dynamics model/vgm would provide more evidence supporting ACT3 architecture.
-
Although an expensive experiment, missing ablation on where this approach would generalize to various sizes of VLM and WM and perform equally well with other foundational models. (least priority)
Questions for authors
- The novelty of ACT3 needs clearer claim. The architectural classification in Fig1 is helpful but still needs more stronger claims. Novelty is mainly architectural about layerwise attention and independent contexts. It would be useful to have a fair comparison on the various architecutural design, e.g. TriVLA vs ACT3 with the same backbones. An experiment comparing ACT3 with the prior works architecure with the same training and evaluation protocol would be very informative to the readers.
- How much does ACT rely on realistic dynamics ? How does action-stream cope with non-plausible generated future frames ?
6 comments
Response to Reviewer [Part 2/3]
[W2&Q1] This work fails to clearly isolate the source of performance gains. ACT3 relies on both a pretrained world-model and pi05. The work further claims generated future obs, trainable backbone, prediction supervision all contribute to improved performance. The real-world finetuning experiments compared with pi05 might not be fairest comparison. A baseline with both a VLA + dynamics model/vgm would provide more evidence supporting ACT3 architecture. The novelty of ACT3 needs clearer claim. The architectural classification in Fig1 is helpful but still needs more stronger claims. Novelty is mainly architectural about layerwise attention and independent contexts. It would be useful to have a fair comparison on the various architecutural design, e.g. TriVLA vs ACT3 with the same backbones. An experiment comparing ACT3 with the prior works architecure with the same training and evaluation protocol would be very informative to the readers.
We thank the reviewer for the thoughtful comment.
(1) We would like to emphasize that our goal is not to develop an entirely new VLA architecture that requires training from scratch, but to explore a more practical yet challenging question: Can we push an already state-of-the-art pretrained VLA to new heights by directly leveraging an existing pretrained world model?
-
This is a very important direction. is widely recognized in the research community as a state-of-the-art VLA, with strong generalization capabilities acquired through large-scale robot pretraining. Further improving such a strong model is far from trivial. Meanwhile, Cosmos has learned rich predictive dynamics from large-scale video pretraining. Rather than relearning these capabilities from scratch, we aim to unlock the complementary strengths of both pretrained models, leveraging the learned dynamics of Cosmos to further enhance .
-
Simplicity is exactly what we strive for, and we believe it is not at odds with novelty. To our knowledge, our action-centric architecture has not been explored before, as highlighted by the comparison with prior methods in Figure 1. Our design preserves the independent forward computations of the pretrained context streams, while allowing the action stream to leverage both representation hierarchies through layerwise attention. Achieving meaningful improvements over an already state-of-the-art VLA through such a lightweight design is valuable in itself. More importantly, our approach provides the community with a simple, effective, and readily adoptable baseline that requires no additional large-scale pretraining and can be directly fine-tuned on downstream tasks after integration.
(2) We have provided an architectural comparison between and existing multi-stream approaches in Figure 1 of the original paper. The key difference from TriVLA [1] lies in how semantic and dynamics contexts are provided to the action policy. TriVLA uses VLM features and dynamics tokens aggregated from multi-level video features as conditioning inputs to a downstream action policy. In contrast, allows action queries to directly access the keys and values from the corresponding VLM and WM layers, enabling layer-wise integration of semantic and dynamics information.
(3) Moreover, we have additionally compared against a TriVLA-style feature-conditioning baseline suggested by the reviewer. Since the official training code and model checkpoints of TriVLA [1] are not publicly available, we implement this baseline using and Cosmos under the same downstream training setup as . Specifically, semantic tokens are extracted from the final vision-language layer of , while predictive tokens are aggregated from selected Cosmos blocks using learned queries. Each action block attends to these two token sets through separate cross-attention modules. As shown in Table R3, achieves a higher success rate than this TriVLA-style baseline, further supporting the effectiveness of our action-centric integration strategy. The additional results have been updated in Appendix C.2 (Page 19) of our revised manuscript.
Table R3. Comparison with the TriVLA-style baseline on RoboCasa.
| Method | Success Rates (%) |
|---|---|
| TriVLA-style baseline | 64.9 |
| 71.2 |
Response to Reviewer [Part 3/3]
[W3] Although an expensive experiment, missing ablation on where this approach would generalize to various sizes of VLM and WM and perform equally well with other foundational models. (least priority)
Thanks for the constructive suggestion. Due to the high experimental cost, we focus on evaluating the generalizability of across different world models. Specifically, we have additionally designed an variant that pairs with Wan2.2-TI2V-5B. We use learned linear projections to align the layerwise keys and values of Wan2.2 with the attention space of the action stream. We evaluate this variant on all 24 kitchen tasks in RoboCasa using the same training and evaluation settings. The results in Table R4 show that with Wan2.2 achieves comparable performance to with the Cosmos world model and outperforms the baseline. These findings suggest that the effectiveness of is not limited to a specific world-model backbone and can generalize across different world models. The additional results have been updated in Appendix C.4 (Page 20) of our revised manuscript.
Table R4. Success rates (%) with different world-model backbones on RoboCasa.
| Method | Dynamics Backbone | Success Rates |
|---|---|---|
| None | 59.4 | |
| Wan2.2-TI2V-5B | 69.1 | |
| Cosmos-Predict2.5-2B | 71.2 |
Reference
[1] TriVLA: A Triple-System-Based Unified Vision-Language-Action Model with Episodic World Modeling for General Robot Control. arXiv 2025.
Response to Reviewer [Part 1/3]
We sincerely thank the reviewer for your insightful comments and for highlighting the strengths of ACT3, including its well motivated design question, success rate improvement, and convincing ablations. Below we address the main concerns raised in the review.
[W1&Q2] This paper is missing analysis on how the dynamics and semantic streams affect the action generations. How much does ACT rely on realistic dynamics? How does action-stream cope with non-plausible generated future frames?
Thank you for this insightful observation.
(1) We would like to clarify that the two streams are designed to provide complementary guidance for action generation.
The semantic stream provides global task guidance from current observations and the task instruction, while the dynamics stream predicts fine-grained local state changes. Both streams are conditioned on the same task instruction and are intended to support the same task objective through complementary global and local information. The evaluated benchmarks further provide clearly specified goals and target objects, with instructions fixed throughout execution. The carrot-versus-potato ambiguity is therefore absent from the task specification, and we expect goal-level conflicts to be limited in this setting.
Even when the two streams provide inconsistent information, the action expert learns to balance their contributions through attention. To better understand how action generation utilizes these contexts, we have analyzed event-dependent attention allocation in Figure 5 of the original paper. We observe that dynamics attention decreases during approach and initial contact, but increases around grasping, object motion, and articulation onset, where local state transitions become more important. In contrast, semantic attention changes more modestly, consistent with providing stable global task guidance as the scene evolves. These observations suggest that the action expert adaptively utilizes semantic and dynamics information across manipulation stages, with greater reliance on dynamics during object interactions.
(2) In Section 4.3 of our original paper, we have conducted ablation experiments to examine whether the benefits of the dynamics stream go beyond simply providing additional features. Specifically, we compare our layerwise access to WM representations with a variant that uses only the final-layer WM representations as additional context for the action expert. For convenience, we present the relevant results from Figure 4(a) of the original paper in Table R1. The results show that layerwise access performs better than using only final-layer WM features. This suggests that the improvements are not simply due to providing additional WM features, but also benefit from integrating dynamics information across layers.
Table R1. Success rates (%) with different WM layer access on RoboCasa.
(3) As suggested, we have additionally conducted experiments to examine the robustness of the action stream to non-plausible generated future frames by introducing different interventions on future representations. Specifically, we introduce three interventions on future representations throughout each rollout, while keeping all other settings fixed: First, we reduce the number of sampling steps used for future generation from the default of two to one. Second, we perturb the default future latent (z) using zα=z+α(σc⊙ϵ),α∈0.25,0.5 where σc is the per-channel standard deviation estimated from independent samples and ϵ∼N(0,I). Both perturbation levels use the same Gaussian noise realization for each corresponding query. Third, we introduce mismatched futures by replacing z with the corresponding latent slice predicted from an image in a normal rollout of the same task, preferably from a different task stage.
The results in Table R2 show that the policy is relatively robust to reduced generation steps and the tested latent perturbations, but is sensitive to mismatched future representations. This contrast suggests that the effectiveness of predictive guidance depends on its alignment with current task progress. The additional results have been updated in Appendix C.6 (Page 21) of our revised manuscript.
Table R2. Success rates (%) under different interventions on future representations.