- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Current transformer models for robotics encode images with text, video, and actions. Pruning attention connectivity between these streams randomly, even at 50%, has little effect on performance. Moreover, if the authors cut all communication past 50% depth, then they can significantly improve latency and peak memory usage, with the second half of certain expert stream entirely unnecessary. This also means that historical KV states are required only for the lower half of the network. Speed-ups range from 13-42% and results are evaluated over four different pre-trained models, with both simulated and real-world tasks, and one evaluation that includes pretraining.
Critical strengths
This paper is clearly written, seeks to answer a simple question, and does so thoroughly and straightforwardly.
Critical weaknesses
I don’t believe there are any critical witnesses. However, I am not as familiar with the related work, and I wonder whether there are existing sparsification methods that should have been compared.
Questions for authors
A lot of related work is cited, but there is almost no commentary on how it relates to this work, and why it was or was not evaluated. Could you explain further?
Minor: What are WAM, Joint Denoising, and Cross-Chunk in Table 1?
6 comments
Rebuttal by Authors (Part 2/2)
Continue on A1:
Furthermore, following the suggestion from Reviewer Y1CS, we conducted additional experiments on InternVLA-A1 with Alternating, Middle-half, and Late-half fusion across RoboTwin and LIBERO. These experiments provide a more comprehensive comparison of connectivity patterns under the same pretrained architecture and downstream training protocol. Detailed results are provided in our response to Reviewer Y1CS. We hope these clarifications and additional experiments help explain the relationship between our work and existing efficiency methods.
Q2: Minor: What are WAM, Joint Denoising, and Cross-Chunk in Table 1?
A2: Thank you for raising this point. In Table 1, we summarize representative pretrained MoT robot policies along four architectural dimensions: expert composition, denoising mechanism, cross-chunk dependency, and backbone parameterization.
WAM (World-Action Model) [9, 10] refers to policies that explicitly incorporate world modeling through future-state or video prediction to support action generation, whereas VLA (Vision-Language-Action) models [11, 12] emphasize pretrained vision-language representations for semantic understanding and action conditioning.
Regarding denoising mechanisms, Joint Denoising [9, 13] refers to simultaneously denoising multiple diffusion experts, which exchange information through joint attention across Transformer layers. In contrast, Causal Denoising [10, 11, 12] conditions action denoising on upstream representations computed beforehand.
Finally, Cross-Chunk describes temporal dependencies across predicted chunks. Specifically, autoregressive (AR) models [10, 13] condition the current chunk on cached keys and values from preceding chunks, whereas non-autoregressive (Non-AR) models [9, 11, 12] operate without such cross-chunk caching.
Further details on these architectural distinctions are provided in Appendix A.
Thank you again for your valuable feedback and hope that all your concerns are properly addressed.
Reference
[1] Weijia Shi, et al. Lmfusion: Adapting pretrained language models for multimodal generation. Neurips, 2026.
[2] Chaorui Deng, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025
[3] Junjie Wen, et al. Tinyvla: toward fast, data-efficient vision-language-action models for robotic manipulation. IEEE Robotics and Automation Letters, 10(4):3988–3995, 2025.
[4] Mustafa Shukor, et al. Smolvla: A vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844, 2025.
[5] Hongyu Wang, et al. Bitvla: 1-bit vision-language-action models for robotics manipulation. arXiv preprint arXiv:2506.07530, 2025b.
[6] Yuhao Xu, et al. Qvla: Not all channels are equal in vision-language-action model’s quantization. arXiv preprint arXiv:2602.03782, 2026b.
[7] Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination?, 2026.
[8] Liheng Ma, et al. Faster-wam: Do world action models need deep action modules? arXiv preprint arXiv:2608.02365, 2026a.
[9] Hongzhe Bi, et al. Motus: A unified latent action world model. CVPR, 2026.
[10] Lin Li, et al. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998, 2026.
[11] Physical Intelligence, et al. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.
[12] Junhao Cai, et al. Internvla-a1: Unifying understanding, generation and action for robotic manipulation. arXiv preprint arXiv:2601.02456, 2026.
[13] Seonghyeon Ye, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026.
Thank you for the clarification and DoT experiment.
An additional question: Looking back at the paper, I noticed few mentions of standard errors and confidence intervals. Is uncertainty quantified across the results tables?
Rebuttal by Authors (Part 1/2)
We thank the reviewer for the insightful comments and valuable feedback. Please see our responses to your concerns point-by-point below.
A1: Thank you for raising this point. Our related work is organized into three parts, covering the architectural background, robot foundation policies, and efficiency-related methods.
The first part introduces the Mixture-of-Transformers (MoT) architecture and its applications to multimodal modeling. It discusses how specialized Transformer streams exchange information through shared attention, with examples such as LMFusion [1] and BAGEL [2], which combine multimodal understanding and generation. These works provide the architectural background for our study of expert connectivity. The second part reviews recent robot foundation policies that connect pretrained vision-language or video representations to action generation, covering both vision-language-action (VLA) models and world-action models (WAMs), including the four representative policies examined in our experiments.
The reviewer may be particularly interested in the third part, which discusses efficient robot policies. We summarize their key technical designs and the components they modify below.
Table R1: Comparison with representative robot-policy efficiency methods.
These methods improve efficiency by using special model architectures, parameter precision, or inference computation, whereas our work directly investigates the redundancy of native expert attention connectivity in pretrained MoT policies. In contrast, our primary research question is how much expert-level fusion is necessary in pretrained MoT robot policies after downstream adaptation. To investigate this, we apply attention masks to pretrained architectures under matched fine-tuning protocols. Dense provides the original connectivity reference, Random-cut examines whether carefully selected connections are necessary, and Early-half evaluates a structured pattern that enables deployment simplification. These controlled comparisons allow us to study connectivity redundancy without redesigning or pretraining a new robot foundation policy.
We have also considered efficient policy designs from the related work in our experiments. In our original submission (Section 5.2), we evaluated our sparsification framework on Fast-WAM [7] using LIBERO, achieving average success rates of 96.8%, 94.4%, and 97.1% for Dense, Random, and Early-half, respectively. To further address the reviewer's concern, we additionally conducted a DoT-inspired experiment during the rebuttal period, following Faster-WAM [8]. Specifically, we reduced InternVLA-A1's 28-layer action expert to a single layer through parameter averaging and DoT-style connections. Under the same RoboTwin fine-tuning protocol, this variant achieves 78.41% average success (77.44% Easy, 79.38% Hard), compared with 88.66% for the original Dense model. Unlike Faster-WAM, which trains a dedicated DoT architecture, our experiment directly compresses an existing pretrained policy, suggesting that such compression may not preserve performance under our setting.