- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents GWM-VLA, a geometry-aware latent world modeling framework for Vision-Language-Action (VLA) robot learning, targeting the well-known robustness degradation of VLA policies under visual and environmental distribution shifts. Departing from the common practice of using geometric foundation models purely as perception feature extractors, this work adopts jointly aggregated multi-view geometric representations from VGGT-Ω as the predictive state space of the latent world model. The framework features three core designs: per-timestep geometry-aware multi-view state encoding, global context-conditioned target-view (wrist-view) latent prediction, and shared latent-action representations that condition both the world model and the flow-matching action head. The method is validated on LIBERO and LIBERO-Plus simulation benchmarks as well as real-world SO-101 robot experiments, with extensive ablation studies verifying the contribution of each component. The authors explicitly acknowledge key limitations, including the requirement of synchronized multi-view inputs.
Critical strengths
The work proposes a fundamentally novel perspective for VLA world modeling by repurposing geometric foundation model outputs as the dynamical state space of latent world models, rather than treating them as passive perceptual frontends. This breaks the prevailing paradigm of using geometric representations only as input features, and opens a new avenue for integrating pre-trained geometric priors into embodied world dynamics learning. This conceptual innovation is the core contribution of the paper and provides strong justification for acceptance.
The evaluation is rigorous, comprehensive, and well-scoped. It spans both in-distribution simulation (LIBERO) and out-of-distribution robustness benchmarks (LIBERO-Plus across seven perturbation dimensions), complemented by real-robot deployment on the SO-101 platform. A full suite of targeted ablation studies systematically validates every core design choice. Critically, the paper clearly delineates the method’s operating boundaries and limitations instead of overstating its generalizability, which strengthens the credibility of its conclusions.
Critical weaknesses
The prediction target view is fixed as the wrist camera throughout training, with no mechanism for adaptive view selection tailored to different task types. The paper provides no empirical evidence that the wrist view remains the optimal choice for other operation modes (e.g., tool use, long-horizon assembly tasks), leaving the generalizability of this design rule largely unsubstantiated. This limits the broader applicability of the proposed framework to diverse robotic manipulation scenarios.
As previously noted in the VLA-JEPA work, the shared latent-action design carries an inherent risk of shortcut learning (information leakage): latent action tokens may directly encode future target-view patch token information rather than learning genuine action-conditioned state transition dynamics. The authors posit that the geometric latent space can mitigate this risk, but provide no diagnostic experiments to verify whether shortcut learning occurs, nor to quantify its magnitude. This methodological concern calls into question whether the observed performance gains truly stem from learned world dynamics.
Questions for authors
How do you verify that the shared latent-action representations learn genuine action-conditioned transition dynamics, rather than directly encoding the patch token information of future target views? Have you conducted diagnostic experiments (e.g., shuffling the order of action tokens, corrupting action inputs, or probing information content) to quantify the degree of potential information leakage?
In the paper, comparison is only made against the VLA-JEPA variant without human video pretraining. What is the rationale for not including the full VLA-JEPA baseline with human video pretraining? Would the robustness improvements of GWM-VLA remain statistically significant when compared against the full VLA-JEPA baseline?
The paper acknowledges the limitation that the method requires synchronized multi-view observations, while most existing large-scale robot datasets are single-view. Do the authors have a concrete research plan to extend this framework to single-view datasets to enable large-scale pretraining, and are there any preliminary results to suggest this extension is feasible?
3 comments
Q3: Applicability to Multi-View and Single-View Data
An expanding collection of large-scale robot datasets supports multi-view learning, including DROID[2], RH20T[3], and AgiBot World[4]. These resources provide a practical foundation for scaling our geometry-aware world-modeling framework.
For single-view data, a straightforward adaptation is to use the frozen VGGT-Ω as a single-view geometric encoder: encode the available image at each timestep and predict its next-step latent features, retaining the shared latent-action representation and joint action/prediction objectives.
Our wrist-only encoding ablation provides preliminary evidence for this approach. Under the same DROID pretraining and LIBERO fine-tuning procedure, it achieves 96.2% on LIBERO and 68.2% on LIBERO-Plus, as shown in Table 3. This experiment uses a single view for world-state encoding, supporting the feasibility of single-view geometric prediction supervision.
We plan to incorporate single-view robot datasets into large-scale pretraining by supplying the available view to both the policy and the geometric encoder, and jointly training on single-view and multi-view trajectories. This would broaden the usable training data while exploiting cross-view geometric information whenever multiple views are available.
Table 3. Success Rates (%) with Single-View and Multi-View World-State Encoding.
| Method | World-state encoding views | LIBERO | LIBERO-Plus |
|---|---|---|---|
| GWM-VLA | Main + wrist | 97.1 | 75.4 |
| Wrist-only encoding | Wrist | 96.2 | 68.2 |
References
[1] Pan Y, Peng H, Zhang Z, et al. World-to-Wrist: Task-conditioned future wrist modeling for fine-grained robot manipulation[J]. arXiv preprint arXiv:2608.05369, 2026.
[2] Khazatsky A, Pertsch K, Nair S, et al. Droid: A large-scale in-the-wild robot manipulation dataset[J]. arXiv preprint arXiv:2403.12945, 2024.
[3] Fang H S, Fang H, Tang Z, et al. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 653-660.
[4] World A B, Bu Q, Cai J, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems[J]. arXiv preprint arXiv:2503.06669, 2025.
We greatly appreciate it if you could take a look at our first-round responses. Please let us know if you have any further comments or suggestions. We would be happy to make additional improvements and engage in further discussion.
We sincerely thank the reviewer for the constructive comments and the appreciation of our work. Our response to your concerns/questions:
W1: Wrist Prediction as a Task-Aligned Target
We select the wrist view because it moves with the robot arm and captures local gripper–object interactions. Predicting this changing viewpoint encourages the world model to use cross-view geometric information to understand how end-effector motion changes the observed scene. Cross-view aggregation provides the global scene context needed to support this prediction, while the wrist target emphasizes manipulation-relevant state changes. The value of focusing on wrist-local interactions is also supported by World-to-Wrist [1], which uses future wrist prediction to improve fine-grained manipulation.
Our target-view ablation on LIBERO-Spatial supports this choice: wrist prediction achieves 92.8%, compared with 87.8% for third-person prediction and 92.4% for mixed target sampling (80% wrist and 20% third-person). These results support wrist prediction as an effective target for learning manipulation-relevant geometric dynamics in the evaluated setting.
W2: Information-Flow Isolation of Latent Actions
GWM-VLA prevents direct future-observation leakage into latent actions by design. Latent actions are computed exclusively from current observations and the task instruction; future observations do not enter this computation.
The leakage mechanism discussed in VLA-JEPA’s Introduction concerns methods that extract latent actions using both current and future observations. In our framework, VGGT-Ω encodes each timestep independently, and the world predictor uses temporally causal attention, preserving the separation between prediction inputs and future targets.
We verified this design by replacing future images while keeping the model parameters and current inputs fixed. Across 1,200 replacements, the latent actions and first-step predictions remained exactly unchanged, even though the future target features changed. This confirms that future images provide prediction-loss targets without entering or influencing the latent-action computation in the tested implementation.
Q1: Latent-Action Perturbation Diagnostic
Following the reviewer’s suggestion, we replaced latent actions with those from other trajectories of the same task, while keeping current-state patch tokens, register tokens, and prediction targets fixed. We selected 400 timesteps from 160 newly collected trajectories for next-state prediction. For each sample, we used latent actions from three other trajectories separately and averaged the resulting prediction errors.
Table 1. Effect of latent-action replacement on next-state prediction.
As shown in Table 1, replacement increased prediction error by 3.24%, corresponding to a mean L1 increase of 0.0404 (95% CI: [0.0349, 0.0460]). This supports the predictor’s use of sample-specific latent actions for state-transition prediction, complementing the future-input isolation verified in W2.
Q2: Fair Comparison with VLA-JEPA
Robot-only VLA-JEPA and GWM-VLA share the same policy-training settings: DROID pretraining, LIBERO fine-tuning, the Qwen3-VL-2B backbone, 50K pretraining steps, 30K fine-tuning steps, and an effective batch size of 256. Their comparison is therefore fair.
Full VLA-JEPA additionally uses Something-Something V2 (SSv2), a single-view human-video dataset without synchronized multi-view observations, for policy pretraining, which is separate from our core contribution: using multi-view geometric representations for latent world modeling to shape shared latent actions.
For completeness, we also report full VLA-JEPA below. It achieves higher average success rates under the additional human-video pretraining setting. Incorporating human videos is a meaningful direction for future work.
Table 2. Average success rates (%) under different policy-pretraining settings.