- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes HyWorldVLA, which is an autonomous driving framework that combines future visual-token prediction with latent-space world modeling. It uses a language-guided video VAE to provide future latent targets, jointly trains a VLA backbone on visual-token and latent prediction, and then co-fine-tunes an action expert conditioned on predicted latents while retaining latent supervision. The main contribution is this hybrid training scheme, intended to combine detailed scene modeling with robustness to visual noise. Experiments on NAVSIM v1/v2, component ablations, and a 655-case rain/fog subset show improved planning performance, with particularly large gains on the adverse-weather subset.
Critical strengths
- Overall, this paper is well-written and easy to follow.
- The designed framework supports both selection-based and flow-matching action experts, demonstrating flexibility in planner design.
- Evaluation is sufficient, especially the robust trajectory planning task under challenging visual conditions (Table 6).
- Qualitative examples and failure cases provide useful context for understanding the model’s behavior and limitations.
Critical weaknesses
-
The novelty relative to closely related work is insufficiently established. Prior methods, such as OneVL, already combine latent representations with future visual prediction supervision. The paper should clarify what its video-VAE latent targets and conditioning mechanism contribute beyond these existing designs, rather than relying primarily on a broad distinction between pixel-based and latent-based world models.
-
The robustness evaluation lacks important experimental details. The substantial gap on the rain/fog subset is difficult to interpret without specifying whether baselines use official or retrained checkpoints, their training data and budgets, and any augmentation or adaptation applied to each method. The selection procedure for the 655 cases and their separation from all training stages also require clarification.
-
The ablations do not sufficiently test the central robustness claim. Table 6 notably omits the Pure LWM variant, leaving unclear whether hybrid modeling improves robustness beyond latent modeling alone. The component ablations (Table 3) should also be repeated on the robustness subset, including removal of language guidance and latent conditioning, to establish which components explain the large gains. Comparisons with relevant latent-based methods, including OneVL, would further strengthen this analysis.
-
The evaluation does not isolate sensitivity to visual corruption. The naturally occurring rain/fog subset differs from the overall test set in potentially more than weather, so the performance gap cannot be attributed solely to noise robustness. Controlled brightness (e.g., use some image style transfer models like FDA [R1] or CycleGAN [R2]), blur, and fog perturbations applied to the same scenes, with results across severity levels, would provide more direct evidence.
-
The proposed explanation for the robustness gains remains largely unverified. A reconstruction-trained video VAE does not inherently guarantee noise-invariant representations, and the paper provides little direct evidence that pixel supervision prevents representation collapse or that latent prediction suppresses nuisance variations. The qualitative trajectory comparisons illustrate behavioral differences but do not explain their cause; paired analyses of future predictions, latent stability, and planning outputs under perturbations would help substantiate the claimed mechanism.
References
[R1] Fda: Fourier domain adaptation for semantic segmentation. CVPR 2020.
[R2] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV 2017.
Questions for authors
Please refer to the weaknesses section.
Response to Reviewer NYLF
Thank you very much for your constructive comments and for taking the time to review our manuscript. We've carefully read your comments and our response is as follows.
Question 1 Response: We have added a detailed comparison with OneVL to Related Work. The core difference is design principle: OneVL uses parallel dual-decoder supervision of latent-CoT tokens in a single stage, whereas HyWorldVLA is built on an asymmetric two-stage Information Bottleneck paradigm. During pre-training, pixel-level reconstruction provides a sufficiency constraint that preserves fine-grained spatiotemporal structure and prevents latent degradation, while future-latent prediction provides a compression constraint that filters pixel-level noise. At co-fine-tuning, the pixel pathway is disabled and the action expert operates purely in the latent space, conditioned on predicted future latents via a joint attention mechanism.
Question 2 Response: (a) All baselines used in our experiments are initialized from official checkpoints, and no data augmentation was applied during the training of our proposed method.
(b) Specifically, we used Qwen-3.6-Plus to filter the test set frame by frame, obtaining 1,317 frames containing rain or fog. However, we found that the LLM struggles to distinguish the severity of noise. We therefore performed a manual second-pass screening on the 1,317 frames and retained the 655 frames with high blur severity as the final test:
(c) Constructed as a subset of the full test set, our noisy test set is inherently isolated from the training data, preventing any data leakage.
Question 3 Response: We have added a full component ablation on the noise-robustness test subset, with results shown below:
The Pure LWM variant scores 68.6 PDMS, 18.3 points below Ours. We attribute this to the representation collapse in pure LWM training(the collapse analysis are provided in our response to Q5).
Question 4 Response:
Robustness evaluation for autonomous driving remains an open problem, and a fully rigorous protocol is beyond the scope of this work. Within this limit, we evaluated synthetic perturbations on the full test set (12,146 frames), covering multiple types and intensities: point-wise blur (4 random point regions per frame with distance-decaying Gaussian blur, simulating raindrops on the lens), Gaussian noise, and brightness shifts (4 levels). Results are shown below:
The gap under synthetic perturbations is much smaller than on the real noise subset, suggesting that temporally coherent real-world noise poses a stricter test of representation consistency — where our mechanism's benefits are most pronounced.
Question 5 Response:
We thank the reviewer for this insightful question. We provide quantitative evidence that pixel-level supervision prevents representation collapse, with results shown in Fig. 4 of the revised Appendix B.3.
LWM (without pixel supervision) exhibits near-complete collapse with cosine similarity of 0.98, where distinct inputs produce nearly identical representations. HyWorldVLA (with pixel supervision) reduces cosine similarity to 0.63 and increases effective rank by 2.2× and feature variance by 33.5× , demonstrating that pixel supervision effectively prevents representation collapse.
Thank you again for the constructive comments. We hope these explanations could address your concerns. Any further questions or suggestions would be greatly appreciated.