acceptodds
Under review as a conference paper at ICLR 2027

HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving

Abstract

Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving. While pixel-level future prediction enables fine-grained spatiotemporal reasoning, it compromises robustness in noisy driving scenarios. Conversely, latent-based world models alleviate this sensitivity but often incur limited interpretability and representational degradation due to absent pixel-level grounding. To reconcile this trade-off, we propose HyWorldVLA, a hybrid world-VLA framework that unifies pixel-level supervision and latent representation learning. In the pre-training stage, HyWorldVLA predicts video latents encoded by a pre-trained video VAE, while simultaneously reconstructing video frames to provide precise pixel-level grounding. During the subsequent co-fine-tuning phase, the model exclusively predicts latent features, which are fed into an action expert to generate trajectories. Extensive experiments on NAVSIM v1 and v2 benchmarks demonstrate that HyWorldVLA outperforms both pixel-based and latent-based world model baselines. Notably, we present the first comprehensive qualitative and quantitative analysis of world model noise robustness in autonomous driving, establishing a new benchmark for evaluating future architectures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, world model, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NYLFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HyWorldVLA, which is an autonomous driving framework that combines future visual-token prediction with latent-space world modeling. It uses a language-guided video VAE to provide future latent targets, jointly trains a VLA backbone on visual-token and latent prediction, and then co-fine-tunes an action expert conditioned on predicted latents while retaining latent supervision. The main contribution is this hybrid training scheme, intended to combine detailed scene modeling with robustness to visual noise. Experiments on NAVSIM v1/v2, component ablations, and a 655-case rain/fog subset show improved planning performance, with particularly large gains on the adverse-weather subset.

Critical strengths
  • Overall, this paper is well-written and easy to follow.
  • The designed framework supports both selection-based and flow-matching action experts, demonstrating flexibility in planner design.
  • Evaluation is sufficient, especially the robust trajectory planning task under challenging visual conditions (Table 6).
  • Qualitative examples and failure cases provide useful context for understanding the model’s behavior and limitations.
Critical weaknesses
  • The novelty relative to closely related work is insufficiently established. Prior methods, such as OneVL, already combine latent representations with future visual prediction supervision. The paper should clarify what its video-VAE latent targets and conditioning mechanism contribute beyond these existing designs, rather than relying primarily on a broad distinction between pixel-based and latent-based world models.

  • The robustness evaluation lacks important experimental details. The substantial gap on the rain/fog subset is difficult to interpret without specifying whether baselines use official or retrained checkpoints, their training data and budgets, and any augmentation or adaptation applied to each method. The selection procedure for the 655 cases and their separation from all training stages also require clarification.

  • The ablations do not sufficiently test the central robustness claim. Table 6 notably omits the Pure LWM variant, leaving unclear whether hybrid modeling improves robustness beyond latent modeling alone. The component ablations (Table 3) should also be repeated on the robustness subset, including removal of language guidance and latent conditioning, to establish which components explain the large gains. Comparisons with relevant latent-based methods, including OneVL, would further strengthen this analysis.

  • The evaluation does not isolate sensitivity to visual corruption. The naturally occurring rain/fog subset differs from the overall test set in potentially more than weather, so the performance gap cannot be attributed solely to noise robustness. Controlled brightness (e.g., use some image style transfer models like FDA [R1] or CycleGAN [R2]), blur, and fog perturbations applied to the same scenes, with results across severity levels, would provide more direct evidence.

  • The proposed explanation for the robustness gains remains largely unverified. A reconstruction-trained video VAE does not inherently guarantee noise-invariant representations, and the paper provides little direct evidence that pixel supervision prevents representation collapse or that latent prediction suppresses nuisance variations. The qualitative trajectory comparisons illustrate behavioral differences but do not explain their cause; paired analyses of future predictions, latent stability, and planning outputs under perturbations would help substantiate the claimed mechanism.

References

[R1] Fda: Fourier domain adaptation for semantic segmentation. CVPR 2020.

[R2] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV 2017.

Questions for authors

Please refer to the weaknesses section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer NYLF

Thank you very much for your constructive comments and for taking the time to review our manuscript. We've carefully read your comments and our response is as follows.


Question 1 Response: We have added a detailed comparison with OneVL to Related Work. The core difference is design principle: OneVL uses parallel dual-decoder supervision of latent-CoT tokens in a single stage, whereas HyWorldVLA is built on an asymmetric two-stage Information Bottleneck paradigm. During pre-training, pixel-level reconstruction provides a sufficiency constraint that preserves fine-grained spatiotemporal structure and prevents latent degradation, while future-latent prediction provides a compression constraint that filters pixel-level noise. At co-fine-tuning, the pixel pathway is disabled and the action expert operates purely in the latent space, conditioned on predicted future latents via a joint attention mechanism.


Question 2 Response: (a) All baselines used in our experiments are initialized from official checkpoints, and no data augmentation was applied during the training of our proposed method.

(b) Specifically, we used Qwen-3.6-Plus to filter the test set frame by frame, obtaining 1,317 frames containing rain or fog. However, we found that the LLM struggles to distinguish the severity of noise. We therefore performed a manual second-pass screening on the 1,317 frames and retained the 655 frames with high blur severity as the final test:

ScenarioProportion
Straight (non-intersection)63%
Straight (intersection)28%
Lane change8%
Left turn0.5%
Right turn0.5%

(c) Constructed as a subset of the full test set, our noisy test set is inherently isolated from the training data, preventing any data leakage.


Question 3 Response: We have added a full component ablation on the noise-robustness test subset, with results shown below:

ConfigNC ↑DAC ↑TTC ↑C. ↑EP ↑PDMS ↑
Pure WAM99.175.794.399.867.870.0
Pure LWM96.575.989.699.265.668.6
w/o language guidance in latent98.876.394.799.568.870.7
w/o latent condition in action expert99.587.997.399.381.183.8
w/o latent supervision in co-fine-tuning99.278.594.899.770.973.2
Ours99.791.196.999.284.486.9

The Pure LWM variant scores 68.6 PDMS, 18.3 points below Ours. We attribute this to the representation collapse in pure LWM training(the collapse analysis are provided in our response to Q5).


Question 4 Response:

Robustness evaluation for autonomous driving remains an open problem, and a fully rigorous protocol is beyond the scope of this work. Within this limit, we evaluated synthetic perturbations on the full test set (12,146 frames), covering multiple types and intensities: point-wise blur (4 random point regions per frame with distance-decaying Gaussian blur, simulating raindrops on the lens), Gaussian noise, and brightness shifts (4 levels). Results are shown below:

PerturbationPure WAMOursΔ
Point-wise blur88.4789.05+0.58
Gaussian noise89.0789.64+0.57
Brightness −0.689.7190.34+0.63
Brightness −2.088.6288.86+0.24
Brightness −4.078.8779.08+0.21
Brightness −5.073.5774.74+1.17

The gap under synthetic perturbations is much smaller than on the real noise subset, suggesting that temporally coherent real-world noise poses a stricter test of representation consistency — where our mechanism's benefits are most pronounced.


Question 5 Response:

We thank the reviewer for this insightful question. We provide quantitative evidence that pixel-level supervision prevents representation collapse, with results shown in Fig. 4 of the revised Appendix B.3.

MetricLWMHyWorldVLAImprovement
Cosine Similarity ↓0.9797 ± 0.00050.6289 ± 0.005335.8% reduction
Feature Variance ↑0.051 ± 0.0021.71 ± 0.1233.5× increase
Effective Rank ↑196 ± 5436 ± 122.2× increase

LWM (without pixel supervision) exhibits near-complete collapse with cosine similarity of 0.98, where distinct inputs produce nearly identical representations. HyWorldVLA (with pixel supervision) reduces cosine similarity to 0.63 and increases effective rank by 2.2× and feature variance by 33.5× , demonstrating that pixel supervision effectively prevents representation collapse.


Thank you again for the constructive comments. We hope these explanations could address your concerns. Any further questions or suggestions would be greatly appreciated.

Reviewer_YRHMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HyWorldVLA, a hybrid world modeling framework for VLA-based autonomous driving. The method combines visual reconstruction and latent prediction during pre-training, while using predicted latent representations for trajectory planning during co-fine-tuning.

Critical strengths

The paper is well motivated and clearly presented. Combining visual reconstruction with latent prediction is a reasonable approach to improving world modeling for autonomous driving.

Critical weaknesses
  1. The proposed hybrid training framework primarily combines visual-token prediction with latent supervision,both of which have already been explored in prior research on world models. Beyond simply integrating these existing training objectives, it remains unclear whether the framework introduces any novel modeling mechanisms.

  2. The authors posit that pixel-level supervision can mitigate the degradation of latent representations; however, they do not provide a direct evaluation of the learned latent representations themselves. Furthermore, the performance gains on the NAVSIM v1 and v2 datasets are relatively modest.

Questions for authors
  1. Could the authors clarify the main technical differences between HyWorldVLA and existing methods that combine future visual prediction with latent world modeling? What is the key methodological contribution beyond joint supervision?

  2. Could the authors provide quantitative evidence that hybrid training mitigates latent representation degeneration and improves the quality of the learned latent representations?

  3. Could the authors clarify the noise robustness evaluation protocol. Also, how do the authors explain the accuracy–robustness trade-off observed under different latent supervision weights?

1 comment
AuthorsOpenReview ↗

Response to Reviewer YRHM

Thank you very much for your constructive comments and for taking the time to review our manuscript. We've carefully read your comments and our response is as follows.


Question 1 Response: We thank the reviewer for this important question. Our contribution lies not in being the first to combine visual prediction with latent modeling, but in organizing them into a two-stage asymmetric training paradigm under the Information Bottleneck (IB) theory [1, 2] — a design that is qualitatively different from existing parallel-supervision approaches. During pre-training, pixel-level reconstruction acts as a sufficiency constraint that preserves fine-grained spatiotemporal structure and prevents latent degradation, while future-latent prediction acts as a compression constraint that filters pixel-level noise. At co-fine-tuning and inference, the pixel pathway is entirely disabled — the action expert operates purely in the robust latent space, conditioned on predicted future latents via a joint attention mechanism.


Question 2 Response: We thank the reviewer for raising this important question. We compare LWM (trained with latent loss only) and HyWorldVLA (trained with latent loss and pixel loss) using the following three metrics, which are evaluated on the full test set (7,605 samples).

MetricLWMHyWorldVLAImprovement
Cosine Similarity ↓0.9797 ± 0.00050.6289 ± 0.005335.8% reduction
Feature Variance ↑0.051 ± 0.0021.71 ± 0.1233.5× increase
Effective Rank ↑196 ± 5436 ± 122.2× increase

From the results we can observe:

  1. Prevents degeneration: LWM exhibits near-complete collapse (cosine similarity 0.98), while HyWorldVLA reduces it to 0.63, demonstrating that hybrid training preserves feature diversity across samples.
  2. Improves quality: Hybrid training increases feature variance by 33.5× and effective rank by 2.2×, indicating richer representations and better latent space utilization.

The results above are obtained by evaluating the models trained for 6,000 steps. We also provide results showing how the metrics change with the number of training steps, which indicates that the improvement persists throughout training. As shown in Fig.4 (B.3 section of the updated manuscript), LWM's cosine similarity remains consistently high (0.96–0.99) across all training steps, never escaping the collapsed state. HyWorldVLA shows a steady decline in cosine similarity from 0.99 to 0.63 as training progresses, indicating that hybrid training continuously improves representation quality rather than providing only initial benefits.


Question 3 Response: (a) The noise robustness evaluation protocol. Specifically, we used Qwen-3.6-Plus to filter the test set frame by frame, obtaining 1,317 frames containing rain or fog. However, we found that the LLM struggles to distinguish the severity of noise. We therefore performed a manual second-pass screening on the 1,317 frames and retained the 655 frames with high blur severity as the final noise-test subset. Its scene distribution is shown in the table below:

ScenarioProportion
Straight (non-intersection)63%
Straight (intersection)28%
Lane change8%
Left turn0.5%
Right turn0.5%

In open-source driving datasets, both lane-change/turning maneuvers and rain/fog frames are individually rare, so their co-occurrence is further attenuated. Despite prioritizing lane-change and turning samples during manual screening, their combined share remains only ~9%. This skew is intrinsic to real-world data collection and highlights a coverage gap in the adverse weather, which merits future community effort in data curation.

(b) The accuracy–robustness trade-off. The results (Tables 5 and 8) exhibit a two-sided failure pattern that can be understood through the IB theory [1, 2] .

  • λ₃ too small (action loss dominant): as training proceeds, the backbone's latent compression and extraction capacity is overridden by action-loss gradients, and the model degrades toward a generic world-model formulation — the representation becomes prone to erroneously attending to high-frequency details that lack temporal stability, echoing the conclusion in Table 6.
  • λ₃ too large (latent-representation loss dominant): the necessary adjustment of backbone representations by action gradients is suppressed — the model loses access to critical details that are decision-relevant in specific scenarios.

References

[1] N. Tishby, F. C. Pereira, and W. Bialek. "The Information Bottleneck Method." Proc. 37th Allerton Conference on Communication, Control and Computing, 2000. arXiv:physics/0004057.

[2] N. Tishby and N. Zaslavsky. "Deep Learning and the Information Bottleneck Principle." IEEE Information Theory Workshop (ITW), 2015. arXiv:1503.02406.

Reviewer_gH4DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HyWorldVLA combines visual-token prediction and future video-latent prediction during world-model pretraining. A language-guided video VAE supplies latent targets, and subsequent co-fine-tuning conditions an action expert on predicted future latents. The paper reports 90.6 PDMS on NAVSIM v1 and 89.7 EPDMS on v2, together with a substantial improvement on a selected set of 655 rain/fog cases.

Critical strengths

The component ablations support combining the two prediction objectives: the full model reaches 90.6 PDMS versus 89.9 for Pure WAM and 87.5 for Pure LWM. Appendix B.1 also provides evidence that the approach extends to a flow-matching action expert.

The robustness results are substantial: Table 6 reports 86.9 PDMS versus 70.0 for Pure WAM. The failure-case analysis is also useful, particularly the examples involving small traffic cues and limited camera coverage.

Critical weaknesses

The robustness study does not fully establish the proposed trade-off. Table 6 omits the Pure LWM variant, although it is central to the argument. The selection criteria for the 655 cases and matched clean/corrupted evaluations are also insufficiently specified, making it difficult to seperate robustness from differences in scene difficulty.

The clean-benchmark improvement over DriveVLA-W0 is modest, 90.6 versus 90.2, without uncertainty estimates or a clear accounting of matched pretraining data and compute. The VAE and world-model pretraining splits also need to be explicitly distinguished from evaluation logs. Additionally, NAVSIM uses non-reactive or pseudo-simulation evaluation, so the paper’s description of the evidence as closed-loop needs qualification.

Questions for authors

Please specify the 655-case selection procedure and provide the Pure LWM result under the same protocol. Are matched clean/corrupted versions available?

Were all VAE and world-model pretraining logs disjoint from evaluation logs? Under matched data and compute, how variable are the clean-benchmark gains across runs?

In Section 3.2, visual patches supply queries and values while text supplies keys. How are the key/value sequence dimensions aligned in the actual attention implementation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer gH4D

We are very grateful for your valuable comments and acknowledgement of our main contributions. We greatly appreciate the time and effort you put into reviewing our paper. Below are our responses to your questions.


Question1 Response: (a) The noise robustness evaluation protocol. We have added an evaluation on real-world weather-noise scenes. Specifically, we used Qwen-3.6-Plus to filter the test set frame by frame, obtaining 1,317 frames containing rain or fog. However, we found that the LLM cannot reliably judge the severity of the noise (i.e., the degree of blurring) — it struggles to distinguish light moisture from significant blur. We therefore performed a manual second-pass screening on the 1,317 frames and retained the 655 frames with high blur severity as the final noise-test subset. Its scene distribution is shown in the table below:

ScenarioProportion
Straight (non-intersection)63%
Straight (intersection)28%
Lane change8%
Left turn0.5%
Right turn0.5%

We note that the low proportion of lane-change and turning scenarios arises from the independent compounding of two low-base-rate attributes: in open-source autonomous driving datasets, lane-change and turning maneuvers are themselves infrequent, and rain/fog frames are likewise infrequent — their joint occurrences are therefore further attenuated in a roughly multiplicative manner. Although we prioritized lane-change and turning samples during manual screening, the combined share of lane-change and turning scenarios remains only ~9% (lane change 8% + left turn 0.5% + right turn 0.5%). This distribution skew reflects an intrinsic property of real-world data collection and highlights a coverage gap of current open-source datasets on the "adverse weather × complex maneuver" combination, which we believe merits joint community effort in future data curation.

(b) We have added a full component ablation on the noise-robustness test subset, with results shown below:

ConfigNC ↑DAC ↑TTC ↑C. ↑EP ↑PDMS ↑
Pure WAM99.175.794.399.867.870.0
Pure LWM96.575.989.699.265.668.6
w/o language guidance in latent98.876.394.799.568.870.7
w/o latent condition in action expert99.587.997.399.381.183.8
w/o latent supervision in co-fine-tuning99.278.594.899.770.973.2
Ours99.791.196.999.284.486.9

Pure LWM is the largest bottleneck: the Pure LWM variant scores 68.6 PDMS, 18.3 points below Ours (86.9). We attribute this to the reppresentation collapse in pure LWM training: the latent representation loses its discriminative structure, preventing the downstream action head from extracting decision-relevant information (shown in Fig.4 of the updated B.3 section). And we provide quantitative evidence demonstrating pixel supervision prevents representation collapse.

MetricLWMHyWorldVLAImprovement
Cosine Similarity ↓0.9797 ± 0.00050.6289 ± 0.005335.8% reduction
Feature Variance ↑0.051 ± 0.0021.71 ± 0.1233.5× increase
Effective Rank ↑196 ± 5436 ± 122.2× increase

LWM (without pixel supervision) exhibits near-complete collapse with cosine similarity of 0.98, where distinct inputs produce nearly identical representations. HyWorldVLA (with pixel supervision) reduces cosine similarity to 0.63 and increases effective rank by 2.2× and feature variance by 33.5× , demonstrating that pixel supervision effectively prevents representation collapse.

(c) Since our noisy test data is collected from the real world, native clean/corrupted pairs are unavailable.


Question2 Response:

(a) Constructed as a subset of the full test set, our noisy test set is inherently isolated from the training data, preventing any data leakage.

(b) Under identical conditions, the variation across runs on the benchmark is on the order of 0.1% (ranging from 0.02% to 0.1%).


Question3 Response: We sincerely thank the reviewer for raising this important technical question. We apologize for the inaccuracy in our original description. The correct description is visual patches serve as queries, while Flan-T5 text embeddings provide both keys and values. Since keys and values both originate from text embeddings, they naturally share the same sequence length, ensuring proper dimension alignment. We will correct Section 3.2 in the future version. We thank the reviewer for this clarification.


Thank you again for the constructive comments. We hope these explanations could address your concerns. Any further questions or suggestions would be greatly appreciated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.