- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
SiWM investigates whether recurrent world modeling can recover useful planning representations after removing explicit perception-task losses from ORION. It integrates visual features and continuous LLM planning features into a multi-token recurrent state, trained through action-conditioned latent prediction, visual/semantic feature reconstruction, trajectory imitation, language supervision, and a collision loss. A conditional VAE generates trajectories from the fused state. The proposed TokenSSM uses 48 visual, eight semantic, and eight shared tokens.
On the 220 Bench2Drive routes, SiWM reports Driving Score (DS) 66.10 and Success Rate (SR) 38.63%, compared with 45.67 and 17.27% for ORION with perception losses removed. These gains occur after four task-specific epochs, compared with 18 for the Direct baseline. Integrating semantics inside the world model also outperforms planner-only semantic fusion. A five-route door-opening subset favors SiWM over full ORION, although full ORION remains stronger on the overall benchmark.
Critical strengths
-
The motivating ablation is compelling: removing perception losses barely changes open-loop trajectory error but substantially reduces closed-loop performance (Figure 1). This directly supports evaluating representations by their effect on driving behavior.
-
The Internal-versus-External comparison addresses a meaningful architectural question. Holding the feature sources, training schedule, and planner framework fixed, internal semantic integration improves DS by 9.20 and SR by 6.82 percentage points (Table 3).
-
The model combines temporal prediction with reconstruction of features before projection. This is a reasonable way to encourage recurrent states to retain scene information while incorporating planning semantics, and the loss definitions and coefficients are supplied in Appendix A.
-
The paper provides full-benchmark closed-loop evaluation, targeted door-opening results, qualitative interaction examples, and representation diagnostics. Reporting 20 hours on eight H20 GPUs and explaining the ten-offset definition of an epoch are also useful for interpreting training cost.
Critical weaknesses
-
The central performance result is recovery relative to an ablated baseline. Full ORION achieves DS 77.74 and SR 54.62%, versus SiWM's 66.10 and 38.63% (Table 1). Removing perception losses may still be attractive, but the overall trade-off between annotation requirements and performance is not quantified. The five door-opening routes support a narrower finding than general superiority to perception-supervised planning.
-
The experiments do not fully identify which world-model mechanisms produce the gains. The Internal/External control is useful, but there are no corresponding closed-loop ablations for recurrence alone, reconstruction without dynamics prediction, removing action conditioning, or removing semantic reconstruction. Consequently, the results do not isolate whether learned dynamics, temporal aggregation, or additional representation supervision is most important.
-
The representation-collapse comparison is incomplete. The RSSM run was stopped after collapse rather than evaluated as a completed, tuned driving baseline (Introduction; Section 4.3.1). Figure 7 also shows substantially different visual-feature similarities for the two configurations before comparing posterior similarity. This complicates attributing the difference specifically to multi-token states. Mean cosine similarity detects directional alignment, but does not establish how much task-relevant information is retained or whether TokenSSM is the essential remedy.
-
The evidence for unusual-event generalization is limited. Table 2 contains five routes for one door-opening scenario, with ORION results taken from public route records. Repeated trials under matched simulator settings are not reported. The examples are encouraging, but variation in traffic execution and scenario difficulty could affect the magnitude of the observed advantage. No uncertainty across training seeds is provided for the main or ablation results either.
-
Removing perception-task losses does not yet establish the claimed annotation advantage. The pipeline retains Chat-B2D question-answer annotations, expert trajectories, pretrained representations, and a collision objective using other participants' future positions (Appendices A.2 and D). These are disclosed, but their provenance and annotation costs are not compared with the removed losses. The distinction matters when assessing whether the method reduces supervision requirements in practice.
-
Training efficiency and deployment behavior need stronger controls. Comparing a four-epoch SiWM endpoint with an 18-epoch Direct endpoint does not provide matched learning curves or matched GPU-hour comparisons. The recurrent model is trained with expert trajectory conditions at one-second intervals, whereas inference uses predicted plans and navigation inputs (Section 3.2). The resulting conditioning shift, state-update frequency during driving, inference latency, and memory cost are not evaluated.
Questions for authors
-
Provide DS/SR learning curves for Direct, External, Internal, and full ORION under matched data exposure and compute. Include the four-epoch Direct result and quantify the annotation/performance trade-off with partially retained perception supervision.
-
Add a focused ablation separating temporal aggregation, latent prediction, reconstruction, and action conditioning. A recurrent planner trained only with the existing planning/language objectives would be an especially useful control.
-
Compare TokenSSM with a tuned, capacity-matched RSSM using the same upstream features and training schedule. Supplement cosine similarity with feature variance/effective rank and probes of planning-relevant scene information, and connect these diagnostics to closed-loop results.
-
Repeat the door-opening evaluation under identical simulator versions and several traffic seeds, and expand it to additional event types. Report per-scenario collision, route-completion, and progress metrics, including failures where SiWM underperforms ORION.
-
Specify which labels and pretrained checkpoints remain necessary, including the source of the collision-loss participant trajectories. Quantify the saved annotation effort and clarify any perception supervision inherited through initialization.
-
Document the recurrent update equations and inference schedule, training sequence length, and exact action-conditioning interface. Test sensitivity to errors in the conditioning trajectory and report complete planning-cycle latency, memory use, and runtime frequency.
3 comments
Response to Reviewer DCU1 (Part 2/3)
Q3. Compare TokenSSM with a tuned, capacity-matched RSSM using the same upstream features and training schedule. Supplement cosine similarity with feature variance/effective rank and probes of planning-relevant scene information, and connect these diagnostics to closed-loop results.
A3. In Figure 7(a), RSSM and TokenSSM use the same upstream input setup, EMA policy, and training schedule. The visual branch is trainable in both runs. We trained the models separately under these same settings. The visual features from the two trained models have mean pairwise cosine similarities of 0.8741 for RSSM and 0.4477 for TokenSSM.
In Figure 7(b), we froze the RSSM visual branch to test whether the posterior states would still align. They quickly converged toward the same direction. This shows that the RSSM state update can produce directional collapse even when the visual features are fixed. We trained this configuration through the full schedule. Figure 7(c) reports the closed-loop results on all 220 Bench2Drive routes. RSSM achieved 45.32 DS and 17.18% SR, close to Direct. SiWM with TokenSSM achieved 66.10 DS and 38.63% SR.
Table 5 in Appendix B reports the posterior state spectrum. We randomly projected the TokenSSM posterior states to the same 1,024 dimensions as RSSM. The effective ranks were 152.76 for TokenSSM and 1.43 for RSSM. The leading singular direction accounted for 53.870% and 99.980% of the energy, respectively. At the same dimensionality, TokenSSM captures differences across scenes along more directions.
We tested and RSSM configurations. The larger configuration has 4,096 state components. It still showed directional collapse and achieved 44.68 DS in closed-loop evaluation. We also applied VICReg-style variance and covariance regularization to the RSSM visual features. At the end of training, the mean pairwise cosine similarity between posterior states from different samples was still 0.9997. Directional collapse persisted. The original 1,024-dimensional RSSM already has more world-model parameters than TokenSSM. When we increased the RSSM state size to match TokenSSM, training ran out of GPU memory on an NVIDIA H20.
Q4. Repeat the door-opening evaluation under identical simulator versions and several traffic seeds, and expand it to additional event types. Report per-scenario collision, route-completion, and progress metrics, including failures where SiWM underperforms ORION.
A4. Thank you for the suggestion. We chose sudden door opening because an open door changes both how the vehicle looks and how much space it occupies. We attribute SiWM’s advantage over ORION in this scene to how it handles this rare visual change. Across a wider range of ordinary driving scenarios, ORION with perception task supervision usually performs better.
We evaluated SiWM on each route in Table 2 with five random seeds. None of the runs had collisions or lane-violation penalties. The mean DS for each route therefore equals its mean route completion on a 0–100 scale.
SiWM scored 100 DS under every seed on three of the five routes. On the remaining two routes, 3476 and 3482, some runs ended before completion. Across all seeds, mean route completion was 64.84% on route 3476 and 85.32% on route 3482.
For comparison, ORION’s public evaluation records show a mean route completion of 52.75% across the same five door-opening routes. Four of the five routes were still incomplete when the simulation reached its step limit.
Response to Reviewer DCU1 (Part 3/3)
Q5. Specify which labels and pretrained checkpoints remain necessary, including the source of the collision-loss participant trajectories. Quantify the saved annotation effort and clarify any perception supervision inherited through initialization.
A5. The collision loss uses CARLA ground-truth positions of other traffic participants at the corresponding future time steps. Appendix A.2 now explains this setup. Direct, External, and Internal use the same collision-loss setup.
We initialize the visual encoder and projector with OmniDrive’s eva02_petr_proj.pth. We initialize the language branch with exiawsh/pretrain_qformer (Appendix C.1). Both checkpoints come from OmniDrive’s 2D vision-language training. That stage used the image–text alignment and visual instruction data from LLaVA v1.5. The visual encoder was frozen during this stage. Its earlier training used general-purpose 2D object detection supervision from Objects365 and COCO.
Direct, External, and Internal use the same initialization weights and the same sources of supervision. All three use Chat-B2D question–answer pairs and Bench2Drive expert ego trajectories. They also use the same CARLA ground-truth participant positions for the collision loss.
During driving-task training, we remove the explicit losses for object detection, lane structure estimation, and motion prediction of surrounding traffic participants. This removes the need for map annotations and motion prediction annotations. The collision loss requires labels for nearby objects that could collide with the ego vehicle. The reduction in 3D object detection annotation effort depends on how many labels this requires. We train and evaluate in simulation. We have yet to quantify how well the method transfers to real-world driving.
Q6. Document the recurrent update equations and inference schedule, training sequence length, and the exact action-conditioning interface. Test sensitivity to errors in the conditioning trajectory and report complete planning-cycle latency, memory use, and runtime frequency.
A6. Appendix C.2 now describes the training sequences, action-conditioning interface, and online inference settings. In Bench2Drive closed-loop evaluation, SiWM reads the hidden state at the planning token directly. At each replanning step, the current observation updates the world state according to Equation (2). The planner then uses to generate the ego trajectory . The dynamics module uses the state and trajectory to predict the next-state prior . The next observation update uses this prior.
During training, the action input is the expert ego trajectory. During inference, it is the trajectory predicted by the planner. Each training sequence contains the current time step and four future steps, spaced one second apart. The action interface takes six 2D displacement increments, spaced 0.5 seconds apart, over a three-second horizon. We flatten the increments into a 12-dimensional vector and pass it to the dynamics module.
SiWM uses the same one-second interval for training state transitions and online replanning. Between replanning steps, the system reuses and reprojects the latest trajectory. The PID controller tracks it at 20 Hz. If the ego vehicle deviates from the planned trajectory, the next replanning step updates the world state using a new observation. The official ORION implementation replans every simulation tick (0.05 seconds). SiWM calls its planner every 20 simulation ticks. Our Bench2Drive results reflect this schedule and the tracking errors that occur during driving.
In open-loop validation with a batch size of one, the logs show an average processing time of 851 ms per sample. This includes data loading and validation-loop overhead. We did not separately time a full closed-loop planning cycle. Inference uses about 32–35 GB of GPU memory.
Response to Reviewer DCU1 (Part 1/3)
Thank you for highlighting the ablation that motivates our study and the value of closed-loop evaluation. Your review also pointed out the need to distinguish the roles of the world model’s training objectives. To address this point, we added targeted ablations to Table 3 and reported closed-loop results for RSSM in the new Figure 7(c). In addition, we expanded Appendix C to document the initialization checkpoints and online runtime settings.
Q1. Provide DS/SR learning curves for Direct, External, Internal, and full ORION under matched data exposure and compute. Include the four-epoch Direct result and quantify the annotation/performance trade-off with partially retained perception supervision.
A1. We added a Bench2Drive closed-loop evaluation of Direct after four training epochs. This configuration uses the same supervision as Direct, retaining language and expert trajectory supervision while dropping explicit perception task losses. It achieves 40.52 DS and 13.85% SR. After the same four task-specific training epochs, External and SiWM score 16.38 and 25.58 DS points above Direct, respectively. The ordering for both closed-loop metrics is SiWM > External > Direct. We did not retrain full ORION and therefore do not have its closed-loop DS/SR learning curves.
Q2. Add a focused ablation separating temporal aggregation, latent prediction, reconstruction, and action conditioning. A recurrent planner trained only with the existing planning/language objectives would be an especially useful control.
A2. We added ablations in Table 3 to examine how future feature prediction, explicit dynamics matching, feature reconstruction, and semantic state input affect closed-loop planning. The Direct baseline scores 45.67 DS.
LAW-style builds on Direct by adding a visual world model that provides a visual state to the trajectory decoder. It uses a recurrent planner trained with the existing planning/language objectives. The world model predicts future visual features from the current visual features and the ego trajectory. LAW-style achieves 47.52 DS, 1.85 points above Direct.
External uses visual dynamics matching, as LAW-style does, and adds a visual feature reconstruction loss. It achieves 56.90 DS, 9.38 points above LAW-style.
Internal w/o L reconstruction uses dynamics matching and visual reconstruction, as External does. This configuration differs from External in two ways. It learns a joint world state from visual and LLM planning features and passes the semantic state Wl to the trajectory decoder. With these two changes, DS rises from 56.90 to 63.25.
Adding language reconstruction to Internal w/o L reconstruction gives full SiWM. DS rises by another 2.85 points to 66.10, the highest in Table 3.
In addition, we use full SiWM as the reference for two ablations. Both configurations use the same temporal aggregation, action-conditioned state prediction, and reconstruction losses for visual and language features.
The first disables only explicit dynamics matching. DS drops from 66.10 to 50.14, a decrease of 15.96 points. Together with the comparison between LAW-style and External, this result shows that combining explicit dynamics matching with visual reconstruction is key to achieving the full closed-loop gain.
The second ablation uses the same training objectives as full SiWM. We remove only the semantic state input Wl from the trajectory decoder. DS is 65.28, just 0.82 points below full SiWM. This shows that the visual world state can convey most of the closed-loop gain to the planner.
Thank you for the suggestion about action conditioning. The ego trajectory specifies the ego motion for future-state prediction. For example, driving straight and turning in the same scene lead to different future observations. All world-model configurations in Table 3 use the ego trajectory as the action input.