acceptodds
Under review as a conference paper at ICLR 2027

SiWM: Learning Planning-Relevant World States via Semantic Integration

Abstract

End-to-end autonomous driving often learns planning representations through predefined perception tasks such as object detection, lane structure estimation, and motion prediction. However, it is difficult for a finite set of tasks and labels to cover all scenes and state changes that affect driving decisions on open roads, and this supervision typically requires extensive manual annotation. We quantify perception task losses' impact on closed-loop driving and investigate how to learn environment representations that support it without these losses. We propose the Semantics-Integrated World Model (SiWM), which integrates visual features and continuous planning semantics from a large language model (LLM) into a shared world state, so that both types of information contribute jointly to state learning and recurrent updates. Through action-conditioned future prediction and feature reconstruction, the model learns how the environment evolves and preserves scene information. It generates ego trajectories from the fused state, incorporating driving task information throughout state learning and planning. After 4 task-specific training epochs, SiWM outperforms the direct planning baseline in closed-loop driving. The gains are 20.43 points in Driving Score (DS) and 21.36 percentage points in Success Rate (SR). In some scenarios involving sudden changes, it also achieves better closed-loop performance than ORION, which uses explicit perception task supervision. Further experiments show that integrating planning semantics into world model learning outperforms fusing planning semantics only at the planner, while multi-token representations substantially alleviate collapse from converging state directions across scenes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, World Models, Vision-Language Models, Representation Learning, Closed-Loop Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DCU1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SiWM investigates whether recurrent world modeling can recover useful planning representations after removing explicit perception-task losses from ORION. It integrates visual features and continuous LLM planning features into a multi-token recurrent state, trained through action-conditioned latent prediction, visual/semantic feature reconstruction, trajectory imitation, language supervision, and a collision loss. A conditional VAE generates trajectories from the fused state. The proposed TokenSSM uses 48 visual, eight semantic, and eight shared tokens.

On the 220 Bench2Drive routes, SiWM reports Driving Score (DS) 66.10 and Success Rate (SR) 38.63%, compared with 45.67 and 17.27% for ORION with perception losses removed. These gains occur after four task-specific epochs, compared with 18 for the Direct baseline. Integrating semantics inside the world model also outperforms planner-only semantic fusion. A five-route door-opening subset favors SiWM over full ORION, although full ORION remains stronger on the overall benchmark.

Critical strengths
  1. The motivating ablation is compelling: removing perception losses barely changes open-loop trajectory error but substantially reduces closed-loop performance (Figure 1). This directly supports evaluating representations by their effect on driving behavior.

  2. The Internal-versus-External comparison addresses a meaningful architectural question. Holding the feature sources, training schedule, and planner framework fixed, internal semantic integration improves DS by 9.20 and SR by 6.82 percentage points (Table 3).

  3. The model combines temporal prediction with reconstruction of features before projection. This is a reasonable way to encourage recurrent states to retain scene information while incorporating planning semantics, and the loss definitions and coefficients are supplied in Appendix A.

  4. The paper provides full-benchmark closed-loop evaluation, targeted door-opening results, qualitative interaction examples, and representation diagnostics. Reporting 20 hours on eight H20 GPUs and explaining the ten-offset definition of an epoch are also useful for interpreting training cost.

Critical weaknesses
  1. The central performance result is recovery relative to an ablated baseline. Full ORION achieves DS 77.74 and SR 54.62%, versus SiWM's 66.10 and 38.63% (Table 1). Removing perception losses may still be attractive, but the overall trade-off between annotation requirements and performance is not quantified. The five door-opening routes support a narrower finding than general superiority to perception-supervised planning.

  2. The experiments do not fully identify which world-model mechanisms produce the gains. The Internal/External control is useful, but there are no corresponding closed-loop ablations for recurrence alone, reconstruction without dynamics prediction, removing action conditioning, or removing semantic reconstruction. Consequently, the results do not isolate whether learned dynamics, temporal aggregation, or additional representation supervision is most important.

  3. The representation-collapse comparison is incomplete. The RSSM run was stopped after collapse rather than evaluated as a completed, tuned driving baseline (Introduction; Section 4.3.1). Figure 7 also shows substantially different visual-feature similarities for the two configurations before comparing posterior similarity. This complicates attributing the difference specifically to multi-token states. Mean cosine similarity detects directional alignment, but does not establish how much task-relevant information is retained or whether TokenSSM is the essential remedy.

  4. The evidence for unusual-event generalization is limited. Table 2 contains five routes for one door-opening scenario, with ORION results taken from public route records. Repeated trials under matched simulator settings are not reported. The examples are encouraging, but variation in traffic execution and scenario difficulty could affect the magnitude of the observed advantage. No uncertainty across training seeds is provided for the main or ablation results either.

  5. Removing perception-task losses does not yet establish the claimed annotation advantage. The pipeline retains Chat-B2D question-answer annotations, expert trajectories, pretrained representations, and a collision objective using other participants' future positions (Appendices A.2 and D). These are disclosed, but their provenance and annotation costs are not compared with the removed losses. The distinction matters when assessing whether the method reduces supervision requirements in practice.

  6. Training efficiency and deployment behavior need stronger controls. Comparing a four-epoch SiWM endpoint with an 18-epoch Direct endpoint does not provide matched learning curves or matched GPU-hour comparisons. The recurrent model is trained with expert trajectory conditions at one-second intervals, whereas inference uses predicted plans and navigation inputs (Section 3.2). The resulting conditioning shift, state-update frequency during driving, inference latency, and memory cost are not evaluated.

Questions for authors
  1. Provide DS/SR learning curves for Direct, External, Internal, and full ORION under matched data exposure and compute. Include the four-epoch Direct result and quantify the annotation/performance trade-off with partially retained perception supervision.

  2. Add a focused ablation separating temporal aggregation, latent prediction, reconstruction, and action conditioning. A recurrent planner trained only with the existing planning/language objectives would be an especially useful control.

  3. Compare TokenSSM with a tuned, capacity-matched RSSM using the same upstream features and training schedule. Supplement cosine similarity with feature variance/effective rank and probes of planning-relevant scene information, and connect these diagnostics to closed-loop results.

  4. Repeat the door-opening evaluation under identical simulator versions and several traffic seeds, and expand it to additional event types. Report per-scenario collision, route-completion, and progress metrics, including failures where SiWM underperforms ORION.

  5. Specify which labels and pretrained checkpoints remain necessary, including the source of the collision-loss participant trajectories. Quantify the saved annotation effort and clarify any perception supervision inherited through initialization.

  6. Document the recurrent update equations and inference schedule, training sequence length, and exact action-conditioning interface. Test sensitivity to errors in the conditioning trajectory and report complete planning-cycle latency, memory use, and runtime frequency.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DCU1 (Part 1/3)

Thank you for highlighting the ablation that motivates our study and the value of closed-loop evaluation. Your review also pointed out the need to distinguish the roles of the world model’s training objectives. To address this point, we added targeted ablations to Table 3 and reported closed-loop results for RSSM in the new Figure 7(c). In addition, we expanded Appendix C to document the initialization checkpoints and online runtime settings.

Q1. Provide DS/SR learning curves for Direct, External, Internal, and full ORION under matched data exposure and compute. Include the four-epoch Direct result and quantify the annotation/performance trade-off with partially retained perception supervision.

A1. We added a Bench2Drive closed-loop evaluation of Direct after four training epochs. This configuration uses the same supervision as Direct, retaining language and expert trajectory supervision while dropping explicit perception task losses. It achieves 40.52 DS and 13.85% SR. After the same four task-specific training epochs, External and SiWM score 16.38 and 25.58 DS points above Direct, respectively. The ordering for both closed-loop metrics is SiWM > External > Direct. We did not retrain full ORION and therefore do not have its closed-loop DS/SR learning curves.

Q2. Add a focused ablation separating temporal aggregation, latent prediction, reconstruction, and action conditioning. A recurrent planner trained only with the existing planning/language objectives would be an especially useful control.

A2. We added ablations in Table 3 to examine how future feature prediction, explicit dynamics matching, feature reconstruction, and semantic state input affect closed-loop planning. The Direct baseline scores 45.67 DS.

LAW-style builds on Direct by adding a visual world model that provides a visual state to the trajectory decoder. It uses a recurrent planner trained with the existing planning/language objectives. The world model predicts future visual features from the current visual features and the ego trajectory. LAW-style achieves 47.52 DS, 1.85 points above Direct.

External uses visual dynamics matching, as LAW-style does, and adds a visual feature reconstruction loss. It achieves 56.90 DS, 9.38 points above LAW-style.

Internal w/o LL reconstruction uses dynamics matching and visual reconstruction, as External does. This configuration differs from External in two ways. It learns a joint world state from visual and LLM planning features and passes the semantic state WlW_l to the trajectory decoder. With these two changes, DS rises from 56.90 to 63.25.

Adding language reconstruction to Internal w/o LL reconstruction gives full SiWM. DS rises by another 2.85 points to 66.10, the highest in Table 3.

In addition, we use full SiWM as the reference for two ablations. Both configurations use the same temporal aggregation, action-conditioned state prediction, and reconstruction losses for visual and language features.

The first disables only explicit dynamics matching. DS drops from 66.10 to 50.14, a decrease of 15.96 points. Together with the comparison between LAW-style and External, this result shows that combining explicit dynamics matching with visual reconstruction is key to achieving the full closed-loop gain.

The second ablation uses the same training objectives as full SiWM. We remove only the semantic state input WlW_l from the trajectory decoder. DS is 65.28, just 0.82 points below full SiWM. This shows that the visual world state can convey most of the closed-loop gain to the planner.

Thank you for the suggestion about action conditioning. The ego trajectory specifies the ego motion for future-state prediction. For example, driving straight and turning in the same scene lead to different future observations. All world-model configurations in Table 3 use the ego trajectory as the action input.

AuthorsOpenReview ↗

Response to Reviewer DCU1 (Part 2/3)

Q3. Compare TokenSSM with a tuned, capacity-matched RSSM using the same upstream features and training schedule. Supplement cosine similarity with feature variance/effective rank and probes of planning-relevant scene information, and connect these diagnostics to closed-loop results.

A3. In Figure 7(a), RSSM and TokenSSM use the same upstream input setup, EMA policy, and training schedule. The visual branch is trainable in both runs. We trained the models separately under these same settings. The visual features from the two trained models have mean pairwise cosine similarities of 0.8741 for RSSM and 0.4477 for TokenSSM.

In Figure 7(b), we froze the RSSM visual branch to test whether the posterior states would still align. They quickly converged toward the same direction. This shows that the RSSM state update can produce directional collapse even when the visual features are fixed. We trained this configuration through the full schedule. Figure 7(c) reports the closed-loop results on all 220 Bench2Drive routes. RSSM achieved 45.32 DS and 17.18% SR, close to Direct. SiWM with TokenSSM achieved 66.10 DS and 38.63% SR.

Table 5 in Appendix B reports the posterior state spectrum. We randomly projected the TokenSSM posterior states to the same 1,024 dimensions as RSSM. The effective ranks were 152.76 for TokenSSM and 1.43 for RSSM. The leading singular direction accounted for 53.870% and 99.980% of the energy, respectively. At the same dimensionality, TokenSSM captures differences across scenes along more directions.

We tested 32×3232\times32 and 64×6464\times64 RSSM configurations. The larger configuration has 4,096 state components. It still showed directional collapse and achieved 44.68 DS in closed-loop evaluation. We also applied VICReg-style variance and covariance regularization to the RSSM visual features. At the end of training, the mean pairwise cosine similarity between posterior states from different samples was still 0.9997. Directional collapse persisted. The original 1,024-dimensional RSSM already has more world-model parameters than TokenSSM. When we increased the RSSM state size to match TokenSSM, training ran out of GPU memory on an NVIDIA H20.

Q4. Repeat the door-opening evaluation under identical simulator versions and several traffic seeds, and expand it to additional event types. Report per-scenario collision, route-completion, and progress metrics, including failures where SiWM underperforms ORION.

A4. Thank you for the suggestion. We chose sudden door opening because an open door changes both how the vehicle looks and how much space it occupies. We attribute SiWM’s advantage over ORION in this scene to how it handles this rare visual change. Across a wider range of ordinary driving scenarios, ORION with perception task supervision usually performs better.

We evaluated SiWM on each route in Table 2 with five random seeds. None of the runs had collisions or lane-violation penalties. The mean DS for each route therefore equals its mean route completion on a 0–100 scale.

SiWM scored 100 DS under every seed on three of the five routes. On the remaining two routes, 3476 and 3482, some runs ended before completion. Across all seeds, mean route completion was 64.84% on route 3476 and 85.32% on route 3482.

For comparison, ORION’s public evaluation records show a mean route completion of 52.75% across the same five door-opening routes. Four of the five routes were still incomplete when the simulation reached its step limit.

AuthorsOpenReview ↗

Response to Reviewer DCU1 (Part 3/3)

Q5. Specify which labels and pretrained checkpoints remain necessary, including the source of the collision-loss participant trajectories. Quantify the saved annotation effort and clarify any perception supervision inherited through initialization.

A5. The collision loss uses CARLA ground-truth positions of other traffic participants at the corresponding future time steps. Appendix A.2 now explains this setup. Direct, External, and Internal use the same collision-loss setup.

We initialize the visual encoder and projector with OmniDrive’s eva02_petr_proj.pth. We initialize the language branch with exiawsh/pretrain_qformer (Appendix C.1). Both checkpoints come from OmniDrive’s 2D vision-language training. That stage used the image–text alignment and visual instruction data from LLaVA v1.5. The visual encoder was frozen during this stage. Its earlier training used general-purpose 2D object detection supervision from Objects365 and COCO.

Direct, External, and Internal use the same initialization weights and the same sources of supervision. All three use Chat-B2D question–answer pairs and Bench2Drive expert ego trajectories. They also use the same CARLA ground-truth participant positions for the collision loss.

During driving-task training, we remove the explicit losses for object detection, lane structure estimation, and motion prediction of surrounding traffic participants. This removes the need for map annotations and motion prediction annotations. The collision loss requires labels for nearby objects that could collide with the ego vehicle. The reduction in 3D object detection annotation effort depends on how many labels this requires. We train and evaluate in simulation. We have yet to quantify how well the method transfers to real-world driving.

Q6. Document the recurrent update equations and inference schedule, training sequence length, and the exact action-conditioning interface. Test sensitivity to errors in the conditioning trajectory and report complete planning-cycle latency, memory use, and runtime frequency.

A6. Appendix C.2 now describes the training sequences, action-conditioning interface, and online inference settings. In Bench2Drive closed-loop evaluation, SiWM reads the hidden state at the planning token directly. At each replanning step, the current observation updates the world state StS_t according to Equation (2). The planner then uses StS_t to generate the ego trajectory ata_t. The dynamics module uses the state and trajectory to predict the next-state prior S^t+1=pθ(St,ϕ(at))\hat S_{t+1}=p_\theta(S_t,\phi(a_t)). The next observation update uses this prior.

During training, the action input is the expert ego trajectory. During inference, it is the trajectory predicted by the planner. Each training sequence contains the current time step and four future steps, spaced one second apart. The action interface takes six 2D displacement increments, spaced 0.5 seconds apart, over a three-second horizon. We flatten the increments into a 12-dimensional vector and pass it to the dynamics module.

SiWM uses the same one-second interval for training state transitions and online replanning. Between replanning steps, the system reuses and reprojects the latest trajectory. The PID controller tracks it at 20 Hz. If the ego vehicle deviates from the planned trajectory, the next replanning step updates the world state using a new observation. The official ORION implementation replans every simulation tick (0.05 seconds). SiWM calls its planner every 20 simulation ticks. Our Bench2Drive results reflect this schedule and the tracking errors that occur during driving.

In open-loop validation with a batch size of one, the logs show an average processing time of 851 ms per sample. This includes data loading and validation-loop overhead. We did not separately time a full closed-loop planning cycle. Inference uses about 32–35 GB of GPU memory.

Reviewer_UNa1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SiWM introduces a multi-token recurrent world model between ORION's vision-language feature extraction and conditional VAE planner. Visual features and continuous LLM planning semantics jointly participate in state updates, action-conditioned prediction, and feature reconstruction; the fused state conditions trajectory generation. Retaining language and planning supervision while removing explicit perception-task losses, SiWM achieves 66.10 DS and 38.63% SR on 220 Bench2Drive routes after four task-specific training epochs. The paper also reports gains over planner-only semantic fusion and higher mean DS on five door-opening routes.

Critical strengths
  1. The internal semantic-integration design has direct closed-loop support. Under the reported common data and four-epoch training setup, Internal improves over External by 9.20 DS and 6.82 percentage points in SR (Table 3). This comparison addresses the central question of whether semantics should participate in state learning rather than only condition the planner.

  2. The recovery after removing perception-task losses is substantial. SiWM improves over Direct by 20.43 DS and 21.36 percentage points in SR with fewer task-specific training epochs. Meanwhile, Avg. L2 slightly worsens from 0.70 m to 0.73 m (Table 1), showing that the closed-loop gains are not explained simply by better fitting of expert trajectories.

Critical weaknesses
  1. The incremental contribution of explicit predictive supervision is not isolated. Table 3 supports the complete internal-integration design, but does not separate dynamics matching from recurrent fusion and feature reconstruction. The semantic-reconstruction configuration for External is also not explicitly specified. A control retaining TokenSSM, semantic inputs, reconstruction objectives, and the training budget while removing only the explicit dynamics-matching loss is missing. Its contribution to the closed-loop gains therefore remains unclear.

  2. The directional-collapse difference is not independently attributable to multi-token modeling. Figure 7 shows less directional convergence with TokenSSM, but the evaluated visual features themselves have different similarities (0.8741 versus 0.4477), and Figure 7(b) explicitly identifies frozen vision only for RSSM. A comparison with shared input features, consistent freezing and EMA settings, and comparable capacity is missing. Lower cosine similarity alone also does not establish that the retained differences are useful for planning. The discontinued RSSM run provides no completed comparison linking this diagnostic to closed-loop outcomes.

Questions for authors
  1. What happens to closed-loop performance when the explicit dynamics-matching loss is removed while retaining TokenSSM, semantic inputs, feature reconstruction, and the training budget? Please also specify how the world-model losses differ between External and Internal.

  2. Do the models in Figure 7 use the same input features, visual-branch freezing rules, and EMA policies? Can a single-vector control with comparable capacity, supplemented by a state-spectrum analysis or a planning-relevant probe, support the information-retention interpretation?

  3. Are the other participants' positions used by the collision loss ground truth or predictions? Which initialization checkpoints are used, and have they been trained with driving perception labels? Are these supervision sources identical across Direct, External, and Internal?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UNa1 (Part 1/2)

Thank you for recognizing the closed-loop gains from internal semantic integration and for pointing out the need to clarify the contribution of dynamics matching and how state stability relates to closed-loop planning performance. To address these points, we added a dynamics matching ablation in Table 3, a closed-loop evaluation of RSSM after its full training schedule in Figure 7(c), and a posterior state spectrum analysis in Table 5 of Appendix B.

Q1. What happens to closed-loop performance when the explicit dynamics-matching loss is removed while retaining TokenSSM, semantic inputs, feature reconstruction, and the training budget? Please also specify how the world-model losses differ between External and Internal.

A1. We added an ablation in Table 3 that removes explicit dynamics matching. With all other settings held fixed, removing explicit dynamics matching from SiWM reduced DS by 15.96 points to 50.14. This comparison shows that explicit dynamics matching is a key training objective for achieving the closed-loop performance of full SiWM.

External and Internal both use a prior–posterior matching loss and a visual feature reconstruction loss. External matches the predicted prior to the observation posterior for the visual state. Internal applies the same matching objective to the joint visual–semantic state. Internal also adds an LLM planning feature reconstruction loss.

Q2. Do the models in Figure 7 use the same input features, visual-branch freezing rules, and EMA policies? Can a single-vector control with comparable capacity, supplemented by a state-spectrum analysis or a planning-relevant probe, support the information-retention interpretation?

A2. In Figure 7(a), RSSM and TokenSSM use the same upstream input setup, EMA policy, and training schedule, with the visual branch trainable in both runs. We measured visual feature similarity on the features learned by each model. The difference in similarity emerged during the respective training runs under these shared settings. The RSSM visual features showed partial directional convergence, and its posterior states showed more pronounced directional collapse. Visual features and posterior states can influence each other during joint training.

To investigate where directional collapse primarily occurs, we froze the RSSM visual branch for the experiment in Figure 7(b). With the visual features fixed, the posterior states still rapidly converged toward the same direction, showing that the RSSM posterior update itself can produce directional collapse.

To quantify the effect on closed-loop driving, we trained RSSM with its visual branch frozen through the full training schedule and added the closed-loop evaluation in Figure 7(c). Its DS and SR were both close to those of Direct, and the closed-loop gains from world modeling largely disappeared.

We also added a posterior state spectrum analysis in Table 5 of Appendix B. We randomly projected the TokenSSM posterior states to 1,024 dimensions, matching RSSM. The effective ranks of TokenSSM and RSSM were 152.76 and 1.43, respectively, and their leading-direction energy shares were 53.870% and 99.980%. At the same analysis dimensionality, cross-scene variation in the TokenSSM posterior states spans more directions.

Then, we attempted to match the dimensionality of RSSM’s single-vector state to that of TokenSSM. The original 1,024-dimensional RSSM already has substantially more world model parameters than TokenSSM, and much of the difference comes from the two models’ network widths and architectures. Increasing the state size to match TokenSSM triggered an out-of-memory error on our NVIDIA H20 GPUs, so we could not complete this control in the current revision. At the largest state size we could train, 64×6464\times64 (4,096 state components), RSSM still exhibited directional collapse and achieved a final closed-loop DS of 44.68.

AuthorsOpenReview ↗

Response to Reviewer UNa1 (Part 2/2)

Q3. Are the other participants’ positions used by the collision loss ground truth or predictions? Which initialization checkpoints are used, and have they been trained with driving perception labels? Are these supervision sources identical across Direct, External, and Internal?

A3. The collision loss uses CARLA ground-truth positions of other traffic participants at the corresponding future time steps. We added this clarification in Appendix A.2. Direct, External, and Internal all use this collision-loss setup.

We initialize the visual encoder and projector with OmniDrive’s official eva02_petr_proj.pth release and the language branch with exiawsh/pretrain_qformer (Appendix C.1). Both checkpoints come from OmniDrive’s 2D vision-language training on LLaVA v1.5 image–text alignment and visual instruction data. The visual encoder remained frozen during this stage, and its upstream weights inherited general-purpose 2D object detection supervision from Objects365 and COCO.

Direct, External, and Internal share the same visual and language initialization weights and supervision sources. All three use Chat-B2D question–answer pairs, Bench2Drive expert ego trajectories, and CARLA ground-truth positions of other traffic participants for the collision loss.

Reviewer_UNa1OpenReview ↗

Final Assessment

The revision and rebuttal substantially alleviate my main concerns. I therefore change my rating from 2 to 3.

  1. Removing explicit dynamics matching while retaining the other components and reconstruction objectives reduces DS from 66.10 to 50.14 (Table 3), providing clear evidence of its incremental value. The loss differences between External and Internal are also clarified. Internal without language reconstruction still exceeds External by 6.35 DS, further strengthening the empirical case for internal semantic modeling.

  2. RSSM achieves only 45.32 DS after its full training schedule, and the added spectrum analysis shows a substantially lower effective rank than TokenSSM (Figure 7 and Table 5). These results make the directional-collapse analysis more convincing. However, matching the analysis dimensionality does not establish matched model capacity, and a comparison under identical frozen-feature conditions remains unavailable. The interpretation specifically linking the multi-token structure to planning-relevant information retention should therefore remain qualified.

  3. The authors specify that the collision loss uses future ground-truth participant positions from CARLA and state that the controls share initialization and supervision sources. This alleviates my concern about comparison fairness. The conclusions should remain scoped to removing explicit perception-task losses during task-specific training, while acknowledging the roles of pretraining, driving QA, and collision supervision.

The remaining issues concern mechanistic attribution and the completeness of some control configurations. The added ablations and closed-loop results now provide sufficient empirical support for the paper's core design.

Reviewer_UaspOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper starts from the idea that perception labels cannot cover every driving situation. So the authors try removing the supervision for these perception tasks and bringing information from the existing LLM into how the model learns about the scene. They train it to predict future states and reconstruct visual and semantic information, then use this to guide driving. We can see that removing the perception losses hurts performance, and their approach brings some of that performance back. But after reading the paper, I’m still not quite sure what the main new insight is. It seems that the main change is letting the LLM participate earlier in learning about the scene. I also found it hard to tell which parts of the method actually make the difference, and the ablation studies did not really clear this up for me. The authors mention LAW and TokenWM, but do not directly compare with them. So what I mainly learned is that this combination works better under the setting used in this paper. The paper does not tell me much about why this combination would be worth using more broadly, or in what situations it would be especially useful.

Critical strengths

The motivation is clear and meaningful. Reducing the reliance on predefined perception tasks is a worthwhile direction for autonomous driving. The experiments suggest that this is a feasible approach.

Critical weaknesses

The main new insight is unclear, and the lack of direct comparisons with LAW and TokenWM makes the added value hard to judge. The ablations do not clearly show what drives the improvement or further explain when this combination would be especially useful.

Questions for authors

1.Does the improvement come from using LLM information earlier, or from the extra training task of reconstructing that information? Can you separate these two effects? 2.What does the baseline fail to capture about the scene, and what does your method add? 3.Could you compare with a LAW-style baseline and clarify how your external baseline differs from TokenWM?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Uasp

Thank you for recognizing the value of reducing reliance on predefined perception-task supervision and for asking us to clarify the source of the closed-loop gains. To address this question, we added ablations in Table 3 to examine how planning semantics influence the learning of visual world states.

Q1. Does the improvement come from using LLM information earlier, or from the extra training task of reconstructing that information? Can you separate the two effects?

A1. The main closed-loop gains come from incorporating planning semantics into the world model, where they guide the learning of visual world states. To distinguish the effects of internal semantic modeling and LLM feature reconstruction, we added an Internal ablation in Table 3. We disable LLM feature reconstruction and use the same joint visual and semantic state learning, dynamics matching, and visual reconstruction as in full SiWM. This configuration achieves 63.25 DS, 6.35 points above External, showing that joint state learning improves closed-loop performance even without LLM feature reconstruction. Adding LLM feature reconstruction back to this configuration further raises DS by 2.85 points to 66.10.

Q2. What does the baseline fail to capture about the scene, and what does your method add?

A2. The Direct baseline already uses LLM planning semantics to guide trajectory generation. SiWM builds on this by providing the planner with a visual world state learned under semantic guidance. We view LLM planning semantics as primarily providing driving intent and behavioral constraints. However, closed-loop planning also requires fine-grained perceptual information.

To examine the role of the visual world state in closed-loop planning, we added the Internal w/o semantic state input ablation in Table 3. This configuration retains all world model training objectives and removes the semantic state input WlW_l from the trajectory decoder. As in Direct, the trajectory decoder receives the original LLM planning features LL. Its only additional input is the visual world state WvW_v. This configuration achieves 65.28 DS, within 0.82 points of full SiWM. The result shows that most of SiWM’s DS gain over Direct can be achieved by adding the semantically guided visual world state to the planner.

Q3. Could you compare with a LAW-style baseline and clarify how your external baseline differs from TokenWM?

A3. We added a LAW-style ablation in Table 3. LAW-style achieves 47.52 DS and 17.45% SR on Bench2Drive. This configuration follows LAW's prediction objective. It predicts future visual features from the current visual features and ego trajectory, and matches the predictions to the encoded features of the future frame. We also disable feature reconstruction in this configuration. With the same ORION backbone and conditional VAE planning framework, External combines visual-state dynamics matching with visual reconstruction and scores 9.38 DS points higher than LAW-style. This result supports the joint use of dynamics matching and visual reconstruction to learn visual world states in our setup.

External draws on TokenWM's recurrent multi-token state structure. TokenWM uses a frozen DINOv2 encoder, which keeps the visual features fixed during training. We think freezing the encoder helps reduce the risk of visual feature collapse during training, but it also keeps the encoder from adapting its features to the task. During joint training, we unfreeze the online visual branch and train the visual representation with both VLM task losses and world-model losses. The visual target branch provides reconstruction targets and is updated by exponential moving average (EMA). This design allows the visual representation to adapt to the closed-loop planning task during joint training.

Thank you for the added experiments.

  1. Even without LLM feature reconstruction, Internal scores 6.35 DS points higher than External, which is encouraging. However, its trajectory decoder also receives a learned semantic state. A cleaner comparison would disable both LLM feature reconstruction and the decoder’s semantic state input in Internal, then compare it with External. Both decoders would receive the original LLM planning features and the visual world state. This would help isolate the effect of incorporating planning semantics into world-state learning.
  2. The 65.28 DS result shows that SiWM retains most of its gain over Direct without feeding the learned semantic state to the decoder. Comparing Direct, External, and the proposed Internal variant on the same door-opening routes would help show how each planner responds when an open door narrows the available driving space.

Though the paper addresses a worthwhile problem, much of the prose, along with several figures and tables, gives the impression of AI-generated draft material that has received little editorial polish. As a result, the manuscript felt unfinished to me. I will keep both my overall score and my confidence rating unchanged.

AuthorsOpenReview ↗

Follow-up Response to Reviewer Uasp

Thank you for the suggestion.

  1. We added another ablation to Table 3, disabling both LLM feature reconstruction and the decoder’s semantic state input in Internal. This configuration scored 63.07 DS, 6.17 points higher than External. This comparison isolates the effect of incorporating planning semantics into world-state learning.

  2. We compiled External’s results for the five door-opening routes listed in Table 2. External achieved a mean DS of 69.91, compared with 90.03 for full SiWM. Its per-route DS values were 100.00, 47.67, 43.23, 58.68, and 100.00, in Table 2 order.

Thanks also for the feedback on the presentation. We revised parts of the text and made the figures and tables more consistent in style and color. Given the time constraints, we’ll continue to polish the manuscript.

Comments on the paper

2 comments

Dear Reviewers,

Please take time to review the responses from the authors and engage with them for further questions and comments if any.

Time left is not much and please do shortly.

Thanks

AC

AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for taking the time to review our paper and for sharing your constructive feedback.

In response to your suggestions, we added several ablations to Table 3 to examine the roles of dynamics matching, LLM feature reconstruction, and the semantic state input to the trajectory decoder. We also added a closed-loop comparison of RSSM and TokenSSM and an analysis of their posterior state spectra (Figure 7 and Table 5). The appendix now provides more detail on the training and inference settings and clarifies the source of supervision for the collision loss.

Thank you again for your feedback. Your suggestions helped us strengthen the experimental evidence and make the text, figures, and tables clearer.

Authors of Submission 18713

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.