acceptodds
Under review as a conference paper at ICLR 2027

PCIWM: A Unified Driving World State for Prediction and Planning

Abstract

We introduce PCIWM, a driving world model that explicitly constructs physical (P), causal-semantic (C), interaction (I), and ego (E) states and maps them jointly into one world-state representation. A geometry-aware composer, shared transition under external ego-motion inputs, and typed decoders turn this definition into candidate-specific predicted futures. We evaluate the architecture through two connected questions: what it predicts, and what decisions those predictions enable. Controlled nuScenes studies test factor inclusion, composition, relation identity, and transition sharing. Historical controls show relation-prediction and long-horizon stability gains; common-budget and three-seed comparisons reveal observable-dependent tradeoffs and remaining single-step prediction errors. On NAVSIM, predicted traffic improves a component readout over static traffic, and a state-motivated candidate library improves coverage and selected utility over equal-count alternatives. Raw consequences yield a positive but uncertain five-seed selection gain. A further control tests the incremental use of later raw-state updates: repeating each candidate's first predicted state outperforms trajectory-only inputs in an exploratory three-seed comparison, while the full multi-step sequence performs worse than that repetition. These cached, annotation-assisted development studies connect an explicit unified-state architecture to controlled prediction-to-decision evidence, rather than equating predictive fidelity with planning utility.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, autonomous driving, structured representations, action-conditioned dynamics, counterfactual planning, decision quality

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A4AyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses which factors a "world state" in a driving world model should contain, and how these factors should co-evolve, and it appends an extremely large number of experimental results (in the appendix). This is the entirety of the summary I can responsibly provide, because I am completely unable to determine what specific problem this paper starts from, nor what its experimental results actually demonstrate. The paper is extremely difficult to read. Structurally, there is almost no definition or articulation of the problem to solve anywhere in the paper, and no explanation of the research motivation.The use of notation is extremely poor: for example, Table 1 contains H6, K11 and many other opaque symbols, and none of these notations is defined anywhere earlier in the paper. The paper contains 54 tables — I genuinely do not think it is appropriate, for a conference paper with a 9-page main-text limit, to discuss a problem so over-ambitious that it requires 54 tables. On the other hand, the paper contains not a single image: every figure is a flowchart or a line plot. This highly anomalous situation leads me to suspect that the entire paper was generated by AI, and that the authors (if they have actually read it) have no idea how an academic paper should be organized. The code provided by the authors is obviously AI-generated.

Critical strengths

None.

Critical weaknesses

Please check summary.

Questions for authors

Do the authors aware that this is a ICLR submission and will be shown to the public?

1 comment
AuthorsOpenReview ↗

Response to Reviewer A4Ay

Thank you for pointing out that the manuscript made the research question and main results difficult to find. We will state the problem at the start of the revision. When an ego vehicle considers different maneuvers, what must a driving world model preserve about nearby vehicles over a rollout, and which parts of those forecasts help select an ego trajectory?

The merge figure is complete. We have prepared an annotated figure from a recorded NAVHARD merge. The front-camera image identifies the neighboring vehicle as it approaches the ego vehicle’s downstream lane. Two bird’s-eye-view panels show the original ego proposal (K0) and its hard-braking variant (K6), along with the frozen Structured-3 model’s forecast for the same neighboring vehicle under each candidate. At 2.5 s, the predicted distance from the ego rear axle to the neighboring vehicle’s center is 2.7 m for K0 and 10.6 m for K6. The model predicts that the neighboring vehicle advances 2.4 m and 0.9 m over four seconds, respectively. The vehicle advances 15.0 m in the recorded continuation. The figure therefore shows both how the forecasts vary with the ego candidate and how substantially the model underestimates the neighboring vehicle’s forward motion in this example.

Revision plan. We will place the figure near the start of the paper and organize the main results around two questions. How do the state and transition choices affect prediction? Which predicted information helps select among fixed candidate trajectories? We will define H6 (the sixth forecast step, about 3 s) and K11 (a pool of 11 candidate trajectories) before their first use. The main text will retain one core prediction table and one core planning table. Detailed ablations and supporting results will move to the appendix.

We understand that the submission and discussion are public. The manuscript’s AI Usage Statement describes assistance with implementation, experiment orchestration and verification, result analysis, and writing. The authors take responsibility for the manuscript, code, and reported results.

Reviewer_cKKsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes PCIWM, a driving world model that explicitly represents physical context, semantic context, interaction structure, and ego state as four factors of a unified world state. A geometry-aware composer maps these factors into shared latent slots, which evolve under candidate ego-motion inputs and are decoded into predicted states. The paper evaluates representation and dynamics through nuScenes architecture controls, and examines prediction-based candidate selection through separate NAVSIM development studies. The results show observable-specific prediction tradeoffs and some benefits from using predicted consequences, while the incremental utility of later raw-state evolution is not demonstrated.

Critical strengths
  1. Useful diagnostic controls at the prediction-to-selection interface.

Within a fixed candidate pool, the paper compares trajectory-only inputs, full predicted-state sequences, and repetition of each candidate’s first predicted state. These comparisons distinguish the question of accessing predicted consequences from that of benefiting from their subsequent temporal evolution. The first-state-repetition result is particularly informative about limitations of the current interface, even though it does not establish the planning value of the PCIE architecture itself.

Critical weaknesses
  1. The motivation and gap relative to prior work are insufficiently developed.

The introduction begins with the PCIE definition and architecture rather than establishing a specific limitation of existing driving world models. The observation that geometry, semantics, interactions, and ego motion are complementary does not sufficiently explain why they require this particular explicit factorization and shared evolution. Moreover, the related-work discussion acknowledges that existing approaches already incorporate structured representations, semantics, action conditioning, and future-based planning, but does not clearly identify the unresolved problem addressed by PCIWM. Consequently, the necessity and significance of the proposed construction are difficult to assess.

  1. The planning studies do not isolate the contribution of the proposed PCIE architecture.

The factor, composition, and transition controls primarily evaluate prediction on nuScenes, whereas the NAVSIM planning studies use separate model and cache lineages. The within-pool consequence controls investigate whether particular predicted inputs help a selector; they do not establish whether the PCIE factorization, composer, or shared transition improves planning over simplified alternatives. Thus, the decision-level value of the paper’s architectural choices remains unresolved. A comparison of architectural variants within a common planning pipeline, with matched input conditions, candidate pools, and selector training, would be needed to support this attribution.

Questions for authors
  1. What specific limitation of existing structured, action-conditioned driving world models motivates the explicit PCIE factorization and shared composition/transition? Which capability or failure mode does this construction address beyond combining the four information types?

  2. Given the separate architecture and planning study lineages, should the contribution be understood as an architectural proposal accompanied by diagnostic studies of consequence readouts, rather than evidence that the PCIE architectural choices improve decisions? Please clarify whether any reported comparison directly supports the latter interpretation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer cKKs

Thank you for asking us to clarify the motivation for PCIWM and what the planning experiments establish.

Motivation. Consider a merge where the ego vehicle can brake or continue. Under either candidate, the neighboring vehicle’s BEV position, semantic attributes, and interaction edges need to remain associated with the same vehicle as its motion and relations change. One concrete failure occurs when an edge is keyed by both its endpoints and its relation label. A label change then changes the edge’s identity even when the two actors remain the same. In our historical edge-identity control, keying edges by ordered actor pairs and predicting the relation label as a time-varying attribute reduced graph-edge validation loss by 14.6%. Reliable prediction of relation changes during autoregressive rollouts remains an open problem in our experiments.

GraphAD models interactions among the ego vehicle, other agents, and the map. WoTE uses predicted BEV futures to evaluate candidate trajectories. Our question is whether geometrically aligning physical, semantic, interaction, and ego information in common slots, then evolving those slots jointly under a candidate ego action, helps preserve actor-linked information over the rollout. Typed decoders let us measure motion and relation errors separately. The prediction controls show gains on some observables and losses on others. We will state this question and the observed tradeoffs before introducing the architecture.

Planning ablation with a fixed pipeline. The original NAVSIM studies tested which predicted information helps the selector. To test the composer at the decision level, we now compare Structured-3 and Monolithic-3 on all 5,912 entries of the official local two-stage NAVHARD evaluator (450 original and 5,462 synthetic). Structured-3 uses factor-specific attention and learned gates. Monolithic-3 concatenates the same input tokens into a single attention pool. Both have three composer layers and share the input schema and decoder architecture. We reran the candidate forecasts for each checkpoint with the same observations, fixed bank of 11 candidates per entry, frozen V23 readout and calibration, and two-stage joint trajectory search.

On the ×100 scale, Structured-3 scored 74.07 / 70.61 / 54.11 for Stage 1 / Stage 2 / combined Extended PDM Score (EPDMS). Monolithic-3 scored 74.95 / 69.34 / 53.53. Structured-3 minus Monolithic-3 was −0.89 / +1.26 / +0.58 points, computed from the unrounded scores. The combined difference has a descriptive 95% paired-bootstrap interval of [−0.38, +1.72] points over 34 source recordings.

Scope of the evidence. Both checkpoints used the same four-epoch schedule and a single paired training seed (1701). Structured-3 has 22.80M parameters and 125 realized AMP optimizer updates, compared with 18.43M and 124 for Monolithic-3 out of 128 attempted updates each. Both use the historical annotation-assisted inputs, including Stage 1 yaw-rate and intent fields that can depend on a frame beyond the observation window. The +0.58-point estimate measures the planning difference between these two trained composers within fixed V23. The interval includes zero, and robustness across training seeds remains untested. The decision-level effects of factor inclusion and transition sharing remain open.

The planning ablation remains inconclusive, and the main concerns about motivation and architectural contribution persist. I will maintain my rating.

Reviewer_UQA3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose PCIWM, a world model for world-state presentation, world future prediction and driving planning whose state is the combination of physical (P), causal-semantic (C), interaction (I), and ego states (E). In the method, those four states are first jointly mapped by geometry-aware cross-attention composer into a set of world-state slots. Then, the world-state slots are propagated by a shared transition transformer under conditioned ego-motion input, and typed decoders predict the future states. Finally, a planning readout computes scores to evaluate those futures. In their experiments, they evaluate the proposed method based on two datasets: nuScenes and NAVSIM/NAVHARD. nuScenes is used to test ablation study of different world-state factors inclusion, transition sharing (shared transition VS independent transition), composition mechanism (structured VS monolithic), and relation identity (baseline VS Persistent identity). NAVSIM/NAVHARD are used to test planning readout and trajectory proposal expansion. Their experiment results show certain improvement using a world-state based planning compared to trajectory-based planning.

Critical strengths
  1. The paper proposes a structured world model that combines physical (P), causal-semantic (C), interaction (I), and ego states (E) into a unified action-conditioned representation. The use of geometry-aware composition and typed decoders makes different aspects of PCIE separately assessable through comparative experiments on world-state factor inclusion, transition sharing (shared transition vs. independent transition), composition mechanisms (structured vs. monolithic), and relation identity (baseline vs. ordered identity). This is a useful contribution, because it makes the evaluation of world-state prediction more systematic.
  2. In their experiments, the authors provide detailed results and transparent analysis. They present ablation studies on world-state factor inclusion, transition sharing (shared transition vs. independent transition), composition mechanisms (structured vs. monolithic), and relation identity (baseline vs. persistent identity), which help identify the contribution of each design choice. The experimental results show improved results compared with the authors’ own baselines in future world-state prediction and driving trajectory planning.
Critical weaknesses
  1. The claims about the proposed method are not fully supported. a. As shown in Tables 2 and 3, the proposed full PCIE model does not always perform the best. Adding factor C (causal-semantic) sometimes increases the mean velocity prediction error, which makes the necessity of factor C questionable. b. The comparison between independent transition and shared transition is not entirely fair because of the different numbers of parameters, which makes the results questionable as well. c. Multi-step predictions result in worse downstream trajectory planning performance than simple repetition of the first predicted state, which makes the effectiveness of the prediction questionable.
  2. Insufficient experiments. a. The paper only provides comparison experiments with internal baselines and lacks direct performance comparisons with state-of-the-art methods, which makes its claims less convincing. b. The experiments in this paper are all based on non-reactive evaluation, where other agents are not influenced by the ego vehicle’s actions. However, evaluation in reactive closed-loop simulation is important for world-model-based driving planning because the method is supposed to predict interactions with other agents and plan appropriate trajectories.
Questions for authors
  1. Some of the inputs, such as BEV feature grids, structured semantic attributes, and directed interaction graphs, may be inaccurate. How sensitive is the proposed method to noisy inputs?
  2. Are all inputs of the proposed method realistic for online deployment? Can all inputs of the proposed method be estimated online in real time?
  3. In the section ‘Training Budgets and Endpoints’, the authors mention that the common architecture comparison only use 10% of the available training data and train for four epochs. Has the model fully converged after four epochs of training? Could the authors provide results using the full training dataset with more training epochs?
  4. The K10 and K11 expansions seem heuristic with fixed transformations. How were the K10 and K11 expansions determined?
  5. The current ablation studies primarily evaluate world-state prediction, which is not the final goal of the proposed method. Could the authors provide an ablation study for the final driving trajectory planning task?
1 comment
AuthorsOpenReview ↗

Response to Reviewer UQA3

Thank you for the detailed questions about the architecture controls, input assumptions, and planning evaluation.

Contribution of C. Across three nuScenes seeds, adding the semantic factor C to PIE increased mean velocity-vector error by 0.0535 m/s at the sixth forecast step (H6, about 3 s). It reduced lateral-velocity and both BEV feature errors in all three seeds. Whether C is necessary for planning remains untested.

Transition sharing. The variants use the same composed input slots and training exposure, with 26.73M and 46.49M total parameters for shared and independent transitions. In the historical comparison, autoregressive H5 ego error amplification relative to teacher forcing was 2.75× and 3.43×, respectively. The parameter difference limits what we can attribute to sharing. At the common training budget, independent transitions reduce velocity-vector error, while shared transitions reduce yaw-rate and lateral-velocity errors.

Training budget. The common comparison uses 1,484 of 14,837 training pairs for four epochs. On the same 149 evaluation chains, seed 7 PCIE's H6 velocity-vector error fell from 1.126 to 0.617 m/s in the final epoch. This comparison does not establish convergence. Historical 12-epoch full-data factor controls used an earlier intent cache. A longer full-data comparison of all nine variants under the revised inputs remains open.

Inputs and noise. We use cached BEV features and actor annotations to ground semantics and build the graph. The new nuScenes comparisons use a history-based intent rule, and other motion fields retain adjacent-frame dependencies. Sensitivity to controlled input noise, sensor-only operation, and end-to-end latency remain unmeasured.

Candidate construction. K10 adds four fixed variants to six learned proposals. Hard and soft braking taper pose increments from anchors 0 and 5, respectively. The curvature variants scale lateral and yaw increments by 0.5 and 1.5, respectively. K11 adds constant velocity. On 482 development entries with 11 candidates per pool, oracle PDM was 92.78 for this library, 89.50 for generic lateral expansion, and 89.17 for generic speed expansion. Perturbation magnitudes differ across libraries.

Planning ablation and reactive evaluation. On all 5,912 entries of the official local two-stage NAVHARD evaluator, we recomputed candidate forecasts with Structured-3 and Monolithic-3 while holding the inputs, K11 candidate bank, V23 readout, and joint search fixed. Structured-3 scored 54.11 combined EPDMS versus 53.53 for Monolithic-3 (+0.58 points on the ×100 scale). The Stage 1 and Stage 2 differences were −0.89 and +1.26 points. The descriptive 95% paired-bootstrap interval over 34 source recordings was [−0.38, +1.72] points for the combined difference. Both checkpoints came from one paired world-training seed, and their parameter counts differ (22.80M versus 18.43M). Both use historical annotation-assisted inputs, including Stage 1 yaw-rate and intent fields that can depend on a frame beyond the observation window.

The official scorer uses reactive IDM traffic agents in both stages. V23 joint search uses benchmark-provided branches and weights each synthetic scene by the distance from its start to the Stage 1 candidate endpoint. End-to-end online closed-loop validation remains open.

Multi-step forecasts and external comparisons. In the original input controls, predicted traffic improved selected PDM over static traffic by 2.83 points [0.79, 5.10], with an uncertain difference against constant-velocity traffic. In the three-seed raw-state control, first-state repetition improved over trajectory-only inputs by 6.95 points [1.11, 13.09] in an exploratory comparison. The full sequence scored 61.27 versus 65.07 for first-state repetition, a difference of −3.80 [−8.98, 1.20]. The later steps showed no added value for that readout. These are 95% recording-group intervals. The MILE-style and OccWorld-style experiments use internal adaptations. A matched comparison with the original external systems remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.