acceptodds
Withdrawn from ICLR 2027

Collective Ego-Agent World Modeling for End-to-End Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World modeling enables end-to-end autonomous driving (E2EAD) to anticipate scene evolution and learn from action consequences. However, existing methods predominantly adopt an ego-centric perspective, treating surrounding agents as scene content rather than explicit rollout subjects. This formulation limits the completeness of future scene representations and leaves predictive consistency across ego and agent perspectives underconstrained. In this paper, we propose CoDreamer, an E2EAD framework for collective ego-agent world dreaming. CoDreamer integrates an ego-centric branch and selected agent-centric branches through a shared latent dynamics model. To improve spatial completeness, each agent branch reconstructs future features over its full spatial window from a coordinate-transformed, partially observed Bird’s-Eye-View (BEV) representation, learning to infer how regions beyond its spatial support evolve. To encourage cross-perspective agreement, ego and agent rollouts are conditioned on their respective logged trajectories and jointly supervised at the prediction horizon by coordinate-aligned views of the same realized future. Ego-Agent Spatial Tokens (EAST) encode relative geometry and entity type, enabling the shared predictor to distinguish the rollout subject from other participants across coordinate frames. This collective learning objective provides additional supervision for planning representations, with future world-model transitions confined to training in the evaluated configuration. On NAVSIMv1, CoDreamer achieves state-of-the-art planning performance among world-model-based methods using the same ResNet-34 backbone, attaining 92.0% PDMS. Ablation studies further demonstrate improved prediction accuracy and scene completion beyond observed regions. These findings underscore the value of agent-centric world modeling for capturing more complete and coherent scene dynamics. Code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, World Models, Multi-Agent World Modeling, Interactive Planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_FXipOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoDreamer, which extends ego-centric world modeling to surrounding agents for end-to-end autonomous driving. Agent-centered BEV features are obtained by transforming the ego BEV into local coordinate frames. A shared predictor then reconstructs future semantic maps using logged trajectories and Ego-Agent Spatial Tokens (EAST). The auxiliary world-model branches are removed at inference. The method achieves 92.0 PDMS on NAVSIM-v1 with a ResNet-34 backbone.

Critical strengths

1.Extending future prediction from the ego vehicle to surrounding agents is a reasonable direction for improving representation learning. 2.The auxiliary world-model branches are used only during training, without requiring future prediction at inference. 3.The camera-only planning results are competitive, and the paper includes component and agent-number ablations.

Critical weaknesses

1.The main motivation is not sufficiently validated. The paper emphasizes spatial completion beyond the ego BEV and cross-perspective consistency, but neither is directly evaluated. Table 4 only compares overall future mIoU with a copy-current baseline. Without results on unobserved regions or cross-view consistency, the central claims remain unsupported by direct experimental evidence.

2.The contribution of agent-centric world modeling is unclear. Table 2 shows only a 0.1 PDMS gain from Agent-WM, while EAST adds another 0.8. Without an ego-only + EAST baseline or controls with matched spatial coverage and supervision, it is difficult to determine whether the improvements actually come from agent-centered prediction.

Questions for authors

1.Can the authors provide an ego-only + EAST ablation to clarify the contribution of agent-centric modeling?

2.Can the authors report prediction accuracy outside the ego BEV support and measure consistency between coordinate-aligned ego and agent predictions?

3.How does the method compare with enlarged ego-BEV or spatial-crop baselines under matched supervision? What is the contribution of the additional agent motion prediction loss?

4.How much does trajectory conditioning contribute? Can the authors evaluate action-free or shuffled-trajectory controls?

Reviewer_aqayOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoDreamer, an end-to-end autonomous driving framework that uses collective ego-agent world dreaming as training-only auxiliary supervision. Ego-centric and agent-centric branches share an action-conditioned scene predictor and a semantic decoder. Full-window agent-centric reconstruction extends future supervision beyond the original ego BEV support, while Ego-Agent Spatial Tokens (EAST) encode participant-relative geometry and entity types. CoDreamer achieves 92.0 PDMS on NAVSIMv1, compared with 91.1 for the ego-centric world-modeling baseline.

Critical strengths
  • The participant-centered formulation is well motivated. Reconstructing future agent neighborhoods provides a useful way to exploit scene-wide annotations beyond a single ego-centered window.
  • The shared predictor and EAST provide a coherent design for handling multiple local reference frames.
  • Collective world dreaming is confined to training, avoiding future-scene prediction during inference.
  • The complete model achieves competitive planning performance, and the manuscript appropriately acknowledges that the main-table comparisons do not follow a common retraining protocol.
Critical weaknesses

The contribution of agent-centric world modeling is not sufficiently isolated.
In Table 2, Agent-WM improves PDMS from 91.1 to 91.2, whereas adding EAST increases it to 92.0. Without an ego-only configuration with EAST, it is unclear whether the improvement requires collective ego-agent world dreaming or mainly follows from explicit spatial conditioning. A supervision-matched local-window reconstruction baseline would also help distinguish participant-centered prediction from the benefits of additional spatial coverage.

  1. The central claims are not directly evaluated.
    Cross-perspective consistency and spatial completion beyond the original ego BEV support are prominent motivations. However, no metric measures agreement between coordinate-aligned predictions, and no results isolate reconstruction outside the original input support. Table 4 establishes an advantage over copy-current prediction, but does not show whether agent-centric supervision improves these particular properties over ego-only world modeling.

  2. The role of participant-conditioned actions remains unclear.
    The action token is constructed using the corresponding ground-truth trajectory, which already specifies the rollout subject’s future position and heading. This is a valid training setup, but the experiments do not establish how much prediction depends on the action token or whether the benefit extends beyond reconstructing the subject itself. Removing action conditioning and replacing logged trajectories with predicted trajectories would clarify this issue.

  3. The empirical evidence is limited.
    The main gain over the baseline is 0.9 PDMS, with no repeated-run statistics. The 0.1-point improvement from Agent-WM alone is therefore difficult to assess. Evaluation on a single benchmark setting also leaves open whether the proposed supervision particularly benefits interaction-intensive scenarios.

Questions for authors
  1. What is the performance of the ego-centric world-modeling baseline when equipped with EAST but without Agent-WM?
  2. Are the auxiliary agent detection and motion prediction branches, their losses, and their training settings identical across all configurations in Table 2?
  3. How does agent-centered reconstruction compare with randomly translated and rotated local windows under a comparable supervision budget?
  4. In Table 4, is future mIoU evaluated on ego-centered windows, agent-centered windows, or both? What are the corresponding results for the ego-only baseline?
  5. Can the authors report future semantic prediction separately inside and outside the original ego BEV support, particularly for dynamic semantic classes?
  6. Does the complete model reduce disagreement between ego- and agent-centered predictions after coordinate alignment? How does EAST affect this agreement?
  7. How do prediction accuracy and planning performance change when action conditioning is removed or logged trajectories are replaced with predicted trajectories?
  8. Does the benefit of action conditioning remain when the rollout subject’s semantic footprint is excluded from the prediction metric?
  9. Figure 2 includes a “Rollout Mask,” while Equation 5 describes entity types and a validity indicator. How is the rollout subject explicitly identified, and how is this mask incorporated into EAST?
  10. Are the reported planning improvements consistent across random seeds? Are larger gains observed in intersections, merging scenarios, or dense traffic?
  11. What are the additional training-time and memory costs of the selected agent branches? Has inference latency been measured against the baseline?
Reviewer_f3hiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

From an implementation perspective, this paper appears to be closer to a planning-oriented representation learning approach assisted by future BEV reconstruction.

Although predicting future BEV representations is not a novel idea in itself, the motivation of this work addresses a genuine and important problem. Treating surrounding agents merely as part of the scene content is indeed a common compromise in existing world-model-based approaches. The paper attempts to address this limitation by explicitly considering the perspectives of surrounding agents.

The authors demonstrate improvements in planning performance on the NAVSIM v1 benchmark.

Critical strengths

The most significant strength of this paper lies in its motivation. The authors explicitly argue that surrounding agents should be modeled as independent decision-making entities rather than merely as components of the scene background. I believe this is an important distinction from many existing world-model-based approaches.

For instance, several related approaches employ video generation backbones, such as Wan 2.2, to generate future driving scenes. However, these generated scenes are typically ego-centric, inevitably mixing decision-relevant dynamic agents with other visual content, including nearby static backgrounds and stationary objects. Such a formulation may introduce implicit difficulties for downstream decision-making, as the model must distinguish information relevant to planning from other scene content.

More generally, generating visually realistic future scenes does not necessarily provide better representations for planning than directly learning task-relevant features. The entanglement of dynamic agents and static scene content may be one possible explanation for this phenomenon.

From this perspective, explicitly modeling surrounding agents from their own viewpoints is a meaningful and potentially valuable direction. I find the motivation particularly compelling because it addresses a fundamental limitation of existing world-model formulations, rather than simply introducing another architectural modification.

Critical weaknesses

Despite the compelling motivation, I have two major concerns regarding the experimental validation and whether the reported results sufficiently support the paper's central claims.

W1. Insufficient evaluation across benchmarks

The experimental evaluation is rather limited, as the paper reports results only on NAVSIM v1.

This is particularly concerning given that several earlier approaches discussed in the related work already provide evaluations on multiple benchmarks. For instance, SeerDrive, published approximately one year earlier, reports results on both NAVSIM and nuScenes.

Furthermore, recent works in this area commonly evaluate their methods on at least two benchmarks selected from NAVSIM (v1/v2), nuScenes, NAVHARD, Bench2Drive, HUGSIM, and others.

While I do not expect every paper to cover all these benchmarks, evaluating the proposed method exclusively on NAVSIM v1 makes it difficult to assess its generalizability and robustness across different driving scenarios and evaluation protocols.

This limitation is especially important because the paper proposes a modeling principle that is intended to address a relatively general issue in autonomous driving world models, rather than a benchmark-specific optimization.

Consequently, I believe the current evaluation is insufficient to fully substantiate the broader claims of the paper.

W2. Insufficient evidence that the improvements originate from the proposed agent-centric modeling

Predicting future BEV representations is not a new research direction. Therefore, the key contribution that distinguishes this paper from previous works should be the explicit treatment of surrounding agents as independent entities rather than background scene content.

Intuitively, this formulation should lead to better modeling of surrounding agents' future behaviors and trajectories, which should subsequently improve ego-vehicle planning.

However, the paper does not provide sufficiently direct evidence supporting this causal relationship.

First, the evaluation of surrounding-agent prediction is insufficient.

Although Table 4 reports future BEV semantic segmentation performance, the evaluation remains rather ambiguous.

In particular, the definition of the eight semantic categories is unclear, and the reported mIoU of 31.8% is not particularly convincing without further analysis.

For reference, SeerDrive reports an mIoU of 38.87%. I fully acknowledge that the experimental settings differ and that these numbers should not be directly compared. Nevertheless, the reported 31.8% does not, by itself, establish that the proposed approach achieves particularly strong future scene prediction.

More importantly, without class-wise results, it is difficult to determine whether the relatively low overall mIoU is primarily caused by challenging or underrepresented categories, such as pedestrians, or whether prediction quality is consistently limited across categories.

Given the proposed formulation, I believe a more informative evaluation would be to directly assess the future trajectories of the 15 selected surrounding agents against their ground-truth trajectories, rather than relying primarily on an aggregated semantic segmentation mIoU.

Such an evaluation would more directly examine whether the proposed method improves the modeling of other agents, which appears to be the central motivation of the paper.

Second, the claims regarding spatial completeness and ego-agent consistency are not sufficiently validated.

For a method that explicitly emphasizes spatial completeness and ego-agent consistency, reporting only an aggregated future BEV mIoU is insufficient to evaluate these properties.

In particular, I would expect a more fine-grained analysis distinguishing:

  1. Regions already observable in the original ego-centric BEV representation from regions beyond its spatial coverage.
  2. Ego-centric future BEV predictions from agent-centric future BEV predictions.
  3. Static semantic categories, such as roads and map elements, from dynamic categories, such as vehicles.
  4. Prediction consistency between ego-centric and agent-centric viewpoints after aligning their predictions to the same physical region in a common coordinate frame.

The fourth point is particularly important.

As explicitly stated by the authors after Eq. (4) on page 5, the method does not introduce an explicit consistency loss. Instead, consistency is encouraged indirectly through shared parameters and consistent ground-truth future labels.

While this design is reasonable, it also makes direct empirical verification of cross-view consistency particularly important.

An improvement in overall future BEV mIoU does not necessarily demonstrate that predictions from different vehicle viewpoints are more consistent with one another.

Therefore, the current results do not sufficiently establish whether the proposed approach achieves the claimed improvements in spatial completeness and ego-agent consistency.

Third, the paper lacks qualitative comparisons that could help explain the source of the planning improvements.

I did not find a qualitative comparison illustrating how the proposed agent-centric modeling changes the resulting planning behavior.

For example, it would be valuable to compare representative scenarios in which surrounding agents are treated as ordinary scene content against scenarios using the proposed agent-centric formulation.

Such visualizations could help demonstrate whether explicitly modeling surrounding agents leads to more reasonable ego-vehicle trajectories, especially in situations involving interactions with other vehicles.

Without these analyses, it remains difficult to determine whether the observed planning improvements genuinely originate from the proposed agent-centric modeling principle or primarily from the additional supervision introduced by future BEV prediction.

This concern becomes even more consequential given the limited benchmark coverage discussed in W1. When cross-benchmark validation is insufficient, detailed analyses of the proposed mechanism become particularly important.

Overall, while I find the motivation compelling, I believe the current experiments do not yet provide sufficiently convincing evidence that the proposed agent-centric formulation, rather than auxiliary future BEV supervision in general, is responsible for the observed improvements.

My initial rating for this paper is 1.

To be honest, I feel that this score is somewhat lower than what the paper deserves in terms of its research motivation. I genuinely appreciate the motivation behind this work, as it addresses a real and challenging problem in autonomous driving world models.

In particular, I believe the distinction between modeling surrounding agents as independent decision-making entities and treating them as ordinary scene content is both meaningful and worth investigating.

However, the weaknesses in the current experimental validation are also substantial. Most importantly, the paper does not yet provide sufficiently direct evidence that its reported improvements originate from the proposed agent-centric modeling principle.

If a finer-grained rating scale were available, I would place this paper somewhere between 1 and 2, perhaps around 1.5, but slightly closer to 2 than my current discrete rating suggests.

I am open to revising my score depending on the authors' responses, particularly if they can provide additional evidence addressing the concerns regarding evaluation coverage, agent trajectory modeling, and cross-view consistency.

Questions for authors

Q1. Could the authors provide additional evaluation using the EPDMS metric?

I would encourage the authors to report EPDMS results by applying the NAVSIM v2 evaluation metric to the NAVSIM test set, as has been done in some recent works.

To my understanding, this evaluation does not require retraining the model and may therefore be feasible within the rebuttal period.

Given the limited benchmark coverage, this additional evaluation would provide useful evidence regarding the robustness of the reported planning improvements.

Q2. Could the authors provide more direct evidence supporting the benefits of agent-centric modeling?

In particular, I would appreciate clarification or additional results regarding the following:

  1. Future agent trajectory prediction: Can the authors directly evaluate the predicted future trajectories of the 15 selected surrounding agents against ground truth? This would help establish whether the proposed representation genuinely improves the modeling of surrounding agents' future behaviors.

  2. Spatial completeness and cross-view consistency: Can the authors provide more fine-grained evaluations of future BEV prediction, particularly for previously unobserved regions and consistency between spatially aligned ego-centric and agent-centric predictions?

  3. Qualitative analysis: Can the authors provide representative visual comparisons between the proposed method and its counterpart without agent-centric modeling, illustrating how the resulting planning trajectories differ in interactive driving scenarios?

These analyses would be particularly valuable in determining whether the improvements arise from the proposed modeling principle or from future BEV auxiliary supervision more generally.

Additional comments and suggestions (non-critical)

The following points are primarily intended for discussion and future improvement. They are not central to my assessment, and I do not expect the authors to address all of them during the rebuttal.

1. Naming of the method

I am curious about the choice of "DREAM" as the method name.

Given the existence of several video-generation-related works using similar naming conventions, particularly DreamAD, the name may lead readers to associate this work with video-generation-based autonomous driving approaches.

However, the technical formulation of this paper appears substantially different from those approaches.

While this is a minor concern, a brief clarification of the naming choice might help avoid unnecessary confusion.

2. Potential evaluation on planning-oriented benchmarks

The core contribution of the paper appears to be closely related to BEV-based modeling and representation learning.

Although the current implementation relies on an image encoder, the underlying idea does not seem fundamentally restricted to a particular end-to-end perception-planning architecture.

From this perspective, a planning-oriented benchmark such as nuPlan could potentially provide a more direct evaluation of the proposed modeling principle.

For example, demonstrating consistent improvements on both NAVSIM and nuPlan, using two relatively lightweight baseline architectures, would substantially strengthen the argument that the proposed approach offers general benefits for planning.

I recognize that such experiments may be difficult to conduct within the rebuttal period. Therefore, I consider this a suggestion for future work rather than a requirement for the current submission.

3. Claims regarding the lightweight backbone and potential scalability

The paper repeatedly emphasizes its use of a lightweight ResNet-34 backbone.

However, based on observations from this benchmark, larger model capacity does not always translate into better performance. Under certain architectures and experimental settings, increasing the model size may even degrade planning performance, although this behavior is not universal and likely depends on the architecture and training configuration.

Therefore, if the authors intend to suggest that the proposed approach has substantial potential for scaling to larger architectures, I believe this claim would be better supported by experiments involving backbones of different capacities, such as ResNet-50 or ViT-Base.

On the other hand, if the intention is simply to emphasize computational efficiency, reporting inference latency or throughput would be more informative than repeatedly highlighting the use of a lightweight backbone.

A performance-versus-inference-speed comparison with recent methods on the same benchmark could provide a clearer picture of the practical efficiency of the proposed approach.

Reviewer_trthOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoDreamer for end-to-end autonomous driving. It extends ego-centered future scene prediction to surrounding agents through coordinate-transformed BEV features and a shared action-conditioned predictor. Ego-Agent Spatial Tokens encode spatial relationships across views. The future prediction branches provide auxiliary supervision during training and are not needed at inference. On NAVSIM v1, the method achieves 92.0 PDMS with a ResNet-34 backbone. The paper also examines individual components, the number of agents, and future semantic prediction.

Critical strengths
  1. Predicting future scenes from both ego and agent perspectives is an interesting idea. It gives the planner richer supervision than ego-only prediction.
  2. The method achieves good planning results. The extra prediction branches are only used during training, which is a practical advantage.
Critical weaknesses
  1. The paper emphasizes cross-view consistency and completion outside the original BEV region, but neither is evaluated separately.
  2. The paper emphasizes cross-view consistency and scene completion, but does not directly evaluate either.
Questions for authors
  1. Do you have results showing better cross-view consistency or prediction outside the original BEV region?
  2. Are ground-truth future trajectories used when evaluating the predictions in Table 4?
Reviewer_6JWAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents CoDreamer, an end-to-end autonomous driving framework that jointly learns ego- and agent-centric future scene prediction through a shared action-conditioned world model. Agent-centric BEV features are constructed by transforming the ego BEV, and full-window semantic reconstruction supervises both supported and missing regions. Ego-Agent Spatial Tokens encode entity types and geometry across coordinate frames. The world-model branches provide training-only auxiliary supervision and are omitted during inference. CoDreamer achieves 92.0 PDMS on NAVSIM v1, compared with 91.1 for its ego-only world-model baseline. The approach is interesting, but the adequacy of the predictive inputs, the interpretation of collective interaction modeling, and the attribution of the gains remain insufficiently established.

Critical strengths

The paper explores a useful extension of ego-centric world modeling by treating neighboring vehicles as explicit prediction subjects. Sharing the predictor and decoder across participant-centered views provides a coherent way to introduce local future supervision, while EAST supplies explicit spatial context for these predictions. The training-only design avoids online future-scene prediction. The reported NAVSIM v1 results show improved planning performance over the ego-only baseline, and the comparisons across agent budgets and against copy-current prediction provide useful initial evidence for the approach.

Critical weaknesses

The predictive formulation leaves important uncertainties. Single-frame observations may be insufficient to identify agent motion states, and transforming a bounded ego BEV cannot recover context outside its support that may be essential to neighboring agents. Each branch is conditioned only on its own subject’s trajectory, leaving the relationship between participant-centered prediction and joint interaction modeling unclear. The evidence does not adequately substantiate the central mechanism claims. Cross-perspective consistency is encouraged indirectly but never measured. Agent-WM alone improves PDMS by only 0.1 points, whereas EAST adds a further 0.8 points, and the missing EAST-only and auxiliary-motion ablations prevent clear attribution. These gaps concern the core formulation and its validation, rather than presentation alone.

Questions for authors
  1. Is an ego-only + EAST result available? Are motion supervision and the other auxiliary objectives identical across the configurations in Table 2?

  2. Does Table 4 evaluate ego views, agent views, or both? How are subjects and spatial regions aggregated?

  3. Can existing predictions demonstrate improvements specifically outside the original BEV support and better agreement between aligned participant views?

  4. Are any existing evaluations available on an independent dataset or a clearly separated domain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.