acceptodds
Under review as a conference paper at ICLR 2027

How Many Futures Are Enough? Redundancy-Aware Effective Multimodality for End-to-End Autonomous Driving

Abstract

Multimodal end-to-end planners can generate multiple plausible trajectories for the same driving scene, yet a larger set of candidates does not necessarily represent a larger set of distinct futures. Numerically different trajectories may repeatedly describe closely related driving decisions and induce similar future evolution, resulting in multimodal redundancy. We introduce effective multimodality to distinguish the nominal number of generated trajectories from the effective future structure represented by the candidate set. We propose AMC-Drive (Action-conditioned Multimodal Consolidation for Driving), which uses a frozen action-conditioned predictive model to establish candidate-level relations beyond trajectory geometry. AMC-Drive reshapes redundant candidate structure through effective-multimodality calibration and topology-aware redundancy correction, while a cross-mode guard preserves distinct decision modes. The physical output cardinality remains unchanged throughout this process. On NAVSIM v1, AMC-Drive achieves 93.9 PDMS, establishing a new state of the art among JEPA-based world-model planners. It further reaches 89.2 EPDMS on NAVSIM v2 and obtains a 0.4308 HD-Score on HUGSIM without HUGSIM-specific fine-tuning. AMC further yields consistent gains across proposal-based, diffusion-based, and flow-matching planners, and transfers to robotic manipulation, demonstrating the broader applicability of redundancy-aware effective multimodality across multimodal decision systems.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Multimodal Planning, Effective Multimodality, Trajectory Redundancy, Action-Conditioned Predictive Representations, Latent World Models, Joint-Embedding Predictive Architectures, Multimodal Decision Making

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4BXXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multimodal end-to-end driving planners that output a fixed number of candidate trajectories but may still suffer from mode collapse or redundant overlap within the candidate set. The proposed AMC-Drive keeps the output cardinality K and inference interface unchanged, and adds training-time regularization: an effective-multimodality calibration loss, a topology-aware realization, and a one-sided cross-mode guard. The experiments on Navsimv1, v2 and Hugsim are given to show its effectiveness.

Critical strengths

The paper addresses an interesting and relevant problem, and the proposed direction is plausible. AMC-Drive reaches 93.9 PDMS on NAVSIM v1, and attains 89.2 EPDMS on NAVSIM v2. It can be combined with proposal-, diffusion-, and flow-based planners.

Critical weaknesses
  1. The novelty is moderate. The main novelty is the use of action-conditioned predictive relations as a candidate-level structural regularizer for driving planners.
  2. Direct evidence for improved effective multimodality is missing. the title asks “How Many Futures Are Enough?”, but the paper does not answer how many futures are enough or how K eff relates to planning performance across budgets.
  3. The ablation suggests the main gain may come from the frozen JEPA backbone.
  4. Cross-generator and cross-domain evidence in Table 14 is too thin for the claims.
  5. The inference FPS should be given and compared.
Questions for authors
  1. What exactly the real-MTD teacher is? Whether it is available during training in a realistic deployment pipeline?
  2. The paper does not compare against simple set-diversity regularizers such as DPP, DLow, repulsion, entropy, or coverage losses.
  3. HUGSIM results contain apparent data-consistency problems.
Reviewer_pHRCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper observes that multimodal end-to-end driving planners emit a fixed number of candidate trajectories, but that numerically distinct trajectories need not represent distinct futures: several candidates may encode closely related driving decisions and induce similar future evolution, which the authors term multimodal redundancy. They introduce effective multimodality, a participation-ratio statistic over a candidate affinity matrix that separates the nominal candidate count from the future structure actually represented. The affinity is built not from trajectory geometry but by mapping each candidate to an action sequence, passing it with the scene through a frozen action-conditioned JEPA predictor, and measuring distances in the resulting predictive space. The proposed method, AMC-Drive, adds two auxiliary training terms to the planner’s original loss: one matching the planner’s effective multimodality to that of a detached teacher proposal set, applied in a topology-aware form that concentrates the correction on strongly related candidate pairs; and a one-sided guard that penalizes the planner only when cross-mode affinity exceeds the teacher’s level, permitting mode separation while limiting excessive contraction. Both terms act only during training, leaving the candidate count and the inference interface unchanged.

Critical strengths
  1. The problem is real and precisely stated. That K numerically distinct trajectories need not represent K distinct futures is a genuine and underexamined issue in multimodal planning, and the paper states it cleanly: redundancy is defined over induced future evolution rather than trajectory geometry, which is the right level of abstraction for the question being asked.

  2. The predictive-space affinity is a reasonable instantiation. Measuring candidate relations through a frozen action-conditioned predictor, rather than by waypoint distance, is a sensible way to capture whether two candidates lead to similar outcomes, and the appendix supports it with a paired action-intervention diagnostic showing the checkpoint is action-sensitive at every rollout horizon.

  3. The evidence boundaries are documented unusually honestly. The appendix states that the effective-multimodality statistic is a known participation ratio rather than a new invention, gives an explicit counterexample showing the statistic is not identifiable, records that the HUGSIM scene sets differ between the local run and the comparison protocol, marks the Push-T provenance as unverifiable, and clarifies that the “+ Predictive” row is a backbone-substitution control with all AMC terms disabled.

Critical weaknesses
  1. The paper never establishes what optimizing effective multimodality buys. K and the inference interface are unchanged, so there is no computational saving; no coverage or recall metric is reported, so improved representation of distinct futures is never shown; and no safety or long-tail analysis is given, even though Appendix A.4 itself notes that the aggregate guard permits offsetting cross-mode merges and that pairwise false-merge and safety diagnostics remain necessary. What remains is a small PDMS gain, which does not establish that the proposed quantity is the right thing to optimize.

  2. The title’s question is never answered, and the central quantity is never characterized. Appendix D.1 states that no sweep over K was performed, and Section 4.3 concedes there is no scene-stratified table of K_eff. The paper therefore cannot say how many candidates are enough, or whether AMC lets a smaller budget match the baseline, and K_eff appears nowhere as a measured distribution or per-scene summary.

  3. The ablation attributes most of the gain to a backbone substitution rather than to AMC. In Table 4, the rise from 91.7 to 93.5 comes entirely from replacing the backbone with the frozen predictive encoder, with all AMC terms disabled. The full method reaches 93.9, a net 0.4 over that control, and the intermediate rows are worse than the control (89.2, then 91.6), so the guard largely repairs damage from the preceding terms. The final NAVSIM v1 margin over the strongest comparator is 0.1, with no standard deviations reported.

Questions for authors
  1. Appendix D.1 states that no sweep over K was performed. Could the authors report PDMS for the baseline and for AMC-Drive at K ∈ {8, 16, 32, 64}, together with the measured distribution of K_eff at each budget? Two outcomes would be informative: whether AMC at a reduced K matches the baseline at K=32, which would demonstrate an actual budget saving, and whether the AMC gain grows with K, which the redundancy argument predicts.

  2. Adding the capacity term alone drops PDMS to 89.2, below both the backbone control (93.5) and the original baseline (91.7). Could the authors explain what degrades? If matching K_eff is itself harmful to planning, how should the subsequent topology and guard terms be interpreted other than as damage control?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pHRC: Summary and Appreciation

We sincerely thank the reviewer for recognizing the significance of our research problem and for the constructive and insightful feedback.

In response to your concerns, we have strengthened our work in three main aspects: (1) Practical utility, through candidate-budget comparisons, effective multimodality analysis, and coverage evaluations; (2) Methodological contribution, by disentangling AMC's improvements from the predictive backbone and clarifying the complementary roles of Capacity, Topology, and Guard; and (3) Generalizability, through additional closed-loop evaluations and experiments across different planning architectures and decision-making tasks.

Importantly, our contribution goes beyond improving benchmark scores. We address a distinct and underexplored question in multimodal planning: whether multiple generated trajectories truly represent meaningfully different predictive futures. AMC provides a practical approach to calibrating this structure while preserving the original inference interface. The additional experiments further support the effectiveness and broader relevance of this perspective.

We sincerely appreciate your valuable suggestions, which have helped us strengthen the experimental evidence, clarify our contributions, and better communicate the significance of this work. Thank you again for your time and thoughtful consideration.

Reviewer_ZnXkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work seeks to better understand how models that can generate up to K various trajectory rollouts translate to diversity and meaningful trajectories. In some cases, the trajectories generated can share a high degree of similarity and the added value of generating multiple candidate trajectories diminishes. As part of that, the work introduces a measure of how effective the top K predictions can be and uses that as part of the formulation. Some important feedback provided below.

Critical strengths
  • The effective modality measure is well introduced and provides valuable insights on effective redundancy. The mathematical properties are interesting (i.e it is permutation invariant and differentiable etc).
  • The training only design can be highlighted more, this has good practical value in real applications as it is minimizes overhead
  • The ablations and benchmarks are comprehensive
Critical weaknesses
  • Authors state that a core focus is designing effective multi-modality. Why is Keff not computed across different models to show this?
  • Following the previous point, it would be good to see some visualizations/quantitative comparison of Keff as a metric as it seems central to this work
  • The predictive component (backbone component) from Table 4 brings most of the performance gain). AMC’s improvement is marginal.
Questions for authors
  • Authors reference "MTD teacher" without proper introduction.
  • How meaningful are the generated rollouts? i.e. a high spread does not guarantee trustworthy or valuable predictions. In such case, it would be good to evaluate how "off" the model is relative to GT and other baselines when an error is made.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Official Comment (2/2): AMC's Methodological Contribution, Empirical Evidence, and Generality

AMC's Methodological Contribution, Empirical Evidence, and Generality (2/2)

We sincerely thank all reviewers for their careful evaluations and constructive feedback. We acknowledge that the initial submission did not sufficiently clarify the broader significance of the problem, the distinction between nominal candidate generation and effective multimodality, or the individual contributions of the proposed objectives. Some experimental results and cross-domain evaluations also needed more systematic presentation and explanation. These comments helped us identify important areas for improvement. In the revised response and manuscript, we have clarified the problem formulation, strengthened the experimental evidence, expanded the comparisons, and improved the explanation of the method and its applicability.

Research Motivation and Problem Significance.

Multimodal end-to-end planning has made substantial progress, with proposal-based, diffusion-based, and flow-matching approaches capable of generating multiple plausible trajectories. However, generating more candidates does not necessarily mean representing more distinct futures. Different trajectories may encode closely related driving decisions and induce similar future evolution, causing redundant candidates to occupy limited candidate capacity without providing additional meaningful alternatives. This is not merely a trajectory duplication problem; it concerns whether a multimodal system effectively represents the diversity of possible future outcomes that matters for decision-making. Under a fixed candidate budget, organizing distinct predictive futures is therefore as important as generating them. Moreover, the required future structure varies across scenes, making effective multimodality a scene-dependent property rather than a quantity determined by the nominal candidate count.

Methodological Contribution.

We propose AMC-Drive, which addresses this problem through action-conditioned predictive relations. Instead of relying solely on geometric distances between trajectories, we use a frozen action-conditioned JEPA world model to characterize candidate relationships in predictive space, capturing how different candidate actions shape future evolution under the same scene context. AMC combines scene-conditioned capacity calibration, topology-aware redundancy correction, and a one-sided cross-mode guard to regulate overall effective multimodality, localize structural adjustments, and preserve distinct decision modes. These objectives operate during training, while the deployed planner retains its original candidate count and inference interface, without requiring the world model or reference bank at inference time.

Experimental Validation and General Applicability.

We evaluate AMC across multiple driving benchmarks, planning architectures, and decision-making tasks. AMC-Drive achieves 93.9 PDMS on NAVSIM v1, 89.2 EPDMS on NAVSIM v2, and 43.22 HD-Score on HUGSIM without HUGSIM-specific fine-tuning. Beyond the primary planner, AMC improves proposal-based, diffusion-based, and flow-matching architectures, demonstrating that its benefits are not tied to a particular candidate-generation mechanism.

We further extend AMC beyond autonomous driving to 3D aerial obstacle-avoidance planning and robotic manipulation (Push-T). In the aerial task, AMC improves physical candidate validity from 91.62% to 95.13% and reference-supported branch coverage from 72.21% to 81.12%, while retaining the same candidate budget. These experiments support the broader applicability of predictive-relation-based candidate calibration to multimodal decision systems.

Open-Source Implementation and Reproducibility.

To facilitate independent verification and future research, we provide the implementation, supplementary materials, checkpoints, configurations, and reproduction scripts:

We hope this work encourages further research into how multimodal decision systems can more effectively represent, organize, and preserve genuinely distinct futures under finite candidate budgets.

We sincerely appreciate the reviewers' feedback. It has helped us improve the technical presentation, strengthen the empirical support, and clarify the broader contribution of this work.

AuthorsOpenReview ↗

Official Comment (1/2): Effective Multimodality as a Learning Problem — A New Perspective for ICLR

Effective Multimodality as a Learning Problem — A New Perspective for ICLR (1/2)

We sincerely appreciate the reviewers' thoughtful feedback. In this first general comment, we would like to emphasize the broader research motivation behind AMC-Drive, its relevance to ICLR, and the new perspective we hope to contribute to multimodal decision-making research.

1. From Multimodal Generation to Effective Multimodality.

End-to-end autonomous driving is increasingly moving toward multimodal trajectory generation. Proposal-based, diffusion-based, and flow-matching planners have substantially advanced the ability to generate multiple plausible driving behaviors. However, as generation becomes increasingly powerful, a fundamental question emerges: do more generated trajectories actually represent more distinct future decisions, or do they repeatedly encode similar predictive outcomes?

Geometrically different trajectories may correspond to closely related driving decisions and induce similar future evolution. Under a fixed candidate budget, such redundancy can limit the representation of meaningful alternatives. We believe that understanding the effective structure of generated futures is therefore an important next step beyond simply increasing the number or diversity of trajectories.

2. Relevance to ICLR and Broader Machine Learning.

We view this problem as lying at the intersection of representation learning, generative modeling, predictive world models, and learning for decision-making, all of which are closely connected to the broader research interests of ICLR.

The central question extends beyond autonomous driving: how can a learning system distinguish, organize, and preserve genuinely different future possibilities when multiple valid decisions coexist? This question is relevant to robotics and other multimodal decision-making systems, where geometric diversity alone may not adequately characterize the underlying decision structure.

3. The New Perspective Offered by AMC-Drive.

AMC-Drive approaches this question through action-conditioned predictive relations. Rather than treating trajectory diversity as an end in itself, we characterize relationships among candidate futures in a frozen predictive representation space and use training-time structural calibration to consolidate redundancy while preserving distinct alternatives.

Our contribution is not a new JEPA backbone or simply another way to improve benchmark scores. Instead, we investigate how predictive representations can be used to understand and improve the effective multimodality of a planner's candidate set. Importantly, this does not imply that smaller effective multimodality is always preferable; the objective is to balance redundancy reduction with the preservation of meaningful future alternatives.

4. Our Intended Contribution to the ICLR Community.

We fully recognize the important progress made by existing world models and multimodal planners. Our intention is to complement these developments with a different research perspective.

As multimodal generation continues to advance, we believe it is increasingly important to ask not only how many futures a model can generate, but also how many meaningfully distinct futures it has learned to represent and preserve.

We hope AMC-Drive provides a useful new perspective for the ICLR community and encourages further research into effective multimodality, predictive structure, and the organization of alternatives in general multimodal decision-making systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.