acceptodds
Under review as a conference paper at ICLR 2027

SparseWAM: Driving World-Action Models are Worth 16 Register Tokens

Abstract

World Action Models (WAMs) transfer scene-dynamics priors to planning by jointly forecasting future scenes and ego trajectories. However, WAMs built on video generators must generate dense future videos at inference, incurring prohibitive latency, while efficient variants discard the world branch at deployment and thus forgo test-time imagination. Even when dense video latents are replaced with patch-level DINO or JEPA features, these representations still involve long token sequences and encode decision-irrelevant information. We therefore propose SparseWAM, which forecasts in a sparse register space of only 16 tokens per frame, compressed from frozen DINOv2 features and shaped by joint planning-and-scoring pretraining to concentrate decision-relevant information. Within this space, SparseWAM employs a modality-decoupled Mixture-of-Transformers with a causal attention mask that lets action tokens condition on imagined future registers, retaining explicit imagine-then-act reasoning at deployment. Its compactness further enables low-cost multimodal sampling. As a result, our SparseWAM achieves 92.1 PDMS on NAVSIM v1, 89.8 EPDMS on NAVSIM v2, and 40.1 on NavHard at only 260 ms latency, roughly 1/3 that of DriveVA, demonstrating the efficiency of the sparse register space. Code will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, World-Action Model, Register Latent, Mixture-of-Transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iEQrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes SparseWAM, which forecasts in a sparse register space of only 16 tokens per frame, compressed from frozen DINOv2 features and shaped by joint planning-and-scoring pretraining to concentrate decision-relevant information. Within this space, SparseWAM employs a modality-decoupled Mixture-of-Transformers with a causal attention mask that lets action tokens condition on imagined future registers, retaining explicit imagine-then-act reasoning at deployment. Its compactness further enables low-cost multimodal sampling. As a result, SparseWAM achieves 92.1 PDMS on NAVSIM v1, 89.8 EPDMS on NAVSIM v2, and 40.1 on NavHard at only 260 ms latency, roughly 1/3 that of DriveVA, demonstrating the efficiency of the sparse register space.

Critical strengths

The paper is well motivated and addresses a problem of practical significance. The overall technical framework is clear and well designed, with each component closely aligned with the research motivation. The manuscript is well organized and easy to follow.

Critical weaknesses
  1. The paper does not directly demonstrate that the action branch benefits from meaningful future predictions. Replacing predicted registers with random or ground-truth registers during inference would provide stronger evidence.

  2. It is unclear whether 16 registers are sufficient for complex traffic scenarios. Ablations with different numbers of registers are recommended.

  3. The individual contributions of planning and scoring supervision are not evaluated.

  4. How does the action-only baseline compare with SparseWAM under the same 64-sample planning and scoring settings?

  5. Since future registers are predicted independently of generated actions, how does the model capture the effects of different ego actions on future scenes?

Questions for authors

Please refer to the weaknesses.

  1. Please replacing predicted registers with random or ground-truth registers during inference and provide stronger evidence to demonstrate that the action branch benefits from meaningful future predictions.
  2. Ablations with different numbers of registers are recommended.
  3. The individual contributions of planning and scoring supervision are recommended.
  4. How does the action-only baseline compare with SparseWAM under the same 64-sample planning and scoring settings?
  5. How does the model capture the effects of different ego actions on future scenes?
5 comments
AuthorsOpenReview ↗

(Part1/5) Response to Reviewer iEQr

We thank Reviewer iEQr for the encouraging assessment of our paper's motivation, technical framework design, and clarity. Below we provide the complete inference-time register replacement experiments, token-budget ablations across scene complexity tiers, Stage-1 pretraining and 64-sample ablations, and a clear explanation of how our causal MoT relates ego intent and future prediction.

1. [W1 & Q1] Replacing Predicted Future Registers at Inference (Shuffled, Random, Pred, and GT Oracle)

To directly verify that the action expert benefits from meaningful future predictions FF rather than ignoring them, we evaluate four inference-time conditioning regimes using the exact same trained Stage-2 MoT model, shared historical register encoding HH, and ego state qtq_t:

  • Controlled protocol: For each candidate rollout, all four conditions share the identical noise pair (εF,εA)(\varepsilon_F, \varepsilon_A) and integrate only the action velocity field vθAv_\theta^A over the 10 Euler steps, reconstructing the future segment at every step σi\sigma_i as Fσ=(1−σi)x0+σiεFF_\sigma = (1 - \sigma_i) x_0 + \sigma_i \varepsilon_F, where x0x_0 is set as:
    1. Pred (Default SparseWAM): Standard joint denoising of FF and AA.
    2. GT (Oracle future frames): Clean registers Ereg(It+2,It+4)E_{\mathrm{reg}}(I_{t+2}, I_{t+4}) extracted from ground-truth future frames (shared across all candidates of a scene).
    3. Random: Pure Gaussian noise x0∼N(0,I)x_0 \sim \mathcal{N}(0, I) of identical shape (sampled once per scene and shared across candidates).
    4. Shuffled (Mismatched scene): Real DINOv2 registers from another scene drawn from a rolling donor bank of up to 16 batches.

Results without Scorer (Single Trajectory, M=1M=1):

Future Register Condition (x0x_0)PDMS↑NC↑DAC↑EP↑TTC↑
Shuffled (Mismatched scene)80.9693.895.176.786.1
Random (N(0,I)\mathcal{N}(0, I))86.8096.797.582.290.7
Pred (SparseWAM default)89.1898.697.183.395.1
GT (Oracle future frames)90.5799.297.484.396.8

Results with Scorer (Multi-Modal Sampling):

Future Register Condition (x0x_0)PDMS↑NC↑DAC↑EP↑TTC↑
Shuffled (Mismatched scene)87.5197.297.382.892.0
Random (N(0,I)\mathcal{N}(0, I))91.5798.799.086.195.6
Pred (SparseWAM default)92.5999.399.086.697.6
GT (Oracle future frames)93.1699.599.186.898.3

Conclusions:

  • Strong causal reliance on future registers (sharp drop under negative replacements): Replacing predicted registers with Random noise degrades PDMS by -2.38 at M=1M=1 (NC 98.6→96.7, TTC 95.1→90.7) and -1.03 with the scorer (92.6→91.57, TTC 97.6→95.6). Injecting mismatched Shuffled scene registers causes an even sharper collapse (-8.22 at M=1M=1 and -5.09 with the scorer, 92.6→87.51), proving that the action expert actively reads and relies on the semantic content of FF.
  • Substantially stronger planning performance with ground-truth futures (positive Oracle gain): Supplying GT Oracle future registers yields a further significant improvement over Pred—raising PDMS from 89.18 to 90.57 (+1.39 PDMS; NC 99.2, EP 84.3, TTC 96.8) without the scorer, and from 92.59 to 93.16 (+0.57 PDMS; NC 99.5, TTC 98.3) with the scorer. This confirms that more accurate future scene representations directly translate into safer and more efficient trajectory planning, establishing a clear positive causal link from future forecasting to action generation.

AuthorsOpenReview ↗

(Part2/5) Response to Reviewer iEQr

2. [W2 & Q2] Sufficiency of 16 Registers on Complex Traffic Scenarios (Nr∈{8,16,32,64}N_r \in \{8, 16, 32, 64\})

Under the exact same MoT architecture and training schedule, we systematically evaluate Nr∈{8,16,32,64}N_r \in \{8, 16, 32, 64\} across both the overall navtest benchmark (with 1× NVIDIA H20 latency, batch size 1) and scene complexity tiers. For the complexity stratification, all 12,146 aligned navtest scenes are partitioned by the score of a representative baseline, DiffusionDrive (Hard: < 0.7, 1,046 scenes; Medium: 0.7–0.9, 2,490 scenes; Easy: ≥ 0.9, 8,610 scenes), and evaluated with the frozen Stage-1 scorer (M=64M=64):

(1) Overall navtest planning performance and inference latency:

Register Tokens (NrN_r)M=64M=64 PDMS↑NC↑DAC↑EP↑TTC↑Latency (M=64M=64, ms)↓
Nr=8N_r = 892.2999.1498.7286.3997.20156.44
Nr=16N_r = 16 (Ours)92.699.398.886.697.5164.32
Nr=32N_r = 3292.0999.1298.4986.4696.95172.49
Nr=64N_r = 6492.2799.0898.6086.8096.94192.10

(2) Stratified performance across scene complexity tiers (Hard / Medium / Easy):

Register Tokens (NrN_r)Hard (1,046 scenes)Medium (2,490 scenes)Easy (8,610 scenes)
Nr=8N_r = 876.5187.6195.56
Nr=16N_r = 16 (Ours)77.9687.6795.73
Nr=32N_r = 3275.6987.2895.47
Nr=64N_r = 6476.5887.8395.46

Why 16 registers suffice even in complex traffic scenarios:

  • Best performance on Hard scenarios: On the Hard subset capturing complex multi-agent interactions and challenging decisions, Nr=16N_r = 16 achieves the highest score (77.96)—outperforming Nr=8N_r=8 (+1.45), Nr=32N_r=32 (+2.27), and Nr=64N_r=64 (+1.38)—while also leading on Easy (95.73) and overall navtest (92.6).
  • Capacity bottleneck vs. target inflation: Reducing to Nr=8N_r = 8 creates a representational bottleneck in dense scenes (overall PDMS drops to 92.29 and Hard drops to 76.51). Conversely, increasing to Nr=32N_r = 32 or 6464 raises latency to 172.49 / 192.10 ms, absorbs redundant background details, and inflates the dimensionality and denoising variance of the Stage-2 rectified-flow future target, yielding no advantage even on complex scenarios.

AuthorsOpenReview ↗

(Part3/5) Response to Reviewer iEQr

3. [W3 & Q3] Individual Contributions of Planning and Scoring Supervision

Keeping the Stage-2 MoT architecture, Nr=16N_r = 16, and training schedule identical, we ablate the Stage-1 pretraining objectives under the same multi-modal evaluation (M=64M=64 sampling):

Stage-1 Pretraining ObjectiveM=64M=64 PDMS↑NC↑DAC↑EP↑TTC↑Comfort
Planning-only (λsc=0\lambda_{\mathrm{sc}} = 0, L1L_1 imitation only)92.099.098.686.296.8100.0
Scoring-only (BCE sub-metric distillation only)92.199.398.586.197.3100.0
Joint Planning + Scoring (Ours, Eq. 4)92.699.398.886.697.5100.0

Complementarity of the two objectives:

  • Planning-only (λsc=0\lambda_{\mathrm{sc}} = 0): L1L_1 imitation captures primary trajectory intent (EP 86.2), yet under-represents off-trajectory collision risks and safety margins, yielding lower NC (99.0 vs. 99.3) and TTC (96.8 vs. 97.5) and 92.0 PDMS overall.
  • Scoring-only: Score distillation preserves safety-critical boundaries (NC 99.3, TTC 97.3), but lacking direct waypoint-level geometric supervision degrades EP (86.1 vs. 86.6) and DAC (98.5 vs. 98.8), reaching 92.1 PDMS.
  • Joint Planning + Scoring (Ours): Combines both complementary signals to achieve the highest performance across every sub-metric (92.6 PDMS).

Reviewer_fV8nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SparseWAM, an efficient World Action Model for autonomous driving planning. To address the computational overhead of future prediction in dense visual latent spaces, the authors introduce a sparse register representation consisting of only 16 tokens per frame. The register encoder is initialized from a frozen DINOv2 backbone and pretrained with joint planning and scoring objectives to encourage the preservation of decision-relevant scene information. Based on this representation,SparseWAM employs a MoT architecture to jointly predict future scene registers and action trajectories, using causal attention to preserve test-time imagination. It achieves 92.1 PDMS on NAVSIM v1 with only 260.4 ms inference latency.

Critical strengths

High inference efficiency. By predicting only 16 register tokens per frame instead of full video latents,SpraseWAM reduces the computational cost of future predictions.

Reasonable task-oriented representation learning. The joint planning and scoring pretraining objectives encourage the register encoder to preserve decision-relevant scene information rather than unnecessary visual details.

Critical weaknesses

Insufficient validation of the sparse representation. The proposed representation is a key contribution, but its advantages are not sufficiently demonstrated. Comparisons with existing token compression methods under the same token budget and ablations on the planning and scoring objectives are needed to justify the design.

Lack of token budget ablations. The paper fixes the number of register tokens to 16 without exploring alternative configurations. An ablation with different numbers of registers would help clarify the trade-off between representation capacity, planning performance, and computational efficiency.

Questions for authors

Comparison with existing token compression methods. How does the proposed register-based representation compare with existing methods under the same token budget and training objectives?

Contribution of pretraining objectives. Could the authors provide ablations comparing planning-only, scoring-only, and joint planning-and-scoring pretraining to clarify the contribution of each objective?

Trade-off between token budget and performance. Why was the number of register tokens fixed at 16? Could the authors evaluate different token budgets (e.g., 8, 16, 32, and 64) to investigate the trade-off between planning performance and computational efficiency?

Reviewer_TFErOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SparseWAM, a world-action model that jointly generates future scene representations and ego trajectories. A DINOv2 encoder is pretrained to capture scence information with planning and scoring tasks. The world-action model predicts register representation per future frame together with trajectory waypoints using a modality-specific MoT. The causal attention mask mechanism ensures that information flows only from the predicted future frame representation to the action representation. During inference, the lightweight denoiser generates 64 samples, significantly improving planning performance at the cost of a relatively small latency.

Critical strengths
  • An efficient predictive planning approach. The model retains a future-prediction branch at deployment while using substantially fewer parameters than the video-based WAMs in Table 5. The 122 ms single-sample and 260.4 ms 64-sample measurements are encouraging.

  • Informative representation analysis. The counterfactual sceneediting attempt to measure the sensitivity of the learned representations to information that influences planning decisions.

Critical weaknesses
  • Ablation studies on the deployed sampling protocol. Table 4 reports 87.9 PDMS for action-only prediction and 89.1 for the full single-sample model. These are useful positive results, but the headline 92.1 additionally uses 64 samples and scorer selection, which adds 3.0 points relative to the full single-sample model. The paper does not report the corresponding 64 sample action-only or isolated-mask results with the same encoder, scorer, and candidate budget. Consequently, it remains unclear how much the deployed system gains from predicted futures over an equally equipped non-imagining planner. A matched comparison with inference costs reported would directly address the central claim. The 0.4 point mask advantage in Table 6 would also benefit from an uncertainty estimate, such as repeated runs or a paired bootstrap over evaluation logs.

  • The counterfactual sensitivity metric does not guarantee true decision alignment. Nighttime and heavy rain are treated as decision-irrelevant background changes, although visibility and road conditions can affect safe driving even if object locations are preserved. This raises concerns about the validity of FBSR as a measure of alignment.

Questions for authors
  • How do the action-only and isolated-mask variants perform with 64 candidates, using the same encoder and scorer as the full model? Please report both planning performance and inference latency. Can you also quantify the uncertainty of the 0.4-point improvement from the causal mask in Table 6?
  • How does SparseWAM compare with SparseDriveV2 and DiffusionDriveV2 in inference latency under comparable hardware and evaluation settings? Please clarify their candidate-generation and scoring protocols where available, so that readers can assess the accuracy–latency trade-off.
  • What justifies treating nighttime and heavy rain as decision-irrelevant changes when they may affect visibility and safe driving behavior? Can you validate the FBSR interpretation using edits that preserve the appropriate driving decision, or provide evidence that the desired trajectory should remain unchanged under the current edits?
3 comments
AuthorsOpenReview ↗

(Part2-1/3) Response to Reviewer TFEr

2. [Q2] Comparable Latency & Protocol Breakdown vs. SparseDriveV2 and DiffusionDriveV2

We benchmark inference latency on the identical hardware setup (1× NVIDIA H20 GPU, batch size 1), contrasting traditional end-to-end (E2E) planners—DiffusionDriveV2 (Zou et al., 2025) and SparseDriveV2 (Sun et al., 2026)—with World Action Models (WAMs) including DriveVA (Liu et al., 2026) and SparseWAM, with full NAVSIM v1 sub-metrics reported below:

MethodParadigmSensorsLatency (ms)NC↑DAC↑EP↑TTC↑C↑PDMS↑
DiffusionDriveV2 (Zou et al., 2025)Traditional E2E3×C + L25.3198.397.987.594.899.991.2
SparseDriveV2 (Sun et al., 2026)Traditional E2E3×C23.2798.598.488.695.099.992.0
DriveVA (Liu et al., 2026)WAM1×C829.099.297.583.598.7100.090.9
SparseWAM (Ours)WAM1×C164.399.398.886.697.5100.092.6

Candidate-Generation & Scoring Protocols:

  • DiffusionDriveV2 (Zou et al., 2025; arXiv:2512.07745): Generates 20 candidates from 20 K-means anchors via 2-step truncated DDIM denoising (fine-tuned with Intra-Anchor and Inter-Anchor Truncated GRPO), followed by a two-stage coarse-to-fine mode selector interacting with BEV, agent, and map features to pick the optimal trajectory.
  • SparseDriveV2 (Sun et al., 2026; arXiv:2603.29163): Adopts a purely scoring-based paradigm without dynamic trajectory generation, pruning a factorized static vocabulary (1,024 path anchors × 256 velocity anchors = 262,144 trajectories) via two coarse scoring stages down to 400 composed candidates (20 paths × 20 velocities), followed by spatiotemporal re-conditioning to predict rule-distilled PDM sub-metrics and aggregate them into the final score.
  • SparseWAM (Ours): Encodes single-front-view history once into sparse registers HH, runs 64 parallel joint rectified-flow rollouts (10-step Euler solver generating 16 future registers per keyframe and 8 trajectory waypoints), and selects the highest-scoring trajectory via the frozen Stage-1 scoring head operating on current-frame registers QtQ_t.
AuthorsOpenReview ↗

(Part2-2/3) Response to Reviewer TFEr

Why SparseWAM (164.3 ms) over ~23–25 ms E2E planners? (Value & Accuracy–Latency Trade-off):

We directly address the central question behind this comparison: when comparing aggregate PDMS (92.6 vs. 92.0 / 91.2), is the higher inference latency justified? Examining sub-metric safety, scalability, and paradigm differences highlights three clear points:

  1. Substantially lower safety-critical failure rates (beyond a +0.6 PDMS headline difference): As shown in the NAVSIM v1 sub-metrics above, SparseDriveV2 (92.0) and DiffusionDriveV2 (91.2) owe much of their aggregate PDMS to higher Ego Progress (EP: 88.6 / 87.5 vs. 86.6), while trailing across safety and compliance sub-metrics. Using only a single front camera (1×C), SparseWAM reduces the collision rate (100 - NC) by >50%—from 1.5% (SparseDriveV2) and 1.7% (DiffusionDriveV2) down to 0.7%—and halves TTC violations (100 - TTC) from 5.0% / 5.2% down to 2.5%, while achieving the best DAC (98.8) and Comfort (100.0). This directly demonstrates the value of explicitly forecasting future scene dynamics for proactive collision avoidance.
  2. Scalability without LiDAR, 3D perception supervision, or RL post-training: DiffusionDriveV2 relies on 3×C+LiDAR, 3D bounding-box and BEV-map perception supervision (inherited from DiffusionDrive), and simulator-based GRPO reinforcement learning; SparseDriveV2 relies on 3×C multi-view inputs and a 262k-trajectory static combinatorial vocabulary—and neither models future scene evolution. In contrast, SparseWAM uses only a single front camera (1×C), with no LiDAR, no 3D detection/map box supervision, and no RL post-training, surpassing both baselines through planning/scoring supervision combined with unlabeled future-frame dynamics priors, while remaining orthogonal to multi-view extensions and RL post-training.
  3. Paradigm difference and fastest runtime within the WAM family: Traditional E2E planners (DiffusionDriveV2, SparseDriveV2) perform static vocabulary scoring or local anchor denoising without forecasting future scenes, whereas WAMs explicitly model future scene dynamics alongside planning, which currently makes WAMs substantially slower than traditional E2E planners as a paradigm. Within the WAM family, however, SparseWAM is already the fastest approach: compared to the dense video-based WAM DriveVA (829.0 ms @ 90.9 PDMS), SparseWAM reduces latency by 5.0× (164.3 ms multi-modal; 6.8× faster at 122.0 ms single-mode) while improving PDMS by +1.7; even compared to SimWAM (400.4 ms @ 90.3 PDMS), which discards its future branch at deployment, SparseWAM is 2.4× faster (3.3× faster single-mode) and +2.3 PDMS higher. In future work, we will further optimize inference speed—via backbone lightweighting, few-step rectified-flow distillation, and kernel fusion—to narrow the remaining latency gap with traditional E2E planners.

(References: Zou et al., "DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving," arXiv:2512.07745, 2025; Sun et al., "SparseDriveV2: Scoring is All You Need for End-to-End Autonomous Driving," arXiv:2603.29163, 2026.)


AuthorsOpenReview ↗

(Part3/3) Response to Reviewer TFEr

3. [W2 & Q3] Validating FBSR with Strictly Decision-Irrelevant Daytime Background Edits

We fully agree with the reviewer's physical judgment: in real closed-loop driving, heavy rain and nighttime conditions alter road-surface friction and visibility, which can legitimately require more conservative driving speeds.

To rigorously evaluate FBSR under strictly invariant dry road friction, daytime visibility, and traffic semantics, we constructed a 400-pair daytime style-transfer counterfactual benchmark on the exact same 400 car-following navtest front-view frames:

  • Decision-preserving protection mask: Using a Cityscapes-trained SegFormer, we segment all driving-relevant regions—roads, sidewalks, traffic lights, pedestrians, cyclists, and all vehicles—together with the 2D projections of 3D ground-truth dynamic-object bounding boxes, and dilate this protected mask outward by 16 pixels.
  • Pure background repainting: The remaining region (sky, buildings, roadside vegetation, and billboards), after removing fragments smaller than 2,000 pixels, forms a single contiguous editing mask. We apply FLUX.2 4-step image inpainting under preserved daytime illumination and camera geometry to repaint this background into a continuous rural/countryside scene, keeping road topology, traffic participants, traffic signals, and the optimal driving decision 100% invariant.
EncoderTokens↓SplanS_{\mathrm{plan}}↑SbgruralS_{\mathrm{bg}}^{\mathrm{rural}} (Daytime Bg)↓New FBSR↑Prev. Night+Rain FBSR
DINOv2 (ViT-S/14)39360.2970.6490.460.43
V-JEPA 2.1 (ViT-B)30240.1890.4320.440.34
Wan2.2 VAE7560.2060.6540.310.19
Sparse Register (Ours)160.7360.3122.361.41

Once physical nighttime and rain factors are excluded, the background sensitivity of our Sparse Register space drops further from 0.520 to 0.312, raising its FBSR to 2.36—which is 5.1×5.1\times–7.6×7.6\times higher than DINOv2 (0.46), V-JEPA 2.1 (0.44), and Wan2.2 VAE (0.31). This provides clean, confounding-free evidence that our sparse registers concentrate sensitivity on decision-critical foreground elements rather than background appearance. We will add this daytime repainting protocol and visualizations of the edited samples to Section 5.4 and Appendix A.1.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.