- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes SparseWAM, which forecasts in a sparse register space of only 16 tokens per frame, compressed from frozen DINOv2 features and shaped by joint planning-and-scoring pretraining to concentrate decision-relevant information. Within this space, SparseWAM employs a modality-decoupled Mixture-of-Transformers with a causal attention mask that lets action tokens condition on imagined future registers, retaining explicit imagine-then-act reasoning at deployment. Its compactness further enables low-cost multimodal sampling. As a result, SparseWAM achieves 92.1 PDMS on NAVSIM v1, 89.8 EPDMS on NAVSIM v2, and 40.1 on NavHard at only 260 ms latency, roughly 1/3 that of DriveVA, demonstrating the efficiency of the sparse register space.
Critical strengths
The paper is well motivated and addresses a problem of practical significance. The overall technical framework is clear and well designed, with each component closely aligned with the research motivation. The manuscript is well organized and easy to follow.
Critical weaknesses
-
The paper does not directly demonstrate that the action branch benefits from meaningful future predictions. Replacing predicted registers with random or ground-truth registers during inference would provide stronger evidence.
-
It is unclear whether 16 registers are sufficient for complex traffic scenarios. Ablations with different numbers of registers are recommended.
-
The individual contributions of planning and scoring supervision are not evaluated.
-
How does the action-only baseline compare with SparseWAM under the same 64-sample planning and scoring settings?
-
Since future registers are predicted independently of generated actions, how does the model capture the effects of different ego actions on future scenes?
Questions for authors
Please refer to the weaknesses.
- Please replacing predicted registers with random or ground-truth registers during inference and provide stronger evidence to demonstrate that the action branch benefits from meaningful future predictions.
- Ablations with different numbers of registers are recommended.
- The individual contributions of planning and scoring supervision are recommended.
- How does the action-only baseline compare with SparseWAM under the same 64-sample planning and scoring settings?
- How does the model capture the effects of different ego actions on future scenes?
5 comments
(Part2/5) Response to Reviewer iEQr
2. [W2 & Q2] Sufficiency of 16 Registers on Complex Traffic Scenarios ()
Under the exact same MoT architecture and training schedule, we systematically evaluate across both the overall navtest benchmark (with 1× NVIDIA H20 latency, batch size 1) and scene complexity tiers. For the complexity stratification, all 12,146 aligned navtest scenes are partitioned by the score of a representative baseline, DiffusionDrive (Hard: < 0.7, 1,046 scenes; Medium: 0.7–0.9, 2,490 scenes; Easy: ≥ 0.9, 8,610 scenes), and evaluated with the frozen Stage-1 scorer ():
(1) Overall navtest planning performance and inference latency:
| Register Tokens () | PDMS↑ | NC↑ | DAC↑ | EP↑ | TTC↑ | Latency (, ms)↓ |
|---|---|---|---|---|---|---|
| 92.29 | 99.14 | 98.72 | 86.39 | 97.20 | 156.44 | |
| (Ours) | 92.6 | 99.3 | 98.8 | 86.6 | 97.5 | 164.32 |
| 92.09 | 99.12 | 98.49 | 86.46 | 96.95 | 172.49 | |
| 92.27 | 99.08 | 98.60 | 86.80 | 96.94 | 192.10 |
(2) Stratified performance across scene complexity tiers (Hard / Medium / Easy):
| Register Tokens () | Hard (1,046 scenes) | Medium (2,490 scenes) | Easy (8,610 scenes) |
|---|---|---|---|
| 76.51 | 87.61 | 95.56 | |
| (Ours) | 77.96 | 87.67 | 95.73 |
| 75.69 | 87.28 | 95.47 | |
| 76.58 | 87.83 | 95.46 |
Why 16 registers suffice even in complex traffic scenarios:
- Best performance on
Hardscenarios: On theHardsubset capturing complex multi-agent interactions and challenging decisions, achieves the highest score (77.96)—outperforming (+1.45), (+2.27), and (+1.38)—while also leading onEasy(95.73) and overallnavtest(92.6). - Capacity bottleneck vs. target inflation: Reducing to creates a representational bottleneck in dense scenes (overall PDMS drops to 92.29 and
Harddrops to 76.51). Conversely, increasing to or raises latency to 172.49 / 192.10 ms, absorbs redundant background details, and inflates the dimensionality and denoising variance of the Stage-2 rectified-flow future target, yielding no advantage even on complex scenarios.
(Part3/5) Response to Reviewer iEQr
3. [W3 & Q3] Individual Contributions of Planning and Scoring Supervision
Keeping the Stage-2 MoT architecture, , and training schedule identical, we ablate the Stage-1 pretraining objectives under the same multi-modal evaluation ( sampling):
| Stage-1 Pretraining Objective | PDMS↑ | NC↑ | DAC↑ | EP↑ | TTC↑ | Comfort |
|---|---|---|---|---|---|---|
| Planning-only (, imitation only) | 92.0 | 99.0 | 98.6 | 86.2 | 96.8 | 100.0 |
| Scoring-only (BCE sub-metric distillation only) | 92.1 | 99.3 | 98.5 | 86.1 | 97.3 | 100.0 |
| Joint Planning + Scoring (Ours, Eq. 4) | 92.6 | 99.3 | 98.8 | 86.6 | 97.5 | 100.0 |
Complementarity of the two objectives:
- Planning-only (): imitation captures primary trajectory intent (EP 86.2), yet under-represents off-trajectory collision risks and safety margins, yielding lower NC (99.0 vs. 99.3) and TTC (96.8 vs. 97.5) and 92.0 PDMS overall.
- Scoring-only: Score distillation preserves safety-critical boundaries (NC 99.3, TTC 97.3), but lacking direct waypoint-level geometric supervision degrades EP (86.1 vs. 86.6) and DAC (98.5 vs. 98.8), reaching 92.1 PDMS.
- Joint Planning + Scoring (Ours): Combines both complementary signals to achieve the highest performance across every sub-metric (92.6 PDMS).
(Part1/5) Response to Reviewer iEQr
We thank Reviewer iEQr for the encouraging assessment of our paper's motivation, technical framework design, and clarity. Below we provide the complete inference-time register replacement experiments, token-budget ablations across scene complexity tiers, Stage-1 pretraining and 64-sample ablations, and a clear explanation of how our causal MoT relates ego intent and future prediction.
1. [W1 & Q1] Replacing Predicted Future Registers at Inference (
Shuffled,Random,Pred, andGTOracle)To directly verify that the action expert benefits from meaningful future predictions F rather than ignoring them, we evaluate four inference-time conditioning regimes using the exact same trained Stage-2 MoT model, shared historical register encoding H, and ego state qt:
Pred(Default SparseWAM): Standard joint denoising of F and A.GT(Oracle future frames): Clean registers Ereg(It+2,It+4) extracted from ground-truth future frames (shared across all candidates of a scene).Random: Pure Gaussian noise x0∼N(0,I) of identical shape (sampled once per scene and shared across candidates).Shuffled(Mismatched scene): Real DINOv2 registers from another scene drawn from a rolling donor bank of up to 16 batches.Results without Scorer (Single Trajectory, M=1):
Shuffled(Mismatched scene)Random(N(0,I))Pred(SparseWAM default)GT(Oracle future frames)Results with Scorer (Multi-Modal Sampling):
Shuffled(Mismatched scene)Random(N(0,I))Pred(SparseWAM default)GT(Oracle future frames)Conclusions:
Randomnoise degrades PDMS by -2.38 at M=1 (NC 98.6→96.7, TTC 95.1→90.7) and -1.03 with the scorer (92.6→91.57, TTC 97.6→95.6). Injecting mismatchedShuffledscene registers causes an even sharper collapse (-8.22 at M=1 and -5.09 with the scorer, 92.6→87.51), proving that the action expert actively reads and relies on the semantic content of F.GTOracle future registers yields a further significant improvement overPred—raising PDMS from 89.18 to 90.57 (+1.39 PDMS; NC 99.2, EP 84.3, TTC 96.8) without the scorer, and from 92.59 to 93.16 (+0.57 PDMS; NC 99.5, TTC 98.3) with the scorer. This confirms that more accurate future scene representations directly translate into safer and more efficient trajectory planning, establishing a clear positive causal link from future forecasting to action generation.