- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The goal of the paper is learning representations suitable for a downstream driving task evaluated on NAVSIM v1. The method encodes image frames via dinov3, each patch gets position, camera-id and time via a pluecker-ray embedding. 8 cameras over 4 timesteps are encoded this way and then fed into a spatio-temporally factorized 12-stage transformer encoder. On top a DPT head decodes dense depth per view and a cross-attn query given a camera, pixel location, source and target time gives the ego-motion-compensated 3D displacement (D4RT inspired) of the point(s). In a next training stage the heads are discarded, the encoder is frozen and a cross-attention adapter with a number of learnable queries is parameterized (compressing the 4 x 8 x 576 tokens into 128 scene tokens). The scene tokens and the ego state are used for DrivoR style planning. During inference the pipeline recieves 8 camera frames across 4 timesteps, poses of all frames (extrinsics), history ego pose and intrinsics to predict future waypoints.
Critical strengths
- S1. The idea to find suitable representations for downstream planning via enforcing depth and flow information to be present in the encoder is interesting.
- S2. It is good that the paper ablates the effect of dropping the dense depth loss signal or the 3D displacement training signal or to keep both.
- S3. The method is well described and the appendix offers useful implementation details.
Critical weaknesses
-
W1 It remains unclear how much the depth head and the 3D displacement head are adding. An ablation on training with the same budget and the same architecture but without the heads (without geometric supervision) e.g, end-to-end with the planning loss or directly using the dinov3 tokens with ray/camera/time embeddings for planning would clarify this.
-
W2. While IMU/ GPS is generally available for many driving systems it can be inaccurate. The proposed method relies more than other methods on proper pose information. The method should ablate how errors in the pose influence downstream performance.
-
W3. The method compares with papers such as VGGT-Omega and DVGT-2 which are rig-agnostic while their method is specifically fit to nuPlan data and evalated on nuPlan data. It remains unclear whether their approach would generalize to different rig configurations, a zero-shot evaluation on another dataset would answer this question.
-
W4. Table 4 needs a matched input, otherwise it remains unclear whether the encoding scheme is better or it only performs better because it obtained richer inputs.
Minor comments:
- W5. in table 1, reporting 1 digit or two digits for all methods for consistency
- W6. In the appendix the first table is above the appendix heading
- W7. The paper opens with the sentence that driving systems need metric, temporal and queryable representations. It is not clear to me that this is true. Many end-to-end models are neither metric nor have queryable representations yet provide good downstream performance.
Questions for authors
- Q1. How does the proposed method perform with noisy poses or rig perturbations?
- Q2. Does the depth/3D displacement score correlates with the NAVSIM score? This could also be shown using ckpts along the training. This would be a strong indication that better 3D understanding actually aids downstream performance. Currently it looks like the results are mixed at least for using depth and 3D displacement jointly.
Check weaknesses section for more.
2 comments
Q2 — does depth/3D-displacement quality correlate with NAVSIM score? Across six frozen encoders (N1, D0, D1, F1, GeoX, DVGT-2; GeoX and DVGT-2 with oracle scale), Spearman ρ between test AbsRel and PDMS is −0.54 (p = 0.27, n = 6), not significant. The geometry-free DINOv3 control scores above all of them. Along a single training trajectory, each checkpoint would require a separate 20-epoch planner, which was not feasible in the rebuttal window. The cross-encoder evidence, the DINOv3 control and DVGT-2 (better geometry than GeoX, oracle AbsRel 0.138 versus 0.146, yet the lowest PDMS of all frozen memories at 90.22) all indicate that, under frozen transfer, better 3D accuracy does not translate into higher nominal PDMS. Its effect appears in robustness instead (W2). The revised paper states this as a finding rather than claiming that geometry aids planning.
We thank the reviewer for the concrete, actionable requests. We ran all of them.
W1 — what do the depth and 3D-displacement heads add? Same budget/architecture without geometric supervision, or DINOv3 tokens with ray/camera/time embeddings.
(b) DINOv3 tokens with the same ray/camera/time embeddings, same adapter, registers and planner: done. 92.16 versus 91.78 PDMS for N1 (CI [−0.02, +0.80]); EPDMS 90.26 versus 89.83 ([−0.02, +0.90]); also at GeoX-matched input (91.41 versus 91.25). Under frozen transfer the heads do not add measurable PDMS.
(a) Same 12-stage architecture and budget, no geometric supervision, planning loss only. We train the N1 architecture with frozen original DINOv3, with the 12-stage encoder and embeddings randomly initialized and trained end-to-end by the planning loss. Its partner is identical except that the encoder is initialized from N1 (including the DINOv3 weights as lightly adapted during geometric pretraining, frozen here). Both use the same 82.3M trainable parameters, recipe, 20 epochs and global batch 256. Their difference isolates geometric pretraining as an initialization. Training is still running; at matched intermediate epochs (fixed before evaluation, full navtest):
The geometric heads therefore add a substantial amount as an initialization for an encoder trained by the planning loss, mostly through ego progress. The learning rate is not yet annealed and the gap is narrowing, so we will post the final epoch-20 results within the discussion period and not over-state the final size.
Taken together: under frozen transfer the heads add no measurable PDMS, but a geometrically pretrained encoder is a much better starting point than a randomly initialized one of the same architecture and budget.
W2 / Q1 — pose errors. At test time we perturb camera extrinsics and the ego-history poses that enter the Plücker rays (images and ego status unchanged). Noise is identical across models, seeded per scene.
Degradation is graceful at mild noise. Under severe noise the geometric memory loses significantly less than the geometry-free control (difference of drops +0.71, CI [+0.15, +1.29]). The method does rely on calibration, but its geometric pretraining makes the planner less sensitive to calibration error than equally calibration-aware features without it.
W3 — rig generalization; zero-shot on another dataset. We evaluated N1 zero-shot on Waymo. N1 was trained on NAVSIM only, with zero Waymo weight in all its stages. Waymo has a different 5-camera rig, intrinsics, mounting and field of view. Test data: 10 held-out Waymo segments, 100 clips × 5 cameras × 4 frames, with the real Waymo calibration used for the rays. Camera-ID embeddings of the Waymo cameras were never trained, so we map each Waymo camera to the NAVSIM camera with the closest heading (fixed a priori, not tuned).
We report this honestly in the revised limitations: the dense reading of the memory transfers across rigs zero-shot, but the query reading and the camera embedding are rig-specific and need target-rig data. We now compare to VGGT-Omega/DVGT-2 only with this caveat.
W4 — Table 4 needs matched input. Done (General response §2). At GeoX's 4 cameras × 1 frame × 64 registers, GeoAct scores 91.25 versus GeoX's 90.83 (+0.42, CI [−0.01, +0.87]), and DINOv3 at the same input scores 91.41. Revised Table 4 reports both input budgets.
W5 / W6 / W7. Table 1 now uses one decimal throughout. The appendix table is placed after its heading. We removed the opening claim that driving systems need metric/queryable representations, and the introduction now states that many end-to-end planners succeed without them and that our own controls confirm geometric pretraining does not drive PDMS here.