acceptodds
Under review as a conference paper at ICLR 2027

GeoAct: Queryable Geometric Representations for Autonomous Driving

Abstract

Driving stacks need scene geometry that is metric at inference time and that downstream modules can read without re-running a reconstruction. We present GeoAct, a calibration-aware encoder that turns multi-camera image histories into one shared geometric memory. The memory is read in two ways: densely, by a DPT-style head predicting per-pixel metric depth and point maps, and sparsely, by a point query asking where the surface seen at a pixel and time lies at a requested time. The encoder has about 100M parameters and is trained from projected LiDAR and monocular pseudo-labels, without distilling a large 3D model. On the NAVSIM/OpenScene test split it reaches 0.1093 AbsRel with no test-time scale alignment, more accurate than GeoX even with a per-scene oracle scale. Behind a frozen encoder, a trained adapter and DrivoR-based planner reach 91.78 PDMS, above a frozen GeoX memory under the same planner budget (90.83). Controlled continuations from one checkpoint reveal a trade-off: dropping dense supervision improves observed-motion prediction and degrades current-frame geometry. Under our frozen-transfer protocol, these two variants show no detectable planning difference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Geometric Representation Learning, Multi-Camera Perception, 3D Reconstruction, Spatiotemporal Correspondence, Depth Estimation, Motion Understanding, Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kGkBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The goal of the paper is learning representations suitable for a downstream driving task evaluated on NAVSIM v1. The method encodes image frames via dinov3, each patch gets position, camera-id and time via a pluecker-ray embedding. 8 cameras over 4 timesteps are encoded this way and then fed into a spatio-temporally factorized 12-stage transformer encoder. On top a DPT head decodes dense depth per view and a cross-attn query given a camera, pixel location, source and target time gives the ego-motion-compensated 3D displacement (D4RT inspired) of the point(s). In a next training stage the heads are discarded, the encoder is frozen and a cross-attention adapter with a number of learnable queries is parameterized (compressing the 4 x 8 x 576 tokens into 128 scene tokens). The scene tokens and the ego state are used for DrivoR style planning. During inference the pipeline recieves 8 camera frames across 4 timesteps, poses of all frames (extrinsics), history ego pose and intrinsics to predict future waypoints.

Critical strengths
  • S1. The idea to find suitable representations for downstream planning via enforcing depth and flow information to be present in the encoder is interesting.
  • S2. It is good that the paper ablates the effect of dropping the dense depth loss signal or the 3D displacement training signal or to keep both.
  • S3. The method is well described and the appendix offers useful implementation details.
Critical weaknesses
  • W1 It remains unclear how much the depth head and the 3D displacement head are adding. An ablation on training with the same budget and the same architecture but without the heads (without geometric supervision) e.g, end-to-end with the planning loss or directly using the dinov3 tokens with ray/camera/time embeddings for planning would clarify this.

  • W2. While IMU/ GPS is generally available for many driving systems it can be inaccurate. The proposed method relies more than other methods on proper pose information. The method should ablate how errors in the pose influence downstream performance.

  • W3. The method compares with papers such as VGGT-Omega and DVGT-2 which are rig-agnostic while their method is specifically fit to nuPlan data and evalated on nuPlan data. It remains unclear whether their approach would generalize to different rig configurations, a zero-shot evaluation on another dataset would answer this question.

  • W4. Table 4 needs a matched input, otherwise it remains unclear whether the encoding scheme is better or it only performs better because it obtained richer inputs.

Minor comments:

  • W5. in table 1, reporting 1 digit or two digits for all methods for consistency
  • W6. In the appendix the first table is above the appendix heading
  • W7. The paper opens with the sentence that driving systems need metric, temporal and queryable representations. It is not clear to me that this is true. Many end-to-end models are neither metric nor have queryable representations yet provide good downstream performance.
Questions for authors
  • Q1. How does the proposed method perform with noisy poses or rig perturbations?
  • Q2. Does the depth/3D displacement score correlates with the NAVSIM score? This could also be shown using ckpts along the training. This would be a strong indication that better 3D understanding actually aids downstream performance. Currently it looks like the results are mixed at least for using depth and 3D displacement jointly.

Check weaknesses section for more.

2 comments

We thank the reviewer for the concrete, actionable requests. We ran all of them.

W1 — what do the depth and 3D-displacement heads add? Same budget/architecture without geometric supervision, or DINOv3 tokens with ray/camera/time embeddings.

  • (b) DINOv3 tokens with the same ray/camera/time embeddings, same adapter, registers and planner: done. 92.16 versus 91.78 PDMS for N1 (CI [−0.02, +0.80]); EPDMS 90.26 versus 89.83 ([−0.02, +0.90]); also at GeoX-matched input (91.41 versus 91.25). Under frozen transfer the heads do not add measurable PDMS.

  • (a) Same 12-stage architecture and budget, no geometric supervision, planning loss only. We train the N1 architecture with frozen original DINOv3, with the 12-stage encoder and embeddings randomly initialized and trained end-to-end by the planning loss. Its partner is identical except that the encoder is initialized from N1 (including the DINOv3 weights as lightly adapted during geometric pretraining, frozen here). Both use the same 82.3M trainable parameters, recipe, 20 epochs and global batch 256. Their difference isolates geometric pretraining as an initialization. Training is still running; at matched intermediate epochs (fixed before evaluation, full navtest):

    Epoch (of 20)Random init, planning loss onlyGeoAct init, planning loss onlyΔ [95% CI]
    485.0588.19+3.15 [+2.54, +3.77]
    887.8190.50+2.69 [+2.10, +3.29]

    The geometric heads therefore add a substantial amount as an initialization for an encoder trained by the planning loss, mostly through ego progress. The learning rate is not yet annealed and the gap is narrowing, so we will post the final epoch-20 results within the discussion period and not over-state the final size.

    Taken together: under frozen transfer the heads add no measurable PDMS, but a geometrically pretrained encoder is a much better starting point than a randomly initialized one of the same architecture and budget.

W2 / Q1 — pose errors. At test time we perturb camera extrinsics and the ego-history poses that enter the Plücker rays (images and ego status unchanged). Noise is identical across models, seeded per scene.

cleanmild (0.5°/2 cm cam, 0.1 m/0.2° ego)severe (1°/5 cm cam, 0.3 m/0.5° ego)
GeoAct N191.7891.54 (−0.24)91.00 (−0.78, CI [0.41, 1.16])
DINOv3 control92.1691.75 (−0.40)90.67 (−1.49, CI [1.01, 2.00])

Degradation is graceful at mild noise. Under severe noise the geometric memory loses significantly less than the geometry-free control (difference of drops +0.71, CI [+0.15, +1.29]). The method does rely on calibration, but its geometric pretraining makes the planner less sensitive to calibration error than equally calibration-aware features without it.

W3 — rig generalization; zero-shot on another dataset. We evaluated N1 zero-shot on Waymo. N1 was trained on NAVSIM only, with zero Waymo weight in all its stages. Waymo has a different 5-camera rig, intrinsics, mounting and field of view. Test data: 10 held-out Waymo segments, 100 clips × 5 cameras × 4 frames, with the real Waymo calibration used for the rays. Camera-ID embeddings of the Waymo cameras were never trained, so we map each Waymo camera to the NAVSIM camera with the closest heading (fixed a priori, not tuned).

  • Dense metric depth transfers. Final-frame AbsRel versus Waymo LiDAR (1–80 m) is 0.127 on the front camera, versus 0.109 in-domain, and 0.162 over all five cameras, with no scale alignment (median scale ratio 1.02). This is worse than MoGe-2 (0.116), which was trained on far more diverse data.
  • The point-query head does not transfer. Current-point EPE median is 7.0 m versus 2.0 m for a model co-trained on Waymo. Its ego-frame 3D regression is tied to the training rig.
  • Camera embeddings matter. Using the untrained Waymo camera IDs raises dense AbsRel to 0.45.

We report this honestly in the revised limitations: the dense reading of the memory transfers across rigs zero-shot, but the query reading and the camera embedding are rig-specific and need target-rig data. We now compare to VGGT-Omega/DVGT-2 only with this caveat.

W4 — Table 4 needs matched input. Done (General response §2). At GeoX's 4 cameras × 1 frame × 64 registers, GeoAct scores 91.25 versus GeoX's 90.83 (+0.42, CI [−0.01, +0.87]), and DINOv3 at the same input scores 91.41. Revised Table 4 reports both input budgets.

W5 / W6 / W7. Table 1 now uses one decimal throughout. The appendix table is placed after its heading. We removed the opening claim that driving systems need metric/queryable representations, and the introduction now states that many end-to-end planners succeed without them and that our own controls confirm geometric pretraining does not drive PDMS here.

Q2 — does depth/3D-displacement quality correlate with NAVSIM score? Across six frozen encoders (N1, D0, D1, F1, GeoX, DVGT-2; GeoX and DVGT-2 with oracle scale), Spearman ρ between test AbsRel and PDMS is −0.54 (p = 0.27, n = 6), not significant. The geometry-free DINOv3 control scores above all of them. Along a single training trajectory, each checkpoint would require a separate 20-epoch planner, which was not feasible in the rebuttal window. The cross-encoder evidence, the DINOv3 control and DVGT-2 (better geometry than GeoX, oracle AbsRel 0.138 versus 0.146, yet the lowest PDMS of all frozen memories at 90.22) all indicate that, under frozen transfer, better 3D accuracy does not translate into higher nominal PDMS. Its effect appears in robustness instead (W2). The revised paper states this as a finding rather than claiming that geometry aids planning.

Reviewer_f5ziOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper porposes GeoAct, which encodes continuous multi-camera frames and decodes into both densely predicted and sparsely queried geometry. Compared to previous works, GeoAct achieves better geometry reconstruction and better planning performance.

Critical strengths
  • Apply queriable 4d tracking and reconstruction on driving tasks and increase the performance
  • Insteresting observation that better geometry does not mean better planning.
Critical weaknesses
  • Lack detailed experimental analysis on why worse geometry still results in the same level of planning performace.
  • The source of the improvement over GeoX remains unclear.
Questions for authors
  • What is the actual motivation or benefit of introducing queriable sparse geometry reconstruction, as the sparse head is not used for planning?
1 comment

We thank the reviewer; both weaknesses are now addressed by new controls.

W2 — source of the improvement over GeoX. Answered directly by two controls (General response §1–2).

  • Matched input (4 cameras, 1 frame, 64 registers): GeoAct 91.25 versus GeoX 90.83, so about half of the original gap was input budget.
  • Same-input DINOv3 without any geometric pretraining: it matches or exceeds GeoAct at both input budgets (92.16 versus 91.78 full; 91.41 versus 91.25 matched).

The improvement over GeoX therefore comes from seeing eight cameras over four frames through calibration-aware embeddings and more registers, not from the geometric memory. The revised abstract, contributions and Section 4.5 state this.

W1 — why does worse geometry give the same planning? We added three analyses (Section 4.5 and a new appendix figure).

  1. Across encoders, geometry does not predict PDMS. Over six frozen encoders, test AbsRel versus PDMS gives Spearman ρ = −0.54, p = 0.27, n = 6. Geometry-free DINOv3 scores highest, and DVGT-2 (1.2B, oracle-scale AbsRel 0.138, better than GeoX's 0.146) under the same planner scores 90.22, below N1 by 1.56 ([0.76, 2.36]).
  2. Nominal PDMS hides the difference. Under calibration perturbation, the geometric memory is significantly more robust than DINOv3 (−0.78 versus −1.49 PDMS under severe noise). Geometry quality shows up in robustness, not in the clean score.
  3. The readout is a bottleneck. The adapter compresses 4×8×576 memory tokens into 128 registers through one cross-attention layer and is trained only by the planning loss. Under frozen transfer, the planner can recover what it needs from either memory. When the same encoder is instead trained by the planning loss, GeoAct initialization beats random initialization by +2.7 to +3.2 PDMS at matched epochs (General response §4, interim). Geometry is therefore useful as a starting point for learning, even though a frozen readout does not exploit it.

We are careful not to over-claim: these are frozen-transfer results with one planner, as stated in the limitations.

Q — motivation of the queryable sparse head if planning does not use it.

  1. It is an inference-time interface in its own right, evaluated on test. Given a pixel and two times, it returns metric position and displacement, independently per query; Table 3 reports current-point and moving-point EPE. Such point-wise space-time queries are what downstream consumers like flow/occupancy probing, sensor simulation or auto-labelling need, without re-running dense reconstruction.
  2. Its supervision changes the shared memory: removing dense supervision improves moving-point EPE by 5.9%, growing with displacement. Removing the query branch leaves dense geometry unchanged.
  3. We do not claim it helps planning. The revised introduction states that the planner reads the memory but never calls the query head.
Reviewer_gm2iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains a small encoder to encode a short horizon of multi-camera from a driving car into a one shared set of memory tokens, given the known calibration and ego motion. The set of memory tokens are used in two ways, one for predicting a metric dense depth for pixels, and another one for surface position at a time query. Both heads are trained from lidar projection. Then the encoder is frozen and a small adapter plus a planner are trained. The findings are that the dense head gives good metric depth with no test time scale alignment, and they plan better than a frozen reference geometry model.

Critical strengths

The paper is well written, easy to follow. The methods and losses are described clearly. The controlled continuation is a good design, so the comparison is meaningful. The geometry results are the main strength of the paper, as the dense head gives metric depth with no test time alignment, from a model multiple times smaller than the baselines. The authors also mention the limitations of the method and of the evaluation, such as single seed and input mismatch with the baseline model and the frozen encoder protocol which helps the reader to understand the results better.

Critical weaknesses

My main issue is that the results do not fully back up the contributions. The planning comparison with the baseline model is not fair on input. this paper uses eight cameras over four frames and twice the registers, while the baseline uses four cameras and one frame, so a good part of the gap could just come from seeing more.

The experiment I would really like to see is missing, which is a frozen DINOv3 encoder with no geometric pretraining, on the same input and with the same adapter and planner. without it we do not know if the geometric memory helps planning at all.

All results come from a single training seed, so the small differences in the supervision trade-off could easily be training noise

The test split used for geometry was also looked at many times during development. The title says queryable, but nothing downstream ever reads the query head, it only works as a training loss.

Table 1 can also be misleading, the planner used here trained end to end with its own encoder, reports a higher PDMS than this paper, as the authors note under Table 4, but it is not listed in Table 1, so the paper's number looks like the new best when a good part of the gain over the listed methods probably comes from the planner itself rather than from the geometric memory.

Finally, the model is built from existing parts (the backbone, the encoder design, the dense head, the query head and the planner are all taken from earlier work), and the authors do not claim otherwise, so the contribution of the paper is in the measurements, and those measurements have the problems I listed above.

Questions for authors
  • how much of the planning gap over the reference model do you think comes from the larger input (more cameras, more frames, more registers) rather than from the representation itself? Is there anything in your current results that can separate the two?

  • do you have any sense of how the frozen memory compares with plain DINOv3 features under the same adapter and planner? Is there a way to see from your setup that the geometric pretraining changes what the planner gets?

  • for the edge-contrast result, how sure are you that edges found from sparse projected lidar are real depth discontinuities and not lidar artifacts near object boundaries? would the dense teacher or a denser ground truth give the same picture?

3 comments

We thank the reviewer for the precise diagnosis; it led to the central result of the revision.

W1 / Q1 — input is not matched; how much of the gap is input? We retrained the planner with GeoX's exact input (4 cameras, final frame, 64 registers; General response §2). N1 at matched input scores 91.25 versus GeoX's 90.83 (+0.42, CI [−0.01, +0.87]). Going from matched to full input adds +0.53 ([+0.11, +0.92]). So roughly half of the gap in the original Table 4 came from seeing more, and the remainder is at the edge of significance. We report this decomposition in the revised Table 4.

W2 / Q2 — same-input frozen DINOv3. Done (General response §1). With the same input, embeddings, adapter, registers and planner, frozen DINOv3 reaches 92.16 versus 91.78 for N1 (CI [−0.02, +0.80]), and 90.26 versus 89.83 on official EPDMS. At matched GeoX input the two are also indistinguishable. We therefore state plainly that under frozen transfer, geometric pretraining does not raise PDMS. Two further results show that the pretraining does change what the planner receives:

  • Under calibration noise, the planner on the GeoAct memory loses 0.78 PDMS versus 1.49 for DINOv3 (difference +0.71, [+0.15, +1.29]).
  • A 1.2B geometry model (DVGT-2) on the same input and planner scores 90.22, significantly below both. The planner is therefore not simply rewarding geometric quality.

We also tested the setting where the encoder is not frozen. We train the same 12-stage architecture with the planning loss only, from random initialization or from GeoAct. At matched intermediate epochs, GeoAct initialization is clearly ahead: +3.15 PDMS at epoch 4 ([+2.54, +3.77]) and +2.69 at epoch 8 ([+2.10, +3.29]) (General response §4; interim, final results to follow). So geometric pretraining does change what the planner gets. Under frozen transfer, the planner can extract an equally useful signal from raw DINOv3 features. When the encoder is learned, geometric pretraining gives a substantially better starting point.

W3 — single seed. A second N1 seed scores 91.93 (Δ = −0.15 versus seed 0, CI [−0.61, +0.30]). This seed-to-seed difference is of the same order as the N1/D0 and N1/DINOv3 differences, which supports reading those as "no detectable difference". We report it that way. A second DINOv3 seed is training and will be added during the discussion period.

W4 — test split inspected during development. We agree and say so in the paper. Mitigations:

  • Every checkpoint is the predefined final step, never selected on test.
  • All arms are reported, including negative ones.
  • Every new comparison in this rebuttal was specified before it was run.
  • We add an evaluation on data never used in GeoAct development: zero-shot Waymo (kGkB W3), on which N1 was never trained or tuned.

W5 — "queryable" but nothing reads the query head. We now define the term narrowly. Queryable means the trained model exposes an inference-time point-query interface: camera, pixel, source time and target time in; metric position and displacement out. We evaluate this interface directly on test (Table 3: current-point EPE, moving-point EPE, I_zero). Its supervision measurably changes the shared memory: dropping the dense branch improves moving-point EPE by 5.9%. We explicitly state that the planner reads the memory but does not call the query head, and we make no claim that querying helps planning.

W6 — Table 1 omits DrivoR. Fixed. Table 1 now lists DrivoR's reported 93.1 PDMS, noting that it uses an end-to-end trained encoder and 25 epochs; our encoders are frozen with 20 planner epochs. It also lists the DINOv3 control. The "new best" wording is removed.

W7 — built from existing parts. We agree; we do not claim architectural novelty. The contribution is the measurement study. The controls the reviewer requested (same-input DINOv3, matched input, seeds, and a dense-teacher edge study) were the missing parts of that study. They change its conclusion, and we report the changed conclusion.

Q3 — are sparse-LiDAR edges real? The reviewer's suspicion was justified. With edges from sparse projected LiDAR, every model, including the sharp dense MoGe-2, scores ≈0.10 edge-contrast fidelity. That protocol therefore cannot discriminate sharpness, and we withdrew the original claim. We re-measured on 1961 test scenes under three stricter definitions (new Table tab:edges):

Edge setGeoActGeoXDVGT-2MoGe-2
Sparse LiDAR (original)0.0990.0940.1050.108
LiDAR, ≥2 returns on each side0.1150.1310.1370.156
LiDAR ∩ MoGe-2 dense edges0.1730.5080.4040.670*
MoGe-2 dense edges0.3330.7410.639—

*Selection-biased toward MoGe-2.

On edges confirmed by the dense teacher, GeoAct is the smoothest model despite the lowest AbsRel. The paper now says "lowest AbsRel but smoothest edges".

We also diagnosed the cause by fine-tuning only the DPT head of the frozen N1 encoder for 6000 steps per variant, against a same-budget control:

  • A near-L1 depth loss plus filtering of LiDAR returns that conflict with the teacher raises LiDAR∩MoGe-2 fidelity from 0.173 to 0.217, with AbsRel improving from 0.1096 to 0.1049.
  • Removing the learned uncertainty weighting has no effect.

The blur is thus partly a loss/label effect, but it remains well below GeoX and DVGT-2, and we report it as a limitation.

Thanks to the authors for their detailed response to the questions I raised during my initial review. I should also say that, regarding weaknesses W3-7, I am satisfied with the authors' reply. However, considering Q1 and Q2, the improvement in the same setup is not significant, given that it is the major claimed contribution of the paper. The gap I see with and without frozen DINOv3 is also not significant. Although I appreciate the authors' efforts to address the reviewers' comments, I'd keep my current assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.