- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Drive3R, a block-wise streaming geometric foundation model for autonomous driving. It combines intra-block bidirectional attention, cross-block KV caching, and confidence-aware re-anchoring to improve long-horizon 3D reconstruction. The experimental results demonstrate strong performance across nuScenes, Waymo, and KITTI.
Critical strengths
- The paper addresses an important and practical challenge in scalable geometric reconstruction for autonomous driving.
- The proposed framework is well-motivated, with a clear architecture and effective integration of temporal context and drift mitigation.
Critical weaknesses
-
Block-wise causal attention, KV caching, and confidence-triggered re-initialization are individually established techniques. The main contribution appears to be their integration. More controlled comparisons are needed to establish the novelty and benefits beyond existing streaming and chunk-based approaches.
-
The proposed SE(3) re-anchoring cannot directly correct accumulated scale inconsistencies, while KITTI trajectory evaluation relies on Sim(3) alignment. This raises questions about whether the method genuinely addresses long-term metric-scale drift.
-
The KV cache continues growing between resets, making computational scalability dependent on the re-anchoring frequency. Moreover, Drive3R has higher inference latency than DVGT-2 (21.10s vs. 12.64s), and the efficiency analysis is limited to relatively short sequences.
Questions for authors
-
Can the authors provide more controlled comparisons to distinguish the contribution of block-wise attention from confidence-aware re-anchoring, using matched architectures and reset budgets?
-
How does SE(3) re-anchoring mitigate accumulated metric-scale drift? Could the authors report unaligned long-horizon trajectory errors and scale-drift measurements?
-
How do inference latency, peak memory, and reset frequency scale with substantially longer sequences?
3 comments
Response 2: Metric-scale consistency without Sim(3) correction
Reviewer's Comment (Critical Weakness 2 / Question 2): SE(3) re-anchoring cannot directly correct accumulated scale inconsistencies, while KITTI evaluation uses Sim(3) alignment. How can metric-scale drift be evaluated without Sim(3) correction?
Response:
The reviewer identifies an important distinction. In Drive3R, each reinitialized segment predicts its own metric scale through the metric-scale branch; scale is neither recovered from nor corrected by the SE(3) stitching transform. The predicted segment is then placed into the global frame using the six-DoF transform estimated on overlapping poses.
Unlike relative-scale models such as VGGT, which are trained without explicit metric-scale supervision, Drive3R directly supervises metric scale during training. We used Sim(3) alignment in the main KITTI table because several compared methods output trajectories only up to relative scale. To evaluate Drive3R independently of this scale-invariant protocol, we report the following metrics:
- Unaligned ATE: compares predicted and ground-truth trajectories without any global alignment.
- SE(3)-aligned ATE: allows global rotation and translation but no scale correction.
- Global scale drift: measures the residual metric-scale discrepancy.
- Boundary RPE: measures local relative-motion error around reset events.
Table R2: Metric-scale consistency and boundary continuity on complete KITTI
| Method | Unaligned ATE (m) ↓ | SE(3)-aligned ATE (m) ↓ | Median global scale error (%) ↓ | Common-window translation RPE (m) ↓ | Common-window rotation RPE (deg) ↓ |
|---|---|---|---|---|---|
| DA3-Streaming | 278.015 | 150.734 | 84.02 | 16.574 | 3.697 |
| DVGT-2 | 190.206 | 104.911 | 18.50 | 3.827 | 4.195 |
| Drive3R | 93.121 | 49.483 | 5.13 | 2.700 | 1.033 |
Without any global alignment, Drive3R reduces ATE by 51.0% relative to DVGT-2 and by 66.5% relative to DA3-Streaming. Under SE(3)-only alignment, the corresponding reductions are 52.8% and 67.2%. Its median global scale error is 5.13%, compared with 18.50% for DVGT-2 and 84.02% for DA3-Streaming.
All three methods use the same fixed reference reset events and temporal windows. Each window contains three frames before and three frames after a Drive3R reset event, yielding 717 valid windows. The RPE therefore measures local relative-motion error over identical intervals; it does not require DVGT-2 to reset at these timestamps or the DA3-Streaming windows to coincide with its chunk boundaries.
On these common windows, Drive3R achieves translation and rotation RPEs of 2.700 m and 1.033 degrees, lower than DVGT-2 (3.827 m and 4.195 degrees) and DA3-Streaming (16.574 m and 3.697 degrees). These results demonstrate lower metric-scale drift and better local consistency across resets without relying on Sim(3) scale correction.
Response 3: Efficiency and scaling on complete sequences
Reviewer's Comment (Critical Weakness 3 / Question 3): The KV cache grows between resets, and the efficiency analysis is limited to short sequences. How do latency, peak memory, and reset frequency scale with substantially longer sequences?
Response:
Table R3 compares runtime and peak GPU memory for DA3-Streaming, DVGT-2, and Drive3R on all 11 complete KITTI sequences, which span 55 to 933 frames and 0.4 to 5.1 km. The longest sequence is 25.9 times longer than the 36-frame efficiency clip. All three methods complete the evaluation without OOM. The internal closed-course sequences in Appendix C.2 also complete without OOM.
Table R3: Efficiency on complete KITTI sequences
| Method | OOM sequences | Mean inference + stitching time (s) | P95 sequence time (s) | Amortized time (s / stereo timestamp) | Mean per-sequence peak GPU memory (GiB) |
|---|---|---|---|---|---|
| DA3-Streaming | 0 | 145.502 | 329.506 | 0.3443 | 20.09 |
| DVGT-2 | 0 | 31.251 | 68.150 | 0.0739 | 19.81 |
| Drive3R | 0 | 38.411 | 84.161 | 0.0909 | 13.60 |
DVGT-2 uses factorized spatial-temporal attention, which separates cross-view spatial attention from temporal attention and reduces the number of tokens involved in each attention operation. DA3-Streaming follows the chunk-then-align paradigm. Drive3R retains joint cross-view and temporal interactions; applying similar factorization reduced reconstruction quality in our experiments.
On complete KITTI logs, Drive3R takes 38.411 s on average, 22.9% longer than DVGT-2 (31.251 s) but 73.6% shorter than DA3-Streaming (145.502 s). Drive3R has the lowest mean peak GPU memory: 13.60 GiB, compared with 19.81 GiB for DVGT-2 and 20.09 GiB for DA3-Streaming, corresponding to reductions of 31.3% and 32.3%.
Across complete KITTI logs, Drive3R performs 65.45 resets per sequence on average. The P95 and global maximum cached-history lengths are 6 and 21 frames, respectively. The revised version will include an optional defensive cache cap B_max, which forces a reset when the cache reaches the bound even if confidence remains above the threshold. This safety guard is not enabled in the experiments reported here.
Taken together, Tables R2 and R3 show a favorable accuracy-efficiency trade-off: Drive3R completes all evaluated long sequences without OOM and uses substantially less average peak memory than both baselines, while achieving lower SE(3)-aligned ATE, global scale error, and boundary RPE. Its runtime is moderately higher than DVGT-2 but substantially lower than DA3-Streaming.
Response to Reviewer 8QuA
We thank the reviewer for the careful evaluation and for recognizing the practical importance of scalable geometric reconstruction for autonomous driving.
Response 1: Problem-driven design and controlled comparison
Response:
Long-horizon surround-view reconstruction in autonomous driving imposes coupled requirements that generic sequence models do not directly address: local geometry must remain consistent across a calibrated multi-camera rig, useful context must persist across blocks, accumulated trajectory and scale drift must be removed before it dominates future predictions, and streaming memory must remain bounded. Generic frame-wise and chunk-based pipelines address only parts of this problem. Drive3R addresses these coupled requirements through a learnable streaming framework, rather than relying on hand-designed long-range heuristics.
The design follows directly from these requirements. Block-wise attention constructs a local multi-camera geometric state; KV memory preserves useful history across blocks; a confidence head trained from trajectory drift determines when that state should be refreshed; and SE(3) alignment reconnects the refreshed segment to the global trajectory. These mechanisms form a coherent state-update pipeline rather than a stack of unrelated generic operations.
Two controlled analyses in the paper isolate local bidirectional context and confidence-aware re-anchoring. We further add a reset-budget-matched comparison to evaluate adaptive reset timing.
Figure 6 isolates the effect of block-wise context by changing only the block size under the same backbone. Table 4 keeps the block-wise setting and removes confidence-aware re-anchoring; this causes ATE to increase from 4.1422 to 18.1769.
Table R1: Controlled reset-policy experiment on 20 nuScenes scenes
The two fixed-interval baselines provide closely matched reset counts or forward time. The confidence-based policy uses 3.80 resets per scene versus 4.00 for the closest reset-count baseline, while reducing ATE from 0.617 to 0.418. At nearly identical forward time (12.063 versus 12.032 s), it reduces ATE from 1.176 to 0.418.
Together, these analyses separate the effect of local bidirectional context from that of learned state resets and show that the latter is not simply due to more frequent resetting.