acceptodds
Withdrawn from ICLR 2027

Drive4RT: Learning Queryable 4D Scene Representations for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Understanding dynamic driving scenes requires more than reconstructing their instantaneous 3D geometry: a scene representation should also capture how the same physical points corresponds across cameras and evolves over time. We present Drive4RT, a queryable 4D scene representation for surround-view autonomous driving. Given a short multi-camera video, Drive4RT encodes the observations into a shared 4D scene representation and predicts the metric 3D location of a physical point conditioned on its source observation, target time, and reference camera frame. By varying the query conditions, 3D reconstruction, cross-view geometric reasoning, and temporal point tracking become different readouts of the same underlying representation. Adapting queryable 4D reconstruction to driving scenes introduces two key challenges: heterogeneous surround-view observations and the coexistence of a largely static environment with independently moving traffic participants. Drive4RT addresses these challenges through structured intra-view, cross-view, and cross-frame reasoning, together with dynamic motion modeling that combines scene-level motion grouping and point-level residuals. To learn point-level 4D geometry without dense trajectory annotations, we further construct complementary supervision from standard driving logs: static correspondences are derived from LiDAR and ego poses, while dynamic trajectories are generated from temporally associated 3D object tracks. Across five large-scale autonomous-driving datasets, Drive4RT achieves strong performance in 3D reconstruction, depth estimation, ego-pose estimation and point-level 4D tracking. These results demonstrate that geometry and motion in dynamic driving scenes can be organized around a shared queryable 4D representation rather than modeled as isolated perception outputs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GA8mOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Drive4RT learns a queryable 4D scene representation from short surround-view driving clips. A query identifies a source pixel, source camera/time, target time, and reference camera, and the model returns the metric 3D position of the corresponding physical point. The encoder factorizes interaction into intra-view, cross-view, and cross-frame stages. A source-anchored decoder retrieves query-specific features, while a structured motion module decomposes trajectories into a static-background slot, foreground motion prototypes, and point-level residuals. Training supervision is derived from standard driving logs: LiDAR and ego poses supervise static geometry, while LiDAR points inside tracked 3D boxes are propagated using box poses to supervise dynamic trajectories. The model is jointly trained on five driving datasets and evaluated on reconstruction, ray depth, ego pose, cross-view queries, and point tracking, with an additional zero-shot KITTI evaluation.

Critical strengths
  1. The conditional point-query interface is appealing: reconstruction, cross-view coordinate transfer, and temporal tracking can be exposed through one representation rather than separate heads.
  2. Factorizing surround-view processing into intra-view, cross-view, and cross-frame interactions is well motivated and substantially outperforms unrestricted global attention in the ablation.
  3. The static/dynamic supervision construction is practical and uses annotations commonly available in autonomous-driving logs instead of requiring dense manual point trajectories.
Critical weaknesses
  1. The quantitative experimental evaluation results are not convincing. E.g., in Tab.1 & Tab.2, the performance of Drive4RT is not as good as the baseline methods, which needs to be discussed. In addition, there is a significant discrepancy between the experimental data of DVGT in Tab1 and the data in its original paper, which fundamentally raises my doubts about the effectiveness of Drive4RT.
  2. The methodological novelty appears limited. The source-point/target-time/reference-frame query and independent decoding closely follow D4RT, while the intra-view/cross-view/cross-frame encoder closely follows DVGT; the geometry-plus-motion querying formulation also substantially overlaps with 4RC. The remaining contribution is mainly a driving-specific integration with motion slots and box-derived supervision. Without a direct comparison to 4RC or controlled D4RT-plus-DVGT baselines, it is unclear whether this constitutes a substantive methodological advance beyond combining existing designs.
  3. It is unclear whether the motion slots learn an actor-specific decomposition. Assigning every dynamic point the same uniform foreground-slot distribution makes the within-object grouping and temporal-consistency losses zero and minimizes the slot-balancing loss, while assigning no distinct slot to any actor. Although the trajectory-position loss and residual penalty may indirectly prevent this behavior, the paper provides no slot-assignment visualization, actor-slot purity/entropy analysis, or slot-versus-residual motion decomposition. Therefore, the interpretation that the slots capture actor-level motion is not sufficiently supported by the current experiments.
Questions for authors
  1. The DVGT results in Tab.1 do not match its original paper and appear misaligned across datasets. Were they re-evaluated under a different protocol or reported incorrectly? Please verify the numbers and revise the related conclusions.
  2. Drive4RT underperforms the strongest baselines on several reconstruction and depth benchmarks in Tab.1 and Tab.2. Please discuss these regressions and clarify the claimed overall advantage.
  3. The slot losses may not explicitly prevent all actors from using the same uniform slot distribution. Additional discussions or ablations are critical to this statement.
  4. It's encouraged to ablate the slot number KK, slot-only versus residual-only motion, and the individual slot losses.
  5. The improvement in tracking capability in Tab.3 may come from unfair comparisons with baseline methods, such as introducing additional training data, extra supervision items, etc. Please clarify the detailed impenmation of those baseline methods to control variable.
  6. The tracking targets are synthesized by rigidly propagating LiDAR points using annotated 3D-box poses, following the same assumption used for training supervision. How does this evaluation distinguish physical point tracking from object-level box-motion estimation? Please include an oracle or predicted-box rigid-motion baseline.
Reviewer_tkX6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Drive4RT extends D4RT-style conditional point querying to surround-view driving videos. A VGGT-based encoder combines intra-view, cross-view, and cross-frame interactions, while motion slots and point-level residuals model dynamics. Queries return a source point’s 3D position at different times and in different camera frames within the observed clip. Static supervision uses LiDAR and ego poses; dynamic training and evaluation targets are generated by rigid propagation using tracked 3D boxes. Experiments cover geometry and pose on five driving datasets, tracking on four, and zero-shot evaluation on KITTI.

Critical strengths
  1. The design addresses concrete characteristics of driving scenes. Surround-view encoding accommodates heterogeneous cameras, while static–dynamic decomposition addresses independently moving actors. Constructing supervision from LiDAR, ego poses, and tracked boxes provides a practical route to learning queryable 4D representations from standard driving logs.

  2. Internal ablations provide preliminary support. Cross-view interaction reduces 3D error from 0.407 m to 0.113 m in Table 5, and motion slots improve APD3D from 0.183 to 0.293 in Table 7. These controls support the proposed choices more directly than comparisons with general-purpose models alone, although their exact configurations need clarification.

Critical weaknesses
  1. The central quantitative results contain unresolved inconsistencies. In Table 3, Drive4RT’s Argoverse 2 AJ3D is 0.247, exceeding its APD3D of 0.221; several SpatialTrackerV2 entries show the same issue. Under the standard TAPVid-3D definitions cited in Appendix C, this cannot occur with identical predictions, alignment, thresholds, valid samples, and aggregation. Furthermore, the DVGT values in Table 1 appear misassigned: its original OpenScene Acc/Comp is 0.402/0.481, whereas the submission lists 1.714/2.216, which corresponds to Waymo in the original paper. These issues affect comparative conclusions and require priority verification.

  2. Architectural gains are not adequately separated from adaptation and input conditions. Drive4RT is trained on five driving datasets and uses calibrated extrinsics for camera-frame readout. However, the implementations, additional driving-domain adaptation, camera inputs, calibration information, and scale handling used for D4RT and SpatialTrackerV2 in Table 3 are insufficiently documented. Identical query points do not establish matched comparison conditions. The internal ablations are useful, but a direct comparison with D4RT adapted using the same driving supervision is still missing.

Questions for authors
  1. Do APD3D and AJ3D use identical predictions, alignment, thresholds, valid samples, and aggregation? Please explain the Table 3 entries where AJ3D exceeds APD3D, provide the evaluation implementation or corrected results, and define Recall.

  2. Are the DVGT numbers in Table 1 copied from the original paper or re-evaluated? Please verify their assignment to Waymo, OpenScene, and DDAD, and clarify which comparative conclusions remain valid.

  3. Which implementations/checkpoints, camera inputs, calibration information, scale alignment, and driving-domain adaptation were used for the Table 3 baselines? On one representative dataset or matched camera subset, do the main gains persist after adapting D4RT with the same driving supervision?

Reviewer_CU8POpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper adapts D4RT to the driving multiview setup. As with D4RT, the model's goal is to return the 3d position of any pixel in any image at any point of time of the observed clip. The model is composed of a VGGT-initialized backbone with attention within each image and across cameras and frames, to extract the information from all the frames, and a decoder that answers each query. the motion module considers 17 groups (1 static background, 16 dynamic) and each point motion is a weighted mix of these groups plus a small per-point correction. During training, the losses push the static lidar points to pick the background group and the dynamic lidar points within moving bounding boxes to share the same assignment. The model is trained on multiple driving datasets (nuScenes, Waymo, OpenScene, DDAD and argoverse 2) and evaluated on reconstruction, depth, ego pose and 3d tracking.

Critical strengths

None

Critical weaknesses
  1. Some reported results in the tables are wrong. In Table 1, DVGT's values are shifted. With the correct scores, the claims flip on OpenScene. Also in Table 3 some AJ values are above APD, which is not possible.
  2. The tracking evaluation is not exactly fair. The way the point tracking is built from box-propagated points here for both training and evaluation means that Drive4RT was trained with this type of tracking, which could explain that it scores much better than the other baselines. The fact that baselines such as D4RT and SpatialTrackerV2 score much better on DriveTrack than here for example is concerning, though might be due to different protocols.
Questions for authors
  1. How were AJ and APD computed in Table 3 given that AJ is higher than APD in 4 cells?
  2. What does the asterisk mean for D4RT? It seems inconsistent across the tables. Also is the D4RT baseline a reimplementation as the code does not seem available?
  3. For the ablations, how was GA trained and why is AUC so much worse than the original VGGT? Also TPE is undefined in the paper, is it temporal positional encoding?
  4. Why are D4RT and SpatialTrackerV2 baselines results so much worse than with DriveTrack?
  5. Could it be possible to provide tracking results on TAPVid-3D DriveTrack split?
  6. Would it be possible to also evaluate tracking on non-rigid actors such as pedestrians, or with ground truth not derived from boxes?
  7. What happens when there are more than 16 moving actors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.