Drive4RT: Learning Queryable 4D Scene Representations for Autonomous Driving

Understanding dynamic driving scenes requires more than reconstructing their instantaneous 3D geometry: a scene representation should also capture how the same physical points corresponds across cameras and evolves over time. We present Drive4RT, a queryable 4D scene representation for surround-view autonomous driving. Given a short multi-camera video, Drive4RT encodes the observations into a shared 4D scene representation and predicts the metric 3D location of a physical point conditioned on its source observation, target time, and reference camera frame. By varying the query conditions, 3D reconstruction, cross-view geometric reasoning, and temporal point tracking become different readouts of the same underlying representation. Adapting queryable 4D reconstruction to driving scenes introduces two key challenges: heterogeneous surround-view observations and the coexistence of a largely static environment with independently moving traffic participants. Drive4RT addresses these challenges through structured intra-view, cross-view, and cross-frame reasoning, together with dynamic motion modeling that combines scene-level motion grouping and point-level residuals. To learn point-level 4D geometry without dense trajectory annotations, we further construct complementary supervision from standard driving logs: static correspondences are derived from LiDAR and ego poses, while dynamic trajectories are generated from temporally associated 3D object tracks. Across five large-scale autonomous-driving datasets, Drive4RT achieves strong performance in 3D reconstruction, depth estimation, ego-pose estimation and point-level 4D tracking. These results demonstrate that geometry and motion in dynamic driving scenes can be organized around a shared queryable 4D representation rather than modeled as isolated perception outputs.