acceptodds
Desk rejected at ICLR 2027

Recon4Drive: Selective and Explicit 4D Priors for VLA Driving

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models reason effectively about driving scenes, but their image-centric representations do not explicitly capture the metric geometry and temporal dynamics required for planning. Pretrained 4D reconstruction models encode such information, yet their dense representations are designed to describe the entire scene. For planning, however, only a small portion of this information is relevant, and the current decision mainly depends on a few ego-centric relations, such as the distance to a lane boundary or the gap to a nearby vehicle, which remain implicit in dense features. To this end, we present Recon4Drive, a VLA framework that converts dense 4D reconstruction features into compact and explicit inputs for planning. The Semantic-Guided 4D Retriever (SG4R) uses image semantics to retrieve planning-relevant geometry and motion cues from the 4D representation, keeping the number of 4D tokens small regardless of its density. Planning-Critical Structured Scene Encoding (PSE) further assigns supervised queries to key relations with lanes and surrounding agents, including their future motion, making these relations directly accessible to reasoning and planning. On the closed-loop Bench2Drive benchmark, Recon4Drive achieves a Success Rate of 62.27% and a Driving Score of 85.45 with camera-only inference, the best results among methods trained on the official Think2Drive-collected data. The code will be made publicly available upon publication.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, Vision-Language-Action

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9sMsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Recon4Drive, a vision-language-action framework that transfers pretrained 4D reconstruction features to autonomous driving. Its Semantic-Guided 4D Retriever (SG4R) uses image-derived queries to compress dense SLARM features into 180 tokens. Planning-Critical Structured Scene Encoding (PSE) introduces 12 supervised queries representing lane boundaries, nearby agents, and their future motion. A flow-matching DiT planner generates trajectories from the resulting VLM hidden states.

Critical strengths

1.Strong closed-loop results in a clearly specified comparison setting. The reported 62.27% SR and 85.45 DS exceed MindDrive by 4.01 percentage points and 4.86 points, respectively. The 91.11% overtaking score is particularly encouraging for a method intended to improve geometric and interaction understanding. Closed-loop evaluation across 220 routes provides consequential evidence beyond open-loop trajectory accuracy. 2.An effective compact interface between reconstruction and planning. SG4R reduces the 4D tokens entering the VLM from 2,592 to 180. With PSE enabled, the compressed model improves SR and DS over the dense counterpart in Table 2 while substantially reducing measured VLM latency. This is a useful practical contribution: pretrained reconstruction representations can be incorporated without passing their full dense token sequence into the language model.

Critical weaknesses

1.How does an image-only model with the same PSE supervision perform? Could you also compare supervised relation queries with the same number of learnable queries without relation supervision, while keeping the planner interface otherwise identical? These comparisons would help determine whether the gains specifically arise from combining 4D features with explicit planning relations, or can largely be explained by auxiliary supervision and additional conditioning capacity. 2.Table 7 compares encoders that differ in architecture and pretraining as well as their treatment of dynamics. What evidence supports attributing SLARM’s advantage specifically to dynamic information? Could you provide a controlled comparison using the same SLARM encoder with and without informative temporal input, while keeping the training setup and token budget consistent? 3.In Table 8, SG4R achieves the same SR as Fixed Image Query and improves DS by 1.08, while its DS advantage over Learnable Query is 0.19. Are these differences consistent across repeated runs? Could you provide evidence that the learned camera aggregation in SG4R offers a reliable benefit over these simpler retrieval alternatives? 4.Appendix B.3 states that the reported latency and GFLOPs cover only the VLM. What are the total inference latency and planning frequency when SLARM encoding, SG4R retrieval, VLM processing, and DiT sampling are included? Please specify the hardware and feature-caching setup, and compare against the image-only baseline under the same conditions to clarify whether the additional system-level overhead is indeed small.

Questions for authors

see Weaknesses

2 comments

Thanks to the author's reply, the author has resolved most of my concerns. I will adjust my score to 3, but similarly, my confidence is 2 because I haven't conducted research directly related to this field.

We thank the reviewer for the careful consideration of our responses. We appreciate the constructive comments, which have helped us clarify our technical contributions and strengthen the experimental validation of our work.

Reviewer_T9qKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces Recon4Drive, a VLA framework for autonomous driving that integrates pre-trained 4D reconstruction features. The framework consists of two core components: SG4R and PSE. SG4R utilizes visual queries to extract compact 4D features, while PSE employs supervised relation queries to encode lane boundaries, surrounding traffic participants, and their future trajectories. These representations are then fed into a flow-matching-based DiT planner to generate driving trajectories. Experimental results on the Bench2Drive benchmark demonstrate that the framework enhances closed-loop driving performance; furthermore, SG4R effectively reduces the computational overhead associated with injecting dense 4D features into VLMs.

Critical strengths
  1. The method achieves strong closed-loop results on Bench2Drive. The improvements over existing methods are encouraging, especially in overtaking scenarios.
  2. The framework is well motivated, and the ablation studies help demonstrate the effectiveness of incorporating 4D features and relation supervision.
Critical weaknesses
  1. The methodological novelty is somewhat limited. Query-based feature retrieval and supervised structured queries have been explored in closely related works, such as DSR Suite (CVPR 2026), VGGDrive (CVPR 2026), and SGDrive (CVPR 2026). The proposed SG4R and PSE share similar ideas with these methods. While the specific designs differ, the paper does not sufficiently clarify its technical contributions beyond existing approaches.
  2. The benefits of the proposed designs are not sufficiently validated. In Table 8, SG4R achieves the same success rate as the simpler Fixed Image Query, with only a small improvement in driving score. Moreover, PSE lacks quantitative evaluation of whether the learned relation states accurately capture geometric and motion information.
Questions for authors
  1. Could the authors clarify the main technical differences between SG4R/PSE and existing methods, including but not limited to DSR Suite, VGGDrive, and SGDrive?
  2. What is the additional benefit of the scene-adaptive camera weighting in SG4R? Could the authors provide a more detailed ablation of the global prior and dynamic weighting components?
  3. Could the authors provide quantitative evaluations of the lane distances, agent distances, and future trajectories predicted by PSE to verify whether the learned relation states capture the intended geometric and motion information?
Reviewer_JNdqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Recon4Drive, a VLA driving framework that incorporates pretrained 4D reconstruction features through two components. The Semantic-Guided 4D Retriever (SG4R) compresses dense features into planning-relevant tokens, while Planning-Critical Structured Scene Encoding (PSE) introduces supervised queries representing ego-centric lane, agent, and future-motion relations. A flow-matching planner generates trajectories from the resulting VLM representations. On Bench2Drive, the method outperforms the others trained on the official Think2Drive-collected data

Critical strengths
  1. Closed-loop evaluation provides relevant evidence for the driving application. The reported improvements extend to overall Multi-Ability performance and open-loop trajectory error.
  2. The compression results are compelling at the VLM interface. SG4R reduces the injected 4D tokens from 2,592 to 180 while largely preserving the benefits of dense features. The appendix also clearly documents implementation details and evaluation protocols.
Critical weaknesses

PSE adds both dedicated relation queries and simulator-derived supervision for lane distances, agent distances, and future trajectories. Table 2 demonstrates that this combination improves planning, but does not establish how much of the improvement comes from exposing structured relations to the planner versus learning better representations through auxiliary perception and prediction losses. This distinction matters because the numerical relation decoders are used only during training; at inference, the planner consumes their associated hidden states. Useful controls would retain the auxiliary supervision while removing the planner’s direct access to relation states, or apply the same supervision through heads attached to existing features.

Temporal VGGT is a useful control for the additional temporal observations. However, comparing VGGT and SLARM also changes the pretrained encoder, including its architecture and training objectives. SLARM additionally incorporates language-aligned semantic features, so its advantage cannot be attributed exclusively to dynamic modeling.

The system-level efficiency benefit remains insufficiently characterized. Appendix B.3 explicitly states that the reported FLOPs and latency cover only the VLM. These measurements demonstrate the efficiency of the compressed interface relative to dense concatenation, but do not quantify the complete overhead relative to the image-only baseline.

The incremental value of the complete SG4R adaptation mechanism could be better established. The compression study supports semantic guidance, but the full SG4R and Fixed Image Query variants achieve the same success rate of 53.64%, with driving scores of 75.40 and 74.32, respectively. These results leave the individual contributions of the global prior A_0 and dynamic adjustment ΔA insufficiently characterized. Separate ablations of these components, accompanied by an analysis of how camera weights change across driving situations, would clarify when the additional adaptation is beneficial.

Questions for authors
  1. With relation supervision held fixed, how much additional benefit comes from allowing the planner to attend directly to the dedicated relation states?
  2. Under controlled temporal ablations of the same pretrained encoder, how much of the planning improvement can be attributed specifically to motion information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.