acceptodds
Withdrawn from ICLR 2027

Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving for their powerful representations, yet it remains unclear when these representations improve driving performance. To investigate this question, we introduce ViRA, a planner-agnostic visual representation alignment framework that keeps the planner architecture and inference cost unchanged. Our study reveals three findings: (1) VFM-guided visual representations consistently improve driving performance across diverse end-to-end planners, with gains extending to zero-shot closed-loop evaluation. (2) The choice of VFM target matters for planning performance, and alignment to a different VFM can further benefit planners with pre-trained VFM encoders. (3) Auxiliary perception supervision reduces sensitivity to VFM target selection, narrowing the EPDMS spread across five targets from 2.7 to 0.5 points and potentially compensating for less effective VFM targets. Guided by these findings, we develop ViRA-Diffusion, a diffusion-based planner trained without auxiliary perception supervision, which achieves 92.3 EPDMS on NAVSIM v2 navtest, outperforming recent methods in our comparison by at least 1.9 points. The results motivate jointly considering target selection and planner supervision when integrating VFMs into end-to-end autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Visual Foundation Models, Visual Representation Learning, Representation Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_76T2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how VFM-guided visual representations affects the driving performance of end-to-end AD planners. While representation alignment generally improves the performance, the authors show that the choice of VFM matters. However, auxiliary perception supervision may compensate for a weaker representation alignment. Finally, a sota diffusion-based planner ViRA-Diffusion is proposed without introducing auxiliary perception supervision.

Critical strengths
  • Extensive experiments have been conducted on various VFMs and End-to-End planners.
  • Figure 4 shows that EPDMS gains can be achieved without additional inference cost.
  • Auxiliary perception supervision is considered for a fair comparison.
Critical weaknesses
  • Limited novelty. Benifits introduced by representation alignment with VFMs are very much expected. The most distinctive finding about auxiliary perception supervision reducing the sensitivity to VFM target selection is yet rather an empirical observation and lacks in-depth analysis.
Questions for authors
  • Do you have a deeper understanding of why auxiliary perception supervison could reduce the sensitivity of VFM selection?
Reviewer_FovTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViRA, a training-time framework that aligns end-to-end driving representations with those of frozen visual foundation models through spatial dependency alignment (SDA) and scene semantic alignment (SSA). Experiments compare planners, teachers, and auxiliary perception supervision settings. The authors find that auxiliary supervision narrows performance differences across teachers. Their final model, ViRA-Diffusion, achieves 92.3 EPDMS on NAVSIM v2 navtest without auxiliary perception supervision.

Critical strengths
  1. The controlled TransFuser comparison is the most interesting part of the paper. Across five teachers, auxiliary perception supervision mainly improves results for less effective teachers, while the best score changes little.
  2. All three planners improve in overall score on navtest, navhard, and HUGSIM. The random-teacher control and loss ablations help substantiate these gains, and the method adds no inference overhead.
Critical weaknesses
  1. The alignment objectives follow established distillation and contrastive learning ideas, making the empirical analysis particularly important. However, the paper offers limited explanation for why some teachers help planning more than others. “Better visual representations” also remains loosely defined; improved BEV segmentation alone does not explain the differences in planning performance.
  2. The teacher spread with auxiliary supervision is only 0.5 EPDMS, and the best scores with and without it differ by 0.2. Repeated-run statistics are needed to assess these finer comparisons. The full auxiliary-supervision experiment is also limited to TransFuser with ResNet-34; The DINOv3-L experiment does not test whether the same findings hold.
  3. Figures 2 and S.2 do not adequately describe feature extraction and processing, particularly whether features are taken before or after projection. If projected features are used, the observed changes may partly reflect adaptation in the projection heads rather than changes in the student encoder.
Questions for authors
  1. Which feature layers are used for each student and teacher, and how are spatial locations matched across views and after resizing? Please clarify the processing used in Figures 2 and S.2, including whether the features are taken before or after projection.
  2. Why does SDA match student-to-teacher spatial relations to the teacher’s self-relations, rather than matching their respective self-relations? Are the teacher-side query and key projections updated during training, and are gradients stopped through the target distribution?
  3. Could the authors provide repeated-run statistics for the key auxiliary-supervision comparisons? Also, what is the result for DiffusionDrive with neither auxiliary perception supervision nor alignment? This missing baseline would help clarify whether alignment compensates for removing auxiliary supervision in ViRA-Diffusion.
Reviewer_uqWEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a very good question: do visual foundation models (VFMs) can provide better representations for end-to-end autonomous driving. They introduce ViRA, which is a representation alignment framework for planning. The results from their multiple experiments can demonstrate that combining the capabilities of visual foundation models can improve the performance of end-to-end autonomous driving.

Critical strengths
  1. They have very solid evaluation of the performance about combining visual representations to end-to-end autonomous driving. Their experiments covers multiple planners and benchmarks which show the effectiveness of their method design.

  2. Their proposed method can transfer knowledge from the visual foundation models without extra cost at test time, which is very efficient and keeps the same budget when doing the real-time inference.

Critical weaknesses
  1. In Figure 5, the results can not prove the representation is better (higher BEV mIoU), the planning performance is better. For example the DINOv3 show the best performance on BEV mIoU, but when adding the perception for planning, the gain from the visual foundation models is the worst one compared to other visual foundation models.
Questions for authors
  1. The results in Table 3 shows the RC and HD-Score in the extreme setting drops when using the method they propose. Can you make some explanation for that phenomenon?

  2. From Figure 5, can you do some extra experiments or make some analysis to show the connection between representation and the real performance improvements on planning across different visual foundation models?

Reviewer_JB58OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ViRA, a framework that aligns driving-planner features with frozen visual foundation models through spatial and semantic objectives. It reports improvements across three planners on NAVSIM v2 and zero-shot evaluation on HUGSIM, and examines how teacher selection interacts with auxiliary perception supervision. While the benchmark gains are encouraging, I have several concerns about the novelty and strength of the conclusions.

  1. The methodological contribution appears incremental. Spatial dependency matching and contrastive semantic alignment largely build on established representation-distillation techniques. The paper does not clearly demonstrate an advantage over closely related distillation approaches under matched experimental conditions.
  2. The experiments do not fully resolve the central question posed by the paper. Comparing different pretrained teachers establishes that teacher selection matters, but does not isolate representation quality from differences in pretraining data, objectives, and architecture. Consequently, the evidence supports the usefulness of feature distillation more directly than a general relationship between better visual representations and better driving.
  3. Some analytical conclusions need stronger validation. The controlled experiment on auxiliary supervision is limited to TransFuser, and the reported differences across teachers lack uncertainty estimates from repeated training runs. Furthermore, the large gains on Rap* use a modified backbone and omit the original augmentation, leaving their magnitude under the official configuration unclear. These concerns limit my confidence that the paper provides sufficiently novel and generalizable insights beyond its reported benchmark improvements.
Critical strengths
  1. ViRA improves performance across regression, diffusion, and scoring planners on NAVSIM v2, with additional gains in zero-shot closed-loop evaluation on HUGSIM. This breadth is a substantive strength because it provides evidence that the benefit extends beyond a single planner or evaluation setting. The randomly initialized teacher controls also help establish that pretrained representations contribute to the improvement.

  2. The framework preserves the planner’s deployed architecture and inference cost by removing the teacher and alignment modules after training. This makes the reported gains practically relevant for driving systems with strict latency constraints and provides a useful alternative to deploying a larger visual backbone.

Critical weaknesses
  1. This paper does not sufficiently distinguish its contribution from established feature distillation, leaving its central scientific question unresolved. ViRA combines spatial dependency matching and contrastive semantic alignment, but lacks a controlled comparison against closely related distillation methods using the same teachers and training budget. Moreover, differences between teachers conflate representation quality with pretraining data, objectives, and architecture. The random-teacher control establishes that pretrained information is useful, but does not explain which representation properties produce better driving. This substantially weakens both the methodological novelty and the explanatory value of a paper framed around understanding the relationship between visual representations and planning.

  2. The evidence supporting the main analytical conclusions is insufficiently robust. The claim that auxiliary perception supervision reduces sensitivity to teacher selection relies on a controlled experiment with only TransFuser, without repeated-seed results or uncertainty estimates. In particular, the reported 0.5-point spread cannot be interpreted confidently without establishing training variability. The largest planner improvements also involve Rap*, which changes the official backbone and removes the original augmentation, leaving their magnitude under the original configuration unverified. These limitations directly affect the paper’s central conclusions: the reported patterns could depend on specific training configurations, and the current experiments do not establish their reproducibility or generality.

Questions for authors
  1. How do you define and independently measure “better visual representations” across the five teachers? Beyond TransFuser’s BEV segmentation results, what evidence separates representation quality from differences in teacher architecture, pretraining data, and objectives?

  2. Table S.2 compares alignment losses, but does ViRA outperform established feature-distillation approaches when the teacher, training budget, and supervision are matched? Which aspect of the proposed formulation accounts for an advantage over these alternatives?

  3. How many independent training runs support the main results? Please provide variability estimates for the teacher comparisons, particularly the conclusion that auxiliary supervision narrows the performance spread from 2.7 to 0.5 EPDMS points. Are these differences reproducible across seeds?

  4. Why does Rap* use a different backbone and omit the original data augmentation? What evidence shows that the large alignment gains persist with the official Rap configuration, rather than primarily compensating for the modified baseline?

  5. For ViRA-Diffusion, how much of the final performance comes from DINOv3 alignment versus removing auxiliary perception supervision? Can you provide the corresponding DiffusionDrive results with and without each component under otherwise identical training settings? This would clarify whether the proposed interaction explains the headline result.

  6. Some aggregate improvements accompany regressions in individual metrics. For example, Rap*’s Extended Comfort decreases from 56.7 to 48.1 after DVGT alignment, and TransFuser’s HUGSIM Extreme HD-Score decreases from 10.9 to 9.9. Are these isolated effects or systematic tradeoffs introduced by alignment?

  7. ViRA-Diffusion’s advantage over the compared methods is particularly pronounced in Ego Progress, while several competing methods achieve higher collision-avoidance and drivable-area compliance scores. Can you provide scenario-level evidence explaining whether the aggregate improvement reflects safer driving, more efficient progress, or a different balance between these objectives?

  8. How sensitive are the teacher rankings to feature-extraction layers, spatial resolution, and alignment-loss weights? Were these settings selected using a consistent protocol across teachers? This matters because the claimed effect of teacher selection could depend substantially on the alignment configuration.

  9. What are the additional training-time costs of ViRA, including teacher feature extraction, GPU memory, and total training time? Under an equivalent computational budget, how does its performance compare with spending the additional resources on training the original planner longer?

Comments on the paper

1 comment
Area_Chair_kxsDOpenReview ↗

Discussion phase

Dear Reviewers, The authors have posted the rebuttal. Please review their responses and engage in the discussion.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.