- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work introduces ScaleDrive, a self-supervised framework that pre-trains visual representations for end-to-end driving models from action-free driving videos. Specifically, the authors train with the V-JEPA 2 algorithm videp encoders using a large dataset of web-based driving videos. The V-JEPA 2 algorithm that they use is modified to also include full frame masking, so as to better learn temporal dynamics. After the pre-training, they freeze the vision encoder, called ScaleDrive, and plug it into several end-to-end driving models, such as LTF, DiffusionDrive, DrivoR, and ChainFlow on Navsim and TransFusion-v6 on Bench2Drive, showing consistent improvements. Furthermore, they show that pretraining with more diverse data leads to better downstream performance.
Critical strengths
The work studies an interesting problem: pretraining visual representations for end-to-end driving models. The authors demonstrate that they can learn strong visual representations that lead to consistent performance improvements on end-to-end driving models while using only driving videos (without action annotations).
Critical weaknesses
- The work lacks technical novelty. It mainly applies the V-JEPA 2 algorithm on driving video, with the only difference being that they also use full-frame masking. However, this modification (1) is very incremental and (2), as seen in Figure 2(a), it offers only a small improvement (increases the PDMS from 93.61 to 93.79), which is not even clear if its statistically significant or consistent across all planners. Moreover, the finding that pre-training with more diverse data improves downstream perfromance, although it is nice to be confirmed in this case, it is somewhat expected.
- I find two issues with the experimental results:
- First, in the results presented in Tables 2-5 it is unclear whether the ScaleDrive can be used as a stand-alone visual backbone. What I mean is that for Drivor and ChainFlow, the ScaleDrive encoder provides only the front-camera visual features while in TFv6, LTF and DiffusionDrive, it is used as an additional visual encoder, on top of what is already there. Is ScaleDrive useful when used in stand-alone fashion in these experiment? Also, it is not clear how the results in Tables 1 and 2 are produced: is there the ScaleDrive encoder used in stand-alone way, or together with other visual encoders?
- It only compares against V-JEPPA 2, and lacks comparison with other state-of-the-art visual encoders, and more specifically DINOv3. How do they results changes if in Tables 1-5 you replace the ScaleDrive with DINOv3?
- To summarize, from the experimental results it is not clear whether ScaleDrive can be used as stand-alone visual encoder and if it is a better alternative to DINOv3.
Questions for authors
- Currently, it is unclear whether the small improvement that full frame masking offers (Figure 2-a) is statistically significant and consistent across all planners. Does it also improve when used with other planners (e.g., ChainFlow, LTF, DiffusionDrive, and TFv6)?
- Clarify whether in Tables 1-2, ScaleDrive is used in stand-alone fashion.
- Why in Tables 3 and 5 ScaleDrive was not used in stand-alone fashion? How do these results change when ScaleDrive is used in stand-alone way (i.e., when you replace the existing image or video encoder with ScaleDrive)? I do not expect results for each planner-benchmark combination; a few combinations would be enough.
- Compare with DINOv3.
Response to Reviewer#uT1k
We are glad that the strengths (pre-trained visual representation for autonomous driving is an interesting problem and the consistent performance improvement) of our work can be acknowledged by the reviewer. We thank the reviewer for the constructive suggestions and valuable time. Detailed responses are as follows.
Q1. Temporal-slice masking gains across planners: We thank the reviewer for this suggestion. We evaluated patch-only and patch w. temporal-slice masking across different planners and computed the gains. These results support the consistency of the benefit across planner architectures. We added these results in the revised manuscript Table 15.
Q2 Stand-alone fashion ScaleDrive We apologize that this was not clear. In Tables 1–2, the ScaleDrive encoder is used as the sole visual encoder, and no additional visual encoder is used. These results therefore evaluate the effectiveness of its representation in a stand-alone fashion. The code for the corresponding model configurations and the ScaleDrive model is provided in the Supplementary Material.
Q3 Evaluating other method with stand-alone fashion. Our original experiments extend each planner’s visual representation to evaluate whether ScaleDrive provides complementary spatiotemporal information that improves existing planning systems. To assess its effectiveness as the sole visual encoder, we additionally replace the original ImageNet-pretrained visual encoders in LTF and DiffusionDrive with ScaleDrive. We construct the BEV representation using ScaleDrive features from the three front cameras, while retaining each planner’s original trajectory prediction head and training objectives.
We further evaluate stand-alone ScaleDrive with LTF on closed-loop simulation HUGSIM to assess its effectiveness across benchmarks. The results are summarized below. E, M, H, and X denote Easy, Medium, Hard, and Extreme, respectively.
Q4. Comparison with DINOv3. We thank the reviewer for suggesting this comparison. We additionally evaluate DINOv3 using DrivoR and ChainFlow. For each planner, we replace ScaleDrive with DINOv3 for comparison. DINOv2*: Reproduced baselines.