acceptodds
Under review as a conference paper at ICLR 2027

Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Video Latent Prediction, Visual Representation, End-to-end Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_uT1kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces ScaleDrive, a self-supervised framework that pre-trains visual representations for end-to-end driving models from action-free driving videos. Specifically, the authors train with the V-JEPA 2 algorithm videp encoders using a large dataset of web-based driving videos. The V-JEPA 2 algorithm that they use is modified to also include full frame masking, so as to better learn temporal dynamics. After the pre-training, they freeze the vision encoder, called ScaleDrive, and plug it into several end-to-end driving models, such as LTF, DiffusionDrive, DrivoR, and ChainFlow on Navsim and TransFusion-v6 on Bench2Drive, showing consistent improvements. Furthermore, they show that pretraining with more diverse data leads to better downstream performance.

Critical strengths

The work studies an interesting problem: pretraining visual representations for end-to-end driving models. The authors demonstrate that they can learn strong visual representations that lead to consistent performance improvements on end-to-end driving models while using only driving videos (without action annotations).

Critical weaknesses
  • The work lacks technical novelty. It mainly applies the V-JEPA 2 algorithm on driving video, with the only difference being that they also use full-frame masking. However, this modification (1) is very incremental and (2), as seen in Figure 2(a), it offers only a small improvement (increases the PDMS from 93.61 to 93.79), which is not even clear if its statistically significant or consistent across all planners. Moreover, the finding that pre-training with more diverse data improves downstream perfromance, although it is nice to be confirmed in this case, it is somewhat expected.
  • I find two issues with the experimental results:
    • First, in the results presented in Tables 2-5 it is unclear whether the ScaleDrive can be used as a stand-alone visual backbone. What I mean is that for Drivor and ChainFlow, the ScaleDrive encoder provides only the front-camera visual features while in TFv6, LTF and DiffusionDrive, it is used as an additional visual encoder, on top of what is already there. Is ScaleDrive useful when used in stand-alone fashion in these experiment? Also, it is not clear how the results in Tables 1 and 2 are produced: is there the ScaleDrive encoder used in stand-alone way, or together with other visual encoders?
    • It only compares against V-JEPPA 2, and lacks comparison with other state-of-the-art visual encoders, and more specifically DINOv3. How do they results changes if in Tables 1-5 you replace the ScaleDrive with DINOv3?
    • To summarize, from the experimental results it is not clear whether ScaleDrive can be used as stand-alone visual encoder and if it is a better alternative to DINOv3.
Questions for authors
  • Currently, it is unclear whether the small improvement that full frame masking offers (Figure 2-a) is statistically significant and consistent across all planners. Does it also improve when used with other planners (e.g., ChainFlow, LTF, DiffusionDrive, and TFv6)?
  • Clarify whether in Tables 1-2, ScaleDrive is used in stand-alone fashion.
  • Why in Tables 3 and 5 ScaleDrive was not used in stand-alone fashion? How do these results change when ScaleDrive is used in stand-alone way (i.e., when you replace the existing image or video encoder with ScaleDrive)? I do not expect results for each planner-benchmark combination; a few combinations would be enough.
  • Compare with DINOv3.
1 comment
AuthorsOpenReview ↗

Response to Reviewer#uT1k

We are glad that the strengths (pre-trained visual representation for autonomous driving is an interesting problem and the consistent performance improvement) of our work can be acknowledged by the reviewer. We thank the reviewer for the constructive suggestions and valuable time. Detailed responses are as follows.

Q1. Temporal-slice masking gains across planners: We thank the reviewer for this suggestion. We evaluated patch-only and patch w. temporal-slice masking across different planners and computed the gains. These results support the consistency of the benefit across planner architectures. We added these results in the revised manuscript Table 15.

PlannerMetricPatch-onlyPatch + TemporalGain
LTFPDMS85.6486.20+0.56
DiffusionDrivePDMS87.6688.11+0.45
ChainFlowPDMS92.9193.24+0.33
TFv6DS95.8596.65+0.80

Q2 Stand-alone fashion ScaleDrive We apologize that this was not clear. In Tables 1–2, the ScaleDrive encoder is used as the sole visual encoder, and no additional visual encoder is used. These results therefore evaluate the effectiveness of its representation in a stand-alone fashion. The code for the corresponding model configurations and the ScaleDrive model is provided in the Supplementary Material.

Q3 Evaluating other method with stand-alone fashion. Our original experiments extend each planner’s visual representation to evaluate whether ScaleDrive provides complementary spatiotemporal information that improves existing planning systems. To assess its effectiveness as the sole visual encoder, we additionally replace the original ImageNet-pretrained visual encoders in LTF and DiffusionDrive with ScaleDrive. We construct the BEV representation using ScaleDrive features from the three front cameras, while retaining each planner’s original trajectory prediction head and training objectives.

PlannerVisual representationNCDACTTCEPPDMSΔPDMS vs. bsl
LTFBEV97.492.892.479.083.8-
LTFScaleDrive stand-alone98.494.397.781.486.5+2.7
DiffusionDriveBEV98.194.894.181.186.6-
DiffusionDriveScaleDrive stand-alone98.995.395.782.087.9+1.3

We further evaluate stand-alone ScaleDrive with LTF on closed-loop simulation HUGSIM to assess its effectiveness across benchmarks. The results are summarized below. E, M, H, and X denote Easy, Medium, Hard, and Extreme, respectively.

PlannerVisualRC (E / M / H / X)RC Avg.HD-Score (E / M / H / X)HD-Score Avg.
LTFBEV0.684 / 0.407 / 0.369 / 0.2550.4100.528 / 0.246 / 0.198 / 0.0810.246
LTFScaleDrive stand-alone0.781 / 0.408 / 0.293 / 0.2870.4220.636 / 0.262 / 0.126 / 0.1300.270

Q4. Comparison with DINOv3. We thank the reviewer for suggesting this comparison. We additionally evaluate DINOv3 using DrivoR and ChainFlow. For each planner, we replace ScaleDrive with DINOv3 for comparison. DINOv2*: Reproduced baselines.

PlannerDINOv2*DINOv3ScaleDrive
DrivoR92.5892.7493.37
ChainFlow92.6292.8693.24
Reviewer_hg19OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ScaleDrive, which uses self-supervised latent-prediction training for the V-JEPA2 encoder on the the OpenDV-YouTube dataset. Complementary temporal-slice masking and conventional patch-level spatial masking are used in the model. Experiments on planning, cross-policy transfer, cross-scenario transfer are conducted to evaluate the performance.

Critical strengths
  1. Comprehensive transfer evaluation. The paper tested on various planner architectures and driving evaluation environments, providing convincing evidence.
  2. Promising results: The paper reports consistent improvements, especially the cross-policy transfer in Table 3.
Critical weaknesses
  1. The major contribution should be elaborated: Drive-JEPA already initializes from V-JEPA2 and uses driving videos for self-supervised pretraining, while LFG [1] studies representation transfer from unposed YouTube driving videos using teacher supervision.
  2. Can the authors report the runtime and memory usage for the proposed method? It would enhance the paper to provide a comparison with other methods. [1] Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos, https://arxiv.org/abs/2602.22091
Questions for authors
  1. LFG [1] from CVPR 2026, is also trained on the OpenDV-YouTube. Can the authors include a comparison against them?
  2. What is the unit for Efficiency in Table 5?
3 comments
AuthorsOpenReview ↗

Response to Reviewer#hg19

We are glad that the strengths (The comprehensive transfer evaluation of ScaleDrive representation and the promising results) of our work can be acknowledged by the reviewer. We thank the reviewer for the constructive suggestions and valuable time. Detailed responses are as follows.

Q1 & W1 Comparison with previous methods. We thank the reviewer for this suggestion. LFG was cited in the original Introduction (L051). We provide a detailed comparison and summarize the differences below:

AspectLFG(CVPR 2026)ScaleDrive (ours)
OpenDV data~2M samples6M clips from 1,156 source videos (~866 raw-video hours)
Initializationπ³ model with a DINOv2V-JEPA2
Parameters1.45B0.33B
Training targetsPoint maps, camera poses, confidence, semantic maps, and motion masksMasked video latents
External pseudo-labelπ³, SegFormer, Grounded SAM2, CoTracker3No pseudo-label

The previous studies address different research questions. Drive-JEPA investigates how driving-video JEPA pretraining, together with multimodal trajectory distillation, benefits an end-to-end planning framework. LFG investigates whether large-scale, in-the-wild driving videos can yield planning-relevant representations with geometry, semantic, and motion pseudo-supervision. ScaleDrive investigates whether increasing the scale of action-free driving-video pretraining, using masked latent prediction without externally generated task-specific pseudo-labels, improves visual representations and their transfer across multiple planning architectures and benchmarks. We added the clarifications in the revised Section 2 and B.2.

Q2 Detailed definition of Efficiency metric We apologize for not specifying the unit in Table 5. Efficiency is reported as a percentage (%), following the official Bench2Drive definition. At each route checkpoint, it measures the ego vehicle’s speed relative to the average speed of nearby vehicles, and the reported value averages these percentages across test routes.

W2 runtime and memory usage Thank you for suggesting this evaluation. On an NVIDIA A800 GPU, batch_size=1 inference takes approximately 110.1 ms, with 73.4 ms spent on the visual encoder and 36.7 ms on the trajectory prediction. The measured GPU memory usage is 1.37 GiB.

Thank you for the clarification and additional runtime results. Overall, the rebuttal helps me better understand the paper. I am raising the score to 3, but remain borderline: although the authors claim that DriveJEPA and LFG address different research question, the ultimate goal is to improve the planning, and both can be evaluated on NAVSIM.

AuthorsOpenReview ↗

Thank you for your constructive feedback

Dear Reviewer,

We sincerely thank you for the time and effort you have dedicated to reviewing our paper and really appreciate your valuable suggestions. Your suggestions have helped us clarify that ScaleDrive improves representation transfer across planners and benchmarks. We are pleased that our rebuttal helped you better understand this paper, and we appreciate your decision to raise your score. We have incorporated the additional clarifications and uploaded the updated PDF.

Best regards,

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer#vQMH

We are glad that the strengths (This representation learning about action-free video is meaningful for autonomous driving, and the broad evaluation strengthens the empirical contribution) of our work can be acknowledged by the reviewer. We thank the reviewer for the constructive suggestions and valuable time. Detailed responses are as follows.

Q1. Novelty: We thank the reviewer for pointing out UI-JEPA and apologize for omitting this relevant work. UI-JEPA and ScaleDrive share the motivation of using complete temporal masking to strengthen temporal modeling. Our focus is on analyzing and evaluating this design in action-free driving-video pretraining. We use Propositions 1 and 2 to characterize a condition under which patch-only masking can admit a same-frame shortcut, and explain why masking an entire temporal slice encourages more reliance on temporal context. In the revised Introduction (L047 and L082), we cite UI-JEPA and clarify that we contribute to a detailed analysis of this design and evaluate its effectiveness in action-free driving-video pretraining.

Q2. Controlled masking ablations: We apologize for not clarifying that the overall masking ratios were not matched in the original comparison. We conducted an additional experiment by increasing the patch-only masking ratio to match the overall ratio of the configuration with the used temporal masking setting. This yields a PDMS of 93.52, compared with 93.61 for the original patch-only configuration. At the same overall masking ratio, temporal-slice masking improves PDMS by 0.27 points. These results suggest that the observed gain is not explained solely by an increased masking ratio.

Masking configurationNCDACEPTTCPDMS
Patch-only (original ratio)99.298.390.796.693.61
Patch-only (ratio matched to Patch + Temporal)99.298.389.996.993.52
Patch + Temporal99.398.490.697.093.79

Q3. Construction of the scaling datasets: The 0.5M, 1M, 3M, and 6M subsets are strictly nested at the source-video level. At each scale, we retain all videos from the smaller subset and add new source videos. Thus, both the number of unique source videos and their raw-video hours increase at every step as follows. We report the corresponding unique video counts and raw-video hours in Table 6 of the revised manuscript.

clipsUnique videosRaw-video hoursNewly added videos
0.5M102~72—
1M197~14495
3M587~467390
6M1,156~866569

To further characterize the source-video pool, we sampled 100 raw driving videos and annotated their geographic region, time of day, and driving scene. The sample covers multiple regions, daytime, dusk, and nighttime recordings, as well as urban, highway, town, and rural settings. This descriptive analysis provides an overview of the diversity of the source videos.

RegionPercentage
North America52%
Asia30%
Europe14%
South America4%
TimePercentage
Daytime87%
Dusk3%
Nighttime10%
ScenePercentage
Urban33%
Highway24%
Town23%
Rural20%

Q4. Scaling behavior of the complete method: We thank the reviewer for suggesting this additional evaluation. We have extended the scaling experiments with temporal-slice masking to the 1M and 3M pretraining subsets. As shown below, PDMS increases consistently with data scale, and temporal-slice masking provides positive gains over patch-only masking at different scales. These results demonstrate that the scaling trend also holds when temporal-slice masking is included. We added these results in the revised manuscript Table 14.

Scale with temporal-slice maskingNCDACEPTTCPDMSΔPDMS vs. patch-only
1M99.3198.1489.5697.1893.33+0.28
3M99.2898.3289.9597.0193.51+0.21
6M99.3098.3690.6396.9993.79+0.18

I sincerely thank the authors for their thoughtful and detailed responses, which have helped me better understand the paper and its contributions. I appreciate the authors' efforts in addressing my concerns and conducting additional experiments. I now find the experimental evaluation to be comprehensive, the motivation to be well-justified, and the proposed method to be well-aligned with its motivation. While I acknowledge the merits of this work, I feel that the methodological novelty may not be sufficiently substantial to warrant a score of 4. Nevertheless, considering the authors' clarifications and additional results, I am happy to raise my score from 2 to 3.

AuthorsOpenReview ↗

Thank you for your constructive feedback

Dear Reviewer,

We sincerely thank you for the time and effort you have dedicated to reviewing our paper and really appreciate your valuable suggestions. Your suggestions have helped us improve the comprehensive evaluation and the clarity of this paper. We are very grateful for your decision to raise the rating. We have incorporated the additional experiments and clarifications and have uploaded the updated PDF.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.