- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether V-JEPA representations can guide future driving-video generation. A frozen V-JEPA encoder defines feature targets from future videos, and a separate Transformer predicts these features from four observed frames. A latent-video diffusion generator, trained with oracle future features, then generates sixteen future frames using predicted features without further fine-tuning. Experiments on nuScenes examine ego-translation accessibility, oracle feature conditioning, and transfer to predicted conditioning. The authors report improvements over context-only generation and a matched VideoMAE pathway using a semantic–depth–flow error aggregate.
Critical strengths
-
The experiments connect representation analysis, oracle conditioning, and context-to-future feature prediction. Replacing oracle features with predicted features while freezing the generator is a useful test of whether the conditioning interface transfers.
-
The paper explicitly identifies evaluator dependence, limited experimental scope, and the inconclusive predicted-state swap result.
Critical weaknesses
-
The manuscript emphasizes relative error reductions rather than absolute generation metrics. This makes the practical significance of the improvements difficult to assess. Figure 5 shows substantial visual degradation across methods, even with oracle conditioning. I cannot distinguish the difference among these output honestly and it looks that the model does not cover. The supplementary numerical artifact reports approximately 4.10% pseudo-mIoU for context-only generation and 5.24% for predicted V-JEPA, suggesting that absolute semantic agreement remains low.
-
Swapping predicted states between scenes produces a 1.68% degradation with a confidence interval spanning zero. The value seems larger than that of VideoMAE. This does not establish that the generator requires the correct scene’s predicted state. Likewise, retaining 95.14% of the oracle’s metric gain does not demonstrate comparable recovery of future information.
-
Comparison coverage and generalization are limited. Generation experiments use one small generator family, one dataset, one camera, and a four-second horizon. There is no experimental comparison with an established driving-video or representation-guided generation pipeline, nor evidence that the advantage survives a stronger context-conditioned generator. Although the author mention it in the limitation, it is still a significant weakness.
-
The representation analysis supports a narrower conclusion than comprehensive dynamics understanding. The probe recovers planar ego translation from features of the actual future video; it does not forecast motion from past observations. Rotation and surrounding-agent motion are not evaluated. Temporal perturbations demonstrate sensitivity to input order but do not uniquely establish physical understanding. Furthermore, comparing pretrained V-JEPA and VideoMAE checkpoints does not isolate their pretraining objectives from differences in data and training history.
Questions for authors
Please refer to the weakness section and also answer the following questions:
-
Can you report absolute semantic, depth, and flow metrics in the manuscript, together with full video examples from the predicted-state pipeline? How should readers interpret the low absolute pseudo-mIoU values and degraded generations in Figure 5?
-
Does the proposed predictor outperform direct V-JEPA context conditioning, persistence of the final context state, and constant or mean-state conditioning under comparable budgets?
Absolute Metrics and the Evidence for Predictive Conditioning
Thank you for recognizing the frozen-generator oracle-to-predicted transition as a useful experimental test. This is our central contribution: predicted features retain generative value without adapting the renderer to their errors.
W1 / Q1: Absolute measurements
The absolute values below come from the numerical evidence. They are medians of seed-wise source-scene means, unscaled.
Pseudo-mIoU compares fixed-model semantic predictions on generated and logged video. The low values indicate limited absolute agreement; statistical improvement alone does not establish practically sufficient fidelity.
The comparative result nevertheless holds across distinct measurements. Against context-only, semantic error decreases by 1.24% [0.98%, 1.56%], depth by 14.31% [7.87%, 20.46%], and flow by 3.98% [3.51%, 4.53%]. All three also improve against predicted VideoMAE, with intervals above zero and positive effects in every seed. Excluding depth retains gains of 2.79% [2.46%, 3.10%] and 1.82% [1.64%, 2.08%] against the two baselines. Our conclusion therefore rests on paired measurements, not subjective ranking of Figure 5.
AbsRel values are raw dimensionless ratios. Generated teacher maps are median-aligned on logged-positive support, without flooring positive denominators. Near-zero targets can amplify the ratio; these values are not percentages or ground-truth metric-depth accuracy. Ratios of absolute medians need not equal the reported median of paired seed-wise effects.
Figure 5 illustrates oracle conditioning, not full predicted-state sequences. Full predicted-state videos are not provided with this response, limiting visual assessment. The quantitative predictive comparison covers all 342 test clips and three seeds.
W2: Swaps and gain retention
The 95.14% figure measures retention of downstream gain: predicted-state improvement divided by oracle-state improvement against context-only. It does not measure the fraction of future information recovered. Its significance is operational: predicted states preserve most of the conditioning benefit while every generator parameter remains fixed.
The predicted swap, 1.68% [-0.74%, 4.52%], leaves scene-specific dependence unresolved. This does not negate the separate positive comparisons of 6.88% [4.32%, 9.10%] against context-only and 4.24% [2.44%, 6.23%] against predicted VideoMAE. Comparing swap point estimates also cannot establish stronger correspondence dependence for one representation without uncertainty for their paired difference.
W3 / Q2: Baselines and generalization
Predicted VideoMAE controls for adding an encoder and predictor: both systems process the same four observed frames with the same predictor architecture, update budget, generator capacity, and sampling noise. V-JEPA wins by 3.50%, 4.24%, and 5.48% across the three seeds. Thus, the selected predictive representation pathway matters under matched downstream resources.
Direct-context, persistence, and mean-state conditioning have not been tested. These would distinguish future-state regression from alternative uses of V-JEPA features. The current result establishes superiority over the evaluated baselines, not that prediction is the only way those features can help.
Generation uses one dataset, camera, horizon, and generator family. Larger models and established driving-video pipelines would test external validity and field-level competitiveness. The controlled comparison and frozen-renderer transfer remain meaningful findings within the evaluated regime; their validity does not require a claim of state-of-the-art synthesis.
W4: Representation interpretation
The readout recovers logged planar translation from features of the actual future clip. Forecasting in our generative pathway is performed by the separate context-to-state predictor. V-JEPA ADE is 2.071 m versus 2.942 m on held-out scenes and 2.359 m versus 3.148 m on independent validation, with the same train-fitted readouts reused without refitting. This independently repeats the accessibility advantage within nuScenes; it is not cross-dataset generation validation.
Temporal perturbations diagnose order sensitivity, not a unique physical mechanism. The checkpoint comparison also includes differences in pretraining data and history. Our contribution concerns the selected representations under a shared downstream interface: their difference remains consequential after matched prediction removes future observations from inference.