acceptodds
Under review as a conference paper at ICLR 2027

From World States to Videos: V-JEPA for Driving Scene Generation

Abstract

Predicting a driving video is not only a problem of extending pixels: the generator must decide how traffic, road geometry, and camera motion evolve after the last observed frame. We ask whether a learned world state can make that evolution explicit. A compact Transformer predicts a future-oriented V-JEPA state from observed video, and a latent-video generator uses the state to synthesize the future. Neither future images nor motion metadata are available at inference. Across 130 unseen nuScenes scenes, predicted states improve a fixed semantic–geometric–motion measure by 6.88% over context-only generation in every seed, retaining more than 95% of the benefit of a teacher state extracted from the logged future. A matched predictive pathway built on VideoMAE uses the same predictor and generator budgets, yet predicted V-JEPA states perform 4.24% better. The gain spans all evaluator components and aggregation rules. Motion readouts explain why this is plausible: V-JEPA exposes temporal dynamics more clearly than VideoMAE on two scene cohorts. Oracle-conditioning experiments then show that the generator uses this information and degrades when state and scene no longer correspond. Together, these results support a context-to-state-to-video path in which predicted world states guide future-video synthesis.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
V-JEPA, video generation, world-state prediction, driving-scene generation, predictive representation learning, latent video diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7GzfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether V-JEPA representations can guide future driving-video generation. A frozen V-JEPA encoder defines feature targets from future videos, and a separate Transformer predicts these features from four observed frames. A latent-video diffusion generator, trained with oracle future features, then generates sixteen future frames using predicted features without further fine-tuning. Experiments on nuScenes examine ego-translation accessibility, oracle feature conditioning, and transfer to predicted conditioning. The authors report improvements over context-only generation and a matched VideoMAE pathway using a semantic–depth–flow error aggregate.

Critical strengths
  1. The experiments connect representation analysis, oracle conditioning, and context-to-future feature prediction. Replacing oracle features with predicted features while freezing the generator is a useful test of whether the conditioning interface transfers.

  2. The paper explicitly identifies evaluator dependence, limited experimental scope, and the inconclusive predicted-state swap result.

Critical weaknesses
  1. The manuscript emphasizes relative error reductions rather than absolute generation metrics. This makes the practical significance of the improvements difficult to assess. Figure 5 shows substantial visual degradation across methods, even with oracle conditioning. I cannot distinguish the difference among these output honestly and it looks that the model does not cover. The supplementary numerical artifact reports approximately 4.10% pseudo-mIoU for context-only generation and 5.24% for predicted V-JEPA, suggesting that absolute semantic agreement remains low.

  2. Swapping predicted states between scenes produces a 1.68% degradation with a confidence interval spanning zero. The value seems larger than that of VideoMAE. This does not establish that the generator requires the correct scene’s predicted state. Likewise, retaining 95.14% of the oracle’s metric gain does not demonstrate comparable recovery of future information.

  3. Comparison coverage and generalization are limited. Generation experiments use one small generator family, one dataset, one camera, and a four-second horizon. There is no experimental comparison with an established driving-video or representation-guided generation pipeline, nor evidence that the advantage survives a stronger context-conditioned generator. Although the author mention it in the limitation, it is still a significant weakness.

  4. The representation analysis supports a narrower conclusion than comprehensive dynamics understanding. The probe recovers planar ego translation from features of the actual future video; it does not forecast motion from past observations. Rotation and surrounding-agent motion are not evaluated. Temporal perturbations demonstrate sensitivity to input order but do not uniquely establish physical understanding. Furthermore, comparing pretrained V-JEPA and VideoMAE checkpoints does not isolate their pretraining objectives from differences in data and training history.

Questions for authors

Please refer to the weakness section and also answer the following questions:

  1. Can you report absolute semantic, depth, and flow metrics in the manuscript, together with full video examples from the predicted-state pipeline? How should readers interpret the low absolute pseudo-mIoU values and degraded generations in Figure 5?

  2. Does the proposed predictor outperform direct V-JEPA context conditioning, persistence of the final context state, and constant or mean-state conditioning under comparable budgets?

1 comment
AuthorsOpenReview ↗

Absolute Metrics and the Evidence for Predictive Conditioning

Thank you for recognizing the frozen-generator oracle-to-predicted transition as a useful experimental test. This is our central contribution: predicted features retain generative value without adapting the renderer to their errors.

W1 / Q1: Absolute measurements

The absolute values below come from the numerical evidence. They are medians of seed-wise source-scene means, unscaled.

ConditioningPseudo-mIoU (higher)Aligned depth AbsRel (lower)Flow EPE / image diagonal (lower)
Context-only0.041021.1770.02004
Predicted VideoMAE0.042720.4650.01976
Predicted V-JEPA0.052418.6680.01932

Pseudo-mIoU compares fixed-model semantic predictions on generated and logged video. The low values indicate limited absolute agreement; statistical improvement alone does not establish practically sufficient fidelity.

The comparative result nevertheless holds across distinct measurements. Against context-only, semantic error decreases by 1.24% [0.98%, 1.56%], depth by 14.31% [7.87%, 20.46%], and flow by 3.98% [3.51%, 4.53%]. All three also improve against predicted VideoMAE, with intervals above zero and positive effects in every seed. Excluding depth retains gains of 2.79% [2.46%, 3.10%] and 1.82% [1.64%, 2.08%] against the two baselines. Our conclusion therefore rests on paired measurements, not subjective ranking of Figure 5.

AbsRel values are raw dimensionless ratios. Generated teacher maps are median-aligned on logged-positive support, without flooring positive denominators. Near-zero targets can amplify the ratio; these values are not percentages or ground-truth metric-depth accuracy. Ratios of absolute medians need not equal the reported median of paired seed-wise effects.

Figure 5 illustrates oracle conditioning, not full predicted-state sequences. Full predicted-state videos are not provided with this response, limiting visual assessment. The quantitative predictive comparison covers all 342 test clips and three seeds.

W2: Swaps and gain retention

The 95.14% figure measures retention of downstream gain: predicted-state improvement divided by oracle-state improvement against context-only. It does not measure the fraction of future information recovered. Its significance is operational: predicted states preserve most of the conditioning benefit while every generator parameter remains fixed.

The predicted swap, 1.68% [-0.74%, 4.52%], leaves scene-specific dependence unresolved. This does not negate the separate positive comparisons of 6.88% [4.32%, 9.10%] against context-only and 4.24% [2.44%, 6.23%] against predicted VideoMAE. Comparing swap point estimates also cannot establish stronger correspondence dependence for one representation without uncertainty for their paired difference.

W3 / Q2: Baselines and generalization

Predicted VideoMAE controls for adding an encoder and predictor: both systems process the same four observed frames with the same predictor architecture, update budget, generator capacity, and sampling noise. V-JEPA wins by 3.50%, 4.24%, and 5.48% across the three seeds. Thus, the selected predictive representation pathway matters under matched downstream resources.

Direct-context, persistence, and mean-state conditioning have not been tested. These would distinguish future-state regression from alternative uses of V-JEPA features. The current result establishes superiority over the evaluated baselines, not that prediction is the only way those features can help.

Generation uses one dataset, camera, horizon, and generator family. Larger models and established driving-video pipelines would test external validity and field-level competitiveness. The controlled comparison and frozen-renderer transfer remain meaningful findings within the evaluated regime; their validity does not require a claim of state-of-the-art synthesis.

W4: Representation interpretation

The readout recovers logged planar translation from features of the actual future clip. Forecasting in our generative pathway is performed by the separate context-to-state predictor. V-JEPA ADE is 2.071 m versus 2.942 m on held-out scenes and 2.359 m versus 3.148 m on independent validation, with the same train-fitted readouts reused without refitting. This independently repeats the accessibility advantage within nuScenes; it is not cross-dataset generation validation.

Temporal perturbations diagnose order sensitivity, not a unique physical mechanism. The checkpoint comparison also includes differences in pretraining data and history. Our contribution concerns the selected representations under a shared downstream interface: their difference remains consequential after matched prediction removes future observations from inference.

Reviewer_ScLSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for generating driving-scene videos based on JEPA representation prediction.

Critical strengths

The research question is clearly defined, and the research methodology is straightforward.

Critical weaknesses
  1. Figure 1 is essentially just a mathematical formula; it fails to illustrate any details regarding modular structure or information flow. Consequently, it lacks the substantive content expected of a methodological figure in a research paper and fails to provide sufficient explanatory power.
  2. There are logical flaws in the writing: the paper repeatedly restates the same research motivations and experimental conclusions, yet it fails to provide a detailed presentation of the model design and methodological specifics.
  3. The experimental design is inadequate, the results are insufficiently comprehensive, and both the data and models employed suffer from significant limitations.
Questions for authors
  1. Could the authors provide experimental results with multiple datasets and models to demonstrate the generalization ability of their research methods?
  2. I have serious concerns about whether the authors have fully disclosed their use of artificial intelligence in the preparation of this manuscript in the AI Use Statement section.
2 comments
AuthorsOpenReview ↗

Methodological Details and Experimental Evidence

Thank you for recognizing that the research question is clearly defined and the methodology is straightforward. We address the presentation and experimental scope separately, because limitations of presentation or breadth should be distinguished from the validity of the measured comparison.

W1–W2: Modular structure, information flow, and implementation

The overview figure is too schematic to carry the full methodological explanation. The method has three separately identifiable components:

  1. A frozen video encoder produces context features and, during training, future-feature targets. V-JEPA and VideoMAE outputs are pooled to a common 4 x 8 x 8 grid with 768-dimensional tokens.
  2. A four-layer block-causal Transformer, with width 256 and eight heads, predicts the future state from context features. It is trained by standardized feature MSE for 6,000 AdamW updates. No generator gradients train this predictor.
  3. A 15.66M-parameter latent-video diffusion generator is trained with oracle future features for 8,000 updates. Visual-context latents and projected state features condition the denoiser. At predictive inference, its parameters remain fixed and the predicted state replaces the oracle state. Four observed frames are the input; sixteen future frames are generated, using 25-step DDIM sampling.

The information flow is therefore:

Inference: observed RGB -> frozen encoder -> state predictor -> state-conditioned generator -> future RGB. Observed RGB also supplies the generator's visual-context input through a frozen image VAE. Training: logged future RGB -> frozen encoder -> target for state regression and oracle conditioning for generator training.

This distinction is essential: predictor and generator are trained separately, and the held-out oracle-to-predicted comparison introduces no generator adaptation. The modular diagram should show both context paths, the future-supervision path, and the frozen components explicitly. The repeated motivation can be condensed to give these implementation details greater prominence. These are specific presentation improvements; they do not require changing the experimental result.

W3 / Q1: Experimental evidence and generalization

The contribution is a controlled test of whether predicted representations retain generative value, rather than a claim of universal driving-video performance. The key evidence is a matched comparison: predicted V-JEPA reduces the fixed semantic/depth/flow error aggregate by 6.88% [4.32%, 9.10%] over context-only and by 4.24% [2.44%, 6.23%] over predicted VideoMAE. Both effects are positive in all three seeds. The VideoMAE pathway uses the same predictor architecture and training budget, matched generator capacity, and shared sampling noise.

The unit of uncertainty is the source scene: 342 clips come from 130 held-out scenes, and clips are averaged within scenes before paired resampling. Neither frames nor clips inflate the effective number of independent observations. All six component-wise comparisons favor predicted V-JEPA, and excluding depth retains a positive result against both baselines. The reported advantage is consequently supported beyond a single seed or a single aggregate component.

Representation analysis also reuses the original train-fitted readouts on 149 independent validation scenes: V-JEPA ADE is 2.359 m versus 3.148 m for VideoMAE. This independently repeats the accessibility contrast within nuScenes; it is not presented as cross-dataset generation validation.

We have no multi-dataset or second-generator-family result to report. These are substantive external-validity extensions. Their absence narrows where the observed effect is established, but does not demonstrate a flaw in the paired comparison or in the frozen-generator transfer experiment. We would welcome identification of a specific remaining issue in that design that would invalidate those conclusions.

Scientific support for the paper rests on the documented experiments and reproducible numerical artifacts. The anonymous code package prepared for review includes the matched predictive comparisons and component/aggregation analyses; its validation records report reproduction of the paired results and consistency with the underlying per-clip aggregation. These artifacts provide an inspectable basis for assessing the reported effects.

I thank the authors for their efforts during the rebuttal period. I will maintain my original rating.

Reviewer_fb8xOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a learned world state can make the evolution of a driving scene explicit, so that a generator does not have to extend pixels alone. It tests this on nuScenes front-camera clips, with training, held-out test, and validation scenes kept disjoint, through three linked experiments. First, motion readouts show that V-JEPA exposes temporal dynamics more clearly than a matched VideoMAE: a frozen linear readout recovers the vehicle’s future motion more accurately, and this advantage weakens when frame order is disrupted. Second, oracle-conditioning experiments show that a latent-video generator conditioned on the future-oriented V-JEPA state, extracted from the logged future, synthesizes better futures than with context-only or VideoMAE conditioning. Quality is judged by frozen semantic, geometric, and motion evaluators. The generator degrades when state and scene no longer correspond. Third, a compact Transformer predicts that state from observed video alone, since neither future images nor motion metadata are available at inference. Fed into the frozen generator without retraining, the predicted state retains most of the benefit of the teacher state and still outperforms a matched predictive pathway built on VideoMAE. The gain spans all evaluator components and aggregation rules, supporting a context-to-state-to-video path in which predicted world states guide future-video synthesis.

Critical strengths

S1. The research question itself is relevant to world-model research. The core question is whether a predicted future representation can guide a generator before the future exists. The paper’s story has three linked steps, and each step has its own check. Reversed, shuffled, and repeat-first frames test whether the motion readout depends on temporal order. A state from another scene tests whether the generator uses scene-specific information. A matched VideoMAE pathway, with the same predictor and generator budgets, is run at both the oracle and predicted stages.

S2. The careful statistical treatment appears to be a strength. Every effect is paired at the source-scene level, and frames, clips, and horizons are never treated as independent. Held-out test and independent validation are reported separately and never pooled. Checkpoints are chosen by training loss only. Every seed has the favorable sign, and qualitative examples were selected by a rule fixed before inspection. The score is split into semantic, depth, and flow components. Both contrasts are recomputed under arithmetic, harmonic, leave-one-component-out geometric, and worst-component reductions. The paper is explicit that this reuses the same generations and tests only the paired comparison.

S3. The authors report their limitations honestly. The predicted-state scene swap is reported as not significant. The single-dataset, single-camera scope and the evaluator-relative nature of the aggregate are stated. Overall, the main text is short and can be followed in one pass. Each section states its claim and the evidence for it.

Critical weaknesses

W1. Most of the gain is not scene-specific. The correct oracle state improves on context-only generation, but a state from an unrelated scene gives up only a small part of that gain. A wrong-scene state therefore still beats context-only by a large margin. For predicted states, the swap effect is indistinguishable from zero, so there is no evidence that the generator uses scene-specific content of the predicted state. What the experiments demonstrate is that conditioning a small generator on some vector in V-JEPA space helps.

W2. Baselines that separate a “future-oriented V-JEPA state” from “V-JEPA features of the context” are missing. The predicted pathway adds an encoder pass over the context and a Transformer on top of it, and the context-only generator gets neither. It seems the retention of the oracle gain is ambiguous.

W3. The generator is small and unrealistic. A small latent diffusion model trained briefly on a small number of clips is far from any generator practitioners would use for driving video. The generated frames are blurry enough that the evaluator teachers run well outside their training distribution. Whether the conclusions hold for a pretrained video model fine-tuned with state conditioning is untested, and the generic component in W1 is what one would expect from an under-trained context-only baseline. A training-length or capacity sweep for the context-only model, or one experiment with a pretrained generator, is needed before the claims generalize beyond this setting.

W4. The evaluation rests on a custom teacher-based aggregate, and no standard metric is reported. Depth error is the dominant term, and removing it shrinks the aggregate gain substantially. Writing that the gain “depends on neither one dominant term” is not accurate. The sign survives, but the magnitude is mostly depth. Teachers applied to blurry frames may partly measure sharpness.

W5. The temporal-perturbation result does not clearly show what it is said to show. Reversal, which flips motion direction, barely hurts V-JEPA, while shuffling, which makes inputs out-of-distribution, hurts it a lot. That pattern is more consistent with the readout capturing speed magnitude and scene layout, plus greater OOD sensitivity of V-JEPA features, than with “temporal organization” of motion.

Questions for authors

Q1. Does the context-only gap shrink when the context-only generator is trained 2-4x longer or with more capacity?

Q2. What does the frozen oracle generator produce when conditioned on the V-JEPA encoding of the context (tiled or last-chunk repeat) instead of the predicted future state?

Q3. How does conditioning on the logged or predicted ego trajectory compare with the V-JEPA state?

Q4. Why does reversal barely degrade the V-JEPA readout while shuffling degrades it strongly, if the signal is temporal motion rather than speed magnitude and layout?

Q5. Can one experiment be run with a small pretrained open video generator fine-tuned with and without the state?

3 comments
AuthorsOpenReview ↗

Matched Comparisons, State Specificity, and Evaluator Robustness

Thank you for recognizing the matched comparisons and scene-level statistics. The central distinction is between usefulness of predicted conditioning, the effect of representation choice, and the necessity of correct scene identity. Our results support the first two; predicted-state swaps leave the third open.

W1: Scene specificity

The predicted swap estimate is 1.68% [95% CI: -0.74%, 4.52%]. This establishes neither scene-specific dependence nor equivalence of correct and swapped states. The oracle swap does show correspondence sensitivity: 2.47% degradation [0.29%, 5.16%].

A separate controlled result goes beyond adding extra conditioning: predicted V-JEPA beats predicted VideoMAE by 4.24% [2.44%, 6.23%]. Both receive a context encoder and the same predictor architecture and budget, with matched generator capacity and sampling. Representation choice therefore matters in this setting, even though its scene-specific and shared contributions remain unresolved. Arbitrary V-JEPA vectors were not tested.

W2 / Q2: Context processing and retention

The matched VideoMAE pathway addresses the extra-computation objection: both pathways use four observed frames and identically configured four-layer predictors trained for 6,000 updates. V-JEPA wins in all three seeds: 3.50%, 4.24%, and 5.48%. An extra predictor on only one side cannot explain this comparison.

Direct-context, persistence, and constant-state controls have not been evaluated. They would isolate future-state regression from other uses of V-JEPA features. Our claim concerns the tested predictive pathway, not every possible conditioning strategy.

The 95.14% figure is predicted-state gain divided by oracle-state gain against context-only. The generator is unchanged between these inputs. This measures transfer through an oracle-trained interface, not the percentage of future information recovered.

W3 / Q1 / Q5: Generator strength

The compact generator provides a controlled representation test. At oracle-to-predicted transfer, generator retraining cannot compensate for predictor errors. Across representations, downstream architecture, optimization budget, seed-wise initialization, and sampling noise are matched.

We have not run 2-4x training, capacity sweeps, or a pretrained-generator comparison. These would test external validity. However, possible under-training of context-only is not an established explanation for V-JEPA's advantage over predicted VideoMAE, which also has learned predictive conditioning. Limited scale bounds the result; it does not negate the matched comparison.

W4: Metrics and depth dominance

Depth contributes most to the magnitude. The precise robustness finding is that the direction survives every tested component exclusion and aggregation rule, while magnitude depends on the components.

Removing depth retains gains of 2.79% [2.46%, 3.10%] over context and 1.82% [1.64%, 2.08%] over predicted VideoMAE, positive in every seed. Semantic and flow also improve individually against both baselines. Thus, the positive result does not require depth. Because the aggregate uses dimensionless error ratios, raw AbsRel's larger numerical scale does not itself increase its weight.

Fixed teachers do not eliminate domain bias. An existing VBench evaluation is mixed: motion-smoothness differences are +0.006907 and +0.004021 against context and predicted VideoMAE, with both intervals above zero; dynamic-degree differences are +0.012821 [-0.007692, 0.029487] and -0.020513 [-0.052564, 0.002564]. These are absolute score differences. Reduced motion remains a possible contributor to smoothness, so we do not use this as evidence of general video-quality improvement.

W5 / Q4: Temporal controls

Reversal raises V-JEPA ADE from 2.071 to 3.237 m; shuffling raises it to 13.197 m, and repeat-first to 12.484 m. Reversal is less damaging than shuffling but causes appreciable degradation.

Reversal preserves the set of frames, so the change contradicts a purely permutation-invariant summary. It does not uniquely separate motion direction, speed, layout, and OOD sensitivity. Preserved neighboring-frame adjacency could explain why reversal is less disruptive than shuffling; this is a hypothesis. The supported diagnostic is order sensitivity, while the positive generation comparisons stand independently.

Q3: Trajectory conditioning

Logged- and predicted-trajectory conditioning have not been tested in the matched experiment. Logged trajectories supply privileged future information; predicted trajectories require a different predictor and state bottleneck. Our demonstrated result uses observed RGB alone to supply useful conditioning to a fixed oracle-trained generator.

Reviewer_fb8xOpenReview ↗

Response to Authors (5115)

I thank the authors for their detailed response. I hope it improves the paper.

Your response appropriately corrects my overstatement in W1, but the scene-specific contribution has not yet been isolated and demonstrated. The matched VideoMAE comparison establishes the importance of representation choice, not scene specificity.

The central issue in W2 is also not whether V-JEPA is better than VideoMAE. The experiments does not isolate the benefit of future-state regression over direct V-JEPA context conditioning.

I understand your explanation regarding the current experimental results, but I would appreciate a more proactive discussion of how generalizability could be assessed to address W3.

My remaining concern in W4 is the appropriateness of the evaluation methodology. As I am less familiar with this area, could you clarify whether similar teacher-based aggregate metrics have been used in prior studies and how their validity has been established?

Regarding W5, I agree that the results demonstrate order sensitivity, but do not uniquely identify the encoded motion information.

Since ICLR allows revised manuscripts to be uploaded, I would appreciate the opportunity to read an updated version incorporating these changes during the rebuttal period.

AuthorsOpenReview ↗

Evaluation Precedents and Manuscript Revision

Thank you for the constructive follow-up. Our demonstrated result is transfer of useful predicted conditioning into a fixed renderer; its mechanism requires further controls.

W1, W2, and W5: Distinct conclusions

We agree that the matched VideoMAE result establishes representation-pathway dependence, not scene specificity. We also agree that it does not isolate future-state regression from direct V-JEPA context conditioning. The revision now distinguishes these questions explicitly.

The positive result remains that predicted V-JEPA improves over the evaluated baselines while retaining most of the oracle gain without generator adaptation. Temporal controls support order sensitivity; they do not uniquely identify the encoded motion variables.

To isolate regression, the informative comparison is predicted state versus pooled context, last-state persistence, and train-mean conditioning. Testing all inputs in the same frozen renderer measures interface transfer. A complementary matched-training direct-context baseline is needed because context features may be out of distribution for an oracle-trained renderer; a poor frozen-renderer result alone would not establish the necessity of prediction.

W3: A concrete generalization assessment

We propose assessing three axes separately:

  1. Optimization and capacity: compare context-only and state-conditioned systems at 1x, 2x, and 4x training budgets, then repeat with a pretrained video generator using matched fine-tuning resources and identical observed-frame access. This tests whether conditioning helps only in the compact, limited-training regime.
  2. Data transfer: freeze the nuScenes-trained pipeline and evaluate on a separate driving dataset with aligned camera, frame-rate, and horizon conventions. Report zero-shot transfer separately from matched target-domain adaptation. Within nuScenes, report weather and lighting strata with uncertainty.
  3. Evaluation transfer: repeat the paired comparison with alternative semantic/depth/flow evaluators and a blinded human study that separates scene consistency from visual fidelity. Check whether method rankings agree across these instruments.

These proposed experiments retain scene-level uncertainty and paired sampling; they have not yet been run.

W4: Precedent and validity of model-based evaluation

Yes, there is precedent for using pretrained models as measurement instruments:

VBench (CVPR 2024) uses model-based dimensions, including RAFT for dynamic degree and interpolation-based motion smoothness. It validates per-dimension scores against human pairwise preferences (Sec. 4.2) [1]. T2V-CompBench (CVPR 2025) uses GroundingDINO, SAM, Depth Anything, and tracking for structured video evaluation. Its metric-selection study compares automatic scores with human judgments using Kendall and Spearman correlations (Secs. 4-5) [2]. WorldScore (ICCV 2025) aggregates normalized controllability, quality, and dynamics measurements. It uses flow/segmentation-based diagnostics and validates the combination used for its subjective-quality component through a human preference study (Sec. 3.3; Appendix C.4) [3].

These are precedents for the measurement approach, not prior validation of our exact semantic-depth-flow geometric aggregate. Their human-alignment evidence also does not automatically transfer to our generated driving videos.

Our endpoint asks a narrower, reference-based question: how closely do generated futures match the logged future in the outputs of fixed evaluators? The aggregate summarizes dimensionless component error ratios with equal log-space weight. It is not a calibrated perceptual-quality scale.

Evidence for our comparative conclusion is available without that aggregate: semantic, depth, and flow errors each favor predicted V-JEPA against both baselines, with positive effects in every seed and paired intervals above zero. Removing depth retains gains of 2.79% over context and 1.82% over predicted VideoMAE. These checks establish robustness of the comparison, not external validity of the teachers. Human alignment and evaluator substitution are the specific outstanding validation steps.

Manuscript revision

The revised manuscript incorporates these distinctions in Sections 6 and 8-10, absolute measurements in Table 4, and a modular method diagram in Figure 1. Section 2 and Appendix E explain evaluation precedents and validity; Section 10 gives the generalization plan. Appendix F reports the existing mixed VBench result. The abstract and conclusion retain the positive measured contribution.

[1] https://openaccess.thecvf.com/content/CVPR2024/papers/Huang_VBench_Comprehensive_Benchmark_Suite_for_Video_Generative_Models_CVPR_2024_paper.pdf [2] https://arxiv.org/abs/2407.14505 [3] https://arxiv.org/abs/2504.00983

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.