- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VR-JEPA predicts task-relevant latent trajectories to guide video generation. It combines localized contrastive-state learning (LCL), which focuses supervision on selected states and tokens, with a skill-routed mixture of experts (SR-MoE). On VBVR-Pro-Bench, it improves the overall score from 50.3% to 56.0%, with stronger gains on ID tasks than on OOD tasks.
Critical strengths
1.Coherent and well-motivated approach. Predicting task-relevant state evolution provides a reasonable way to address the gap between visually plausible generation and task correctness. LCL and SR-MoE contribute complementary supervision and modeling capacity.
2.Useful experimental support. The full model improves overall performance by 5.7 percentage points and ID performance by 10.5 points. Component ablations support both LCL and SR-MoE. The LoRA-only comparison and cross-sample trajectory replacement experiment further support the contribution of trajectory guidance.
Critical weaknesses
Cross-task generalization gains are limited. The OOD score improves only from 45.1% to 46.1%, with mixed category-level results, including a decline in spatial reasoning from 72.2% to 65.2%. The authors acknowledge the smaller OOD benefit. Without uncertainty estimates, these small differences remain difficult to interpret. This limits the breadth of the contribution but does not invalidate the stronger ID results.
Questions for authors
1.Could the authors provide per-task OOD score differences or a paired uncertainty analysis using the existing evaluation results?
2.Clarify the contribution of local selection and weighting. Table 2 supports LCL as a complete component. A simplified comparison retaining the same sample pairs and contrastive loss, but applying supervision uniformly across states and tokens, would help isolate the benefit of the proposed localization mechanism.
3.Verify Figure 2. The rightmost frame of the bottom-right “Reasoning Video” appears to repeat the corresponding “Generated Candidate Video” frame above, showing two purple agents rather than the successful reference’s final state. Please check whether this is an illustration error or clarify its meaning.
5 comments
Response to Reviewer 1PBB (2/3)
(Continued from A2, point 1.) Paired task-level scores on all 50 original OOD tasks, five instances per task.
Higher scores (23 tasks)
| Task | VBVR | VR-JEPA | Task | VBVR | VR-JEPA | Task | VBVR | VR-JEPA |
|---|---|---|---|---|---|---|---|---|
| O-11 | 32.7 | 79.0 | G-240 | 67.1 | 71.3 | O-56 | 50.2 | 51.0 |
| G-202 | 13.3 | 53.7 | O-43 | 65.2 | 69.0 | O-6 | 77.5 | 78.3 |
| O-9 | 56.5 | 85.9 | O-39 | 95.1 | 98.5 | O-5 | 61.4 | 62.0 |
| G-161 | 20.0 | 47.5 | G-140 | 58.8 | 62.0 | O-60 | 21.4 | 21.7 |
| O-62 | 3.3 | 20.2 | O-54 | 32.1 | 35.2 | G-24 | 95.9 | 96.0 |
| G-223 | 49.8 | 62.0 | O-58 | 41.9 | 43.2 | O-2 | 50.2 | 50.3 |
| O-27 | 88.6 | 97.2 | O-64 | 51.6 | 52.9 | G-54 | 1.3 | 1.3 |
| O-65 | 13.7 | 21.6 | O-59 | 14.6 | 15.4 |
Equal scores (14 tasks)
| Task | VBVR | VR-JEPA | Task | VBVR | VR-JEPA | Task | VBVR | VR-JEPA |
|---|---|---|---|---|---|---|---|---|
| G-135 | 0.0 | 0.0 | G-189 | 19.4 | 19.4 | G-222 | 0.0 | 0.0 |
| G-136 | 0.0 | 0.0 | G-206 | 40.0 | 40.0 | G-273 | 40.0 | 40.0 |
| G-160 | 40.0 | 40.0 | G-212 | 20.0 | 20.0 | O-22 | 0.0 | 0.0 |
| G-167 | 0.0 | 0.0 | G-217 | 0.0 | 0.0 | O-49 | 100.0 | 100.0 |
| G-169 | 100.0 | 100.0 | G-218 | 0.0 | 0.0 |
Lower scores (13 tasks)
| Task | VBVR | VR-JEPA | Task | VBVR | VR-JEPA | Task | VBVR | VR-JEPA |
|---|---|---|---|---|---|---|---|---|
| G-248 | 57.3 | 19.7 | G-193 | 83.6 | 69.9 | O-85 | 60.2 | 59.5 |
| G-47 | 64.8 | 36.6 | G-174 | 72.2 | 60.4 | G-221 | 74.4 | 73.9 |
| G-219 | 40.0 | 20.0 | O-46 | 92.4 | 80.9 | O-61 | 35.3 | 35.0 |
| G-247 | 80.0 | 60.0 | G-250 | 3.8 | 1.2 | G-168 | 92.5 | 92.2 |
| G-147 | 78.6 | 58.9 |
-
Generation-Seed Variability.
To assess variability due to generation randomness, we evaluated VR-JEPA on the same original OOD evaluation instances using five generation seeds. The OOD score is 46.2 ± 0.5 (mean ± sample standard deviation). We report the per-seed results and category-level statistics in the table below. All five runs score above the 45.1 obtained in the existing VBVR-Wan2.2 evaluation, providing complementary evidence that the observed advantage over this baseline score is not confined to a single VR-JEPA generation seed.
Method / run OOD Avg. Abstraction Knowledge Perception Spatial reasoning Transformation VBVR-Wan2.2 45.1 35.8 42.7 34.0 72.2 77.7 VR-JEPA (run 1) 47.1 41.4 36.6 39.2 66.2 75.9 VR-JEPA (run 2) 46.1 41.5 42.1 35.2 65.2 77.5 VR-JEPA (run 3) 46.3 43.5 38.2 35.0 64.7 79.4 VR-JEPA (run 4) 45.7 39.7 38.6 35.7 66.1 78.6 VR-JEPA (run 5) 46.0 44.4 34.1 36.6 65.6 72.8 VR-JEPA (five-run mean) 46.2 42.1 37.9 36.3 65.5 76.8 Sample standard deviation 0.5 1.8 2.9 1.7 0.6 2.6
Q2. Local Selection and Weighting in LCL
A3: Thank you for suggesting this comparison. To clarify the contribution of local selection and weighting in LCL, we compared VR-JEPA with Local CL against a Uniform CL variant. Uniform CL applies contrastive supervision uniformly across states and tokens. Both variants use the same positive–negative sample pairs and contrastive loss, including its hyperparameters; they differ only in whether supervision is locally selected and weighted or applied uniformly across states and tokens. We also include the w/o CL (+ SR-MoE) configuration in Table 3 of the revised manuscript as a reference.
On VBVR-Pro-Bench, Local CL scores 65.9 in-domain and 46.1 out-of-domain, compared with 62.0 and 44.9 for Uniform CL. The gains on both splits suggest that selecting and weighting states and tokens improves contrastive supervision. Uniform CL also scores below w/o CL (65.2 in-domain and 45.9 out-of-domain) on both splits. A generated negative can still contain correct states, and an error within a state may be confined to a few tokens. Applying the contrastive loss uniformly can therefore push predictions away from valid parts of the candidate. Together, the results suggest that focusing contrastive supervision on informative discrepancies helps preserve useful shared content while directing learning toward the parts that need correction.
Response to Reviewer 1PBB (3/3)
(Continued from A3.) Local CL versus Uniform CL on VBVR-Pro-Bench, with w/o CL as a reference; all three rows are from Table 3 of the revised manuscript.
| Variant | Overall (0–100) | ID Avg. | ID Abst. | ID Know. | ID Perc. | ID Spat. | ID Trans. | OOD Avg. | OOD Abst. | OOD Know. | OOD Perc. | OOD Spat. | OOD Trans. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| w/o CL | 55.5 | 65.2 | 54.0 | 61.4 | 60.7 | 69.8 | 95.6 | 45.9 | 42.8 | 42.3 | 34.3 | 64.2 | 77.3 |
| Uniform CL | 53.5 | 62.0 | 52.8 | 56.5 | 50.3 | 70.4 | 94.7 | 44.9 | 45.0 | 40.7 | 32.8 | 62.0 | 73.8 |
| Local CL | 56.0 | 65.9 | 56.3 | 63.8 | 61.8 | 69.1 | 91.4 | 46.1 | 41.5 | 42.1 | 35.2 | 65.2 | 77.5 |
Q3. Clarification of the Video Frame in Figure 2
A4: Thank you for your careful attention to detail. We have replaced the rightmost frame in the bottom-right “Reasoning Video” panel of Figure 2 with the correct final frame from the corresponding reasoning video.
Response to Reviewer 1PBB (1/3)
Thank you for your careful review and constructive suggestions. We are encouraged that you recognized our coherent and well-motivated approach, the complementary roles of LCL and SR-MoE, and the experimental support for our performance gains and trajectory guidance. To address the concerns about OOD generalization and variability, we provide expanded OOD and spatial evaluations (A1, points 1–2), paired task-level comparisons (A2, point 1), and results across five generation seeds (A2, point 2). We also assess local selection and weighting through a controlled comparison with uniform supervision (A3), and have corrected the video frame in Figure 2 (A4). We address each point below.
W1 / Q1. OOD Generalization and Uncertainty
W1.1. OOD Generalization and Spatial Reasoning
A1:
Expanded OOD Evaluation.
To examine whether the observed OOD advantage persists across reasoning categories with broader test coverage, we additionally evaluated both methods on 20 new instances for each of the 50 OOD tasks, totaling 1,000 additional instances. Both methods were evaluated on the same instances to enable a paired comparison. On this additional test set, VR-JEPA achieves an OOD average of 45.4, compared with 39.8 for VBVR-Wan2.2, an improvement of 5.6 points. The category-level scores also improve across all five categories. These results provide additional evidence of OOD benefits beyond the original small test set, with positive differences in both the overall average and all five category averages.
Spatial Reasoning Performance.
We first analyzed the original spatial-reasoning results, which motivated us to expand the paired evaluation to reduce sensitivity to individual outcomes. The expanded evaluation provides a more representative assessment and stronger evidence of VR-JEPA’s advantage over the baseline. The original OOD spatial subset contains only six tasks with five instances each, totaling 30 instances. As illustrated in Appendix D.4, Figure 13, VBVR-Pro-scorer can occasionally penalize visually correct reasoning outcomes because of errors in pixel-level object extraction and matching. With such a small test set, individual outcomes, including occasional scoring errors, can noticeably affect the category average. To reduce this sensitivity and reassess spatial reasoning performance, we evaluated both methods on the same 100 instances for each of the six tasks, totaling 600 paired instances. On this larger set, VR-JEPA achieves 73.5, compared with 72.7 for VBVR-Wan2.2. We believe that the original 7.0-point gap may partly reflect limited test coverage and occasional scoring errors, whose influence on the category average can be pronounced when only 30 instances are evaluated.
W1.2 / Q1. Task-Level Performance and Generation-Seed Variability
A2: Thank you for suggesting a closer examination of the OOD results. We provide the requested paired task-level comparisons on the original OOD evaluation set to clarify how the gains are distributed across unseen tasks. We complement this analysis with an evaluation across five generation seeds to assess VR-JEPA’s sensitivity to generation randomness.
Paired Task-Level Comparison.
We provide the complete paired task-level comparison across all 50 unseen tasks on the original OOD test set, with five instances per task, to make the modest average improvement easier to interpret. VR-JEPA improves on 23 tasks and matches the baseline on 14. Most tied scores occur on tasks that appear either particularly difficult or relatively easy for both models on the evaluated instances. At the category level, the average score increases by 5.72 points in abstraction and 1.14 points in perception. These comparisons suggest that task-relevant latent guidance can benefit different types of unseen reasoning problems, while revealing how the magnitude of the benefit varies across tasks.