acceptodds
Under review as a conference paper at ICLR 2027

VR-JEPA: Learning Contrastive-State Latent Guidance for Generation-based Video Reasoning

Abstract

Reasoning through video generation offers a promising path toward visual intelligence by modeling latent visual states and their dynamics. However, current video generation models often lack explicit guidance on how these states should evolve, leaving generated trajectories prone to physical and structural inconsistencies that undermine reasoning reliability. While the Video Joint-Embedding Predictive Architecture (V-JEPA) provides rich spatiotemporal priors learned through latent prediction, these general priors do not naturally adapt to the logical reasoning capabilities required for complex visual tasks. To bridge this gap, we propose VR-JEPA, a framework that aligns the V-JEPA predictor with task-specific reasoning logic through localized contrastive-state learning and uses its predicted latent trajectories to guide video generation for visual reasoning. Specifically, (i) we pair successful trajectories with generated alternatives under the same input conditions and use discrepancies in their V-JEPA representations to identify informative states and tokens for localized contrastive supervision. (ii) We further equip the V-JEPA predictor with skill-specific experts trained on anchor-task data, allowing the model to adaptively specialize its shared spatiotemporal priors across diverse cognitive domains. Together with skill-specific experts, this contrastive supervision enables VR-JEPA to predict latent trajectories that provide task-specific logical guidance for video generation. Comprehensive experiments on the large-scale VBVR-Pro-Bench dataset demonstrate that VR-JEPA achieves an relative improvement over cutting-edge generation-based reasoning baselines, significantly mitigating physical artifacts and enhancing logical consistency. Our code and data will be released upon acceptance.

Primary area:
generative models
Keywords:
Visual Reasoning, Joint-Embedding Predictive Architecture, Video Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_1PBBOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VR-JEPA predicts task-relevant latent trajectories to guide video generation. It combines localized contrastive-state learning (LCL), which focuses supervision on selected states and tokens, with a skill-routed mixture of experts (SR-MoE). On VBVR-Pro-Bench, it improves the overall score from 50.3% to 56.0%, with stronger gains on ID tasks than on OOD tasks.

Critical strengths

1.Coherent and well-motivated approach. Predicting task-relevant state evolution provides a reasonable way to address the gap between visually plausible generation and task correctness. LCL and SR-MoE contribute complementary supervision and modeling capacity.

2.Useful experimental support. The full model improves overall performance by 5.7 percentage points and ID performance by 10.5 points. Component ablations support both LCL and SR-MoE. The LoRA-only comparison and cross-sample trajectory replacement experiment further support the contribution of trajectory guidance.

Critical weaknesses

Cross-task generalization gains are limited. The OOD score improves only from 45.1% to 46.1%, with mixed category-level results, including a decline in spatial reasoning from 72.2% to 65.2%. The authors acknowledge the smaller OOD benefit. Without uncertainty estimates, these small differences remain difficult to interpret. This limits the breadth of the contribution but does not invalidate the stronger ID results.

Questions for authors

1.Could the authors provide per-task OOD score differences or a paired uncertainty analysis using the existing evaluation results?

2.Clarify the contribution of local selection and weighting. Table 2 supports LCL as a complete component. A simplified comparison retaining the same sample pairs and contrastive loss, but applying supervision uniformly across states and tokens, would help isolate the benefit of the proposed localization mechanism.

3.Verify Figure 2. The rightmost frame of the bottom-right “Reasoning Video” appears to repeat the corresponding “Generated Candidate Video” frame above, showing two purple agents rather than the successful reference’s final state. Please check whether this is an illustration error or clarify its meaning.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1PBB (1/3)

Thank you for your careful review and constructive suggestions. We are encouraged that you recognized our coherent and well-motivated approach, the complementary roles of LCL and SR-MoE, and the experimental support for our performance gains and trajectory guidance. To address the concerns about OOD generalization and variability, we provide expanded OOD and spatial evaluations (A1, points 1–2), paired task-level comparisons (A2, point 1), and results across five generation seeds (A2, point 2). We also assess local selection and weighting through a controlled comparison with uniform supervision (A3), and have corrected the video frame in Figure 2 (A4). We address each point below.

W1 / Q1. OOD Generalization and Uncertainty

W1.1. OOD Generalization and Spatial Reasoning

A1:

  1. Expanded OOD Evaluation.

    To examine whether the observed OOD advantage persists across reasoning categories with broader test coverage, we additionally evaluated both methods on 20 new instances for each of the 50 OOD tasks, totaling 1,000 additional instances. Both methods were evaluated on the same instances to enable a paired comparison. On this additional test set, VR-JEPA achieves an OOD average of 45.4, compared with 39.8 for VBVR-Wan2.2, an improvement of 5.6 points. The category-level scores also improve across all five categories. These results provide additional evidence of OOD benefits beyond the original small test set, with positive differences in both the overall average and all five category averages.

    MethodOOD Avg.AbstractionKnowledgePerceptionSpatial reasoningTransformation
    VBVR-Wan2.239.842.237.526.067.357.6
    VR-JEPA45.450.143.030.277.259.0
  2. Spatial Reasoning Performance.

    We first analyzed the original spatial-reasoning results, which motivated us to expand the paired evaluation to reduce sensitivity to individual outcomes. The expanded evaluation provides a more representative assessment and stronger evidence of VR-JEPA’s advantage over the baseline. The original OOD spatial subset contains only six tasks with five instances each, totaling 30 instances. As illustrated in Appendix D.4, Figure 13, VBVR-Pro-scorer can occasionally penalize visually correct reasoning outcomes because of errors in pixel-level object extraction and matching. With such a small test set, individual outcomes, including occasional scoring errors, can noticeably affect the category average. To reduce this sensitivity and reassess spatial reasoning performance, we evaluated both methods on the same 100 instances for each of the six tasks, totaling 600 paired instances. On this larger set, VR-JEPA achieves 73.5, compared with 72.7 for VBVR-Wan2.2. We believe that the original 7.0-point gap may partly reflect limited test coverage and occasional scoring errors, whose influence on the category average can be pronounced when only 30 instances are evaluated.

W1.2 / Q1. Task-Level Performance and Generation-Seed Variability

A2: Thank you for suggesting a closer examination of the OOD results. We provide the requested paired task-level comparisons on the original OOD evaluation set to clarify how the gains are distributed across unseen tasks. We complement this analysis with an evaluation across five generation seeds to assess VR-JEPA’s sensitivity to generation randomness.

  1. Paired Task-Level Comparison.

    We provide the complete paired task-level comparison across all 50 unseen tasks on the original OOD test set, with five instances per task, to make the modest average improvement easier to interpret. VR-JEPA improves on 23 tasks and matches the baseline on 14. Most tied scores occur on tasks that appear either particularly difficult or relatively easy for both models on the evaluated instances. At the category level, the average score increases by 5.72 points in abstraction and 1.14 points in perception. These comparisons suggest that task-relevant latent guidance can benefit different types of unseen reasoning problems, while revealing how the magnitude of the benefit varies across tasks.

AuthorsOpenReview ↗

Response to Reviewer 1PBB (2/3)

(Continued from A2, point 1.) Paired task-level scores on all 50 original OOD tasks, five instances per task.

Higher scores (23 tasks)

TaskVBVRVR-JEPATaskVBVRVR-JEPATaskVBVRVR-JEPA
O-1132.779.0G-24067.171.3O-5650.251.0
G-20213.353.7O-4365.269.0O-677.578.3
O-956.585.9O-3995.198.5O-561.462.0
G-16120.047.5G-14058.862.0O-6021.421.7
O-623.320.2O-5432.135.2G-2495.996.0
G-22349.862.0O-5841.943.2O-250.250.3
O-2788.697.2O-6451.652.9G-541.31.3
O-6513.721.6O-5914.615.4

Equal scores (14 tasks)

TaskVBVRVR-JEPATaskVBVRVR-JEPATaskVBVRVR-JEPA
G-1350.00.0G-18919.419.4G-2220.00.0
G-1360.00.0G-20640.040.0G-27340.040.0
G-16040.040.0G-21220.020.0O-220.00.0
G-1670.00.0G-2170.00.0O-49100.0100.0
G-169100.0100.0G-2180.00.0

Lower scores (13 tasks)

TaskVBVRVR-JEPATaskVBVRVR-JEPATaskVBVRVR-JEPA
G-24857.319.7G-19383.669.9O-8560.259.5
G-4764.836.6G-17472.260.4G-22174.473.9
G-21940.020.0O-4692.480.9O-6135.335.0
G-24780.060.0G-2503.81.2G-16892.592.2
G-14778.658.9
  1. Generation-Seed Variability.

    To assess variability due to generation randomness, we evaluated VR-JEPA on the same original OOD evaluation instances using five generation seeds. The OOD score is 46.2 ± 0.5 (mean ± sample standard deviation). We report the per-seed results and category-level statistics in the table below. All five runs score above the 45.1 obtained in the existing VBVR-Wan2.2 evaluation, providing complementary evidence that the observed advantage over this baseline score is not confined to a single VR-JEPA generation seed.

    Method / runOOD Avg.AbstractionKnowledgePerceptionSpatial reasoningTransformation
    VBVR-Wan2.245.135.842.734.072.277.7
    VR-JEPA (run 1)47.141.436.639.266.275.9
    VR-JEPA (run 2)46.141.542.135.265.277.5
    VR-JEPA (run 3)46.343.538.235.064.779.4
    VR-JEPA (run 4)45.739.738.635.766.178.6
    VR-JEPA (run 5)46.044.434.136.665.672.8
    VR-JEPA (five-run mean)46.242.137.936.365.576.8
    Sample standard deviation0.51.82.91.70.62.6

Q2. Local Selection and Weighting in LCL

A3: Thank you for suggesting this comparison. To clarify the contribution of local selection and weighting in LCL, we compared VR-JEPA with Local CL against a Uniform CL variant. Uniform CL applies contrastive supervision uniformly across states and tokens. Both variants use the same positive–negative sample pairs and contrastive loss, including its hyperparameters; they differ only in whether supervision is locally selected and weighted or applied uniformly across states and tokens. We also include the w/o CL (+ SR-MoE) configuration in Table 3 of the revised manuscript as a reference.

On VBVR-Pro-Bench, Local CL scores 65.9 in-domain and 46.1 out-of-domain, compared with 62.0 and 44.9 for Uniform CL. The gains on both splits suggest that selecting and weighting states and tokens improves contrastive supervision. Uniform CL also scores below w/o CL (65.2 in-domain and 45.9 out-of-domain) on both splits. A generated negative can still contain correct states, and an error within a state may be confined to a few tokens. Applying the contrastive loss uniformly can therefore push predictions away from valid parts of the candidate. Together, the results suggest that focusing contrastive supervision on informative discrepancies helps preserve useful shared content while directing learning toward the parts that need correction.

AuthorsOpenReview ↗

Response to Reviewer 1PBB (3/3)

(Continued from A3.) Local CL versus Uniform CL on VBVR-Pro-Bench, with w/o CL as a reference; all three rows are from Table 3 of the revised manuscript.

VariantOverall (0–100)ID Avg.ID Abst.ID Know.ID Perc.ID Spat.ID Trans.OOD Avg.OOD Abst.OOD Know.OOD Perc.OOD Spat.OOD Trans.
w/o CL55.565.254.061.460.769.895.645.942.842.334.364.277.3
Uniform CL53.562.052.856.550.370.494.744.945.040.732.862.073.8
Local CL56.065.956.363.861.869.191.446.141.542.135.265.277.5

Q3. Clarification of the Video Frame in Figure 2

A4: Thank you for your careful attention to detail. We have replaced the rightmost frame in the bottom-right “Reasoning Video” panel of Figure 2 with the correct final frame from the corresponding reasoning video.

Reviewer_SRaNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VR-JEPA, a framework for generation-based visual reasoning. A frozen V-JEPA 2.1 encoder provides visual representations, while a newly trained autoregressive predictor produces instruction-conditioned latent trajectories that guide a VBVR-Wan2.2 video generator. The method introduces two main components: localized contrastive-state learning (LCL), which weights contrastive supervision using feature discrepancies between successful reference videos and generated candidates; and a skill-routed mixture of experts (SR-MoE), whose experts are initialized on manually selected anchor tasks and subsequently combined through learned routing.

Critical strengths

The paper presents a coherent framework that separates task-conditioned latent state prediction from video synthesis. Predicting intermediate states before rendering provides a plausible way to guide generation toward task completion, addressing the distinction between visually plausible motion and correct execution. The evaluation covers a broad range of visual reasoning tasks and uses task-grounded scoring. The authors evaluate 100 tasks, report separate ID/OOD and category-level results, and use rule-based evaluators that assess task completion more directly than generic visual-quality or text–video similarity metrics.

Critical weaknesses
  1. Comparison with related V-JEPA methods is insufficient. Without controlled comparisons to approaches such as WMReward or OMR, the advantage of the proposed rollout-based design over existing JEPA-guidance mechanisms remains unclear.
  2. The skill decomposition is insufficiently justified. Existing routing ablations do not distinguish meaningful skill specialization from additional capacity or staged training. Capacity-matched dense models and alternative expert initialization strategies are missing.
  3. LCL does not reliably distinguish reasoning errors from valid trajectory differences. Feature discrepancies may reflect alternative correct solutions, timing differences, or appearance changes, potentially encouraging reference matching rather than task correctness.
  4. Generalization evidence is limited. Evaluation covers only one benchmark with five instances per task. OOD gains are small, and replacing rollouts across samples barely changes OOD performance, leaving their task-specific contribution uncertain.
  5. The computational trade-off is unclear. Parameter counts alone do not capture candidate generation, expert training, autoregressive prediction, and JEPA encoding costs. End-to-end latency, memory usage, and training compute are not reported.
Questions for authors
  1. How does VR-JEPA compare with existing V-JEPA guidance methods, such as WMReward or OMR, under the same generator, evaluation protocol, and comparable inference budgets?
  2. Does SR-MoE outperform a capacity- and training-budget-matched dense predictor or an MoE without manually assigned skill anchors? How sensitive are the results to anchor-task selection?
  3. What proportion of LCL-selected candidates actually violates task constraints? How does the method distinguish reasoning errors from alternative correct solutions or temporal misalignment?
  4. Why does cross-sample rollout substitution barely affect OOD performance? Can evaluation on an independent benchmark establish whether task-specific latent rollouts improve generalization?
  5. What are the total training GPU-hours, inference latency, and peak memory relative to VBVR-Wan2.2, including candidate generation and JEPA processing? Does sparse routing reduce actual expert computation?
7 comments
AuthorsOpenReview ↗

Response to Reviewer SRaN (1/4)

Thank you for your careful review and constructive suggestions. We are encouraged that you recognized our coherent framework, the rationale for predicting latent states to guide task completion, and our broad evaluation with task-grounded scoring. Our additional comparisons show higher ID and OOD averages than both WMReward variants with lower inference latency under common settings (A1). Controlled comparisons support the contribution of skill-anchor initialization beyond predictor capacity and training budget, with similar performance under the tested anchor-task replacements (A2), while LCL diagnostics link selected states and tokens to task violations and characterize valid trajectory differences (A3). VR-JEPA achieves higher scores than VBVR-Wan2.2 on the expanded test sets and independent benchmarks (A4). Cross-sample rollout comparisons further support the contribution of sample-specific rollout content to reasoning performance (A5). We also document training and inference costs and verify that sparse routing reduces actual expert-adapter computation (A6).

W1 / Q1. Comparison with Existing JEPA Guidance Methods

A1: Thank you for suggesting comparisons with related V-JEPA methods to clarify the advantages of our rollout-based design. As OMR’s code and trained adapter weights are not yet publicly available, we evaluate WMReward using its official implementation.

WMReward guides generation using predictive consistency from a frozen V-JEPA model. We evaluate two strategies: BoN-2, which generates two candidates and selects the one with the lowest prediction error, and gradient guidance, which uses gradients of this error to refine samples during denoising. Both are compared with VR-JEPA using the same VBVR-Wan2.2 base generator and VBVR-Pro-Bench evaluation protocol.

We use a common sampling configuration: one H100 80GB GPU, batch size 1, 512 × 512 resolution, 61 frames, 50 denoising steps, and CFG 5. VR-JEPA achieves higher overall, ID, and OOD average scores than both WMReward variants. Its overall score reaches 56.0, compared with 50.8 for BoN-2 and 50.1 for gradient guidance, with measured generation times of 201 s, 402 s, and 712 s, respectively. These results suggest that task-conditioned latent rollouts provide effective guidance for visual reasoning while achieving higher scores and lower inference latency than the evaluated WMReward strategies.

MethodOverall (0–100)ID Avg.ID Abst.ID Know.ID Perc.ID Spat.ID Trans.OOD Avg.OOD Abst.OOD Know.OOD Perc.OOD Spat.OOD Trans.Time (s) ↓
WMReward (BoN-2)50.857.136.556.153.667.291.744.535.745.132.469.878.6402.0
WMReward (gradient guidance)50.155.734.055.049.867.791.744.535.843.233.269.177.7712.0
VR-JEPA56.065.956.363.861.869.191.446.141.542.135.265.277.5201.0

W2 / Q2. Capacity, Skill-Anchor Initialization, and Anchor-Task Sensitivity

A2:

  1. Capacity and Initialization Comparisons.

    To assess the contribution of skill-anchor initialization, we compared SR-MoE with a capacity-matched dense predictor and an MoE without skill-anchor initialization. The dense control replaces the routed experts and router with a wider dense MLP of matched parameter capacity. The unanchored MoE retains the same seven experts and router. It also undergoes predictor-initialization pretraining on the same data as VR-JEPA. During pretraining, however, it does not restrict training samples from particular tasks to designated experts. This comparison therefore controls for the pretraining data and budget while removing the prescribed task-to-expert assignments. All three configurations use the same number of training steps, amount of training data, and computational budget, with LCL retained throughout. VR-JEPA achieves an overall score of 56.0, compared with 54.2 for the capacity-matched dense predictor and 54.5 for the unanchored MoE, with higher ID and OOD averages in both comparisons. These results support the benefit of skill-guided expert organization under matched capacity and training budgets, suggesting that the gains are not explained by additional capacity or staged training alone.

AuthorsOpenReview ↗

Response to Reviewer SRaN (2/4)

  1. Anchor-Task Sensitivity.

    We further examined whether these benefits depend on the specific anchor tasks used for pretraining. The anchors provide an initial skill structure, while subsequent routing is learned across all training tasks. For expert pretraining, we applied anchor-task replacement to each of the seven experts: one of each expert’s seven anchor tasks was randomly replaced with another representative task of the same reasoning skill, while the other six anchors were retained. The replaced-anchor variant achieves an overall score of 55.9 and ID/OOD averages of 65.6/46.2, close to the original configuration’s 56.0 and 65.9/46.1. Both configurations outperform the dense and unanchored MoE controls on these averages, suggesting that aggregate performance is robust to the tested anchor-task substitution.

VariantOverall (0–100)ID Avg.ID Abst.ID Know.ID Perc.ID Spat.ID Trans.OOD Avg.OOD Abst.OOD Know.OOD Perc.OOD Spat.OOD Trans.
Predictor design comparisons
Matched dense predictor54.263.651.858.459.269.197.444.942.146.132.964.072.4
MoE w/o skill anchors54.564.252.662.555.571.890.944.840.245.731.667.576.5
Sensitivity to anchor-task selection
VR-JEPA (replaced anchors)55.965.654.461.259.872.994.446.240.640.437.862.275.9
VR-JEPA56.065.956.363.861.869.191.446.141.542.135.265.277.5

W3 / Q3. LCL Selection: Task Violations and Valid Trajectory Differences

A3: Thank you for raising these concerns. To examine how LCL’s selections relate to task violations, we analyzed generated candidates from our training dataset. Of the 100 training tasks, 50 have corresponding task-specific scorers in VBVR-Pro-Bench. We randomly sampled 100 generated candidates from each of these 50 tasks, yielding 5,000 candidates. We applied VR-JEPA’s existing LCL selection and weighting rules to these candidates, then assessed the resulting selections at the candidate, state, and token levels.

At the candidate level, 35.14% of generated candidates met our task criterion based on VBVR-Pro-scorer. LCL retained 50.32% of candidates, of which 72.34% did not meet the same criterion. At the state level, 44.28% of evaluated states received nonzero LCL weights. GPT-6.1 Sol assessed weighted states using the task instruction and surrounding trajectory, with a reported violation rate of 66.87%. At the token level, 43.62% of tokens within nonzero-weight states also received nonzero token weights. In states with confirmed, spatially localized violations, 74.03% of weighted token projections overlapped the annotated violation regions. These results suggest that LCL can select states containing task violations and emphasize regions associated with those violations.

To assess how often LCL assigns weight to valid trajectory differences, we analyzed alternative correct solutions and temporal misalignment. Some tasks, such as shortest-path navigation and grouping or sorting, allow different valid routes or operation orders. In our analysis, 0.83% of weighted states were associated with such alternative correct solutions. We also found that 9.20% of weighted states came from fully correct candidates and differed from the reference solely in execution timing. The observed proportions of weighted states associated with valid alternatives or pure timing differences were both relatively small.

For appearance variation, visual reasoning tasks typically require objects to retain their identity and task-relevant visual attributes. For example, a maze agent is expected to preserve its appearance during motion, so a change in its shape violates the task requirements. Discrepancies arising from such changes can therefore help LCL select task-violating states and localize the affected regions. LCL also subtracts a within-state quantile of dense-feature discrepancies before computing local weights to reduce the influence of scene-wide appearance variation.

Together, the selection statistics and analyses of valid trajectory differences provide evidence that LCL can focus supervision on states and regions associated with task violations. Our ablation results further support the practical benefits of this localized supervision for visual reasoning.

AuthorsOpenReview ↗

Response to Reviewer SRaN (3/4)

W4 / Q4. Generalization and Sample-Specific Rollout Guidance

W4.1. Broader Evaluation of Generalization

A4:

  1. Evaluation on Independent Benchmarks.

    To broaden the evaluation beyond VBVR-Pro-Bench, we evaluated VR-JEPA and VBVR-Wan2.2 on two additional benchmarks, VR-Bench and VideoThinkBench, neither of which was used to train VR-JEPA. Our evaluation subsets cover five VR-Bench tasks and two VideoThinkBench tasks, with 100 samples per task. For VR-Bench, Success Rate (SR) measures goal attainment; Precision Rate (PR) measures correct-prefix alignment with the optimal path; Step Deviation (SD) measures relative path-length redundancy; and Maze Fidelity (MF) measures preservation of the static maze layout. For VideoThinkBench, we report accuracy separately for Eyeballing and Visual Puzzles. Compared with VBVR-Wan2.2, VR-JEPA improves VR-Bench SR from 64.40% to 80.00%, Eyeballing accuracy from 45.00% to 54.00%, and Visual Puzzles accuracy from 15.00% to 25.00%. These results provide additional evidence of VR-JEPA’s generalization beyond the original benchmark on the evaluated subsets.

    MethodVR-Bench SR (%) ↑VR-Bench PR (%) ↑VR-Bench SD ↓VR-Bench MF (%) ↑VideoThinkBench Eyeballing Acc. (%) ↑VideoThinkBench Visual Puzzles Acc. (%) ↑
    VBVR-Wan2.264.4011.140.578581.4645.0015.00
    VR-JEPA80.0011.830.541581.5054.0025.00
  2. Expanded ID/OOD Evaluation.

    To complement the evaluation on additional benchmarks, we expanded instance coverage on both the ID and OOD splits of VBVR-Pro-Bench. Each split contains 50 tasks, and we generated 20 additional test instances per task, totaling 1,000 new instances per split. Both methods were evaluated on the same instances for paired comparisons, excluding the original five instances per task. On these additional instances, VR-JEPA achieves an ID average of 67.1, compared with 58.0 for VBVR-Wan2.2, and an OOD average of 45.4, compared with 39.8. VR-JEPA also obtains higher scores across all five reasoning categories in both splits. These results provide complementary evidence that the observed benefits extend beyond the original small test set across both ID and OOD tasks.

    MethodID Avg.ID Abst.ID Know.ID Perc.ID Spat.ID Trans.OOD Avg.OOD Abst.OOD Know.OOD Perc.OOD Spat.OOD Trans.
    VBVR-Wan2.258.040.260.049.764.491.039.842.237.526.067.357.6
    VR-JEPA67.153.266.467.371.191.845.450.143.030.277.259.0

W4.2 / Q4. Sample-Specific Rollout Guidance

A5: To further examine the original OOD result, we extended the cross-sample rollout comparison to the additional OOD instances and independent benchmark subsets described in A4, points 1–2. For each input, we replaced its predicted rollout with that of a randomly selected different sample from the same task, testing the importance of correspondence between the rollout and the current sample.

On the expanded VBVR-Pro-Bench OOD set, cross-sample replacement reduces the average score from 45.4 to 40.4, a decrease of 5.0 points. On VR-Bench, it reduces SR from 80.00% to 70.60%. On VideoThinkBench, Eyeballing and Visual Puzzles accuracy decrease from 54.00% to 45.00% and from 25.00% to 23.00%, respectively. These results provide complementary evidence that matching the predicted rollout to the current sample contributes useful guidance across the evaluated OOD instances and independent benchmark subsets.

MethodExpanded OOD Avg. ↑Expanded OOD Abst. ↑Expanded OOD Know. ↑Expanded OOD Perc. ↑Expanded OOD Spat. ↑Expanded OOD Trans. ↑VR-Bench SR (%) ↑VR-Bench PR (%) ↑VR-Bench SD ↓VR-Bench MF (%) ↑VideoThinkBench Eyeballing Acc. (%) ↑VideoThinkBench Visual Puzzles Acc. (%) ↑
Cross-sample40.444.637.626.067.257.670.6011.030.663281.3945.0023.00
VR-JEPA45.450.143.030.277.259.080.0011.830.541581.5054.0025.00
Reviewer_KGxjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VR-JEPA addresses visual reasoning through video generation: an initial scene and instruction specify the required state changes. It predicts a latent rollout using a frozen V-JEPA 2.1 encoder (Mur-Labadia et al. 2026), then conditions a VBVR-Wan2.2 generator through semantic-memory and modulation paths. Appendix A.2 states that the state pooler and six-layer task predictor are trained for this framework without inheriting pretrained predictor weights. Localized contrastive-state learning (LCL) weights discrepancies between successful and generated trajectories; seven routed experts adapt prediction across skills. On VBVR-Pro-Bench, the reported overall score rises from 50.3 to 56.0, with substantially larger in-domain than out-of-domain gains. Related designs include internal latent hierarchies for visual reasoning (Qian et al. 2026) and frozen-JEPA conditioning for traffic video generation (Nguyen et al. 2026).

Critical strengths
  1. Figure 2 explains how predicted trajectories condition generation, and Appendix Table 6 tests whether the generator uses their sample-specific content. With the initial scene, instruction, and sampling seed fixed, cross-sample rollout replacement reduces the in-domain score from 65.9 to 58.5; removing rollout conditioning reduces it to 51.5. These interventions support the usefulness of the predicted content within familiar task families. Together with the clear training description, they make the proposed interface an informative contribution beyond a single benchmark improvement.

  2. Table 1 improves the in-domain average from 55.4 to 65.9 under the paper’s common preprocessing and scoring pipeline. Table 2 compares rollout-conditioned variants: the dense baseline scores 53.9 overall, LCL alone 55.1, routing alone 55.5, and the full model 56.0. Appendix Table 4 separately examines interface and LoRA tuning. These results support a credible, useful improvement on held-out instances of represented task families. This evidence supports my weak-acceptance recommendation, with the scope of the conclusions limited by the weaknesses below.

Critical weaknesses
  1. Table 1 improves the out-of-domain average by only 1.0 point, from 45.1 to 46.1, while spatial reasoning drops from 72.2 to 65.2. Cross-sample rollout replacement in Appendix Table 6 barely changes the out-of-domain score, from 46.1 to 45.9. Each of the 50 unseen tasks has five evaluated instances, without paired task-level uncertainty or repeated generation seeds. The evidence therefore supports familiar-task improvement more strongly than broad skill transfer. Also, unless an additional binary threshold was applied, “task success” denotes mean rule scores on a percentage scale, rather than the proportion of fully solved problems, because the benchmark permits partial credit (Xu et al. 2026). Clear score semantics and uncertainty would materially sharpen the contribution’s scope.

  2. Section 3.2 acknowledges that generated LCL candidates can satisfy the task. Feature discrepancies can reflect timing or appearance, and Figure 3 does not establish that weighted states and tokens identify rule violations. Appendix Table 5 supports learned routing over uniform or random routing, but Figure 12’s seven selected expert-removal examples do not establish transferable skill specialization. The abstract’s physical-artifact claim likewise lacks a separate physical-consistency measure. These are fixable limits on interpretation: the measured gains remain useful, while claims about error localization, specialist skills, and physical consistency should match the available tests. I consequently regard informativeness and soundness as qualified, despite the positive overall recommendation.

References

Mur-Labadia, Lorenzo, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. 2026. “V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning.” https://arxiv.org/abs/2603.14482.

Nguyen, Trinh Tra Giang, Thanh Nguyen Vo, Nguyen Hoai Thuong Bui, and Ha Duc Bui. 2026. “JEPA Guided Diffusion: Predictive Vision-Language Conditioning for Generative Traffic Forecasting.” https://arxiv.org/abs/2609.21379.

Qian, Zezhong, Xiaowei Chi, Chak-Wing Mak, Tianze Zhou, Ruibin Yuan, Yuhan Rui, Hengzhe Sun, et al. 2026. “Hierarchical Denoising for Multi-Step Visual Reasoning.” https://arxiv.org/abs/2607.15278.

Xu, Junxiang, Ruisi Wang, Fanyi Pu, Maijunxian Wang, Ran Ji, Tongxi Zhou, Chenyang Gu, et al. 2026. “VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning.” https://arxiv.org/abs/2608.26105.

Questions for authors
  1. Were the reported percentages obtained directly by averaging rule scores, or was a binary success threshold applied? Can you provide paired per-task differences for the 50 unseen tasks and, if already available, variability across generation seeds? This would clarify whether the small unseen-task gain is stable; a new seed study is a suggested strengthening experiment if these results do not yet exist.

  2. What fraction of the generated LCL candidates already satisfy the task, and how many states and tokens receive contrastive weight? Does any existing analysis link the selected regions to actual rule violations? These answers could strengthen or narrow the LCL interpretation. Excluding successful candidates is a suggested follow-up experiment if no such analysis is available.

5 comments
AuthorsOpenReview ↗

Response to Reviewer KGxj (1/4)

Thank you for your careful review and constructive suggestions. We are encouraged that you recognized our informative rollout-conditioning interface, the usefulness of sample-specific rollout content within familiar task families, the performance gains from LCL and routing, and our clear training description. To address your concern about limited OOD generalization, we show that VR-JEPA outperforms the baseline on the expanded spatial-reasoning subset and maintains its advantage on the expanded OOD evaluation set (A1, points 2 and 6). We further examine how the gains vary across individual tasks and five generation seeds to clarify their consistency and scope (A1, points 4–5). Cross-sample rollout comparisons on the expanded OOD set, VR-Bench, and VideoThinkBench further support the contribution of sample-specific rollout content (A1, point 3). Our analyses link LCL’s weighted states and tokens to task violations (A3), while the expert-removal analysis provides additional evidence of skill specialization beyond the anchor tasks (A4). Finally, we clarify score semantics and align our claims with the evaluated reasoning capabilities (A2, A5).

W1 / Q1. OOD Evaluation and Score Interpretation

W1.1. OOD Evaluation

A1:

  1. OOD Performance Gains.

    We acknowledge that the original evaluation shows smaller gains on OOD tasks than on ID tasks. Our central goal is to use V-JEPA’s pretrained spatiotemporal priors for task-specific reasoning, using predicted latent trajectories to guide video generation toward task completion. We view this approach as a step toward extending these priors to reasoning beyond familiar task families. To further assess OOD performance, we provide paired task-level comparisons of the original results (point 4), evaluations across generation seeds (point 5), and an expanded evaluation with additional OOD test instances (point 6).

  2. Spatial Reasoning Performance.

    We first analyzed the original spatial-reasoning results, which motivated us to expand the paired evaluation to reduce sensitivity to individual outcomes. The expanded evaluation provides a more representative assessment and stronger evidence of VR-JEPA’s advantage over the baseline. The original OOD spatial subset contains only six tasks with five instances each, totaling 30 instances. As illustrated in Appendix D.4, Figure 13, VBVR-Pro-scorer can occasionally penalize visually correct reasoning outcomes because of errors in pixel-level object extraction and matching. With such a small test set, individual outcomes, including occasional scoring errors, can noticeably affect the category average. To reduce this sensitivity and reassess spatial reasoning performance, we evaluated both methods on the same 100 instances for each of the six tasks, totaling 600 paired instances. On this larger set, VR-JEPA achieves 73.5, compared with 72.7 for VBVR-Wan2.2. We believe that the original 7.0-point gap may partly reflect limited test coverage and occasional scoring errors, whose influence on the category average can be pronounced when only 30 instances are evaluated.

  3. Cross-Sample Rollout Replacement on OOD Tasks.

    To further assess whether matching predicted rollouts to the current sample improves performance on unseen tasks, we conducted additional comparisons on VR-Bench and VideoThinkBench, neither of which was used to train VR-JEPA. We evaluated five VR-Bench tasks and two VideoThinkBench tasks, with 100 samples per task. For each sample, we replaced its predicted rollout with that of a randomly selected different sample from the same task.

    For VR-Bench, we report Success Rate (SR; goal attainment), Precision Rate (PR; correct-prefix alignment with the optimal path), Step Deviation (SD; relative path-length redundancy), and Maze Fidelity (MF; preservation of the static maze layout). For VideoThinkBench, we report accuracy separately for Eyeballing and Visual Puzzles.

    Cross-sample replacement reduces VR-Bench SR from 80.00% to 70.60%, Eyeballing accuracy from 54.00% to 45.00%, and Visual Puzzles accuracy from 25.00% to 23.00%. These results provide complementary evidence that sample-specific rollout correspondence contributes useful guidance on the independent benchmark subsets.

    MethodVR-Bench SR (%) ↑VR-Bench PR (%) ↑VR-Bench SD ↓VR-Bench MF (%) ↑VideoThinkBench Eyeballing Acc. (%) ↑VideoThinkBench Visual Puzzles Acc. (%) ↑
    Cross-sample70.6011.030.663281.3945.0023.00
    VR-JEPA80.0011.830.541581.5054.0025.00
AuthorsOpenReview ↗

Response to Reviewer KGxj (2/4)

We additionally conducted the same cross-sample rollout comparison on the expanded VBVR-Pro-Bench OOD set, using 20 new instances for each of the 50 tasks (point 6). Cross-sample replacement reduces the OOD average from 45.4 to 40.4, a decrease of 5.0 points. These results further support the contribution of sample-specific rollout correspondence on the additional OOD instances.

MethodOOD Avg. ↑Abstraction ↑Knowledge ↑Perception ↑Spatial Reasoning ↑Transformation ↑
Cross-sample40.444.637.626.067.257.6
VR-JEPA45.450.143.030.277.259.0
  1. Paired Task-Level Comparison.

    We provide the complete paired task-level comparison across all 50 unseen tasks on the original OOD test set, with five instances per task, to make the modest average improvement easier to interpret. VR-JEPA improves on 23 tasks and matches the baseline on 14. Most tied scores occur on tasks that appear either particularly difficult or relatively easy for both models on the evaluated instances. At the category level, the average score increases by 5.72 points in abstraction and 1.14 points in perception. These comparisons suggest that task-relevant latent guidance can benefit different types of unseen reasoning problems, while revealing how the magnitude of the benefit varies across tasks.

    Higher scores (23 tasks)

    TaskVBVRVR-JEPATaskVBVRVR-JEPATaskVBVRVR-JEPA
    O-1132.779.0G-24067.171.3O-5650.251.0
    G-20213.353.7O-4365.269.0O-677.578.3
    O-956.585.9O-3995.198.5O-561.462.0
    G-16120.047.5G-14058.862.0O-6021.421.7
    O-623.320.2O-5432.135.2G-2495.996.0
    G-22349.862.0O-5841.943.2O-250.250.3
    O-2788.697.2O-6451.652.9G-541.31.3
    O-6513.721.6O-5914.615.4

    Equal scores (14 tasks)

    TaskVBVRVR-JEPATaskVBVRVR-JEPATaskVBVRVR-JEPA
    G-1350.00.0G-18919.419.4G-2220.00.0
    G-1360.00.0G-20640.040.0G-27340.040.0
    G-16040.040.0G-21220.020.0O-220.00.0
    G-1670.00.0G-2170.00.0O-49100.0100.0
    G-169100.0100.0G-2180.00.0

    Lower scores (13 tasks)

    TaskVBVRVR-JEPATaskVBVRVR-JEPATaskVBVRVR-JEPA
    G-24857.319.7G-19383.669.9O-8560.259.5
    G-4764.836.6G-17472.260.4G-22174.473.9
    G-21940.020.0O-4692.480.9O-6135.335.0
    G-24780.060.0G-2503.81.2G-16892.592.2
    G-14778.658.9
  2. Generation-Seed Variability.

    To assess variability due to generation randomness, we evaluated VR-JEPA on the same original OOD evaluation instances using five generation seeds. The OOD score is 46.2 ± 0.5 (mean ± sample standard deviation). We report the per-seed results and category-level statistics in the table below. All five runs score above the 45.1 obtained in the existing VBVR-Wan2.2 evaluation, providing complementary evidence that the observed advantage over this baseline score is not confined to a single VR-JEPA generation seed.

    Method / runOOD Avg.AbstractionKnowledgePerceptionSpatial reasoningTransformation
    VBVR-Wan2.245.135.842.734.072.277.7
    VR-JEPA (run 1)47.141.436.639.266.275.9
    VR-JEPA (run 2)46.141.542.135.265.277.5
    VR-JEPA (run 3)46.343.538.235.064.779.4
    VR-JEPA (run 4)45.739.738.635.766.178.6
    VR-JEPA (run 5)46.044.434.136.665.672.8
    Sample standard deviation0.51.82.91.70.62.6
    VR-JEPA (five-run mean)46.242.137.936.365.576.8
AuthorsOpenReview ↗

Response to Reviewer KGxj (3/4)

  1. Expanded OOD Evaluation.

    To address the limited coverage of the original evaluation and examine whether the observed OOD advantage persists across reasoning categories, we additionally evaluated both methods on 20 new instances for each of the 50 OOD tasks, totaling 1,000 additional instances. Both methods were evaluated on the same instances to enable a paired comparison. On this additional test set, VR-JEPA achieves an OOD average of 45.4, compared with 39.8 for VBVR-Wan2.2, an improvement of 5.6 points. The category-level scores also improve across all five categories. These results provide additional evidence of OOD benefits beyond the original small test set, with positive differences in both the overall average and all five category averages.

    MethodOOD Avg.AbstractionKnowledgePerceptionSpatial reasoningTransformation
    VBVR-Wan2.239.842.237.526.067.357.6
    VR-JEPA45.450.143.030.277.259.0

W1.2 / Q1. Score Interpretation

A2: Thank you for requesting clarification of the score definition. On VBVR-Pro-Bench, each evaluated instance receives a task-specific rule score between 0 and 1, with partial credit where applicable. We compute overall, ID/OOD, and category-level scores by averaging the corresponding instance scores and multiplying by 100, without applying a binary success threshold. These scores therefore represent mean normalized rule scores, rather than the proportion of fully solved problems. To clarify this distinction, we now use “normalized task score (0–100)” for VBVR-Pro-Bench results in the text and tables and have explicitly described the scoring and aggregation procedure in Section 4.1 of the revised manuscript.

W2 / Q2. LCL, Expert Skills, and Claim Scope

W2.1 / Q2. LCL Candidate Correctness and Localization

A3: Thank you for requesting a quantitative examination of LCL’s selections. To complement the qualitative examples in Figure 3, we analyzed generated candidates from our training dataset. Of the 100 training tasks, 50 have corresponding task-specific scorers in VBVR-Pro-Bench, so we randomly sampled 100 candidates from each of these tasks, yielding 5,000 candidates. We applied VR-JEPA’s existing LCL selection and weighting rules, then assessed candidate correctness and the relationship between weighted states, tokens, and task violations.

At the candidate level, 35.14% of generated candidates met our task criterion based on VBVR-Pro-scorer. LCL retained 50.32% of candidates, of which 72.34% did not meet the same criterion. At the state level, 44.28% of evaluated states received nonzero LCL weights. Among the weighted states assessed by GPT-6.1 Sol using the task instruction and surrounding trajectory, 66.87% were classified as containing task violations. At the token level, 43.62% of tokens within nonzero-weight states also received nonzero token weights. In states with confirmed, spatially localized violations, 74.03% of weighted token projections overlapped the annotated violation regions. These results suggest that LCL can select states containing task violations and emphasize regions associated with those violations.

As the reviewer notes, feature discrepancies can also reflect timing or appearance differences. Our timing analysis found that 9.20% of weighted states came from candidates assessed as correct and differed from the reference solely in execution timing. Such timing-only differences accounted for a relatively small proportion of weighted states in this analysis. Our ablation experiments further suggest that LCL can help improve the model’s visual reasoning performance. Appearance changes also need to be interpreted in relation to task requirements. In tasks where object identity or shape must be preserved, changes that break these constraints constitute invalid outcomes—for example, a maze agent changing shape when shape preservation is required. Feature discrepancies arising from such changes can therefore provide useful signals for LCL selection.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response

Dear AC and Reviewers,

We sincerely thank all reviewers for their constructive feedback. We are encouraged that they recognized our coherent and well-motivated framework (SRaN, 1PBB), our informative rollout-conditioning interface (KGxj), the complementary roles of LCL and SR-MoE (1PBB), the useful overall performance gains (KGxj, 1PBB), and our broad evaluation with task-grounded scoring (SRaN).

Below, we summarize the additional experiments addressing the reviewers’ main questions.

  1. Expanded benchmark evaluation and OOD analysis (KGxj W1 / Q1; SRaN W4 / Q4; 1PBB W1 / Q1). To assess OOD generalization with broader test coverage and examine the consistency of the observed gains, we add four complementary evaluations: (1) Expanded benchmark evaluation: Using 20 additional instances per task, VR-JEPA achieves an OOD score of 45.4 versus 39.8 for VBVR-Wan2.2, with gains across all five categories. (2) Larger spatial evaluation: Evaluating the same six spatial tasks on 600 instances also favors VR-JEPA, with 73.5 versus 72.7. (3) Generation-seed evaluation: Five generation seeds on the original OOD set further support the consistency of the observed gains, with VR-JEPA averaging 46.2 ± 0.5 (mean ± sample standard deviation). (4) Independent benchmark evaluation: VR-JEPA also outperforms VBVR-Wan2.2 on VR-Bench SR (80.00% vs. 64.40%) and VideoThinkBench Eyeballing accuracy (54.00% vs. 45.00%) and Visual Puzzles accuracy (25.00% vs. 15.00%). We also provide complete task-level comparisons and scorer diagnostics.

  2. Sample-specific latent guidance (KGxj W1; SRaN W4 / Q4). To further validate the benefits of sample-specific latent guidance, we extend our original cross-sample rollout experiments with two complementary evaluations: (1) Expanded VBVR-Pro-Bench OOD: On 20 additional instances per task, mismatched latent guidance lowers the OOD score from 45.4 to 40.4. (2) Two independent benchmarks: Mismatched latent guidance reduces VR-Bench success rate from 80.00% to 70.60% and decreases VideoThinkBench Eyeballing and Visual Puzzles accuracy by 9.00 and 2.00 percentage points, respectively. These findings reinforce the importance of matching latent guidance to the current sample.

  3. Rollout-based design versus existing JEPA guidance mechanisms (SRaN W1 / Q1). We add two WMReward comparisons under the same VBVR-Wan2.2 generator and evaluation protocol to clarify the advantage of our rollout-based design: (1) Best-of-two selection (BoN-2): VR-JEPA improves the overall score by 5.2 points over candidate selection using frozen V-JEPA prediction error. (2) Gradient guidance: VR-JEPA improves the overall score by 5.9 points over prediction-error-based denoising guidance. The improved reasoning scores and reduced inference latency support the effectiveness and efficiency of VR-JEPA’s rollout-based design.

  4. Effectiveness of local contrastive learning (KGxj W2 / Q2; SRaN W3 / Q3; 1PBB Q2). (1) Local CL selection and localization: Among LCL-retained candidates, 72.34% did not meet the scorer-based task criterion. Among the assessed weighted states, 66.87% were classified as containing task violations. Within states with confirmed, spatially localized violations, 74.03% of weighted token projections overlapped the annotated violation regions. (2) Comparing Local CL with global Uniform CL: VR-JEPA’s Local CL scores 56.0 overall, compared with 53.5 for global Uniform CL, which applies contrastive supervision uniformly across states and tokens. These results suggest that LCL can select states containing task violations and emphasize associated regions, while its gains over Uniform CL support the benefit of local selection and weighting.

  5. SR-MoE design and expert specialization (KGxj W2; SRaN W2 / Q2). We add three complementary studies of SR-MoE’s skill-oriented design: (1) Capacity and initialization comparisons: VR-JEPA outperforms both the capacity-matched dense predictor and the MoE without skill anchors, achieving 56.0 overall compared with 54.2 and 54.5, respectively. (2) Anchor-task sensitivity: Using alternative representative anchors preserves overall performance, supporting robustness to anchor-task selection. (3) Transferable expert skills: We complement the qualitative examples with quantitative analysis on non-anchor tasks, showing that removing task-relevant experts generally causes larger reasoning-score drops than removing other experts. This provides further support for transferable skill specialization.

  6. Computational cost and sparse routing (SRaN W5 / Q5). We summarize VR-JEPA’s computational cost across candidate generation, training, and inference. We also quantify actual expert computation under sparse routing.

AuthorsOpenReview ↗

Summary of Revisions

We have revised the manuscript to incorporate the clarifications and additional experiments presented in our rebuttal. The main additions are highlighted in red in the revised PDF and summarized below:

  1. Sections 4.1–4.2 & Tables 1–2 (Evaluation and Comparisons): Clarified score definitions and aggregation, added comparisons with WMReward using BoN-2 and gradient guidance, and evaluated generalization on two independent benchmarks, VR-Bench and VideoThinkBench.
  2. Section 4.3 & Table 3 (Controlled Ablations): Added a capacity-matched dense predictor, an MoE without skill-anchor initialization, and a Uniform CL variant to assess skill-guided expert organization and local contrastive selection and weighting.
  3. Appendix A.5 (Computational Cost): Reported candidate-generation and training costs, JEPA encoding time, inference latency and peak memory, and sparse expert execution statistics.
  4. Appendices D.1–D.5 (Generalization and Rollout Analysis): Added paired ID/OOD evaluations with 20 additional instances per task, complete task-level OOD comparisons, VR-JEPA’s OOD results across five generation seeds, a 600-instance paired spatial evaluation with scorer diagnostics, and cross-sample rollout interventions on expanded OOD and independent benchmarks.
  5. Appendices E.1–E.2 (Expert Analysis): Added anchor-task replacement experiments and quantitative expert-removal interventions on seven non-anchor tasks to examine anchor sensitivity and transferable skill specialization.

We sincerely thank the reviewers and AC for their constructive feedback, which has helped us improve the clarity and completeness of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.