- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LEAP, a framework for improving the physical plausibility of T2V generation using latent-space rewards derived from a frozen V-JEPA 2 model. The framework first trains an Aligner to map VAE latents into V-JEPA 2 representations. The Aligner is then frozen and used in the proposed LPFL to fine-tune the video generator by minimizing predictive surprise directly in latent space. The authors propose EMA velocity regularization to stabilize training. The paper also examines the computational benefits of passing pixel-space reward computation.
Critical strengths
The paper is well written and logically organized, making the overall framework easy to follow. It clearly explains the prior ideas and pretrained models on which the method builds. Using the predictive consistency of a pretrained V-JEPA 2 model as a reward provides a promising way to leverage physical priors into video generation. The Aligner makes this signal available directly in the generator’s latent space, avoiding pixel-space decoding during reward computation.
Critical weaknesses
- The masking design has a potential information leakage issue in the physics objective. In Section 3.2.1, the Q-Former attends to flattened VAE latent tokens before masked target tokens are removed for the subsequent self-attention blocks. Dropping output tokens afterward does not enforce a past-only context. It may contain information from future regions.
- Minimizing future prediction error can potentially favor reduced motion or simpler interactions, while physically valid but stochastic dynamics may still incur high surprise. The paper reports improved physical plausibility, but does not establish whether this improvement is maintained at comparable levels of motion and action complexity. Controlled comparisons involving static, simplified, and physically inconsistent versions of the same event would help determine what the reward actually captures.
Questions for authors
- In Equation (7) of Section 3.2.3, the three loss terms are summed with equal weights. Given that feature alignment, surprise-score matching, and predictor consistency each constrain different aspects of the Aligner, is it necessary to introduce adjustable weights to control the relative contribution of each term?
- In the context branch of the Aligner, are the Q-Former’s keys and values restricted to visible source tokens before cross-attention, or does each query attend to the entire video latent? I am concerned that, in the latter case, the retained contextual representations may already encode information about the masked targets. Discarding the corresponding output tokens after the cross-attention mechanism does not necessarily eliminate this information.
3 comments
Weakness 2. Whether minimizing prediction error favors reduced motion or simpler interactions.
We thank the reviewer for raising this important question. To examine whether the improvement in physical plausibility is accompanied by a shift toward static videos, we measured Dynamic Degree (VBench 0.1.5) on videos generated from the 599 VideoPhy2 prompts.
Table R3. Dynamic Degree on videos generated from the 599 VideoPhy2 prompts. Dynamic and Static are video counts.
| Model | Dynamic | Static | Dynamic Degree (%) |
|---|---|---|---|
| Wan2.1-1.3B (baseline) | 512 | 87 | 85.48 |
| SFT | 552 | 47 | 92.15 |
| LEAP | 567 | 32 | 94.66 |
LEAP generates the fewest static videos (32, compared with 87 for the baseline and 47 for SFT) while achieving the highest PC (Table 1). These results show that the PC improvement is accompanied by a higher proportion of dynamic videos, rather than a shift toward static outputs.
We also examined semantic adherence (SA) to assess whether the generated videos continue to follow the requested actions and interactions. As shown in Table 1, LEAP improves SA from 48.26% to 49.13% on VideoPhy and from 29.55% to 31.55% on VideoPhy2. Together with the Dynamic Degree results, this provides evidence against a general shift toward static or semantically simplified outputs, although it does not directly measure interaction complexity.
Question 1. Loss weights in Eq. (7).
We chose equal weights for the three objectives in Eq. (7) to keep the tuning burden manageable. We also checked their relative magnitudes: after training, , and account for 56%, 33% and 11% of the total loss, respectively, indicating that all three terms contribute to the objective under equal weighting.
Table 4(d) further evaluates the extreme setting , which reduces from 0.74 to −0.62. This supports including the predictor-based objectives alongside embedding alignment. We appreciate the suggestion and will explore tuning these weights as time and resources permit.
Thank you for the response and the additional experiments and results. The new comparisons about the aligner and dynamic degree are helpful and support the strengths of LEAP. I will keep my positive rating.
We thank the reviewer for the careful reading and constructive feedback. Below, we address the comments and present additional experimental results.
Weakness 1 and Question 2. Information leakage through Q-Former cross-attention.
We thank the reviewer for raising this concern. The Q-Former's keys and values cover all VAE latent tokens. Applying the mask after cross-attention may not fully prevent target information from entering the retained context representations.
Why the Q-Former attends to the whole latent. The Q-Former bridges differences in frame sampling, resolution, crops and token structure between the VAE and V-JEPA 2. Its queries map the full VAE latent to the V-JEPA token grid, where the teacher's masks are then applied.
Experiment 1: past-only keys and values at evaluation without retraining. Using the original Aligner and the paper's sliding-window protocol on Video-Bench (85 human-labeled pairs), pairwise accuracy increases from 54.1% to 71.8% (McNemar p = 0.006). Surprise rises more for less realistic videos (+0.034 vs. +0.020), increasing the pairwise surprise gap from 0.0036 to 0.0175. However, agreement with V-JEPA 2 surprise decreases (Table R1). Since this restriction changes the original Aligner's evaluation conditions, Experiment 2 also retrains it.
Experiment 2: retraining and LPFL with past-only keys and values. We retrained the Aligner with the same data, objectives and hyperparameters, a comparable number of iterations, and no re-tuning. At matched late-training checkpoints, pairwise accuracy improves (64.7% vs. 58.5%), but agreement with V-JEPA 2 surprise remains lower (Table R1); even the retrained run's best checkpoint reaches only 0.496 / 0.336 / 0.475 in SRCC / KRCC / PLCC. LPFL using the retrained Aligner scores lower on all three metrics of both benchmarks (Table R2).
Table R1. Aligner evaluation on Video-Bench. Correlations are between Aligner and V-JEPA 2 surprise; PLCC uses a 4-parameter logistic mapping. The paper's checkpoint was selected by SRCC, not pairwise accuracy.
Table R2. LPFL results on VideoPhy and VideoPhy2 (%).
Temporal restriction improves pairwise accuracy but reduces agreement with the teacher's surprise and downstream LPFL performance. These results support retaining full-latent cross-attention in our current design, where the Aligner serves as a differentiable surrogate of V-JEPA surprise for LPFL. We will clarify this design choice in Section 3.2.1 and add both experiments to the appendix.