acceptodds
Under review as a conference paper at ICLR 2027

LEAP: Latent Encoder Alignment for Physics Rewards in Video Generation

Abstract

Reward Feedback Learning is a powerful paradigm for generative modeling, yet its extension to Text-to-Video (T2V) generation is hindered by intractable computational overhead and the scarcity of differentiable reward signals. Although mapping rewards to the latent space reduces computational burdens, existing latent rewards rely heavily on human preference datasets, inherently lacking the rich physical and structural priors of foundational models. In this work, we introduce a novel framework that constructs a latent physics reward by extracting and leveraging priors from a latent world model. We overcome two fundamental hurdles to achieve this integration: 1) developing an alignment mechanism with a lightweight Aligner to connect the generative VAE latent space with the target prior embedding space, and 2) adapting the feedback learning algorithm to stably incorporate these latent priors during training. Our method is the first to demonstrate the viability of projecting T2V latent representations into the V-JEPA 2 vision embedding space for enforcing physical plausibility in generated videos. Empirical results on the VideoPhy and VideoPhy2 datasets validate that our lightweight Aligner helps latent reward substantially improves the physical realism and temporal consistency of generated videos, while training within 12.9GB, where backpropagating the same reward through pixel space would require hundreds of GB.

Primary area:
reinforcement learning
Keywords:
reward feedback learning, latent reward model, text-to-video generation, video generation model, reinforcement learning, computer vision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XCeqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LEAP, a framework for improving the physical plausibility of T2V generation using latent-space rewards derived from a frozen V-JEPA 2 model. The framework first trains an Aligner to map VAE latents into V-JEPA 2 representations. The Aligner is then frozen and used in the proposed LPFL to fine-tune the video generator by minimizing predictive surprise directly in latent space. The authors propose EMA velocity regularization to stabilize training. The paper also examines the computational benefits of passing pixel-space reward computation.

Critical strengths

The paper is well written and logically organized, making the overall framework easy to follow. It clearly explains the prior ideas and pretrained models on which the method builds. Using the predictive consistency of a pretrained V-JEPA 2 model as a reward provides a promising way to leverage physical priors into video generation. The Aligner makes this signal available directly in the generator’s latent space, avoiding pixel-space decoding during reward computation.

Critical weaknesses
  • The masking design has a potential information leakage issue in the physics objective. In Section 3.2.1, the Q-Former attends to flattened VAE latent tokens before masked target tokens are removed for the subsequent self-attention blocks. Dropping output tokens afterward does not enforce a past-only context. It may contain information from future regions.
  • Minimizing future prediction error can potentially favor reduced motion or simpler interactions, while physically valid but stochastic dynamics may still incur high surprise. The paper reports improved physical plausibility, but does not establish whether this improvement is maintained at comparable levels of motion and action complexity. Controlled comparisons involving static, simplified, and physically inconsistent versions of the same event would help determine what the reward actually captures.
Questions for authors
  • In Equation (7) of Section 3.2.3, the three loss terms are summed with equal weights. Given that feature alignment, surprise-score matching, and predictor consistency each constrain different aspects of the Aligner, is it necessary to introduce adjustable weights to control the relative contribution of each term?
  • In the context branch of the Aligner, are the Q-Former’s keys and values restricted to visible source tokens before cross-attention, or does each query attend to the entire video latent? I am concerned that, in the latter case, the retained contextual representations may already encode information about the masked targets. Discarding the corresponding output tokens after the cross-attention mechanism does not necessarily eliminate this information.
3 comments

We thank the reviewer for the careful reading and constructive feedback. Below, we address the comments and present additional experimental results.


Weakness 1 and Question 2. Information leakage through Q-Former cross-attention.

We thank the reviewer for raising this concern. The Q-Former's keys and values cover all VAE latent tokens. Applying the mask after cross-attention may not fully prevent target information from entering the retained context representations.

Why the Q-Former attends to the whole latent. The Q-Former bridges differences in frame sampling, resolution, crops and token structure between the VAE and V-JEPA 2. Its queries map the full VAE latent to the V-JEPA token grid, where the teacher's masks are then applied.

  • Spatial masks. Per-frame self-attention in the VAE mixes information across the frame, so selecting latent tokens cannot exclude a masked spatial region. These masks are used only in Aligner training, where embedding alignment and predictor consistency provide soft constraints, not structural guarantees.
  • Temporal masks. The WanX VAE is temporally causal, allowing keys and values to be restricted to past latent frames. We tested this restriction in two experiments.

Experiment 1: past-only keys and values at evaluation without retraining. Using the original Aligner and the paper's sliding-window protocol on Video-Bench (85 human-labeled pairs), pairwise accuracy increases from 54.1% to 71.8% (McNemar p = 0.006). Surprise rises more for less realistic videos (+0.034 vs. +0.020), increasing the pairwise surprise gap from 0.0036 to 0.0175. However, agreement with V-JEPA 2 surprise decreases (Table R1). Since this restriction changes the original Aligner's evaluation conditions, Experiment 2 also retrains it.

Experiment 2: retraining and LPFL with past-only keys and values. We retrained the Aligner with the same data, objectives and hyperparameters, a comparable number of iterations, and no re-tuning. At matched late-training checkpoints, pairwise accuracy improves (64.7% vs. 58.5%), but agreement with V-JEPA 2 surprise remains lower (Table R1); even the retrained run's best checkpoint reaches only 0.496 / 0.336 / 0.475 in SRCC / KRCC / PLCC. LPFL using the retrained Aligner scores lower on all three metrics of both benchmarks (Table R2).

Table R1. Aligner evaluation on Video-Bench. Correlations are between Aligner and V-JEPA 2 surprise; PLCC uses a 4-parameter logistic mapping. The paper's checkpoint was selected by SRCC, not pairwise accuracy.

AlignerAcc. (%)SRCCKRCCPLCC
Original aligner, full-latent K/V (paper)54.10.7370.5300.719
Exp. 1: original aligner, past-only K/V at evaluation only71.80.6280.4400.605
Exp. 2: aligner retrained with past-only K/V (comparable iteration)64.70.4200.2810.426
V-JEPA surprise (reference)57.6–––

Table R2. LPFL results on VideoPhy and VideoPhy2 (%).

Reward model in LPFLVideoPhy PCVideoPhy SAVideoPhy JointVideoPhy2 PCVideoPhy2 SAVideoPhy2 Joint
Original aligner, full-latent K/V (paper)29.3649.1315.1268.4531.5527.88
Exp. 2: aligner retrained with past-only K/V26.4548.5513.6666.9430.7227.38

Temporal restriction improves pairwise accuracy but reduces agreement with the teacher's surprise and downstream LPFL performance. These results support retaining full-latent cross-attention in our current design, where the Aligner serves as a differentiable surrogate of V-JEPA surprise for LPFL. We will clarify this design choice in Section 3.2.1 and add both experiments to the appendix.

Weakness 2. Whether minimizing prediction error favors reduced motion or simpler interactions.

We thank the reviewer for raising this important question. To examine whether the improvement in physical plausibility is accompanied by a shift toward static videos, we measured Dynamic Degree (VBench 0.1.5) on videos generated from the 599 VideoPhy2 prompts.

Table R3. Dynamic Degree on videos generated from the 599 VideoPhy2 prompts. Dynamic and Static are video counts.

ModelDynamicStaticDynamic Degree (%)
Wan2.1-1.3B (baseline)5128785.48
SFT5524792.15
LEAP5673294.66

LEAP generates the fewest static videos (32, compared with 87 for the baseline and 47 for SFT) while achieving the highest PC (Table 1). These results show that the PC improvement is accompanied by a higher proportion of dynamic videos, rather than a shift toward static outputs.

We also examined semantic adherence (SA) to assess whether the generated videos continue to follow the requested actions and interactions. As shown in Table 1, LEAP improves SA from 48.26% to 49.13% on VideoPhy and from 29.55% to 31.55% on VideoPhy2. Together with the Dynamic Degree results, this provides evidence against a general shift toward static or semantically simplified outputs, although it does not directly measure interaction complexity.


Question 1. Loss weights in Eq. (7).

We chose equal weights for the three objectives in Eq. (7) to keep the tuning burden manageable. We also checked their relative magnitudes: after training, LMSE\mathcal{L}_{\text{MSE}}, LSS\mathcal{L}_{\text{SS}} and Lpred\mathcal{L}_{\text{pred}} account for 56%, 33% and 11% of the total loss, respectively, indicating that all three terms contribute to the objective under equal weighting.

Table 4(d) further evaluates the extreme setting (λMSE,λSS,λpred)=(1,0,0)(\lambda_{\text{MSE}}, \lambda_{\text{SS}}, \lambda_{\text{pred}}) = (1, 0, 0), which reduces ρ\rho from 0.74 to −0.62. This supports including the predictor-based objectives alongside embedding alignment. We appreciate the suggestion and will explore tuning these weights as time and resources permit.

Thank you for the response and the additional experiments and results. The new comparisons about the aligner and dynamic degree are helpful and support the strengths of LEAP. I will keep my positive rating.

Reviewer_XVCqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Latent Encoder Alignment for Physics Rewards (LEAP), a framework that aims to improve physical plausibility in text-to-video generation using reward feedback learning. The authors argue that pretrained world models, such as V-JEPA 2, already capture useful physical priors that can be leveraged to improve video generation. To achieve this, they train an aligner that maps the T2V model's latent representations into the embedding space of the world model, allowing rewards/likehoods to be computed directly in latent space rather than pixel space, significantly reducing computational costs. They also introduce EMA-based regularization to prevent reward hacking and preserve generation quality during fine-tuning.

Critical strengths
  1. The paper computes rewards directly in latent space rather than pixel space, significantly reducing peak memory usage and making reward feedback learning more computationally efficient.

  2. The paper demonstrates that pretrained world models can provide meaningful physics-based reward signals without relying on human preference annotations.

  3. The proposed method consistently improves physical plausibility and achieves better overall performance than the baselines on VideoPhy and VideoPhy2.

Critical weaknesses
  1. The paper assumes that a lower V-JEPA 2 prediction error corresponds to better physical correctness, which may not always hold. For instance, V-JEPA 2 might favor predictable or low-motion videos rather than genuinely realistic physical dynamics. An ablation investigating whether the reward favors such behaviors would help validate this assumption and strengthen the paper.

  2. The experiments are conducted exclusively on Wan2.1-1.3B, and the evaluation is limited to VideoPhy and VideoPhy2. The authors could consider additional benchmarks such as PhyGenBench [1], which covers a broader range of physical laws, including optics, thermal phenomena, and material properties. This would help determine whether LEAP improves general physical-law adherence beyond action-level plausibility. Evaluating on larger generators, such as Wan2.1-14B, would also provide evidence of the method's scalability.

  3. The paper lacks comparisons with closely related representation-alignment methods such as VideoREPA, which also transfers physical priors from pretrained video encoders to video generators. Since LEAP uses these priors as differentiable rewards on generated samples, a controlled comparison with representation alignment or direct feature distillation using the same V-JEPA 2 backbone would help isolate the benefits of reward feedback learning over simply transferring pretrained representations.

[1] Meng, F., Liao, J., Tan, X., Lu, Q., Shao, W., Zhang, K., Cheng, Y., Li, D. & Luo, P.. (2025). Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation. <i>Proceedings of the 42nd International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 267:43781-43806 Available from https://proceedings.mlr.press/v267/meng25c.html.

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for the careful reading and constructive feedback. Below, we address the comments and present additional experimental results.

Weakness 1. Whether the V-JEPA 2 reward favors predictable or low-motion videos.

To examine whether the improvement in physical plausibility is accompanied by a shift toward static videos, we measured Dynamic Degree (VBench 0.1.5) on videos generated from the 599 VideoPhy2 prompts.

Table R1. Dynamic Degree on videos generated from the 599 VideoPhy2 prompts. Dynamic and Static are video counts.

ModelDynamicStaticDynamic Degree (%)
Wan2.1-1.3B (baseline)5128785.48
SFT5524792.15
LEAP5673294.66

LEAP generates the fewest static videos (32, compared with 87 for the baseline and 47 for SFT) while achieving the highest PC (Table 1). These results show that the PC improvement is accompanied by a higher proportion of dynamic videos, rather than a shift toward static outputs.


Weakness 2. Evaluation on an additional benchmark and a larger backbone.

PhyGenBench. We evaluated the baseline, SFT, and LEAP on PhyGenBench, which covers mechanics, optics, thermal phenomena, and material properties. Following the official protocol, each video was scored by GPT-4o (gpt-4o-2024-08-06) on a 0–3 scale.

Table R2. PhyGenBench results.

ModelMean Overall Score (0–3)Normalized Score (%)
Wan2.1-1.3B (baseline)1.512550.42
SFT1.593853.13
LEAP1.625054.17

LEAP achieves the highest overall score, improving over the baseline by 3.75 percentage points (50.42% →\rightarrow 54.17%) and over SFT by 1.04 percentage points. These results extend the evidence for LEAP beyond VideoPhy and VideoPhy2 to a benchmark covering a broader range of physical phenomena.

Wan2.1-14B. We additionally applied LPFL to Wan2.1-14B, reusing the Aligner and all training hyperparameters from the 1.3B experiments. Given the computational cost at this scale and the limited time and compute resources available during the rebuttal period, we conducted a single training run without hyperparameter tuning. Wan2.1-14B generates videos at 720P, whereas the Aligner was trained on 480P VAE latents, so this experiment also tests its reuse across resolutions.

Table R3. LPFL results on VideoPhy2 with Wan2.1-14B (%). The Δ\Delta row reports changes in percentage points.

ModelPCSAJoint
Wan2.1-14B (baseline)59.2727.7124.04
LEAP59.7728.2124.04
(LEAP − baseline)+0.50+0.50+0.00

In this single training run, LPFL improves PC and SA by 0.50 percentage points each, while Joint remains unchanged. These preliminary results demonstrate that LPFL can be applied to a larger generator with the existing Aligner, although the performance gains are modest. We will include these results in the revised paper.

Weakness 3. Comparisons with VideoREPA and direct V-JEPA 2 feature alignment.

We agree VideoREPA is a relevant baseline. The two methods differ in what they transfer and where it is applied. VideoREPA aligns the token relations of the generator's intermediate features with those of a pretrained video encoder (VideoMAEv2) on real training videos. LEAP instead uses the prediction error of the frozen V-JEPA 2 predictor as a reward on the generator's own samples.

The published VideoREPA results considered here use CogVideoX, whereas our method is trained on Wan2.1-1.3B. The baselines already differ substantially (CogVideoX-2B: 13.3 VideoPhy PC; Wan2.1-1.3B: 23.55), so their absolute scores are not directly comparable. We first provide two comparisons based on published results, then report a controlled direct-alignment ablation using the same V-JEPA 2 backbone.

First, we compare the gain each method obtains over its own baseline, on the same benchmark and with the same evaluator:

Table R4. VideoPhy2 PC gains over each method's baseline (percentage points). Parentheses show baseline and final scores (%).

Method (backbone)VideoPhy2 PC gain over own baseline
VideoREPA-2B (CogVideoX-2B)+4.57 (67.97 72.54)
ProPhy (Wan2.1-1.3B)+2.56 (62.44 65.00)
LPFL (ours, Wan2.1-1.3B)+6.01 (62.44 68.45)

Second, ProPhy's Table 1 already compares VideoREPA and ProPhy directly, both on CogVideoX-5B, which removes the backbone difference from that comparison:

Table R5. VideoREPA and ProPhy on CogVideoX-5B, as reported in ProPhy's Table 1 (%).

MethodPCSAJoint
CogVideoX-5B67.229.022.3
+ VideoREPA72.524.222.0
+ ProPhy72.532.226.7

On CogVideoX-5B, ProPhy matches VideoREPA on PC and achieves higher SA and Joint. VideoREPA raises PC but lowers SA (29.0 →\rightarrow 24.2), with Joint remaining close to the baseline, while ProPhy improves all three metrics. Our Table 1 separately compares LPFL with ProPhy on Wan2.1-1.3B. These published comparisons provide context, but do not establish the relative performance of LEAP and VideoREPA under matched training conditions.

Third, as suggested, we compared LEAP with direct feature distillation using the same V-JEPA 2 backbone. We trained a direct-alignment variant of LEAP that uses the same generator, Aligner, and training data, but replaces the physics reward with an MSE loss that aligns the Aligner output of the generator's predicted clean latent to the V-JEPA 2 encoder features of the real training video with the same caption.

Table R6. Direct feature alignment and LEAP on VideoPhy and VideoPhy2 (%).

MethodVideoPhy PCVideoPhy SAVideoPhy JointVideoPhy2 PCVideoPhy2 SAVideoPhy2 Joint
Wan2.1-1.3B (baseline)23.5548.2612.2162.4429.5525.38
Direct alignment19.7750.2910.1758.9333.0628.55
LEAP29.3649.1315.1268.4531.5527.88

Direct alignment raises SA but lowers PC below the baseline on both benchmarks (23.55% to 19.77% on VideoPhy and 62.44% to 58.93% on VideoPhy2), whereas LEAP raises PC by 5.81 and 6.01 percentage points, respectively. These results show that transferring V-JEPA 2 features through direct alignment alone does not reproduce the PC improvements obtained with our predictor-based reward. Direct alignment achieves higher SA and Joint on VideoPhy2, indicating a trade-off across evaluation metrics. We will add this ablation to the revised paper.

I thank the authors for their detailed clarifications, which have addressed most of my concerns. I therefore increase my score to 3 and incline towards acceptance of the paper.

Reviewer_rGkBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LEAP, a framework for improving physical plausibility in text-to-video generation through latent-space reward feedback learning. The method introduces a lightweight Aligner that maps video VAE latents into the V-JEPA 2 embedding space, using prediction consistency as a differentiable physics reward. The authors further propose Latent Physics Feedback Learning (LPFL), which incorporates EMA velocity regularization to stabilize training. Experiments demonstrate improvements in physical plausibility with reduced training memory.

Critical strengths
  • The paper writing is clear, and the motivation of leveraging pretrained model priors for efficient physics-aware video generation is reasonable.
  • The proposed latent-space reward mechanism avoids expensive pixel-space decoding during reward optimization. The efficiency analysis demonstrates reductions in training memory and computational cost.
  • The experimental results show improvements over existing baselines on VideoPhy and VideoPhy2. The authors also provide component ablations and human evaluations supporting the effectiveness of the proposed approach.
Critical weaknesses
  • The methodological novelty is limited, and the authors do not clarify their distinctions from existing related approaches. In particular, the paper's claim of being the "first" in the abstract is misleading, as it overlooks and fails to discuss closely-related recent works on latent reward models, such as:
    • VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward (ECCV 2026)
    • Stitched Value Model for Diffusion Alignment (NeurIPS 2026)
  • The proposed physics reward relies on V-JEPA 2 prediction consistency, which does not necessarily correspond to physical correctness. Predictable but physically implausible motions may also receive high rewards. It would be useful to show the correlation between the reward and physical plausibility.
  • The experimental evaluation is limited to a single video generation backbone (Wan2.1-1.3B). Experiments on larger or different architectures would strengthen the proposed approach.
  • The ablation studies do not fully isolate the contributions of latent feature alignment and physics reward optimization. A comparison against direct V-JEPA 2 feature alignment during generator training would help clarify the benefits of the proposed formulation.
Questions for authors

Please refer to the weaknesses above. I would be willing to reconsider my rating based on the responses.

3 comments

We thank the reviewer for the careful reading and constructive feedback. Below, we address the comments and present additional experimental results.

Weakness 1. Novelty and distinctions from contemporaneous latent reward methods.

We thank the reviewer for highlighting these related works. According to the ICLR 2027 Reviewer Guidelines, VGGRPO (ECCV 2026) and the Stitched Value Model (NeurIPS 2026) are considered contemporaneous work, since their publication dates are later than July 17, 2026: ECCV 2026 was held in September, and NeurIPS 2026 will be held in December. Although authors are not required to compare against contemporaneous work, we will include a comparison with both papers in the Related Work section of the revised paper, following your suggestion.

VGGRPO connects video diffusion latents to geometry foundation models and uses camera-motion smoothness and geometry-reprojection rewards for latent-space GRPO. LEAP instead derives its reward from a frozen video world model’s predictive consistency and optimizes the generator by directly backpropagating this reward through a lightweight Aligner. The methods therefore differ in the pretrained prior, reward construction, and optimization procedure.

The Stitched Value Model attaches a frozen diffusion backbone to a truncated pixel-space reward model through a lightweight stitching layer to estimate values on noisy latents, with experiments focused on text-to-image generation. LEAP shares the motivation of making pretrained feedback accessible in latent space, but uses an Aligner to map the generator’s clean latent estimates into the V-JEPA 2 embedding space, then computes a reward from the discrepancy between the frozen predictor’s future predictions and the corresponding Aligner-produced embeddings.

Our contribution is the combination of an Aligner trained to reproduce the teacher’s representations and predictive behavior, and a latent feedback-learning procedure that applies this surrogate reward to the generator’s own rollouts. The “first” claim in our abstract is specifically scoped to projecting T2V latents into the V-JEPA 2 embedding space to derive a physics reward. Neither of the two works uses V-JEPA 2, so we believe the claim remains accurate within this scope. We will make this scope and the relationship to these contemporaneous methods clearer in the revised paper.

Weakness 2. Correlation between the V-JEPA 2 reward and physical plausibility.

To examine the relationship between the reward and physical plausibility, we measured the ranking agreement between V-JEPA 2 surprise and human Physical Commonsense (PC) scores on the VideoPhy2 test set. We scored 498 human-annotated videos from 350 prompts with the original V-JEPA 2 (ViT-G/384). For every pair of videos whose PC scores differ by at least 2 points, we checked whether the video with the higher PC score has the lower surprise.

Across the 54,080 eligible pairs, the ranking accuracy is 60.16% (95% CI 55.02–65.06%, prompt-level bootstrap), above the 50% chance level. The accuracy increases with the PC score difference.

Table R1. Ranking agreement by human PC score difference.

Absolute PC differenceEligible pairsCorrect rankingsRanking accuracy (%)
2 points30,24017,42357.6
3 points17,44010,63561.0
4 points6,4004,47970.0
Total54,08032,53760.2

This shows that V-JEPA 2 surprise agrees with human judgments of physical plausibility, and the agreement grows stronger as the difference in plausibility increases. We will add this analysis to the revised paper.

Weakness 3. Evaluation on a larger backbone (Wan2.1-14B).

We additionally applied LPFL to Wan2.1-14B, reusing the Aligner and all training hyperparameters from the 1.3B experiments. Given the computational cost at this scale and the limited time and compute resources available during the rebuttal period, we conducted a single training run without hyperparameter tuning. Wan2.1-14B generates videos at 720P, whereas the Aligner was trained on 480P VAE latents, so this experiment also tests its reuse across resolutions.

Table R2. LPFL results on VideoPhy2 with Wan2.1-14B (%). The Δ\Delta row reports changes in percentage points.

ModelPCSAJoint
Wan2.1-14B (baseline)59.2727.7124.04
LEAP59.7728.2124.04
(LEAP − baseline)+0.50+0.50+0.00

In this single training run, LPFL improves PC and SA by 0.50 percentage points each, while Joint remains unchanged. These preliminary results demonstrate that LPFL can be applied to a larger generator with the existing Aligner, although the performance gains are modest. We will include these results in the revised paper.

Weakness 4. Isolating the contribution of the predictor-based reward.

To isolate the contribution of the reward, we trained a direct-alignment variant of LEAP. It uses the same generator, Aligner, and training data, but replaces the physics reward with an MSE loss that aligns the Aligner output of the generator's predicted clean latent to the V-JEPA 2 encoder features of the real training video with the same caption. This corresponds to direct V-JEPA 2 feature distillation without the predictor.

Table R3. Direct feature alignment and LEAP on VideoPhy and VideoPhy2 (%).

MethodVideoPhy PCVideoPhy SAVideoPhy JointVideoPhy2 PCVideoPhy2 SAVideoPhy2 Joint
Wan2.1-1.3B (baseline)23.5548.2612.2162.4429.5525.38
Direct alignment19.7750.2910.1758.9333.0628.55
LEAP29.3649.1315.1268.4531.5527.88

Direct alignment raises SA but lowers PC below the baseline on both benchmarks (23.55% to 19.77% on VideoPhy and 62.44% to 58.93% on VideoPhy2), whereas LEAP raises PC by 5.81 and 6.01 percentage points, respectively. These results show that transferring V-JEPA 2 features through direct alignment alone does not reproduce the PC improvements obtained with our predictor-based reward. Direct alignment achieves higher SA and Joint on VideoPhy2, indicating a trade-off across evaluation metrics. We will add this ablation to the revised paper.

Thank you for the responses, which have mostly addressed my concerns. I will adjust my rating to 3, and the authors should better clarify the methodological novelty and position their contributions relative to existing work in the future version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.