acceptodds
Under review as a conference paper at ICLR 2027

Physics Before the First Step: Rectifying the Text Embedding of Frozen Video Generators

Abstract

Video generators trained at scale acquire a prior over natural motion, yet they still fail on actions whose outcome is dictated by physics. We observe that as flow matching and DiT backbones replace earlier designs, the structure of the generated video is decided earlier and earlier in the denoising process. This undermines methods that correct the latent during sampling, which need a window in which an error is visible and still open to repair. We turn instead to the text embedding, the one input read clean at every layer and every step, and find two properties that make it a sufficient and learnable target: an embedding fitted to a single real clip reproduces that clip under every initial noise, and the displacement to it is spread across the whole embedding, so a prediction that lands near it suffices. Building on these, we propose PhysRect: for physical events cut from real video, it fits the embedding under which a frozen generator reproduces the clip, keeping the fits of different mechanisms apart with a separability term. The fits are then distilled into a VLM-gated mixture of mechanism experts that corrects the embedding once before sampling. On VideoPhy-2, PhysRect reduces the physics error of Wan2.2 by 51%, matches a 14B model fine-tuned on physical video at one fourteenth of its generation time with no inference overhead.

Primary area:
generative models
Keywords:
generative model, video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tVBSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PhysRect, which corrects a frozen Wan2.2 generator's text embeddings before sampling. Embeddings fitted to real physical events are distilled into seven mechanism experts gated by a VLM.

Critical strengths

Mean physics error decreases from 1.431 to 0.705 on VideoPhy-2. Supervision comes from real events, and the correction avoids additional operations at each sampling step.

Critical weaknesses
  1. The benefit of selecting experts is not established. The paper does not directly compare selecting experts with averaging all seven outputs. If each expert improves a different aspect of physical consistency, the paper needs to explain why all experts should not be used together. Table 2 compares against a smaller single network, which changes both the architecture and capacity. This does not show why selecting experts is necessary.
  2. The choice of correction strength is insufficiently documented. Figure 6 evaluates alpha on the test prompts and reports the best result at 0.5, also used in Table 1. The paper does not explain whether this value was fixed on independent validation data, leaving possible use of test feedback unresolved.
  3. The cost and necessity of Gemini are unclear. Table 1 does not specify whether timing includes a Gemini call for each new prompt or uses precomputed labels. The full inference cost is therefore unclear. There is also no comparison with a standard open-source VLM to justify this dependency.
Questions for authors
  1. Does selecting experts outperform averaging all seven outputs under the same settings?
  2. Was alpha=0.5 selected on independent validation data before evaluating the test prompts?
  3. Does runtime include Gemini for a new prompt? Can an open-source VLM replace it for mechanism labeling, and how does this affect video quality, latency, and cost?
2 comments

W1 / Q1: Selecting experts vs. averaging all seven

Selecting vs. averaging. On VideoPhy-2, averaging all seven experts with equal weights and selecting them by ℓ(y)\ell(y) differ by only 0.04. VideoPhy-2 prompts usually involve several mechanisms (3.1 on average), so the selected experts and the full set give close corrections. To isolate the effect of selection, we took 105 held-out VideoPhy-2 prompts, disjoint from the test captions, whose labels have a single primary mechanism, 15 per mechanism. On these prompts, selection is ahead of averaging by 0.105:

105 single-mechanism promptsMean ↓Quality ↑error reduction vs. no correction ↑
Wan2.2, no correction1.3053.48–
all seven experts averaged1.0153.860.290 [0.015, 0.552]
PhysRect, experts selected by ℓ(y)\ell(y)0.9103.980.394 [0.103, 0.667]

Capacity. The gain does not come from the added parameters. A single network without mechanism labels, widened to 73.2M parameters to match the mixture's 73.5M and trained on the same fits, reaches 57% of the gain, and its error is higher than ours by 0.236:

variantparamsMean ↓Quality ↑
Wan2.2, no correction–1.7592.98
single network, no mechanisms73.2M1.4423.29
PhysRect, experts selected by ℓ(y)\ell(y)73.5M1.2073.58

W2 / Q2: Was α=0.5\alpha = 0.5 selected on independent data?

In the paper, α\alpha was chosen from the sweep in Fig. 6a, which uses the test prompts. We have now repeated the sweep on a separate validation set of 300 prompts, drawn from the VideoPhy-2 training split and disjoint from the test captions:

α\alpha0 (Wan2.2)0.250.50.751.0
Mean ↓1.7041.1501.2381.3681.413
Quality ↑3.023.633.533.363.34

The gain is robust to α\alpha, the difference between α=0.25\alpha = 0.25 and 0.50.5 is within noise (0.088 [−0.081, 0.255]), and larger values are worse. We will describe this protocol in Sec. 4.1.

W3 / Q3: Cost and necessity of Gemini

The label is read from the prompt text alone during inference: one text-only call per prompt, made once before sampling, with no effect on the cost per denoising step. With Gemini 3 Flash it takes 1.31 s and $0.00049 per prompt on average. The 38.3 s in Table 1 does not include it, since labels are computed before sampling. Adding it gives 39.6 s per video against 38.9 s for the uncorrected backbone, 0.7 s (1.8%) more.

Open-source labeller. Gemini can be replaced by a local open-source model without retraining and without loss. We relabelled the 591 test prompts with Qwen3-VL-8B-Instruct, with the same instruction, and regenerated them with the same experts:

labeller at inferenceMean ↓Quality ↑label latency / cost per prompt
Gemini 3 Flash (paper)1.2073.581.31 s / $0.00049
Qwen3-VL-8B-Instruct, one GPU1.2033.590.24 s / –

The two differ by 0.003 [−0.102, 0.106], and Qwen3-VL-8B gives the same primary mechanism as Gemini on 97% of the prompts.

Thanks for the response and the new experiments. My main concerns about the capacity-matched baseline, α validation, and the Gemini dependency are mostly addressed, so I'm glad to raise my score.

Reviewer_hQMHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PhysRect, a framework for improving the physical consistency of frozen video generators through text embedding rectification. Rather than modifying the pretrained generator or intervening during the denoising process, PhysRect directly adjusts the text conditioning before video synthesis. Specifically, the method first optimizes event-specific text embeddings to reconstruct real physical events using a frozen video generator, with an additional separability objective encouraging corrections associated with different physical mechanisms to remain distinguishable. These optimized corrections are subsequently distilled into a mechanism-aware mixture of experts, which predicts text embedding adjustments from input prompts.

Critical strengths
  • A novel and well-motivated perspective on physics-aware video generation. The paper explores an interesting alternative to existing approaches for improving physical consistency in video generation. Instead of fine-tuning the pretrained video generator or introducing additional interventions during denoising, PhysRect directly modifies the text embedding before generation. This design is motivated by the observation that important aspects of video structure emerge early in the denoising process, potentially limiting the effectiveness of methods that attempt to correct physical inconsistencies during sampling. By operating on the conditioning representation, PhysRect provides a computationally efficient mechanism for influencing generation throughout the denoising trajectory while preserving the pretrained generator.
  • Strong empirical improvements supported by meaningful ablation studies. PhysRect achieves substantial improvements over the original Wan2.2 backbone on VideoPhy-2. In particular, the reported mean physics error decreases from 1.431 to 0.705, while the overall quality score increases from 3.18 to 3.99. The ablation studies further support the effectiveness of the proposed architecture. Comparisons with a single correction network, prompt-independent adjustments, and token-averaged corrections demonstrate the importance of mechanism-specific and prompt-dependent embedding rectification. Additional experiments investigate the influence of correction strength and the separability objective. These results provide meaningful empirical evidence for the practical effectiveness of the proposed approach.
Critical weaknesses
  • Insufficient evidence that the learned embedding corrections specifically encode transferable physical mechanisms. Although PhysRect demonstrates substantial improvements in physical plausibility metrics, the underlying interpretation of the learned corrections remains insufficiently established. The event-specific embeddings are optimized to reconstruct complete video clips using a frozen generative model. Consequently, these optimized representations inevitably contain information about appearance, camera configuration, scene composition, and motion patterns, in addition to physical dynamics. The learned corrections may therefore capture correlations between visual characteristics and physical outcomes rather than explicitly representing transferable physical principles. Furthermore, the separability objective encourages embedding displacements associated with different predefined mechanisms to become distinguishable. However, the resulting separation in embedding space does not necessarily establish that these representations disentangle physical properties from appearance-related factors.
  • Limited evidence of generalization and insufficient diversity in the experimental evaluation. Although the proposed PHYSEVENTS pipeline collects 32,828 physical events, the mixture-of-experts model is trained using only 576 fitted events. While this relatively small training set makes the reported improvements interesting, the paper does not sufficiently investigate how the number and diversity of fitted events influence the learned corrections or their generalization capabilities. Moreover, the main experiments are conducted using a single frozen video generation backbone, Wan2.2-TI2V-5B, and evaluated primarily on VideoPhy-2 through automated vision-language model judgments. Although the selected evaluation protocol provides a systematic assessment of physical plausibility, such judgments do not directly establish adherence to quantitative physical laws. Consequently, it remains unclear whether PhysRect generalizes effectively to unseen physical mechanisms, more complex interactions, or different pretrained video generation architectures. Additional experiments involving cross-backbone transfer, varying training data scales, and independent human or quantitative physics-based evaluation would substantially strengthen the evidence for broader applicability.
Questions for authors
  • How can the authors distinguish improvements in physical consistency from changes in appearance, scene composition, or memorized motion patterns introduced by the optimized text embeddings? In particular, can the authors provide controlled experiments demonstrating that the learned corrections remain effective when the same physical mechanism is presented under substantially different visual conditions?
  • Why are only 576 fitted events used to train the mechanism experts despite the collection of more than 32K physical events? How does performance scale with the number and diversity of fitted events, and to what extent can the learned corrections generalize across different video generation backbones or previously unseen physical scenarios?
4 comments

W1 / Q1: Physics vs. appearance, scene composition and memorised motion

Appearance. The gain does not come from changes in appearance. Appearance is one of the seven mechanisms, with its own expert, so we split the 591 test prompts of Table 1 by whether their label includes it. The reduction is slightly larger on the prompts where it does not:

promptsnWan2.2PhysRecterror reduction ↑
appearance expert inactive2861.4260.6440.782 [0.634, 0.933]
appearance expert active3051.4350.7610.674 [0.529, 0.816]

A shared change in appearance. A change in appearance shared across prompts accounts for only 16% of the gain. Table 2 includes this control: the mean displacement of all training fits, added to every prompt. Because it is the same for every prompt, it can carry an overall change of style, appearance or camera, but not the physics of a particular prompt. It lowers the error from 1.431 to 1.317, against 0.705 for PhysRect. The rest of the gain requires a correction that depends on the prompt.

Same mechanism, different visual conditions. We drew 50 VideoPhy-2 test prompts at random and had an LLM rewrite each into three new scenes, changing the location, time of day, lighting, colour palette, background and camera framing while keeping the interacting objects and the action. Wan2.2 and PhysRect were run on all 200 prompts:

promptsnWan2.2PhysRecterror reduction ↑
original scenes501.8361.2640.572 [0.168, 0.964]
rewritten scenes1501.8931.4490.444 [0.216, 0.661]

On the rewritten scenes, PhysRect still reduces the error by 0.444, so the gain holds under substantially different visual conditions.

Memorisation. The experts never see a test prompt: VideoPhy-2 prompts are written independently of our stock-footage events, and the rewritten scenes above are new to both the generator and the experts.

W2 / Q2: Only 576 fitted events: scale, diversity and other backbones

Selection and diversity. We fit the events that the frozen generator reconstructs worst, where a correction is most needed. For every candidate event we measure how far a renoise-denoise round trip through the generator moves its latent, and keep the events whose latent changes markedly. The fitted events cover all seven mechanisms, and an event carries 3–4 of them on average. Counts by dominant mechanism, for the 600 fitted events and for the VideoPhy-2 test prompts:

rigid motionbody motionfluidgranulardeformationfractureappearancetotal
fitted events2831586923282712600
VideoPhy-2 test3671392110341010591

Number of fitted events. PhysRect adds only 73.5M parameters to a 5B generator and is data-efficient: with 144 events the gain is already as large as with all 576. We retrained it on mechanism-stratified subsets of the training events and scored all models on the 591 VideoPhy-2 prompts (same setting as Table 1):

training events144 (25%)289 (50%)576 (100%)
Mean ↓1.0561.1861.087
Quality ↑3.703.583.68
Error reduction vs. no correction ↑0.630.500.60

Other backbones. We adapted the experts trained on Wan2.2-5B to Wan2.1-T2V-1.3B. The two models share the umT5 text encoder, so the corrected embedding lives in the same space, while their DiT and VAE differ. We add a rank-16 LoRA to the experts (2.87M trainable parameters, with the experts and the generator frozen) and train it for 500 steps with Wan2.1's own flow-matching loss on the same training clips, about one hour on four GPUs. With this adaptation, PhysRect also lowers the physics error of Wan2.1:

Wan2.1-1.3B, 591 promptsMean ↓Quality ↑error reduction ↑
no correction1.4763.26–
PhysRect, adapted from Wan2.2-5B1.3513.360.125 [0.005, 0.243]

Complex scenarios. The VideoPhy-2 prompts are written independently of our training events, and 96% of them combine two or more mechanisms (3.1 on average), so the gains in Table 1 are measured on prompts with interacting mechanisms.

W2 (evaluation): Automated VLM judgments only

JudgeFit (Cao et al., 2026) is not an uncalibrated VLM: it calibrates the judge against human physics annotations and keeps, per judge, only the error dimensions it assesses reliably. We use its configuration with the highest human agreement, and the judge was not used at any stage of training.

We are also running a blind user study of PhysRect against Wan2.2 and will post here once it is complete.

W2 (evaluation): User study results

We conducted a blind user study with 15 participants on 20 of the 591 VideoPhy-2 prompts, randomly sampled and stratified by physics category so that every category is represented. Each pair shows PhysRect (ours) and Wan2.2 generated from the same prompt and the same initial noise, with the left and right position randomized per participant. Participants chose the video with more plausible physics or marked the two as about the same.

On the pairs where more participants preferred one video, PhysRect wins 86% (sign test, p = 0.01). Excluding ratings of about the same, participants preferred PhysRect 73% of the time, and each of the 15 participants individually favored PhysRect.

Dear Reviewer hQMH,

Thank you again for your review. As the discussion period ends soon, we would like to check whether our responses have addressed your concerns. Since our first reply we have added the completed user study (W2), which shows a clear human preference for PhysRect over Wan2.2, together with the new results on other backbones and the additional seeds.

We would be glad to answer any further questions or run additional analyses you think would help. If our responses have resolved your concerns, we would be grateful if you would consider updating your score.

Best regards,

The authors

Dear Reviewer hQMH,

We understand this is a busy time, and thank you again for your detailed feedback. As the discussion period ends shortly, we wanted to follow up once more in case our responses and the new experiments are helpful for your assessment. We would be glad to clarify anything that remains unclear.

Best regards,

The authors

Reviewer_ZquMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PhysRect, which improves the physical consistency of a frozen video generator by correcting its text embedding once before sampling. In training, the embedding which the generator reproduces the clip keeps different mechanisms apart using separation term. The fitted embeddings are then used into a mixture of mechanism experts gated by a VLM-labeled mechanism vector. At inference, the prompt embedding is corrected as with the original text embedding from the description. Experiments are conducted on VideoPhy-2 with Wan2.2-5B as the backbone.

Critical strengths
  1. Splitting the experts by physical mechanism makes the intent of the design clear, and Table 2 supports that the correction should be mechanism-specific, per token, and prompt-dependent.

  2. Since the generator is frozen and the embedding is corrected only once before sampling, there is no extra cost per step, and it still performs comparably to or better than fine-tuning based methods on VideoPhy2.

  3. The paper builds a pipeline, PhysEvents, that extracts physical events from real videos and validates each stage against human labels.

Critical weaknesses
  1. There seems to be a mismatch between training and inference. During training, the mechanism vector is labeled from the clip, mask, and description, but at inference it is read from the prompt alone, and the gating weights are average of the binary in training but graded-only at inference. The paper argues that the two agree because the description is written from the clip, but this only holds for the training descriptions, not for the test prompts.

  2. Although the paper presents a pipeline that produces 32,828 events, only 600 of them are actually fitted and used for training for experts. It is not explained how these events are selected or how they are distributed across mechanisms.

  3. It is not clear whether the improvement really comes from physics. The authors already note that the fitted displacement also carries appearance and camera in limitation section. However, the appearance is even one of the mechanisms, so the gain may partly come from changes in appearance or style.

  4. (Minor) The experts are trained by matching velocities instead of regressing Δ∗\Delta^* directly, but there is no comparison with direct regression.

Questions for authors

Refer to the weaknesses.

2 comments

W1: Mismatch between training and inference labels

Label consistency. We re-ran the labeller, with the same instruction, on the 653 training descriptions and the 591 VideoPhy-2 prompts, and compared every new label with the one used in the paper. The per-mechanism agreement (macro F1) is 0.966 on the training descriptions and 0.968 on the test prompts, so test prompts are labelled as reliably as the training descriptions.

Binary vs. graded weights. We regenerated all 591 VideoPhy-2 prompts with the training-time weights wc=1[ℓc>0]w_c = \mathbf{1}[\ell_c > 0] in place of the graded wc=ℓcw_c = \ell_c, with the same experts, seeds and sampler. Binary weights change the mean error by −0.017 [−0.126, 0.094], so the mismatch does not affect the gain. We keep graded weights because they preserve the labeller's distinction between primary and secondary mechanisms.

W2: How the fitted events were selected and distributed

Selection. We fit the events where a correction is most needed. For every candidate event we measure how far a renoise-denoise round trip through the generator moves its latent, and keep the events whose latent changes markedly. Counts by dominant mechanism, for the 600 fitted events and for the VideoPhy-2 test prompts:

rigid motionbody motionfluidgranulardeformationfractureappearancetotal
fitted events2831586923282712600
VideoPhy-2 test3671392110341010591

Number of events. PhysRect adds only 73.5M parameters to a 5B generator and is data-efficient: with 144 events the gain is already as large as with all 576. We retrained it on mechanism-stratified subsets of the training events and scored all models on the 591 VideoPhy-2 prompts (same setting as Table 1):

training events144 (25%)289 (50%)576 (100%)
Mean ↓1.0561.1861.087
Quality ↑3.703.583.68
Error reduction vs. no correction ↑0.630.500.60

W3: Is the gain physics, or appearance and style?

The appearance expert. The gain does not come from the appearance expert: splitting the 591 prompts of Table 1 by whether it is active, the reduction (↑) is slightly larger when it is inactive:

promptsnWan2.2PhysRecterror reduction ↑
appearance expert inactive (ℓapp=0\ell_{\mathrm{app}} = 0)2861.4260.6440.782 [0.634, 0.933]
appearance expert active3051.4350.7610.674 [0.529, 0.816]

Same physics, different appearance. We drew 50 VideoPhy-2 test prompts at random and had an LLM rewrite each into three new scenes, changing the location, time of day, lighting, colour palette, background and camera framing while keeping the interacting objects and the action. Wan2.2 and PhysRect were run on all 200 prompts:

promptsnWan2.2PhysRecterror reduction ↑
original scenes501.8361.2640.572 [0.168, 0.964]
rewritten scenes1501.8931.4490.444 [0.216, 0.661]

On the rewritten scenes, PhysRect still reduces the error by 0.444, so the gain holds when the appearance changes and the physics stays the same.

A shared change in appearance. A change in appearance shared across prompts accounts for only 16% of the gain. Table 2 includes this control: the mean displacement of all training fits, added to every prompt. Because it is the same for every prompt, it can carry an overall change of style, appearance or camera, but not the physics of a particular prompt. It lowers the error from 1.431 to 1.317, against 0.705 for PhysRect. The rest of the gain requires a correction that depends on the prompt.

W4: Velocity matching vs. direct regression of Δ∗\Delta^\ast

We trained the same seven experts on the same fits with the loss replaced by ∥f(C0,ℓ)−Δ∗∥2\|f(C_0, \ell) - \Delta^\ast\|^2, everything else fixed, and generated the 591 VideoPhy-2 prompts (same setting as Table 1) with both:

distillation targetMean ↓Quality ↑
none (Wan2.2)1.7592.98
regress Δ∗\Delta^\ast1.5213.24
match velocity (ours)1.2073.58

Velocity matching (ours) lowers the error by 0.315 over regression. Its loss is measured in the generator's output rather than in the embedding, so differences in Δ∗\Delta^\ast that leave the video unchanged are not penalised. We will add this comparison to Table 2.

Thank you for the detailed response and the additional experiments. My concerns are almost addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.