acceptodds
Under review as a conference paper at ICLR 2027

Physics-AdaNet: Learning Physical Laws via Adaptive Architecture Alignment across Diverse Scenarios

Abstract

Recent advances in video generation aim to produce physically realistic content. However, existing methods often fail to maintain physical consistency under Out-of-Distribution (OOD) environmental changes. Motivated by this observation, we study the task of OOD generalization of physical laws in video generation, where a model must infer physically plausible motion in unseen environments by recomposing invariant physical laws from previously observed motion primitives, rather than extrapolating memorized visual patterns or numerical parameters. This setting poses two key challenges: extracting transferable physical structures from high-dimensional visual observations and adapting the alignment mechanism to diverse physical environments. To address these challenges, we propose Physics-AdaNet. Specifically, our approach employs Adaptive Physics Alignment (AdaAlign), which bridges the gap between latent representations and physical priors via temporal masking, fostering structural physical understanding rather than relying on statistical patterns. Meanwhile, to accommodate diverse physical environments, we incorporate Neural Architecture Search (NAS) to adaptively tune the alignment architecture for each scenario, further enhancing generalization capacity. Experiments on multiple benchmark tasks with structural OOD shifts demonstrate that Physics-AdaNet achieves the best kinematic accuracy across all OOD scenes, where the invariant governing law must be re-composed rather than recalled, while remaining competitive with state-of-the-art baselines on In-Distribution (ID) scenes and standard visual-fidelity metrics. More visualization results, frame-level qualitative results on real-world footage, and video demonstrations are provided in the appendix and on our project website at https://physicsadanet.github.io/Physics-AdaNet/.

Primary area:
generative models
Keywords:
Video Generation, OOD Generalization, Physical Laws, Neural Architecture Search

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dTnFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies OOD generalization of physical laws in video generation: a model trained on a restricted "primitive" regime of a physical law (e.g., purely vertical projectile motion, or collision with a stationary target) must generalize to a structurally different regime governed by the same invariant law (arbitrary-direction projectile motion, or collision between two moving balls), as formalized in Definition 3.1. To address this, the paper proposes Physics-AdaNet, combining (1) Adaptive Physics Alignment (AdaAlign), a spatial+temporal Token Relation Distillation loss (Eq. 9-10) that aligns the diffusion backbone's intermediate features with relational structure extracted by a pretrained visual encoder from ground-truth video, with a temporal masking mechanism that exempts the I2V conditioning frames from this alignment; and (2) a NAS-based Adaptive Projector that searches (via a weight-sharing supernet + evolutionary search, Eq. 12-14) a scenario-specific architecture for the feature projector used in the alignment, rather than using a fixed projector architecture.

The method is evaluated on two synthetic benchmark tasks built on CogVideoX-5B (Newton's law of universal gravitation, and conservation of momentum), each with an ID split (training-distribution dynamics) and a structural-OOD split (Table 1). Physics-AdaNet outperforms CogVideoX, PISA, and VideoREPA on OOD kinematic error in both tasks while remaining competitive on ID scenes and standard visual-fidelity metrics (FVD/SSIM/PSNR/LPIPS). An ablation (Table 2, Appendix C) attributes gains to both AdaAlign and the NAS projector, and the paper reports that a projector architecture searched on one task degrades when transferred to the other task, which is used to argue the searched architectures are scenario-dependent.

Critical strengths

The problem formulation (Definition 3.1) cleanly isolates structural OOD shift from the scalar-parameter OOD shift studied in prior work (Kang et al., 2024), and the two benchmark tasks operationalize this distinction concretely and verifiably via closed-form kinematics (Eq. 5-8).** Because ground-truth trajectories are known exactly, the paper can report a well-defined kinematic error metric rather than relying only on pixel-level fidelity, which is the right evaluation axis for the claim being made (physical-law generalization, not just visual quality), and the CogVideoX failure cases in Fig. 2(b) make the targeted failure mode concrete.

Critical weaknesses

W1. The benchmark consists of only two synthetic, low-dimensional kinematic scenarios (single ball under gravity; two-ball 1D elastic collision), and all headline claims rest entirely on this benchmark.

Both tasks have closed-form, low-order-polynomial solutions and a handful of scalar parameters (position, velocity, radius, color). This is a reasonable minimal testbed to demonstrate the structural-OOD concept, but it is a narrow basis for the paper's broader framing (Introduction, Conclusion) that the method addresses "OOD generalization of physical laws" in video generation generally. There is no scenario with more than two interacting bodies, no rotational dynamics, no scenario where the structural shift could plausibly be confused with a simple interpolation of training trajectories, and no systematic variation of how large the structural gap is. The real-world qualitative result (Fig. 3b, one scene) is explicitly preliminary and not quantified. Given this, the quantitative superiority in Table 1 is good evidence for the two tasks as constructed, but weaker evidence for the paper's stated general claim than the framing suggests.

W2. The NAS component's claimed benefit (scenario-specific architecture adaptation) is undercut by the paper's own cross-task transfer result, and no comparison is made against simpler alternatives to NAS.

Section 4.3 reports that a projector searched on one task "degrades when retrained on the other task," which the paper frames as confirming architectures are scenario-dependent. An equally plausible reading is that the search is overfitting to per-task noise in the small AdaAlign-loss proxy signal rather than discovering a genuinely different and better-matched architecture per scenario; the paper does not report variance across repeated searches, nor compare against a plausible non-NAS baseline (e.g., a single reasonably-sized fixed projector, or a random architecture of matched size) that would show the search is doing more than selecting a well-sized network. Without that control, the ablation (ID/OOD with and without the NAS projector, Table 2) establishes that some projector configuration helps, but not that the evolutionary search over the specific space in Table 3 (Appendix E) is necessary.


Minor Comments

  • The baselines PISA and VideoREPA are adapted from 3D-physical-video and general T2V alignment settings respectively (Sec. 4.1); PISA in particular drops its optical-flow/semantic-segmentation supervision since depth maps are unavailable in 2D. This is disclosed, which is appreciated, but it leaves open whether the baselines are tuned as well for this specific benchmark as Physics-AdaNet.
  • λ\lambda (the AdaAlign weight) is fixed via small-scale preliminary experiments rather than jointly searched with the architecture (Sec. 3.3); this is a reasonable practical simplification but means the reported configuration may not be at the actual optimum for either component.
Questions for authors
  1. NAS vs. simpler baselines. Does the searched projector outperform a fixed, reasonably-sized projector of comparable parameter count, or a few manually chosen configurations, on the same two tasks? This would clarify whether the evolutionary search is finding something non-trivial versus mainly selecting adequate capacity.
  2. Robustness of the "scenario-dependent" claim. Is the degradation observed when transferring the searched architecture across tasks consistent across repeated independent searches (different random seeds for the evolutionary search), or could it reflect search variance rather than a genuine scenario-specific optimum?
5 comments
AuthorsOpenReview ↗

Response to Reviewer dTnF (1)

Response to Reviewer dTnF

We thank the reviewer for the thoughtful assessment. We narrow the scope of our physical-law claims, add manual, random, and reduced-grid architecture-selection controls, analyze search-seed sensitivity conditional on a fixed supernet, and clarify PISA adaptation and λ\lambda selection. We also report guidance-referenced velocity measurements for the real-world generations, while acknowledging their limitations as independent real-world physical validation.

W1. Narrow benchmark scope and the broader “physical-law” framing

We agree that the phrase “learning physical laws” is broader than what our experiments establish. Our study does not recover explicit governing equations, identify symbolic physical laws, or demonstrate general-purpose physical reasoning. We also do not evaluate interactions involving more than two bodies, rotational dynamics, interpolation-controlled structural shifts, or systematic variation of the structural gap; these remain outside the scope of our current empirical evidence.

Our evidence supports a narrower claim: when the governing mechanism remains fixed but a condition variable frozen during training becomes active at test time, Physics-AdaNet improves the transfer of motion regularities consistent with task-specific physical constraints. The reduced OOD velocity errors indicate improved law-consistent motion generation under the evaluated structural shifts, rather than explicit equation recovery. We will therefore replace broad instances of "learning physical laws" with "learning transferable physical priors for structural OOD generalization". We will also limit our empirical claims to the two controlled benchmarks and describe the real-world results as preliminary qualitative and guidance-referenced quantitative evidence beyond synthetic appearance.

Nevertheless, we believe the two controlled settings remain meaningful. Although their governing equations are simple, the compared baselines exhibit substantially larger OOD kinematic errors on these structural shifts:

  • In Task 1, the best baseline OOD vx,errv_{x,\mathrm{err}} is 7.2038, compared with 4.3210 for Physics-AdaNet.
  • In Task 2, the best baseline errors are 6.5456 for vb,pre,errv_{b,\mathrm{pre,err}} and 9.1287 for va,post,errv_{a,\mathrm{post,err}}, compared with 5.1170 and 7.5075 for Physics-AdaNet.

Thus, the results support the narrower conclusion that Physics-AdaNet effectively addresses these two nontrivial structural-motion shifts, without implying a general solution to arbitrary physical laws.

To assess how faithfully the real-world generations preserve the supplied motion guidance, we are adding quantitative velocity measurements for the generations in Figure 3(b) and Figures 14–16. Object velocities will be extracted from the generated real-world videos and compared with the corresponding 2D ground-truth velocity sequences.

Table R1a. Quantitative velocity errors for real-world gravitation scenes, using the corresponding 2D ground-truth velocities as references.

MethodBall: vx,err↓v_{x,\mathrm{err}}\downarrowCube: vx,err↓v_{x,\mathrm{err}}\downarrow
CogVideoX7.7227 ± 0.18507.9426 ± 0.1536
PISA7.5492 ± 0.17798.1159 ± 0.2064
VideoREPA7.2530 ± 0.14647.8372 ± 0.2197
Physics-AdaNet4.3116 ± 0.13335.3161 ± 0.1152

Table R1b. Quantitative velocity errors for real-world collision scenes, using the corresponding 2D ground-truth velocities as references.

MethodBalls: vb,pre,err↓v_{b,\mathrm{pre,err}}\downarrowBalls: va,post,err↓v_{a,\mathrm{post,err}}\downarrowCubes: vb,pre,err↓v_{b,\mathrm{pre,err}}\downarrowCubes: va,post,err↓v_{a,\mathrm{post,err}}\downarrow
CogVideoX6.6790 ± 0.205110.1940 ± 0.28028.0017 ± 0.17229.6915 ± 0.2539
PISA6.4972 ± 0.14279.2825 ± 0.26106.1896 ± 0.14378.5050 ± 0.2345
VideoREPA7.0784 ± 0.180310.7201 ± 0.21676.8201 ± 0.266310.0827 ± 0.3397
Physics-AdaNet5.2197 ± 0.21087.8693 ± 0.31995.2962 ± 0.17347.2537 ± 0.2527

These results show that the improvements in both sustained horizontal motion and collision-induced velocity changes persist across different object appearances. The overall error magnitudes and relative method rankings are broadly consistent with the measurements on the 2D videos. This similarity is expected because the real-world generation process uses the corresponding 2D videos as motion maps: their trajectories guide the warping of the initial noise, while each model generates subsequent frames from the same warped noise and initial frame. The dominant motion structure is therefore shared between the 2D guidance and the real-world generation process, even though their visual appearances differ. Nevertheless, the consistent relative improvements across balls and cubes support the robustness of Physics-AdaNet’s kinematic gains under changes in object morphology and visual appearance.

AuthorsOpenReview ↗

Response to Reviewer dTnF (2)

At the same time, we acknowledge that precise physical parameters of the real-world footage remain unavailable and that these measurements use the corresponding 2D velocity sequences as references. We will therefore present the combined qualitative and quantitative results as preliminary transfer evidence beyond synthetic appearance, rather than as comprehensive validation of unrestricted real-world physical dynamics.

W2 and Q1. Architecture-selection controls: manual, random, and grid alternatives

We agree that the current cross-task degradation alone cannot distinguish genuine scenario dependence from variance in the evolutionary search or its LscoreL_{\mathrm{score}} proxy. Similarly, comparison with only the heavier hand-crafted three-layer MLP does not establish that evolutionary search is necessary.

To address this issue, we add the following controls:

  1. A manually designed, parameter-matched two-layer MLP projector, which provides a shallow manual alternative with manually chosen hyperparameters following VideoREPA, except for the number of layers and the feature dimensions.
  2. Random search, which uniformly samples 455 candidate architectures from the same search space and selects the candidate with the lowest LscoreL_{\mathrm{score}}. Unlike evolutionary search, it does not use previous evaluation results to guide subsequent sampling.
  3. Grid search with a depth prior, which exhaustively evaluates 576 architectures with the projector depth fixed to c=2c=2. Specifically, the reduced grid consists of alignment layers a∈{15,21}a\in\lbrace15,21\rbrace; normalization methods n∈{None,BatchNorm,LayerNorm,RMSNorm}n\in\lbrace\mathrm{None},\mathrm{BatchNorm},\mathrm{LayerNorm},\mathrm{RMSNorm}\rbrace; and all ordered module-type combinations (type1,type2)∈T2(\mathrm{type}_1,\mathrm{type}_2)\in\mathcal{T}^{2}, where T={Linear,Self−Attn,Conv1D}\mathcal{T}=\lbrace\mathrm{Linear},\mathrm{Self-Attn},\mathrm{Conv1D}\rbrace, allowing different module types in the two layers. The hidden dimension satisfies dim∈{2048,4096}\mathrm{dim}\in\lbrace2048,4096\rbrace. The four activation-function configurations are given by (act1,act2)∈A×{Identity}(\mathrm{act}_1,\mathrm{act}_2)\in\mathcal{A}\times\lbrace\mathrm{Identity}\rbrace, where A={Identity,GeLU,SiLU,SwiGLU}\mathcal{A}=\lbrace\mathrm{Identity},\mathrm{GeLU},\mathrm{SiLU},\mathrm{SwiGLU}\rbrace.

This yields 2×4×32×2×4=5762\times4\times3^2\times2\times4=576 candidates. The candidate with the lowest LscoreL_{\mathrm{score}} is selected for standalone training. To keep the evaluation budget broadly comparable, we use a reduced grid with a two-layer depth prior and the restricted axis choices specified above. Exhaustively extending this grid to the full depth range and all per-layer architecture combinations would be computationally prohibitive due to its exponential growth with depth.

All variants use the same masking strategy, alignment objective, value of λ\lambda, training data, and 8,000 final standalone-training steps. Only the architecture-selection strategy changes.

Table R2. Comparison of architecture-selection strategies on OOD scenes. All searched and manually designed alternatives use the same final training protocol.

Selection strategyEvaluated candidatesTask 1 vx,err↓v_{x,\mathrm{err}}\downarrowTask 2 vb,pre,err↓v_{b,\mathrm{pre,err}}\downarrowTask 2 va,post,err↓v_{a,\mathrm{post,err}}\downarrow
Manually designed three-layer MLP16.8020 ± 0.15017.4039 ± 0.24758.2445 ± 0.1951
Manually designed two-layer MLP16.5141 ± 0.14686.7932 ± 0.18848.0492 ± 0.2467
Random search4556.0361 ± 0.17366.3685 ± 0.16978.4528 ± 0.2379
Grid search5764.9208 ± 0.16125.7320 ± 0.18137.9244 ± 0.2635
Evolutionary search4554.3210 ± 0.13555.1170 ± 0.15937.5075 ± 0.2703

The shallow manual control improves over the three-layer default on all three metrics, while its comparison with random search is mixed. This suggests that projector depth and manual configuration explain part of the gain, but do not account for the consistent further improvements achieved by evolutionary search. Grid search is the strongest non-evolutionary alternative. Importantly, its evaluation cost is kept manageable by using the reduced grid described above, including the two-layer depth prior and the restricted axis choices. Extending the exhaustive grid to deeper projectors would be computationally prohibitive due to the exponential growth of per-layer architecture combinations. Even with this depth prior and a slightly larger candidate-evaluation budget than evolutionary search (576 versus 455), grid search remains inferior to evolutionary search on all three final OOD metrics. These results indicate that the gains are not attributable merely to selecting a suitably sized two-layer network or to a larger search budget; the architecture-selection strategy itself contributes to identifying a projector better suited to each scenario.

AuthorsOpenReview ↗

Response to Reviewer dTnF (3)

Q2. Search-seed sensitivity on a fixed supernet

To directly assess the variability introduced by evolutionary search itself, we repeat the search five times on the same trained supernet, using the same held-out validation set and 455 candidate evaluations per run. Only the random seed controlling population initialization, parent selection, crossover, and mutation changes. Tables R3a and R3b report the selected architectures and their corresponding best LscoreL_{\mathrm{score}} values.

Table R3a. Architectures selected by five independent evolutionary searches on the same supernet for Task 1.

Search runAlignment layerNormalizationDepthHidden dimensionModule sequenceActivation sequenceBest Lscore,1↓L_{\mathrm{score},1}\downarrow
Run 118BatchNorm21024(Linear, Linear)(SiLU, Identity)0.0574
Run 218BatchNorm21024(Linear, Linear)(SwiGLU, Identity)0.0576
Run 318BatchNorm21024(Linear, Linear)(SwiGLU, Identity)0.0576
Run 418BatchNorm21024(Linear, Linear)(SiLU, Identity)0.0574
Run 518BatchNorm21024(Linear, Linear)(SiLU, Identity)0.0574

Table R3b. Architectures selected by five independent evolutionary searches on the same supernet for Task 2.

Search runAlignment layerNormalizationDepthHidden dimensionModule sequenceActivation sequenceBest Lscore,2↓L_{\mathrm{score},2}\downarrow
Run 112RMSNorm22048(Linear, Linear)(SwiGLU, Identity)0.0593
Run 212RMSNorm22048(Linear, Linear)(SiLU, Identity)0.0597
Run 312RMSNorm22048(Linear, Linear)(SwiGLU, Identity)0.0593
Run 412RMSNorm22048(Linear, Linear)(SwiGLU, Identity)0.0593
Run 512RMSNorm22048(Linear, Linear)(SwiGLU, Identity)0.0593

Across five independent evolutionary searches on the same trained supernet, the selected architectures exhibit strong stability in their core structural choices. For Task 1, all runs select layer 18, BatchNorm, a two-layer Linear projector, and a hidden dimension of 1024. Only the first-layer activation varies between SiLU and SwiGLU, with the originally reported SiLU configuration recovered in three of the five runs. For Task 2, all runs select layer 12, RMSNorm, a two-layer Linear projector, and a hidden dimension of 2048. The originally reported SwiGLU configuration is recovered in four of the five runs, while the remaining run differs only in using SiLU as the first-layer activation. The two-layer Linear design is preserved in 5/5 runs for both tasks, and all architectural choices other than the first-layer activation remain unchanged across the repeated searches.

The best LscoreL_{\mathrm{score}} values also show limited dispersion. Task 1 scores range from 0.05740.0574 to 0.05760.0576, and Task 2 scores range from 0.05930.0593 to 0.05970.0597. Conditional on the same trained supernet and the same held-out validation set, these results show that the core architecture-selection outcomes are insensitive to the random seed of the evolutionary search. In particular, each task repeatedly recovers the same alignment layer, normalization method, hidden dimension, and shallow Linear projector family, with variation limited to the first-layer activation. We therefore interpret the selected projectors as repeatedly selected candidate designs under the fixed-supernet protocol.

Reviewer_EonbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies compositional generalization in video generation on two motion transfer types: from vertical motion to projectile motion and from stationary-target collisions to collisions between two moving objects. It proposes spatial-temporal alignment with features of VideoMAEv2 and task-specific neural architecture search for the alignment projector. Physics-AdaNet shows better accuracy for out-of-domain motion prediction than CogVideoX, PISA, and VideoREPA.

Critical strengths
  1. The paper demonstrates clear advantages in the proposed test dataset
  2. Ablations show that the alignment design does affect generalization rather than merely improving appearance.
Critical weaknesses
  1. Although the compared models share the same base model of CogVideoX, they don't share the same training cost (e.g., model parameter count, total training steps), which makes the comparison unfair.
  2. The test dataset has only two motion transfer types, so it's hard to say whether the model "learns physical laws" rather than fits better on those specific motion types.
  3. The test dataset has only synthetic data, which has a large distribution difference with real-world data. So the generalization to real-world data requires further evidence, both qualitative and quantitative.
  4. There is not enough ablation on the design of NAS, for example, compared with an equally budgeted shallow-projector sweep or random search.
Questions for authors
  1. Does NAS perform better than other designs like shallow projectors with the same temporal masking and alignment objective?
  2. How to prove the model learns physical laws rather than fitting the two motion types?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Eonb (1)

Response to Reviewer Eonb

We thank the reviewer for the careful assessment. We distinguish the shared final-model training budget from the additional NAS development cost, narrow our physical-law claims, evaluate guided real-world generations against their corresponding 2D motion references, and add manual, random, and reduced-grid architecture-selection controls.

W1. Final-model training budget and fairness of baseline comparisons

We agree that the total development compute is not identical across methods because Physics-AdaNet includes an offline architecture-search stage. We will distinguish this cost explicitly from the training and inference cost of the final compared models.

For the final standalone models, all methods:

  1. use the same CogVideoX-5B backbone;
  2. are fine-tuned on the same task-specific training data;
  3. use full-parameter supervised fine-tuning of the transformer backbone;
  4. undergo 8,000 final training steps.

Thus, the dominant backbone parameter count and final optimization budget are shared. The NAS search is an architecture-selection procedure conducted before this final training, rather than additional gradient steps given to the final Physics-AdaNet model during the comparison.

We provide training loss curves to demonstrate that, at the common 8,000-step budget, all compared methods have reached convergence. The comparison is therefore made between converged final models under the same final training protocol, which provides a reasonable basis for a fair comparison. The corresponding loss curves are shown in Figure 17. Specifically, the losses of CogVideoX, PISA, VideoREPA, and Physics-AdaNet all decrease rapidly during the early stage, continue to decline more gradually through approximately 4,000–5,000 steps, and then stabilize with small fluctuations in the later stage.

For transparency, we will retain the separate NAS overhead analysis in Appendix D: the complete architecture-selection and final-training pipeline costs approximately 25,500 equivalent training steps. We do not claim equal total architecture-development compute. Nevertheless, the final model introduces no additional inference cost over the baseline. Our intended fairness claim is therefore limited to the comparison of converged final models trained with the same backbone and final optimization budget, not identical total research cost.

W2 and Q2. Limits of claiming physical-law learning from two tasks

We agree that the two tasks and their quantitative metrics cannot establish explicit physical-law learning or fully distinguish it from improved fitting of motion patterns. Our study does not recover explicit governing equations, identify symbolic physical laws, or demonstrate general-purpose physical reasoning.

Our evidence supports a narrower claim: when the governing mechanism remains fixed but a condition variable frozen during training becomes active at test time, Physics-AdaNet improves the transfer of motion regularities consistent with task-specific physical constraints. The reduced OOD velocity errors indicate improved law-consistent motion generation under the evaluated structural shifts, rather than explicit equation recovery. We will therefore replace broad instances of "learning physical laws" with "learning transferable physical priors for structural OOD generalization". We will also limit our empirical claims to the two controlled benchmarks and describe the real-world results as preliminary qualitative and guidance-referenced quantitative evidence beyond synthetic appearance.

AuthorsOpenReview ↗

Response to Reviewer Eonb (2)

W3. Guided real-world evaluation and its limitations

We agree that synthetic-to-real generalization requires careful qualification. In addition to the synthetic benchmarks, the current paper contains an initial real-world exploration:

  • Figure 3(b): real-world universal-gravitation footage with a ball;
  • Appendix H, Figure 14: real-world universal-gravitation footage with a cube;
  • Appendix H, Figure 15: real-world collision footage with balls;
  • Appendix H, Figure 16: real-world collision footage with cubes.

These frame-level comparisons use the same initial frame across methods and examine whether the generated videos preserve plausible projectile or collision dynamics. They therefore represent an initial exploration beyond purely synthetic visuals, rather than experiments restricted only to the Box2D appearance domain.

To assess how faithfully the real-world generations preserve the supplied motion guidance, we are adding quantitative velocity measurements for the generations in Figure 3(b) and Figures 14–16. Object velocities will be extracted from the generated real-world videos and compared with the corresponding 2D ground-truth velocity sequences.

Table R1a. Quantitative velocity errors for real-world gravitation scenes, using the corresponding 2D ground-truth velocities as references.

MethodBall: vx,err↓v_{x,\mathrm{err}}\downarrowCube: vx,err↓v_{x,\mathrm{err}}\downarrow
CogVideoX7.7227 ± 0.18507.9426 ± 0.1536
PISA7.5492 ± 0.17798.1159 ± 0.2064
VideoREPA7.2530 ± 0.14647.8372 ± 0.2197
Physics-AdaNet4.3116 ± 0.13335.3161 ± 0.1152

Table R1b. Quantitative velocity errors for real-world collision scenes, using the corresponding 2D ground-truth velocities as references.

MethodBalls: vb,pre,err↓v_{b,\mathrm{pre,err}}\downarrowBalls: va,post,err↓v_{a,\mathrm{post,err}}\downarrowCubes: vb,pre,err↓v_{b,\mathrm{pre,err}}\downarrowCubes: va,post,err↓v_{a,\mathrm{post,err}}\downarrow
CogVideoX6.6790 ± 0.205110.1940 ± 0.28028.0017 ± 0.17229.6915 ± 0.2539
PISA6.4972 ± 0.14279.2825 ± 0.26106.1896 ± 0.14378.5050 ± 0.2345
VideoREPA7.0784 ± 0.180310.7201 ± 0.21676.8201 ± 0.266310.0827 ± 0.3397
Physics-AdaNet5.2197 ± 0.21087.8693 ± 0.31995.2962 ± 0.17347.2537 ± 0.2527

These results show that the improvements in both sustained horizontal motion and collision-induced velocity changes persist across different object appearances. The overall error magnitudes and relative method rankings are broadly consistent with the measurements on the 2D videos. This similarity is expected because the real-world generation process uses the corresponding 2D videos as motion maps: their trajectories guide the warping of the initial noise, while each model generates subsequent frames from the same warped noise and initial frame. The dominant motion structure is therefore shared between the 2D guidance and the real-world generation process, even though their visual appearances differ. Nevertheless, the consistent relative improvements across balls and cubes support the robustness of Physics-AdaNet’s kinematic gains under changes in object morphology and visual appearance.

At the same time, we acknowledge that precise physical parameters of the real-world footage remain unavailable and that these measurements use the corresponding 2D velocity sequences as references. We will therefore present the combined qualitative and quantitative results as preliminary transfer evidence beyond synthetic appearance, rather than as comprehensive validation of unrestricted real-world physical dynamics.

AuthorsOpenReview ↗

Response to Reviewer Eonb (3)

W4 and Q1. Architecture-selection controls: manual, random, and grid alternatives

We agree that the current hand-crafted three-layer MLP comparison does not fully isolate the effect of NAS. To address this issue, we add the following controls:

  1. A manually designed, parameter-matched two-layer MLP projector, which provides a shallow manual alternative with manually chosen hyperparameters following VideoREPA, except for the number of layers and the feature dimensions.
  2. Random search, which uniformly samples 455 candidate architectures from the same search space and selects the candidate with the lowest LscoreL_{\mathrm{score}}. Unlike evolutionary search, it does not use previous evaluation results to guide subsequent sampling.
  3. Grid search with a depth prior, which exhaustively evaluates 576 architectures with the projector depth fixed to c=2c=2. Specifically, the reduced grid consists of alignment layers a∈{15,21}a\in\lbrace15,21\rbrace; normalization methods n∈{None,BatchNorm,LayerNorm,RMSNorm}n\in\lbrace\mathrm{None},\mathrm{BatchNorm},\mathrm{LayerNorm},\mathrm{RMSNorm}\rbrace; and all ordered module-type combinations (type1,type2)∈T2(\mathrm{type}_1,\mathrm{type}_2)\in\mathcal{T}^{2}, where T={Linear,Self−Attn,Conv1D}\mathcal{T}=\lbrace\mathrm{Linear},\mathrm{Self-Attn},\mathrm{Conv1D}\rbrace, allowing different module types in the two layers. The hidden dimension satisfies dim∈{2048,4096}\mathrm{dim}\in\lbrace2048,4096\rbrace. The four activation-function configurations are given by (act1,act2)∈A×{Identity}(\mathrm{act}_1,\mathrm{act}_2)\in\mathcal{A}\times\lbrace\mathrm{Identity}\rbrace, where A={Identity,GeLU,SiLU,SwiGLU}\mathcal{A}=\lbrace\mathrm{Identity},\mathrm{GeLU},\mathrm{SiLU},\mathrm{SwiGLU}\rbrace.

This yields 2×4×32×2×4=5762\times4\times3^2\times2\times4=576 candidates. The candidate with the lowest LscoreL_{\mathrm{score}} is selected for standalone training. To keep the evaluation budget broadly comparable, we use a reduced grid with a two-layer depth prior and the restricted axis choices specified above. Exhaustively extending this grid to the full depth range and all per-layer architecture combinations would be computationally prohibitive due to its exponential growth with depth.

All variants use the same masking strategy, alignment objective, value of λ\lambda, training data, and 8,000 final standalone-training steps. Only the architecture-selection strategy changes.

Table R2. Comparison of architecture-selection strategies on OOD scenes. All searched and manually designed alternatives use the same final training protocol.

Selection strategyEvaluated candidatesTask 1 vx,err↓v_{x,\mathrm{err}}\downarrowTask 2 vb,pre,err↓v_{b,\mathrm{pre,err}}\downarrowTask 2 va,post,err↓v_{a,\mathrm{post,err}}\downarrow
Manually designed three-layer MLP16.8020 ± 0.15017.4039 ± 0.24758.2445 ± 0.1951
Manually designed two-layer MLP16.5141 ± 0.14686.7932 ± 0.18848.0492 ± 0.2467
Random search4556.0361 ± 0.17366.3685 ± 0.16978.4528 ± 0.2379
Grid search5764.9208 ± 0.16125.7320 ± 0.18137.9244 ± 0.2635
Evolutionary search4554.3210 ± 0.13555.1170 ± 0.15937.5075 ± 0.2703

The shallow manual control improves over the three-layer default on all three metrics, while its comparison with random search is mixed. This suggests that projector depth and manual configuration explain part of the gain, but do not account for the consistent further improvements achieved by evolutionary search. Grid search is the strongest non-evolutionary alternative. Importantly, its evaluation cost is kept manageable by using the reduced grid described above, including the two-layer depth prior and the restricted axis choices. Extending the exhaustive grid to deeper projectors would be computationally prohibitive due to the exponential growth of per-layer architecture combinations. Even with this depth prior and a slightly larger candidate-evaluation budget than evolutionary search (576 versus 455), grid search remains inferior to evolutionary search on all three final OOD metrics. These results indicate that the gains are not attributable merely to selecting a suitably sized two-layer network or to a larger search budget; the architecture-selection strategy itself contributes to identifying a projector better suited to each scenario.

Reviewer_SEuXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies OOD generalization of physical dynamics in video generation. It introduces two synthetic benchmarks based on projectile motion and elastic collision, where test scenarios contain motion configurations not seen during training. The proposed Physics-AdaNet aligns CogVideoX intermediate features with the spatiotemporal relation features of VideoMAEv2 and uses NAS to select a task-specific alignment projector. Experiments show improved OOD kinematic accuracy over CogVideoX, PISA, and VideoREPA while maintaining competitive visual quality.

Critical strengths
  1. Meaningful OOD evaluation. The benchmark goes beyond changing scalar parameters and instead introduces unseen motion configurations. The use of trajectory-based metrics is also more relevant to physical consistency than relying only on FVD or perceptual quality metrics.

  2. Consistent empirical improvements. Physics-AdaNet improves the reported OOD kinematic metrics on both tasks, with results averaged over three runs. The paper also includes component ablations, cross-task projector transfer, and morphological-shift experiments, providing a reasonably broad empirical analysis within its synthetic setting.

Critical weaknesses
  1. The two toy tasks show that CogVideoX struggles with the proposed OOD settings, but they do not show that a fixed architecture is the cause. The failure may instead come from the backbone, conditioning design, or limited training data. Results on another backbone and data/training-scale controls are needed before claiming that NAS is necessary.

  2. Table 2 does not fairly isolate the benefit of NAS. The searched projectors in Table 4 are simple two-layer Linear networks, while the hand-designed baseline is a heavier three-layer MLP. A parameter-matched two-layer projector, random/grid search, and a shared fixed projector across both tasks should be compared.

Questions for authors
  1. How many conditioning frames are used in the experiments, and what information is included in the text prompts? Can the method still generalize when only one initial frame is provided?

  2. Does LscoreL_{\text{score}} correlate with the final OOD kinematic accuracy across candidate architectures? Please report rank correlations using independently trained candidates.

  3. How does NAS compare with a manually designed, parameter-matched two-layer Linear projector or simple random/grid search?

  4. What evidence shows that the VideoMAEv2 relation matrix captures physical laws rather than appearance or generic motion? How do spatial-only and temporal-only alignment perform?

  5. Can one shared architecture generalize across both tasks or to a new physical scenario without rerunning NAS and retraining?

  6. Are the gains consistent on a different video-generation backbone or with more training data and longer training?

6 comments
AuthorsOpenReview ↗

Response to Reviewer SEuX (1)

Response to Reviewer SEuX

We thank the reviewer for the constructive questions. We narrow our claims to learning transferable physical priors for structural OOD generalization; add architecture-selection controls, spatial- and temporal-only alignment ablations, and an exploratory score–OOD analysis; clarify the conditioning protocol and the limits of shared-architecture reuse; and explicitly restrict our empirical claims where backbone and data/training-scale evidence is unavailable.

W1. Scope of the causal claim: fixed architectures and OOD failure

We agree that the current experiments do not establish that a fixed architecture causes CogVideoX’s OOD failure. The backbone, conditioning design, training data, and optimization process may all affect OOD performance. We will revise the paper to remove wording that implies such a causal conclusion.

We would also like to clarify the distinct roles of the two components:

  • AdaAlign is designed to improve structural-OOD generalization by transferring spatiotemporal relational structure from ground-truth videos into the generator’s intermediate representations.
  • NAS is not intended to solve structural OOD by itself. Its role is to adapt the alignment architecture to scenario-dependent physical priors, thereby improving the effectiveness of AdaAlign across different physical scenarios.

Therefore, our supported claim is not that “NAS is necessary for OOD generalization,” but that scenario-dependent architecture adaptation improves the effectiveness of physical alignment in the two studied settings. The CogVideoX failure cases motivate the benchmark, while the comparison between fixed-architecture and NAS-selected alignment under the same backbone, data, conditioning, and final training protocol evaluates the incremental contribution of architecture adaptation within our framework, rather than identifying the root cause of the baseline failure.

AuthorsOpenReview ↗

Response to Reviewer SEuX (2)

W2 and Q3. Architecture-selection controls: manual, random, and grid alternatives

We agree that Table 2 does not fully disentangle the benefit of architecture search from projector capacity: the searched projectors are two-layer MLPs, whereas the hand-crafted VideoREPA baseline uses a heavier three-layer MLP.

To address this issue, we add the following controls:

  1. A manually designed, parameter-matched two-layer MLP projector, which provides a shallow manual alternative with manually chosen hyperparameters following VideoREPA, except for the number of layers and the feature dimensions.
  2. Random search, which uniformly samples 455 candidate architectures from the same search space and selects the candidate with the lowest LscoreL_{\mathrm{score}}. Unlike evolutionary search, it does not use previous evaluation results to guide subsequent sampling.
  3. Grid search with a depth prior, which exhaustively evaluates 576 architectures with the projector depth fixed to c=2c=2. Specifically, the reduced grid consists of alignment layers a∈{15,21}a\in\lbrace15,21\rbrace; normalization methods n∈{None,BatchNorm,LayerNorm,RMSNorm}n\in\lbrace\mathrm{None},\mathrm{BatchNorm},\mathrm{LayerNorm},\mathrm{RMSNorm}\rbrace; and all ordered module-type combinations (type1,type2)∈T2(\mathrm{type}_1,\mathrm{type}_2)\in\mathcal{T}^{2}, where T={Linear,Self−Attn,Conv1D}\mathcal{T}=\lbrace\mathrm{Linear},\mathrm{Self-Attn},\mathrm{Conv1D}\rbrace, allowing different module types in the two layers. The hidden dimension satisfies dim∈{2048,4096}\mathrm{dim}\in\lbrace2048,4096\rbrace. The four activation-function configurations are given by (act1,act2)∈A×{Identity}(\mathrm{act}_1,\mathrm{act}_2)\in\mathcal{A}\times\lbrace\mathrm{Identity}\rbrace, where A={Identity,GeLU,SiLU,SwiGLU}\mathcal{A}=\lbrace\mathrm{Identity},\mathrm{GeLU},\mathrm{SiLU},\mathrm{SwiGLU}\rbrace.

This yields 2×4×32×2×4=5762\times4\times3^2\times2\times4=576 candidates. The candidate with the lowest LscoreL_{\mathrm{score}} is selected for standalone training. To keep the evaluation budget broadly comparable, we use a reduced grid with a two-layer depth prior and the restricted axis choices specified above. Exhaustively extending this grid to the full depth range and all per-layer architecture combinations would be computationally prohibitive due to its exponential growth with depth. All variants use the same masking strategy, alignment objective, value of λ\lambda, training data, and 8,000 final standalone-training steps. Only the architecture-selection strategy changes.

Table R1. Comparison of architecture-selection strategies on OOD scenes. All searched and manually designed alternatives use the same final training protocol.

Selection strategyEvaluated candidatesTask 1 vx,err↓v_{x,\mathrm{err}}\downarrowTask 2 vb,pre,err↓v_{b,\mathrm{pre,err}}\downarrowTask 2 va,post,err↓v_{a,\mathrm{post,err}}\downarrow
Manually designed three-layer MLP16.8020 ± 0.15017.4039 ± 0.24758.2445 ± 0.1951
Manually designed two-layer MLP16.5141 ± 0.14686.7932 ± 0.18848.0492 ± 0.2467
Random search4556.0361 ± 0.17366.3685 ± 0.16978.4528 ± 0.2379
Grid search5764.9208 ± 0.16125.7320 ± 0.18137.9244 ± 0.2635
Evolutionary search4554.3210 ± 0.13555.1170 ± 0.15937.5075 ± 0.2703

The shallow manual control improves over the three-layer default on all three metrics, while its comparison with random search is mixed. This suggests that projector depth and manual configuration explain part of the gain, but do not account for the consistent further improvements achieved by evolutionary search. Grid search is the strongest non-evolutionary alternative. Importantly, its evaluation cost is kept manageable by using the reduced grid described above, including the two-layer depth prior and the restricted axis choices. Extending the exhaustive grid to deeper projectors would be computationally prohibitive due to the exponential growth of per-layer architecture combinations. Even with this depth prior and a slightly larger candidate-evaluation budget than evolutionary search (576 versus 455), grid search remains inferior to evolutionary search on all three final OOD metrics. These results indicate that the gains are not attributable merely to selecting a suitably sized two-layer network or to a larger search budget; the architecture-selection strategy itself contributes to identifying a projector better suited to each scenario.

AuthorsOpenReview ↗

Response to Reviewer SEuX (3)

Q1. Conditioning frames, prompt content, and the limits of one-frame conditioning

For the synthetic benchmark experiments, all methods use the same three conditioning frames, and this conditioning protocol is shared by Physics-AdaNet and the compared baselines. Here, the three conditioning frames refer to raw input frames. By contrast, the temporal-mask parameter kk in Section 3.2 is defined over latent frames after temporal encoding. The conditioning prefix corresponds to the first latent frame in the alignment sequence; accordingly, k=1k=1 excludes this latent prefix from the AdaAlign objective and does not imply that only one raw conditioning frame is used. This masking operation affects only the alignment loss and leaves the conditioning inputs unchanged. All compared methods receive the same three raw conditioning frames.

The text prompts are fixed within each task and do not contain sample-specific velocity values:

  • Task 1: "[On a blank 2D background, a solid-colored rigid ball with a certain initial velocity moves downward under the influence of gravity. It accelerates uniformly, gradually increasing in speed. The motion is confined strictly to the 2D plane with no perspective, depth, or change in the ball's size or shape. Clean, sharp, and stable 3-second video at 10 fps.]"
  • Task 2: "[On a blank 2D background, two solid-colored rigid balls with given initial velocities move in a vacuum. They travel in straight lines at constant speeds until they may undergo an elastic collision, after which their velocities change and they continue moving. The motion is confined strictly to the 2D plane with no perspective, depth, or change in the balls' size or shape. Clean, sharp, and stable 3-second video at 10 fps.]"

Thus, the prompts provide only generic qualitative descriptions of the tasks. With one initial frame alone, without sample-specific velocity text or motion guidance, the initial velocity required by the target dynamics is not identifiable from the available input. Our current results therefore do not establish reliable ground-truth-consistent generalization in that setting. This limitation concerns missing input information rather than something that alignment alone can resolve.

Q2. Exploratory correlation between standalone LscoreL_{\mathrm{score}} and OOD kinematic performance

We agree that it is important to examine whether LscoreL_{\mathrm{score}} is a meaningful performance proxy rather than merely a supernet-internal objective. We provide a preliminary analysis using five independently trained architectures per task: the manual three-layer default, the manual two-layer control, and the architectures selected by random, grid, and evolutionary search. Each architecture is trained for 8,000 steps under the identical final training protocol.

At five common checkpoints, [s1,s2,s3,s4,s5][s_1,s_2,s_3,s_4,s_5], we recompute LscoreL_{\mathrm{score}} using Eq. (14) on the same held-out validation set and evaluate the corresponding OOD kinematic performance. At each checkpoint, we compute Spearman’s rank correlation across the five architectures and then average the five step-wise coefficients. Since both LscoreL_{\mathrm{score}} and the OOD errors are lower-is-better, a positive coefficient indicates that lower scores tend to correspond to lower errors.

Table R2. Step-wise and step-averaged Spearman rank correlations between LscoreL_{\mathrm{score}} and OOD kinematic performance. Each step-wise coefficient is computed across the five architecture-selection strategies.

Training checkpointTask 1: ρ\rho for vx,errv_{x,\mathrm{err}}Task 2: ρ\rho for vb,pre,errv_{b,\mathrm{pre,err}}Task 2: ρ\rho for va,post,errv_{a,\mathrm{post,err}}
s1=1000s_1=10000.800.900.80
s2=2000s_2=20000.900.900.50
s3=4000s_3=40000.900.90−0.10
s4=6000s_4=60000.901.000.90
s5=8000s_5=80000.900.701.00
Mean over five checkpoints, ρ‾\overline{\rho}0.880.880.62

The step-averaged correlation coefficients for the three OOD metrics are 0.88, 0.88, and 0.62, respectively. The correlations for Task 1 and Task 2’s pre-collision velocity error remain strong and stable throughout training. In contrast, the correlation for Task 2’s post-collision velocity error is weaker and more variable across intermediate checkpoints. This may be because post-collision dynamics are more complex, involving collision timing and velocity transfer between the two objects, making them harder for the alignment score alone to track during optimization. Nevertheless, its step-averaged coefficient of 0.62 still indicates a clear positive association, which becomes stronger near convergence. Overall, these results support LscoreL_{\mathrm{score}} as an imperfect but useful proxy for selecting architectures with better final OOD kinematic performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and the Reviewer Discussion

Dear AC,

Thank you and the reviewers for your time and effort in reviewing our paper.

We have carefully addressed all reviewer comments through detailed responses, additional experiments, and manuscript revisions. So far, we have not yet received responses from any of the three reviewers regarding our rebuttal and revised manuscript.

With approximately one day remaining in the discussion period, we would be grateful if you could kindly encourage the reviewers to review our latest responses and revised manuscript. We would welcome any further feedback and remain happy to respond promptly to any remaining questions or concerns.

Thank you very much for your assistance and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.