acceptodds
Under review as a conference paper at ICLR 2027

Sim-∞: Simple Simulations Improve Physics in Photorealistic Video Generation

Abstract

Can deliberately simple, non-photorealistic simulations improve physics in photorealistic video generation? In this work, we show that a carefully designed fine-tuning recipe substantially improves the physical fidelity of a pretrained image-to-video model across multiple challenging physical skills, while preserving photorealism and general-domain video quality. We find that visually abstract, minimally textured simulations paired with detailed captions effectively teach targeted physical skills. Meanwhile, co-training with randomly sampled real videos preserves the model’s photorealism. Our experiments investigate three challenging physical skills: object permanence under occlusion, rigid-body collision dynamics, and deformable-body wave propagation. On a benchmark with independently constructed photorealistic physical scenes, a single model trained jointly on all three skills significantly outperforms a real-video-only baseline across all physics types. Physics-IQ-Verified increases from 47.98 to 49.80, while VBench++ scores remain closely matched. Ablations examine how the simulation-to-real-video ratio and caption design affect skill transfer and appearance retention. The method requires neither expensive photorealistic simulation rendering nor a simulator at inference time, offering a scalable way to expand the physical capabilities of video world models.

Primary area:
generative models
Keywords:
Sim-to-Photorealistic, Image-to-Video generation, Physics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PgHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper works on whether simple and non-photorealistic simulations can improve physical behavior in photorealistic video generation. The authors construct about 297K simulation clips covering object permanence, collision dynamics, and wave propagation. They fine-tune Cosmos3-Super using 30% simulation data and 70% real videos, with detailed captions describing the scenes and events. They evaluate specialist models and a joint model across simulation, synthetic, and photorealistic domains. The results show improvements on the three target tasks and Physics-IQ-Verified, while VBench++ scores remain similar to the baseline.

Critical strengths

S1. The paper addresses a useful question with a simple but efficient approach. Improving physical behavior without expensive photorealistic simulation or additional inference-time components is valuable. The results suggest that simple simulation data can improve the selected skills in photorealistic scenes. The method also keeps the standard image-to-video inference process.

S2. The experiments cover several complementary settings. The authors evaluate three physicial skills across three visual domains and compare specialist training with joint training. The joint model improves all nine task/domain combinations over real-only-SFT. The human comparison, confidence intervals, and failure analysis provide additional support for the observed improvements.

Critical weaknesses

W1. The idea of mixture training recipe is similar as Zhao et al., Synthetic Video Enhances Physical Fidelity in Video Synthesis (ICCV 2025). Zhao et al. already study synthetic-real video mixing, detailed captions, synthetic-domain tags, and the trade-off between physical fidelity and synthetic appearance. Several parts of the proposed recipe closely follow the same approach. This paper appears to extend an existing approach to additional physical scenarios, and I am not yet convinced that the novelty is sufficient.

Reference: Zhao et al., Synthetic Video Enhances Physical Fidelity in Video Synthesis, ICCV 2025. https://arxiv.org/abs/2503.20822

W2. The scope of the learned physical skills is not sufficiently verified. The VLM judge agrees well with human rankings, but agreement on score-gap magnitude is only moderate (Spearman 0.42–0.51; MAE 0.17–0.23). Robustness to another judge is also not reported, so the meaning of the numerical gains remains unclear. Generalization is mainly tested within the same activity families. Despite parameter variation in training, systematic tests on unseen physical properties, such as mass ratios, restitution, or rope tension, are limited. The recipe is also tested on only one backbone. These results support task-specific improvements, but provide limited evidence for broader physical generalization.

Questions for authors

Q1: Evaluation robustness. Results from another VLM judge, such as Qwen or Gemini, would help check whether the rankings and score gains are consistent. The number of human annotators and agreement between them also need clarification.

Q2: Physical generalization. The train/test split should show which object properties and physical parameters are unseen during training. Tests on new mass ratios, restitution values, or rope configurations would help separate visual transfer from physical generalization. The paper should also state whether hidden physical properties are given in the prompt.

Q3: Applicability. The recipe is tested on only Cosmos3-Super. Results on another video model would help show whether it works more generally.

4 comments
AuthorsOpenReview ↗

W1: The pipeline is shared; the task and research question are not, and Zhao et al.'s results do not predict ours.

Thank you for taking the time to review our paper and for your feedback and suggestions. We address each weakness and question below.

W1: The pipeline is shared; the task and research question are not, and Zhao et al.'s results do not predict ours.

Synthetic–real mixing, detailed captions, and domain tags describe a broad pipeline used across synthetic-data fine-tuning (e.g., Zhao et al., 2025; Bai et al., 2025). A shared pipeline does not carry over across tasks by default: in Zhao et al.'s own experiments, low-quality assets or low-cost rendering sharply reduce transfer (their Table 2).

1. Task and research question. Zhao et al. ask whether high-fidelity CGI improves text-to-video human-pose integrity, 3D consistency under camera orbits, and layer decomposition, with a single object per scene (their Sec. 2), leaving multi-object interactions to future work (their Sec. 6). We ask whether simple, non-photorealistic simulation, which their Table 2 would predict to fail, improves physics (e.g., object permanence under occlusion (OP), rigid-body collision dynamics (CD), and wave propagation along deformable ropes (WP)) in photorealistic image-to-video generation; our experiments show that it does.

2. Step-by-step differences. The pipeline is shared, but each step is implemented differently from Zhao et al. (Table R6):

StepZhao et al.Sim-∞ (ours)
VideosHigh-fidelity CGI (curated 3D assets, production renderers); one object per sceneSimple, non-photorealistic simulations of primitive shapes with multiple interacting objects (Appendix A.1)
CaptionsEach scene element captioned separately, then merged (their Sec. 3.2)The physical event over time (initial state, motion, contacts, outcome) from simulator metadata
Domain tagsCritical: dance success 12.5% → 90% with "rendered"/"animated" tags (their Table 7)Not critical: removing "simulation" changes photorealistic OP only from 0.77 to 0.75 (Table 6)
TrainingSynthetic–real mixing with a tuned ratio still leaves synthetic artifacts, so they additionally train SimDrop to alleviate these artifacts during inference (their Sec. 3.3)Real–simulation mixture alone (30% simulation, 70% real), no extra model: manual inspection of every 30%-mixture run, including all 2,287 photorealistic and general-benchmark videos of the final model (OP 300, CD 900, WP 534, Physics-IQ-Verified 198, VBench++ 355), found no simulation textures, primitive-shaped objects, or flat shading
Evaluation40 text prompts (10 each for gymnastics, dancing, camera motion, layer decomposition); human success rates, pose confidence, COLMAP; no general-quality benchmark (their Sec. 4)Simulation, synthetic, and photorealistic domains with a VLM judge checked against human annotators; Physics-IQ-Verified and VBench++; specialist versus joint training and cross-skill transfer

Table R6: Step-by-step comparison with Zhao et al. (2025).

References
Qi Zhao, Xingyu Ni, Ziyu Wang, Feng Cheng, Ziyan Yang, Lu Jiang, and Bohan Wang. Synthetic video enhances physical fidelity in video synthesis. In Proceedings of the International Conference on Computer Vision (ICCV), 2025.
Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Zuozhu Liu, Haoji Hu, Pengfei Wan, and Di Zhang. SynCamMaster: Synchronizing multi-camera video generation from diverse viewpoints. In International Conference on Learning Representations (ICLR), 2025.

AuthorsOpenReview ↗

W2 (judge) & Q1: Robustness to another judge and human agreement.

1. Second VLM judge. We evaluated the same photorealistic videos of the five models in Table 2 (excluding the added Wan2.2 rows) with an independent VLM judge from another model family, Gemini 4 Argon (Argon), using the same rubrics. All rankings align with the original VLM judge (GPT): the five-model ranking is identical on every task (Kendall τ = 1 for object permanence under occlusion (OP), rigid-body collision dynamics (CD), and wave propagation along deformable ropes (WP)), and the gains of Sim-∞ over Pretrained remain significant on all three tasks (GPT: +0.18 / +0.49 / +0.59, 95% CIs [0.10, 0.25] / [0.45, 0.53] / [0.54, 0.62]; Argon: +0.11 / +0.42 / +0.46, 95% CIs [0.05, 0.18] / [0.39, 0.44] / [0.43, 0.50]; Table R7).

ModelOP (GPT / Argon)CD (GPT / Argon)WP (GPT / Argon)
MiniMax-H30.73 / 0.620.53 / 0.410.74 / 0.46
Gemini Omni Flash0.45 / 0.410.38 / 0.300.49 / 0.42
Cosmos3-Super (Pretrained)0.54 / 0.480.28 / 0.230.17 / 0.11
real-only-SFT0.48 / 0.440.32 / 0.250.29 / 0.19
Sim-∞0.72 / 0.590.77 / 0.650.76 / 0.57

Table R7: Both VLM judges scored the same videos from one seed (seed 16); the paper's Table 2 averages three seeds, so values differ from it.

2. Human annotators. We added a second independent human annotator, who rated the same 190 videos (95 scene pairs) blind to the first annotator and to all VLM judge scores. The two human annotators prefer the same video in every pair where both express a preference (24 of 30 OP, 29 of 30 CD, and 35 of 35 WP pairs), and their per-video scores agree closely (Krippendorff's α = 0.82 / 0.90 / 0.91 for OP / CD / WP).

3. The meaning of the numerical gains. The size of a single-pair gap is noisy for any two evaluators, including the two humans: the VLM judge scores each video independently, and gap size is inherently subjective. The moderate agreement between the VLM judge and a human on single-pair gap size in Appendix B.1 is at the same level as the agreement between the two humans (Table R8). The performance gains and conclusions of this paper, however, are drawn from scores averaged over all cases. At that level, the gains measured by both VLM judges are equal to or smaller than those measured by the human mean (Table R8, last column: 0.00 to −0.26).

A vs. B (OP / CD / WP)Same preferred video (non-tied pairs)Single-pair gap: SpearmanSingle-pair gap: MAEMean-gain difference (A − B)
Human 2 vs. Human 1100 / 100 / 100%0.57 / 0.44 / 0.270.22 / 0.20 / 0.18−0.13 / −0.11 / −0.05
VLM judge (GPT) vs. Human 1 (Appendix B.1)100 / 96 / 97%0.47 / 0.51 / 0.420.23 / 0.17 / 0.21−0.06 / −0.10 / −0.16
VLM judge (GPT) vs. human mean95 / 97 / 97%0.56 / 0.66 / 0.290.20 / 0.14 / 0.210.00 / −0.04 / −0.13
VLM judge (Argon) vs. human mean91 / 90 / 94%0.62 / 0.56 / 0.410.17 / 0.19 / 0.28−0.14 / −0.05 / −0.26

Table R8: Agreement between evaluators. Mean gain: an evaluator's average score gain of Sim-∞ over real-only-SFT across the 95 pairs; the last column is A's mean gain minus B's. Human mean: the average of the two human annotators' scores.

We have added this analysis to Appendix B.1 (marked in red). The remaining points of W2 (unseen physical properties and broader physical generalization; a single backbone) are addressed in Q2 and Q3.

AuthorsOpenReview ↗

W2 (unseen physical properties) & Q2: Unseen properties and physical information in the prompt.

1. Train/test split. Every photorealistic evaluation case contains configurations that never appear in simulation training (Table R9).

SkillSimulation trainingPhotorealistic evaluation (cases)
Object permanence under occlusion (OP)One ball size per scene: diameter ratio 1.0, equal massesDifferent ball sizes and masses in a scene, diameter ratio 1.8–3.7 (60/100); configurations absent from all training rollouts (37/100); ball fully hidden at frame 0 (3/100)
Rigid-body collision dynamics (CD)40–60 cm balls; 2.2 m Newton's cradle2–24 cm everyday balls; tabletop Newton's cradle (300/300)
Wave propagation along deformable ropes (WP)Rope driven by a rigid controllerRope swung by a person (178/178)
AllPlain materials and floorReal objects and surfaces (all cases)

Table R9: Simulation training versus photorealistic evaluation. Sizes are the values requested when constructing the first frames. We have added this table to Appendix A.3 (marked in red).

2. Physical generalization. The photorealistic scenes use real objects, materials, and physical scales, so material-dependent properties such as restitution and friction differ from those in simulation, although we do not measure their exact values. On these scenes, Sim-∞ still improves OP, CD, and WP by +0.28, +0.47, and +0.50 over real-only-SFT (Table 2), so the learned skills generalize to real objects of different materials and to physical scales absent from simulation. Where a hidden property is needed to define the event (e.g., the ball-to-block mass ratio, the rope's wave speed), the prompt states it; material-dependent properties such as restitution and friction are not given in the prompt.

AuthorsOpenReview ↗

W2 (single backbone) & Q3: Applicability to another video model.

We applied the same recipe to Wan2.2-TI2V-5B (Team Wan et al., 2025), a 5B model from a different family, without backbone-specific tuning: the same simulation data, the same real-video pool, the same joint training on all three skills, the same number of steps (7,500), and our 480p, 189-frame setting rather than Wan's native 720p and 121 frames. Over Pretrained Wan2.2-TI2V-5B, Sim-∞ improves all three skills by +0.41 / +0.46 / +0.58 on OP / CD / WP and Physics-IQ-Verified by +3.64 (33.40 to 37.04), while VBench++ stays at 89.43 vs. 89.10 (Table R10).

ModelOPCDWPPhysics-IQ-VerifiedVBench++ (overall)
Wan2.2-TI2V-5B (Pretrained)0.200.120.1133.4089.10
+ Sim-∞0.610.580.6937.0489.43

Table R10: Photorealistic physics scores (OP/CD/WP) are averaged over three seeds, as in Table 2. We have added these results to Table 2 (marked in red).

Reference
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.

Reviewer_ZPNaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a simple pipeline that injects simulation data into the post-training framework. This stage requires joint training of real and simulated videos. This paper addresses three physics events, including object permanence, rigid-body collision dynamics, and wave propagation. On physics IQ and VBench++, the method shows improvement over the pretrained and real-only pretrained models.

Critical strengths

This paper shows that when injecting simulation videos into the joint training pipeline with real videos, it improves the video generators in physical realism and maintains photorealistic properties.

I take a look at the generated videos in the supplement, and the qualitative comparison shows clear improvement compared to pretrained/oracle video generation models.

Critical weaknesses

The technical contribution is limited to just adding simulation videos to the training dataset.

Even though the qualitative/quantitative results are great, comparing with only pretrained SFT baselines is not convincing enough. The paper proposes a post-training pipeline. I think the authors should include at least one post-training method for comparison. I suggest including this post-training framework: "What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards", Le et al., 2025. The authors should take their current base models (for ex Cosmos3-Super) and post-training with NewtonRewards, for ex V-JEPA rewards can help improve object permanence, and Newton's laws reward can lead to better physical motions in rigid-body collisions.

The authors should discuss the generalizability of the designed pipeline, whether it can scale to general physical laws. And how?

Questions for authors

Right now i put the score as weak rejection due to a lack of comparisons with other strong post-training pipelines. I'm leaning towards increasing the score if the authors can discuss and compare with the NewtonRewards baseline. Otherwise, it's hard to convince of the effectiveness of the proposed framework by just comparing with pretrained/SFT baselines with limited technical contributions.

3 comments
AuthorsOpenReview ↗

W1: Technical contribution.

Thank you for taking the time to review our paper and for your feedback and suggestions. We address each weakness and question below.

W1: Technical contribution. Injecting simulation videos into joint training improves physical realism and maintains photorealistic properties even though the simulations themselves are deliberately abstract and non-photorealistic. That such simplified renders suffice to transfer physical skills (object permanence under occlusion, OP; rigid-body collision dynamics, CD; and wave propagation along deformable ropes, WP) to photorealistic image-to-video generation is our primary finding. This is not expected a priori: prior work reports that low-quality assets or low-cost rendering sharply reduce the success of synthetic-data transfer, and therefore relies on high-fidelity assets and rendering (Zhao et al., 2025, Table 2). Our recipe combines physics-targeted scene design, captions generated from simulation metadata, and a real–simulation mixture that preserves general video quality, with the mixture ratio validated by ablation (Table 5). The simplicity is deliberate: Sim-∞ needs no simulator, extra inputs, or architectural change at inference, so it applies to other backbones. Sim-∞ improves all three skills and Physics-IQ-Verified (PIQ) while keeping VBench++ quality, whereas post-training with NewtonRewards improves OP and CD substantially but WP only slightly, and degrades PIQ and VBench++ (W2).

Reference

Qi Zhao, Xingyu Ni, Ziyu Wang, Feng Cheng, Ziyan Yang, Lu Jiang, and Bohan Wang. Synthetic video enhances physical fidelity in video synthesis. In Proceedings of the International Conference on Computer Vision (ICCV), 2025.

AuthorsOpenReview ↗

W2: Comparison with a post-training method (NewtonRewards).

As suggested, we compared against NewtonRewards (NR; Le et al., 2025). Its code is not released, so we implemented it from the paper with both NR rewards as defined there (Eqs. 10–12): the kinematic reward, the constant-acceleration residual of RAFT optical flow, and the mass reward, the V-JEPA 2 feature distance to the ground-truth simulation video. Starting from Cosmos3-Super, we report the checkpoint with the best NR reward on 50 held-out prompts (10 per family; step 100) and steps 500 and 1,000 (Table R4).

ModelOPCDWPPIQVBench++ (overall)VBench++ (dynamic degree)VBench++ (aesthetic)
Pretrained (Cosmos3-Super)0.540.280.1746.2389.6236.9965.36
+ NR (step 100)0.260.220.0939.5788.8910.9865.78
(Δ)(−0.28)\scriptsize\color{red}(-0.28)(−0.06)\scriptsize\color{red}(-0.06)(−0.08)\scriptsize\color{red}(-0.08)(−6.66)\scriptsize\color{red}(-6.66)(−0.73)\scriptsize\color{red}(-0.73)(−26.01)\scriptsize\color{red}(-26.01)(+0.42)\scriptsize\color{green}(+0.42)
+ NR (step 500)0.710.680.2434.1784.640.4156.83
(Δ)(+0.17)\scriptsize\color{green}(+0.17)(+0.40)\scriptsize\color{green}(+0.40)(+0.07)\scriptsize\color{green}(+0.07)(−12.06)\scriptsize\color{red}(-12.06)(−4.98)\scriptsize\color{red}(-4.98)(−36.58)\scriptsize\color{red}(-36.58)(−8.53)\scriptsize\color{red}(-8.53)
+ NR (step 1,000)0.810.720.2924.6985.730.8157.77
(Δ)(+0.27)\scriptsize\color{green}(+0.27)(+0.44)\scriptsize\color{green}(+0.44)(+0.12)\scriptsize\color{green}(+0.12)(−21.54)\scriptsize\color{red}(-21.54)(−3.89)\scriptsize\color{red}(-3.89)(−36.18)\scriptsize\color{red}(-36.18)(−7.59)\scriptsize\color{red}(-7.59)
+ Sim-∞0.720.770.7649.8089.6645.1265.39
(Δ)(+0.18)\scriptsize\color{green}(+0.18)(+0.49)\scriptsize\color{green}(+0.49)(+0.59)\scriptsize\color{green}(+0.59)(+3.57)\scriptsize\color{green}(+3.57)(+0.04)\scriptsize\color{green}(+0.04)(+8.13)\scriptsize\color{green}(+8.13)(+0.03)\scriptsize\color{green}(+0.03)

Table R4: NewtonRewards (NR) versus Sim-∞. Δ: change from Pretrained. PIQ: Physics-IQ-Verified. Photorealistic physics scores (OP/CD/WP) use seed 16 for all models; the paper's Table 2 averages three seeds, so values differ from it.

1. Gains on OP and CD. As the reviewer anticipated, NR's rewards improve OP and CD substantially over Pretrained (up to +0.27 and +0.44).

2. Limited gain on WP. The kinematic reward assumes constant acceleration, which fits falling or sliding objects but not oscillating rope waves; WP improves only slightly (up to +0.12, reaching 0.29 vs. 0.76 for Sim-∞).

3. Cost to general quality. NR degrades general physics and quality: PIQ falls by up to 21.54 points; on the VBench++ prompts, the videos become nearly static (dynamic degree < 1), the background flickers, objects morph, and aesthetic quality drops by up to 8.53. NR's higher OP score at step 1,000 therefore comes with degraded video quality. Sim-∞ improves all three skills and PIQ (+3.57) while keeping VBench++ quality.

We have added this comparison to Appendix B.7 (marked in red).

Reference
Minh-Quan Le, Yuanzhi Zhu, Vicky Kalogeiton, and Dimitris Samaras. What about gravity in video generation? Post-training Newton's laws with verifiable rewards. arXiv preprint arXiv:2512.00425, 2025.

AuthorsOpenReview ↗

W3: Generalizability to general physical laws.

1. Scaling to new physics. Our three skills span distinct physical mechanisms — state tracking through occlusion (object permanence under occlusion, OP), multi-body contact with abrupt velocity changes at impact (rigid-body collision dynamics, CD), and continuous deformation (wave propagation along deformable ropes, WP) — all produced by the same recipe and learned by a single joint model, which transfers to held-out photorealistic configurations. We treat them as atomic physical behaviors. General physical laws can be approached bottom-up by defining further atomic behaviors and their combinations, and applying to each the pipeline shown to work here: define the success criterion of the behavior or combination (e.g., occlusion followed by contact), design minimal scenes that expose it, simulate varied configurations, generate captions from simulator metadata, and add it to the same joint training with the mixed-data recipe.

2. Generalization to physics not simulated in our data. On Physics-IQ-Verified, Sim-∞ improves over real-only-SFT overall (+1.82, 95% CI [0.49, 3.20]) and scores highest among Pretrained, real-only-SFT, and Sim-∞ in three of the five categories (Table R5): fluid dynamics (51.37) and magnetism (56.74), which are not simulated in our training data, and solid mechanics (52.45), which our simulations only partly cover.

Physics-IQ-VerifiedPretrainedreal-only-SFTSim-∞Sim-∞ − real-only-SFT
Overall46.2347.9849.80+1.82
Solid mechanics47.7750.8952.45+1.56
Fluid dynamics48.5449.4451.37+1.93
Optics34.2034.4032.32−2.08
Thermodynamics51.8245.7950.34+4.55
Magnetism39.4039.3556.74+17.39

Table R5: Physics-IQ-Verified scores by category. Our simulations cover only part of solid mechanics; fluid dynamics, optics, thermodynamics, and magnetism are not simulated.

3. Generalization to another backbone. Because the recipe changes only the training data (a new behavior needs neither curated realistic 3D assets nor photorealistic rendering, and no simulator is needed at inference), it also carries over to other backbones: on Wan2.2-TI2V-5B, it improves OP, CD, and WP by +0.41, +0.46, and +0.58 (Table 2).

We have added this discussion at the end of Section 3.3 (marked in red).

Reviewer_igFrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method to do supervised finetuning on video generative models using both synthetic physics data and real data. The resulting model can achieve good physics consistency on given inputs. The authors evaluate their method on existing benchmarks and using GPT-as-judge, which demonstrates the effectiveness of their training recipe.

Critical strengths
  • The authors demonstrate that synthetic data mixed with real data training can maintain video generation quality and improve video model’s physics plausibility.
  • The generated videos look realistic and physics plausible.
  • The authors provided sufficient details for data generation and evaluation details and results analysis.
Critical weaknesses
  • The paper only focuses on three very specific categories of physics and trained on a lot of data. It is not clear how it can be transferred to various and generic physics scenarios.
  • The quality improvement is still mainly on simulation inputs (or style transferred inputs from simulation) that is quite close to the simulation training data. Results on general benchmarks (Table 2) seems to have limited improvement on the baselines and not comparable with other models.
  • The authors didn’t compare with other physics-grounded video generation methods, including PhysGen, PhysGen3D, Goal Force etc.
Questions for authors
  • Why 30% simulation data is used instead of other ratios?
  • Lack of details of GPU hours of training and data generation.
4 comments
AuthorsOpenReview ↗

W1: Generalization beyond the three physics categories.

Thank you for taking the time to review our paper and for your feedback and suggestions. We address each weakness and question below.

W1: Generalization beyond the three physics categories.

1. Scaling to new physics. Our three skills span distinct physical mechanisms — state tracking through occlusion (object permanence under occlusion, OP), multi-body contact with abrupt velocity changes at impact (rigid-body collision dynamics, CD), and continuous deformation (wave propagation along deformable ropes, WP) — all produced by the same recipe and learned by a single joint model, which transfers to held-out photorealistic configurations. We treat them as atomic physical behaviors. General physical laws can be approached bottom-up by defining further atomic behaviors and their combinations, and applying to each the pipeline shown to work here: define the success criterion of the behavior or combination (e.g., occlusion followed by contact), design minimal scenes that expose it, simulate varied configurations, generate captions from simulator metadata, and add it to the same joint training with the mixed-data recipe.

2. Generalization to physics not simulated in our data. On Physics-IQ-Verified, Sim-∞ improves over real-only-SFT overall (+1.82, 95% CI [0.49, 3.20]) and scores highest among Pretrained, real-only-SFT, and Sim-∞ in three of the five categories (Table R1): fluid dynamics (51.37) and magnetism (56.74), which are not simulated in our training data, and solid mechanics (52.45), which our simulations only partly cover.

Physics-IQ-VerifiedPretrainedreal-only-SFTSim-∞Sim-∞ − real-only-SFT
Overall46.2347.9849.80+1.82
Solid mechanics47.7750.8952.45+1.56
Fluid dynamics48.5449.4451.37+1.93
Optics34.2034.4032.32−2.08
Thermodynamics51.8245.7950.34+4.55
Magnetism39.4039.3556.74+17.39

Table R1: Physics-IQ-Verified scores by category. Our simulations cover only part of solid mechanics; fluid dynamics, optics, thermodynamics, and magnetism are not simulated.

3. Generalization to another backbone. Because the recipe changes only the training data (a new behavior needs neither curated realistic 3D assets nor photorealistic rendering, and no simulator is needed at inference), it also carries over to other backbones: on Wan2.2-TI2V-5B, it improves OP, CD, and WP by +0.41, +0.46, and +0.58 (Table 2).

We have added this discussion at the end of Section 3.4 (marked in red).

AuthorsOpenReview ↗

W2: Gains on simulation-like inputs and general benchmarks.

1. Photorealistic evaluation. Our gains are not limited to simulation-like inputs. The synthetic set is the image-to-image translation of simulation frames, preserving their task-relevant layout and physical setup. The photorealistic set, in contrast, is not close to the simulation training data: its scenes are newly constructed with everyday objects that never appear in the simulated training data. For example, collisions use marbles, billiard balls, and basketballs on tabletops and floors; Newton's cradles are tabletop apparatus rather than 2.2 m pendulums; ropes are swung by a person rather than a rigid controller; and 60 of the 100 object-permanence scenes mix ball sizes, which never occurs in simulation. Overall, Sim-∞ improves photorealistic OP, CD, and WP by +0.28, +0.47, and +0.50 over real-only-SFT, with all 95% confidence intervals above zero (Appendix B.3). Compared to other models on these benchmarks, Sim-∞ also outperforms Gemini Omni Flash on all three skills by +0.28 (OP), +0.38 (CD), and +0.27 (WP); it is comparable to MiniMax-H3 on OP (−0.01) and WP (+0.02) and higher on CD (+0.23).

2. General benchmarks. Physics-IQ-Verified compares generated dynamics with real experimental videos, and VBench++ checks that general video quality is retained. On Physics-IQ-Verified (Table 2), Sim-∞ reaches 49.80 (Pretrained: 46.23; real-only-SFT: 47.98), closing 91% of the gap between Pretrained and MiniMax-H3 (50.15) and exceeding Gemini Omni Flash (45.67). Adding simulation data is commonly expected to hurt general visual quality, since fine-tuning on synthetic or rendered videos carries their visual style into generated videos (Yu et al., 2025; Jin et al., 2026). Sim-∞ does not: on VBench++ (Table 2), where all compared models score between 88.35 and 90.01, it retains the quality of Pretrained (89.66 vs. 89.62), whereas training on simulation data alone lowers it to 89.40 (Table 5).

References
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. GameFactory: Creating new games with generative interactive videos. In Proceedings of the International Conference on Computer Vision (ICCV), 2025.

Wonjoon Jin, Jiyun Won, Janghyeok Han, Qi Dai, Chong Luo, Seung-Hwan Baek, and Sunghyun Cho. Learning to generate highly dynamic videos using synthetic motion data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

AuthorsOpenReview ↗

W3: Comparison with PhysGen, PhysGen3D, and Goal Force.

These methods target physics-conditioned, controllable generation and rely on task-specific inputs or inference-time simulation (Table R2). Sim-∞ is complementary: it improves a general-purpose image-to-video model under its standard image-and-text interface, with no simulator or extra inputs at inference. We have added the following to Appendix B.6 and PhysGen3D to the related work (marked in red).

MethodExtra inputs at inferenceSimulator at inferencePhysics covered
PhysGenForce or velocity; per-object parametersYes (2D rigid-body)2D rigid bodies
PhysGen3DObjects to move; per-object velocity and materialYes (MPM)Rigid, soft, granular
Goal ForceOne force or target forceNoBall collisions, dominoes, swaying flowers
Sim-∞ (ours)NoneNoObject permanence under occlusion (OP); rigid-body collision dynamics (CD); wave propagation along deformable ropes (WP)

Table R2: Inputs and scope of physics-grounded methods.

Rigid-body collision dynamics is the skill covered by all three methods (Table R2), so we evaluated PhysGen3D and Goal Force on our 300 photorealistic rigid-body collision dynamics cases with the same VLM judge. PhysGen3D extends PhysGen from 2D to 3D simulation (Chen et al., 2025, Sec. 3), so we did not run PhysGen separately.

PhysGen3D (using the released pipeline; given our scene-specified velocities and assigned materials): its design cannot represent Newton's cradle or movers outside the first frame, so only 165 of the 300 cases can be expressed. None of these 165 produces the specified collision:

  • 151 fail at single-image reconstruction: objects are either small (median box ≈41 px) or have plain, untextured surfaces (e.g., plain ivory domino blocks up to 270 px tall yield 0–7 feature matches, below the minimum PnP requires), so they are undetected or only partly detected, or their pose and scale cannot be registered correctly.
  • 13 stall in the MPM solver: for 12, the paper's stiffness categories with the released solver's fixed sub-step violate the Courant–Friedrichs–Lewy stability condition; for the other, the object is registered 0.08 units from the camera while the depth map places it at 0.35, and its particles scatter during simulation.
  • 1 completes, but the moving ball deforms and stops after about 0.3 ball diameters, short of the target 5.9 diameters away.

Goal Force (using the released model and its default force magnitude) receives a force on each moving ball, pointing toward the object it should strike; we located both in every first frame (or the frame edge where an off-screen ball enters) and verified each case manually. Sim-∞ uses only the image and text prompt. Goal Force scores below Sim-∞ on every family (Table R3): the gap is small on ball impacts (0.55 vs. 0.62) and large where momentum passes through several bodies (ball-induced domino toppling: 0.66 vs. 0.81; Newton's cradle: 0.20 vs. 0.89).

FamilyGoal ForceSim-∞Δ [95% CI]
Ball impacts0.550.62−0.07 [−0.15, 0.00]
Ball-induced domino toppling0.660.81−0.15 [−0.23, −0.06]
Newton's cradle0.200.89−0.69 [−0.75, −0.63]
All0.470.77−0.30 [−0.35, −0.26]

Table R3: Both methods use one seed (seed 16); the paper averages three seeds, so Sim-∞ values differ slightly from Table 2. Δ = Goal Force − Sim-∞ (paired bootstrap).

Reference: B. Chen et al. PhysGen3D: Crafting a Miniature Interactive World from a Single Image. CVPR 2025.

Q1: Why 30% simulation data? We chose 30% as a working mixture for two reasons. First, since the goal is a photorealistic world model, we kept simulation as a minority of the training mixture so that real data continue to dominate appearance. Second, we wanted a share large enough to test appearance retention under substantial abstract-simulation exposure and to accommodate several skills — the joint model splits it into 10% per skill. The ablation (Table 5) supports this choice: at matched simulation exposure, 30% and 100% simulation achieve completion scores within 0.02 across all three domains, whereas the 100% setting lowers VBench++ from 89.74 to 89.40, sharply reduces motion (dynamic degree from 46.75 to 10.16; appendix Table 12), and makes newly entering objects more simulation-like (Fig. 9).

Q2: GPU hours.

  • Training. Sim-∞ is fine-tuned on 32 nodes with 4 GB200 GPUs each (128 GPUs; batch size 128) for 7,500 steps, which takes about 37.7 hours of optimization time, i.e., about 4.8K GB200 GPU-hours.

  • Data generation. Simulating and rendering the 297K training clips with Blender EEVEE takes about 1.1K L40S GPU-hours (OP ≈ 65, CD ≈ 640, WP ≈ 400), run as parallel L40S workers, and captioning them with Qwen3.8-27B takes about 0.4K A100 GPU-hours (OP ≈ 75, CD ≈ 190, WP ≈ 130), both for a single pass at our measured throughput.

We have added these compute details to Appendix A.2 (marked in red).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.