- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SimForcing, a framework for transferring simulation-derived motion priors to action-conditioned robot video prediction. A simulation world model is first trained on replayed and perturbed robot trajectories, then used both to initialize a student and as a frozen teacher. During adaptation, the student jointly learns simulation- and real-domain prediction, with a latent-motion distillation objective matching adjacent-frame latent differences. The method additionally introduces simulation-latent conditioning with corruption and dropout, and a classifier-free guidance formulation to control reliance on this condition. At inference, the same student first generates simulation latents and then uses them to condition real-domain video generation. Experiments evaluate video prediction on Bridge and InternData-A1, component ablations, and downstream policy initialization on LIBERO.
Critical strengths
The complete framework improves over a real-only baseline using the same backbone. On Bridge, SimForcing improves PSNR from 22.751 to 25.077 and reduces LPIPS from 0.0926 to 0.0673 relative to Wan2.2 SFT. Improvements over the real-only baseline are also observed on InternData-A1 for most metrics. These comparisons provide useful evidence that simulation-derived supervision can improve robot video prediction.
The downstream transfer results provide evidence beyond video reconstruction quality. World-model initialization improves average LIBERO success from 82.6% to 89.2% under action-only training, with smaller improvements under the other two settings. Although the contribution of individual pretraining components remains unclear, the action-only result suggests that the learned initialization contains information useful for policy learning.
Critical weaknesses
-
The additional complexity is not sufficiently justified by the incremental gains. SimForcing introduces simulation pretraining, a frozen teacher, multiple training objectives, and simulation-conditioning pathways. Full world-model inference requires sequential simulation-latent generation and real-video generation, with two prediction branches per real-video denoising step under the default guidance setting. Table 1 reports 4.1 seconds per sample, compared with 1.8 seconds for the real-only Wan2.2 baseline. However, Table 2 shows that joint flow matching with trajectory augmentation already achieves much of the improvement, and the complete method provides mixed additional benefits. The paper therefore does not convincingly establish that the proposed complexity offers a favorable performance–cost trade-off.
-
The substantial degradation in FID and FVD is insufficiently explained. Adding simulation conditioning to configuration (d) improves PSNR, SSIM, and LPIPS, but worsens FID/FVD from 19.18/100.27 to 25.21/137.49. The paper acknowledges a trade-off between prediction fidelity and distributional quality, but does not analyze the cause of this degradation. A clearer account of what each mechanism improves and sacrifices is needed to assess the complete framework.
-
The LIBERO comparisons do not isolate the contribution of the proposed training mechanisms. They compare SimForcing-pretrained initialization against Wan2.2 initialization, leaving the effects of additional robot-domain pretraining confounded with those of motion distillation and simulation conditioning. Downstream comparisons against real-only world-model pretraining and simple simulation–real joint pretraining are needed to establish the specific value of the proposed method. And training and evaluation protocol is incompletely specified(training config, evaluation rollouts per task)
Questions for authors
-
At table 2, what causes the substantial FID/FVD degradation from configuration (d) to the complete model? Can the authors explain which aspects of the generated videos improve or deteriorate, beyond describing the result as a metric trade-off?
-
How does LIBERO performance change when the policy is initialized from a real-only world model or a simulation–real joint-training checkpoint without the proposed distillation and conditioning mechanisms, using the same downstream protocol and comparable pretraining budgets?
-
Please specify the LIBERO protocol: demonstrations per task, training steps, batch size, learning rate, frozen/trainable modules, action prediction and execution horizons, sampling steps, and evaluation rollouts per task. Please also clarify whether the FastWAM reference result uses the same protocol.
Further clarification and additional key experiments could substantially influence my score.
4 comments
Official Comment by Authors (Part 2/2)
The full model, jointly trained with MD and SC, effectively combines motion prediction and simulation guidance to generate more accurate videos in terms of action execution and visual details, leading to better PSNR and SSIM. The VLA success rates reported above also collectively demonstrate the effectiveness of MD and SC.
W3&Q2: LIBERO pretraining baselines
We add LIBERO experiments using real-only world-model pretraining and simulation–real joint pretraining, with comparable pretraining budgets. These experiments use the action-only setting.
| Init. | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| Wan2.2 | 88.0 | 95.6 | 82.8 | 64.0 | 82.6 |
| Real-FM | 90.2 | 98.6 | 91.2 | 73.6 | 88.4 |
| Joint-FM | 91.6 | 97.2 | 91.8 | 71.6 | 88.1 |
| Full | 89.4 | 95.6 | 92.4 | 79.2 | 89.2 |
Both Real-FM and Joint-FM substantially improve over the Wan2.2 baseline. Real-FM achieves an average success rate 0.3 percentage points higher than Joint-FM, possibly due to the domain gap between simulated and real videos, particularly the lack of object interactions in the simulation data. The two models have different strengths across task suites, while our full model achieves the highest average success rate of 89.2%.
Training and evaluation protocol is provided in our response to Q3 below.
Q1: Simulation guidance from (d) to the full model
Compared with configuration (d), the full model introduces simulation conditioning with dropout during training, allowing it to learn how to adaptively use simulation guidance. During inference, the full model first predicts simulation latents, then uses them as conditioning to predict real-video latents, which are finally decoded into a real video. The full model's FID/FVD deteriorate because the simulation latents, injected through residual connections into the real-video prediction branch, affect the predicted distribution of real-video latents. Its PSNR/SSIM improve because it combines real-world motion prediction with adaptive simulation guidance, allowing simulation to provide more accurate action information. We compare the effects of different guidance strengths on PSNR/SSIM.
| cfg | PSNR↑ | SSIM↑ |
|---|---|---|
| =0.0 | 24.620 | 0.854 |
| =0.3 | 25.014 | 0.857 |
| =0.6 | 25.077 | 0.858 |
| =1.0 | 24.857 | 0.856 |
At =0, the model uses only its real-world motion prediction capabilities; at =1, it relies solely on simulation guidance. At =0.6, it combines real-world motion prediction and simulation guidance, leveraging the adaptive guidance capabilities learned during training.
Q3: LIBERO training and evaluation protocol
We fine-tune on 40 LIBERO tasks using 1,712 demonstrations (32–50 per task) for 20,000 steps, with a batch size of 16 per GPU and AdamW (learning rate 1e-4, weight decay 1e-2, and a cosine learning rate schedule). We consider three settings:
- Action-only: train only the action expert, keeping the video expert frozen.
- Unconditional: jointly train both experts with video and action losses; action prediction attends only to the observed first-frame video tokens.
- Video+action: jointly train both experts with video and action losses; action prediction attends to all video tokens, including future frames.
The VAE remains frozen and language embeddings are precomputed in all settings. We predict 32 actions using 10 sampling steps and execute the first 10 before replanning, without temporal ensembling. Evaluation uses 50 rollouts per task (2,000 total). Our reproduced FastWAM baseline uses the same training and evaluation protocol for each corresponding setting.
References:
[1] IRASim: Learning Interactive Real-Robot Action Simulators. ICCV 2025.
[2] Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling. ICLR 2026.
Thank you for adding the LIBERO baselines and clarifying the evaluation protocol. On Bridge, the full model improves PSNR and LPIPS over joint training with trajectory augmentation. On InternData-A1, however, the gains over joint training are more limited: PSNR improves slightly, SSIM is unchanged at the reported precision, and LPIPS worsens. The new LIBERO comparison shows a meaningful gain on Long (79.2% vs. 73.6% with real-only pretraining), but a smaller gain on average (89.2% vs. 88.4%). I remain unconvinced that these incremental gains justify the added complexity of the full method, so I will maintain my original score.
Thank you for your follow-up feedback and for acknowledging the additional experiments.
1. SOTA PSNR/SSIM on both datasets
We acknowledge that the gains over joint training are smaller on InternData-A1. Nevertheless, among methods without embodied pretraining, our full model achieves SOTA PSNR and SSIM on both Bridge and InternData-A1, with SSIM tied on InternData-A1 at the reported precision. This demonstrates consistent competitiveness on the two primary action-to-video prediction metrics emphasized in our work.
2. LIBERO as complementary evidence of downstream transfer
Regarding LIBERO, the full model provides a modest average improvement over real-only pretraining, alongside a more substantial gain on Long. Our motion distillation (MD) and simulation conditioning (SC) primarily target video prediction, and world-model pretraining does not include a loss supervising action outputs. The LIBERO experiments therefore provide complementary evidence of the downstream transfer value of the learned representations, while our core objective and primary evaluation remain focused on world-model video prediction.
3. Pretraining complexity versus downstream deployment cost
Regarding the added complexity, we would like to distinguish pretraining costs from deployment costs. The teacher model used by MD is required only during pretraining and adds no inference overhead. SC increases inference costs for full video prediction in exchange for better PSNR/SSIM. In the reported action-only LIBERO comparisons, however, the video expert is frozen and used as a feature extractor. All models follow the same downstream training protocol, without additional simulation-video generation. These downstream gains therefore do not incur the video-generation overhead of SC.
We appreciate your careful evaluation and hope these clarifications help contextualize the contributions and practical trade-offs of our method.
Official Comment by Authors (Part 1/2)
We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.
W1: Primary evaluation metrics and performance–cost trade-off
(1) PSNR/SSIM/LPIPS as the primary metrics for action-to-video prediction
PSNR, SSIM, and LPIPS compare each predicted video with its corresponding ground-truth (GT) video frame by frame. They measure pixel-level error, local structural similarity, and perceptual distance in deep feature space, respectively, and are averaged over frames and samples. FID and FVD compare feature distributions between generated and real data, computing the Fréchet distance using the means and covariances of the two feature distributions. They therefore require no one-to-one correspondence between predicted and GT videos and do not directly measure whether each prediction matches its corresponding real future.
IRASim [1] notes that action-to-video prediction requires strict adherence to the input actions, in contrast to text-to-video generation, and should therefore prioritize similarity metrics such as PSNR. It also reports that PSNR aligns better with human preferences, while FID/FVD do not align sufficiently well. LPWM [2] likewise uses only PSNR/SSIM/LPIPS for action-to-video tasks and only FVD for stochastic video generation. These findings indicate that PSNR/SSIM/LPIPS are the primary metrics for action-to-video prediction, while FID/FVD provide a complementary assessment of the overall distributional quality of generated videos.
To provide a comprehensive assessment of these components, we report results on the downstream VLA task.
(2) Inference cost and prediction gains
In Tab. 2, models without SC have the same inference cost as the Wan2.2 (SFT) baseline, approximately 1.8 s/sample. FM+TA+MD achieves better prediction metrics than SFT at the same inference cost. MD introduces a teacher model only during training and adds no inference overhead compared with FM+TA. It further improves FID/FVD, although some similarity metrics deteriorate, possibly because motion distillation emphasizes motion consistency without sufficiently optimizing visual details. Adding SC requires the full model to additionally generate simulation latents and perform guided prediction, increasing the total inference time to 4.1 s/sample. This improves PSNR/SSIM but worsens FID/FVD. A possible explanation is that the action information provided by simulation guidance improves video prediction accuracy, while simulation prediction errors and the domain gap between simulated and real videos may also affect the feature distribution of generated videos. Thus, SC trades additional inference cost for improvements in PSNR/SSIM, the primary evaluation metrics in this work, while configurations without SC retain the same inference efficiency as the baseline. The VLA success rates reported above also collectively demonstrate the effectiveness of MD and SC.
W2: Component contributions and fidelity–distribution trade-offs
We use the results in Tab. 2 to explain what the proposed mechanisms improve and sacrifice. First, joint flow matching (FM) and trajectory augmentation (TA) have clear effects: both consistently improve prediction fidelity and distributional quality. When the simulation world model is trained without TA, the teacher has weaker motion prediction capabilities and may provide incorrect guidance to the student, resulting in worse distributional quality. With TA, the teacher's motion prediction capabilities improve substantially, leading to better distributional quality.
For configuration (e), which uses simulation conditioning, we mistakenly reported the results with cfg=1. For a fair comparison, we should use cfg=0.6, the same hyperparameter setting as the full model. We provide results for different cfg values below.
These results show a clear trade-off between prediction fidelity (PSNR/SSIM) and distributional quality (FID). Different cfg values for simulation guidance yield different results. With cfg=1 (conditional generation), the predicted distribution is closer to the real distribution, resulting in better FID. Mixing the predictions with cfg=0.6 effectively combines action adherence with a natural video distribution, yielding better PSNR and SSIM.