acceptodds
Under review as a conference paper at ICLR 2027

SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models

Abstract

Action-conditioned robot world models must respond precisely to robot trajectories while preserving realistic visual dynamics, yet learning both from heterogeneous robot videos remains challenging. Simulation offers structured motion supervision, but appearance differences hinder direct transfer, and inaccurate simulation predictions can misguide real-video generation. We present SimForcing, a simulation-guided framework that uses simulation both as a source of transferable motion knowledge and as a controllable reference for prediction. First, we transfer motion knowledge from a simulation teacher through latent-motion distillation, aligning temporal changes in latent space to internalize motion priors while mitigating the influence of appearance differences. Second, we introduce multi-block simulation conditioning with condition dropout to exploit predicted simulation trajectories without relying excessively on their accuracy. Our simulation-conditioning classifier-free guidance scheme unifies these two ideas by balancing predictions based on internalized motion knowledge with those additionally guided by simulation latents. The jointly trained student generates both simulation conditions and real-domain videos, requiring no additional world model at inference. On Bridge, SimForcing achieves the best PSNR, SSIM, LPIPS, and FVD among the compared methods without external embodied pretraining. Evaluation on InternData-A1 further supports its applicability across robot datasets. Moreover, using our trained world model to initialize a vision-language-action model improves LIBERO success, suggesting its utility for downstream policy learning.

Primary area:
generative models
Keywords:
world model, action-condition video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SMbYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SimForcing, a framework for transferring simulation-derived motion priors to action-conditioned robot video prediction. A simulation world model is first trained on replayed and perturbed robot trajectories, then used both to initialize a student and as a frozen teacher. During adaptation, the student jointly learns simulation- and real-domain prediction, with a latent-motion distillation objective matching adjacent-frame latent differences. The method additionally introduces simulation-latent conditioning with corruption and dropout, and a classifier-free guidance formulation to control reliance on this condition. At inference, the same student first generates simulation latents and then uses them to condition real-domain video generation. Experiments evaluate video prediction on Bridge and InternData-A1, component ablations, and downstream policy initialization on LIBERO.

Critical strengths

The complete framework improves over a real-only baseline using the same backbone. On Bridge, SimForcing improves PSNR from 22.751 to 25.077 and reduces LPIPS from 0.0926 to 0.0673 relative to Wan2.2 SFT. Improvements over the real-only baseline are also observed on InternData-A1 for most metrics. These comparisons provide useful evidence that simulation-derived supervision can improve robot video prediction.

The downstream transfer results provide evidence beyond video reconstruction quality. World-model initialization improves average LIBERO success from 82.6% to 89.2% under action-only training, with smaller improvements under the other two settings. Although the contribution of individual pretraining components remains unclear, the action-only result suggests that the learned initialization contains information useful for policy learning.

Critical weaknesses
  1. The additional complexity is not sufficiently justified by the incremental gains. SimForcing introduces simulation pretraining, a frozen teacher, multiple training objectives, and simulation-conditioning pathways. Full world-model inference requires sequential simulation-latent generation and real-video generation, with two prediction branches per real-video denoising step under the default guidance setting. Table 1 reports 4.1 seconds per sample, compared with 1.8 seconds for the real-only Wan2.2 baseline. However, Table 2 shows that joint flow matching with trajectory augmentation already achieves much of the improvement, and the complete method provides mixed additional benefits. The paper therefore does not convincingly establish that the proposed complexity offers a favorable performance–cost trade-off.

  2. The substantial degradation in FID and FVD is insufficiently explained. Adding simulation conditioning to configuration (d) improves PSNR, SSIM, and LPIPS, but worsens FID/FVD from 19.18/100.27 to 25.21/137.49. The paper acknowledges a trade-off between prediction fidelity and distributional quality, but does not analyze the cause of this degradation. A clearer account of what each mechanism improves and sacrifices is needed to assess the complete framework.

  3. The LIBERO comparisons do not isolate the contribution of the proposed training mechanisms. They compare SimForcing-pretrained initialization against Wan2.2 initialization, leaving the effects of additional robot-domain pretraining confounded with those of motion distillation and simulation conditioning. Downstream comparisons against real-only world-model pretraining and simple simulation–real joint pretraining are needed to establish the specific value of the proposed method. And training and evaluation protocol is incompletely specified(training config, evaluation rollouts per task)

Questions for authors
  1. At table 2, what causes the substantial FID/FVD degradation from configuration (d) to the complete model? Can the authors explain which aspects of the generated videos improve or deteriorate, beyond describing the result as a metric trade-off?

  2. How does LIBERO performance change when the policy is initialized from a real-only world model or a simulation–real joint-training checkpoint without the proposed distillation and conditioning mechanisms, using the same downstream protocol and comparable pretraining budgets?

  3. Please specify the LIBERO protocol: demonstrations per task, training steps, batch size, learning rate, frozen/trainable modules, action prediction and execution horizons, sampling steps, and evaluation rollouts per task. Please also clarify whether the FastWAM reference result uses the same protocol.

Further clarification and additional key experiments could substantially influence my score.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.

W1: Primary evaluation metrics and performance–cost trade-off

(1) PSNR/SSIM/LPIPS as the primary metrics for action-to-video prediction
PSNR, SSIM, and LPIPS compare each predicted video with its corresponding ground-truth (GT) video frame by frame. They measure pixel-level error, local structural similarity, and perceptual distance in deep feature space, respectively, and are averaged over frames and samples. FID and FVD compare feature distributions between generated and real data, computing the Fréchet distance using the means and covariances of the two feature distributions. They therefore require no one-to-one correspondence between predicted and GT videos and do not directly measure whether each prediction matches its corresponding real future.

IRASim [1] notes that action-to-video prediction requires strict adherence to the input actions, in contrast to text-to-video generation, and should therefore prioritize similarity metrics such as PSNR. It also reports that PSNR aligns better with human preferences, while FID/FVD do not align sufficiently well. LPWM [2] likewise uses only PSNR/SSIM/LPIPS for action-to-video tasks and only FVD for stochastic video generation. These findings indicate that PSNR/SSIM/LPIPS are the primary metrics for action-to-video prediction, while FID/FVD provide a complementary assessment of the overall distributional quality of generated videos.

To provide a comprehensive assessment of these components, we report results on the downstream VLA task.

Init.SpatialObjectGoalLongAvg.
Wan2.288.095.682.864.082.6
Joint-FM91.697.291.871.688.1
Joint-FM + MD93.098.890.673.689.0
Joint-FM + MD + SC (Full)89.495.692.479.289.2

(2) Inference cost and prediction gains
In Tab. 2, models without SC have the same inference cost as the Wan2.2 (SFT) baseline, approximately 1.8 s/sample. FM+TA+MD achieves better prediction metrics than SFT at the same inference cost. MD introduces a teacher model only during training and adds no inference overhead compared with FM+TA. It further improves FID/FVD, although some similarity metrics deteriorate, possibly because motion distillation emphasizes motion consistency without sufficiently optimizing visual details. Adding SC requires the full model to additionally generate simulation latents and perform guided prediction, increasing the total inference time to 4.1 s/sample. This improves PSNR/SSIM but worsens FID/FVD. A possible explanation is that the action information provided by simulation guidance improves video prediction accuracy, while simulation prediction errors and the domain gap between simulated and real videos may also affect the feature distribution of generated videos. Thus, SC trades additional inference cost for improvements in PSNR/SSIM, the primary evaluation metrics in this work, while configurations without SC retain the same inference efficiency as the baseline. The VLA success rates reported above also collectively demonstrate the effectiveness of MD and SC.

W2: Component contributions and fidelity–distribution trade-offs

We use the results in Tab. 2 to explain what the proposed mechanisms improve and sacrifice. First, joint flow matching (FM) and trajectory augmentation (TA) have clear effects: both consistently improve prediction fidelity and distributional quality. When the simulation world model is trained without TA, the teacher has weaker motion prediction capabilities and may provide incorrect guidance to the student, resulting in worse distributional quality. With TA, the teacher's motion prediction capabilities improve substantially, leading to better distributional quality.

For configuration (e), which uses simulation conditioning, we mistakenly reported the results with cfg=1. For a fair comparison, we should use cfg=0.6, the same hyperparameter setting as the full model. We provide results for different cfg values below.

Additional results for (e)PSNR↑SSIM↑FID↓FVD↓LPIPS↓
cfg=022.0690.83633.73257.310.1050
cfg=0.323.5920.84637.74253.290.0925
cfg=0.624.0860.85026.07144.710.0762
cfg=1.023.7320.84821.83145.400.0772

These results show a clear trade-off between prediction fidelity (PSNR/SSIM) and distributional quality (FID). Different cfg values for simulation guidance yield different results. With cfg=1 (conditional generation), the predicted distribution is closer to the real distribution, resulting in better FID. Mixing the predictions with cfg=0.6 effectively combines action adherence with a natural video distribution, yielding better PSNR and SSIM.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

The full model, jointly trained with MD and SC, effectively combines motion prediction and simulation guidance to generate more accurate videos in terms of action execution and visual details, leading to better PSNR and SSIM. The VLA success rates reported above also collectively demonstrate the effectiveness of MD and SC.

W3&Q2: LIBERO pretraining baselines

We add LIBERO experiments using real-only world-model pretraining and simulation–real joint pretraining, with comparable pretraining budgets. These experiments use the action-only setting.

Init.SpatialObjectGoalLongAvg.
Wan2.288.095.682.864.082.6
Real-FM90.298.691.273.688.4
Joint-FM91.697.291.871.688.1
Full89.495.692.479.289.2

Both Real-FM and Joint-FM substantially improve over the Wan2.2 baseline. Real-FM achieves an average success rate 0.3 percentage points higher than Joint-FM, possibly due to the domain gap between simulated and real videos, particularly the lack of object interactions in the simulation data. The two models have different strengths across task suites, while our full model achieves the highest average success rate of 89.2%.

Training and evaluation protocol is provided in our response to Q3 below.

Q1: Simulation guidance from (d) to the full model

Compared with configuration (d), the full model introduces simulation conditioning with dropout during training, allowing it to learn how to adaptively use simulation guidance. During inference, the full model first predicts simulation latents, then uses them as conditioning to predict real-video latents, which are finally decoded into a real video. The full model's FID/FVD deteriorate because the simulation latents, injected through residual connections into the real-video prediction branch, affect the predicted distribution of real-video latents. Its PSNR/SSIM improve because it combines real-world motion prediction with adaptive simulation guidance, allowing simulation to provide more accurate action information. We compare the effects of different guidance strengths ww on PSNR/SSIM.

cfgPSNR↑SSIM↑
ww=0.024.6200.854
ww=0.325.0140.857
ww=0.625.0770.858
ww=1.024.8570.856

At ww=0, the model uses only its real-world motion prediction capabilities; at ww=1, it relies solely on simulation guidance. At ww=0.6, it combines real-world motion prediction and simulation guidance, leveraging the adaptive guidance capabilities learned during training.

Q3: LIBERO training and evaluation protocol

We fine-tune on 40 LIBERO tasks using 1,712 demonstrations (32–50 per task) for 20,000 steps, with a batch size of 16 per GPU and AdamW (learning rate 1e-4, weight decay 1e-2, and a cosine learning rate schedule). We consider three settings:

  • Action-only: train only the action expert, keeping the video expert frozen.
  • Unconditional: jointly train both experts with video and action losses; action prediction attends only to the observed first-frame video tokens.
  • Video+action: jointly train both experts with video and action losses; action prediction attends to all video tokens, including future frames.

The VAE remains frozen and language embeddings are precomputed in all settings. We predict 32 actions using 10 sampling steps and execute the first 10 before replanning, without temporal ensembling. Evaluation uses 50 rollouts per task (2,000 total). Our reproduced FastWAM baseline uses the same training and evaluation protocol for each corresponding setting.

References:
[1] IRASim: Learning Interactive Real-Robot Action Simulators. ICCV 2025.
[2] Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling. ICLR 2026.

Thank you for adding the LIBERO baselines and clarifying the evaluation protocol. On Bridge, the full model improves PSNR and LPIPS over joint training with trajectory augmentation. On InternData-A1, however, the gains over joint training are more limited: PSNR improves slightly, SSIM is unchanged at the reported precision, and LPIPS worsens. The new LIBERO comparison shows a meaningful gain on Long (79.2% vs. 73.6% with real-only pretraining), but a smaller gain on average (89.2% vs. 88.4%). I remain unconvinced that these incremental gains justify the added complexity of the full method, so I will maintain my original score.

Thank you for your follow-up feedback and for acknowledging the additional experiments.

1. SOTA PSNR/SSIM on both datasets
We acknowledge that the gains over joint training are smaller on InternData-A1. Nevertheless, among methods without embodied pretraining, our full model achieves SOTA PSNR and SSIM on both Bridge and InternData-A1, with SSIM tied on InternData-A1 at the reported precision. This demonstrates consistent competitiveness on the two primary action-to-video prediction metrics emphasized in our work.

2. LIBERO as complementary evidence of downstream transfer
Regarding LIBERO, the full model provides a modest average improvement over real-only pretraining, alongside a more substantial gain on Long. Our motion distillation (MD) and simulation conditioning (SC) primarily target video prediction, and world-model pretraining does not include a loss supervising action outputs. The LIBERO experiments therefore provide complementary evidence of the downstream transfer value of the learned representations, while our core objective and primary evaluation remain focused on world-model video prediction.

3. Pretraining complexity versus downstream deployment cost
Regarding the added complexity, we would like to distinguish pretraining costs from deployment costs. The teacher model used by MD is required only during pretraining and adds no inference overhead. SC increases inference costs for full video prediction in exchange for better PSNR/SSIM. In the reported action-only LIBERO comparisons, however, the video expert is frozen and used as a feature extractor. All models follow the same downstream training protocol, without additional simulation-video generation. These downstream gains therefore do not incur the video-generation overhead of SC.

We appreciate your careful evaluation and hope these clarifications help contextualize the contributions and practical trade-offs of our method.

Reviewer_B6u5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimForcing transfers simulation-derived robot motion priors to real-domain action-conditioned video prediction. It combines distillation of adjacent-frame latent differences with multi-block simulation conditioning, condition dropout, and adjustable inference-time guidance. The student generates both simulation conditions and real videos without retaining the teacher at inference. Experiments on Bridge, InternData-A1, and LIBERO demonstrate improvements in video prediction and downstream policy initialization.

Critical strengths
  1. The framework provides complementary ways to exploit simulation priors. Parameter distillation and explicit conditioning transfer motion information, while adjustable guidance controls reliance on potentially inaccurate simulation predictions. The teacher is unnecessary at inference.
  2. Validation covers both prediction and policy transfer. The complete model improves several Bridge prediction metrics. Under action-only adaptation, LIBERO success increases from 82.6% to 89.2%, suggesting usefulness beyond visual generation.
Critical weaknesses
  1. The independent benefit of motion distillation is inconsistent. In Table 2, adding distillation to joint training and trajectory augmentation reduces PSNR from 24.523 to 24.310 and worsens LPIPS from 0.0702 to 0.0769, although FID/FVD improve. In Table 4, joint training alone also achieves better LPIPS than the full model. These results suggest complementary effects and trade-offs, but do not clearly establish when the central distillation mechanism is necessary or reliably beneficial.

  2. The evaluation does not sufficiently distinguish visual improvements from better interaction dynamics. The simulation teacher excludes object interactions, while motion matching applies to the full latent sequence, potentially imposing inappropriate constraints on real object motion. Most metrics measure video similarity, and the LIBERO comparison does not isolate distillation from other pretraining benefits.

Questions for authors
  1. Contribution clarification: Are the metric-dependent effects of motion distillation in Tables 2 and 4 stable across repeated training or evaluation? Can existing experiments separate distillation from simulation initialization and additional data?

  2. Method clarification: How does full-latent motion matching avoid suppressing real object motion when the teacher excludes object interactions? Is there an analysis focused on object regions or contact phases?

  3. Validation: Are matched-budget LIBERO comparisons available for joint training alone, joint training with distillation, and the full model? These would materially affect my assessment of distillation’s practical contribution.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.

W1: PSNR/SSIM/LPIPS as the primary metrics for action-to-video prediction

(1) Explanation of the importance of the evaluation metrics
PSNR, SSIM, and LPIPS compare each predicted video with its corresponding ground-truth (GT) video frame by frame. They measure pixel-level error, local structural similarity, and perceptual distance in deep feature space, respectively, and are averaged over frames and samples. FID and FVD compare feature distributions between generated and real data, computing the Fréchet distance using the means and covariances of the two feature distributions. They therefore require no one-to-one correspondence between predicted and GT videos and do not directly measure whether each prediction matches its corresponding real future.

IRASim [1] notes that action-to-video prediction requires strict adherence to the input actions, in contrast to text-to-video generation, and should therefore prioritize similarity metrics such as PSNR. It also reports that PSNR aligns better with human preferences, while FID/FVD do not align sufficiently well. LPWM [2] likewise uses only PSNR/SSIM/LPIPS for action-to-video tasks and only FVD for stochastic video generation. These findings indicate that PSNR/SSIM/LPIPS are the primary metrics for action-to-video prediction, while FID/FVD provide a complementary assessment of the overall distributional quality of generated videos.

To provide a comprehensive assessment of these components, we report results on the downstream VLA task.

Init.SpatialObjectGoalLongAvg.
Wan2.288.095.682.864.082.6
Joint-FM91.697.291.871.688.1
Joint-FM + MD93.098.890.673.689.0
Joint-FM + MD + SC (Full)89.495.692.479.289.2

(2) Analysis of component effects
In Table 2, motion distillation (MD) in configuration (d) improves learning of the overall video distribution, particularly motion, yielding better FID and FVD. However, using MD alone causes the model to prioritize motion consistency and neglect visual details, leading to lower PSNR and SSIM. Simulation conditioning (SC) alone introduces errors from simulation guidance, reducing performance. Joint training with MD and SC enables the model to adapt to this guidance, ultimately improving PSNR and SSIM.

For Table 4, InternData-A1 differs from Bridge, which contains videos of real robots in real-world scenes. The original annotated videos in InternData-A1 are rendered by a simulator, as are the simulated robot-arm videos constructed in our work. Joint-FM may fit these rendered videos more directly during training, resulting in the best FID and LPIPS, although its detail fidelity (PSNR) and motion quality (FVD) are not the best. MD improves the details of rendered videos. Joint training with MD and SC mitigates the degradation in FVD and LPIPS caused by SC alone, ultimately achieving the best PSNR and FVD.

We mistakenly reported the results with cfg=1 in the “With simulation conditioning” row of Table 4. For a fair comparison, we should report results with cfg=0.6, using the same setting as the full model. We provide results for additional cfg values below.

Additional results with simulation conditioningPSNR↑SSIM↑FID↓FVD↓LPIPS↓
Joint-FM25.3820.81330.30317.050.0742
cfg=025.2430.81132.11316.080.0784
cfg=0.325.4490.81332.29321.020.0772
cfg=0.625.4710.81332.30351.150.0769
cfg=1.025.0290.80832.06318.780.0806

Using cfg=0.6 achieves higher PSNR and the same SSIM as Joint-FM, demonstrating the effectiveness of simulation conditioning.

The VLA results above show that both MD and SC improve success rates, validating their effectiveness. The full method also achieves the best PSNR and SSIM, our primary metrics.

W2&Q2: Action and object motion evaluation

(1) We add action-sensitive metrics from WorldArena [3] on the Bridge dataset, including Trajectory Accuracy (TA), Dynamic Time Warping Distance (DTW), Average Displacement Error (ADE), Final Displacement Error (FDE), Motion Smoothness, Interaction Quality, and Instruction Following.

ModelTA↑DTW↓ADE↓FDE↓Motion smoothness↑Interaction Quality↑Instruction Following↑
SFT (Baseline)60.70740.067950.085900.108251.760333.05262.7368
Joint-FM (Ours)79.13810.037120.051250.077871.756023.12282.7368
Full (Ours)88.02670.030920.044670.062141.772133.15792.8596

These results demonstrate the effectiveness of our full method on action-sensitive metrics, with substantially better scores than both Wan2.2 SFT and Joint-FM.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

(2) Although the reference motion from the teacher model includes only robot-arm motion, we also include a real-video prediction loss during training, which preserves the prediction of other objects' motion in real videos. The action-sensitive metrics evaluated on real videos in the table above also show improvements over the baselines in modeling the full motion in real videos, including both the robot arm and other objects.

The LIBERO comparisons are provided in our response to Q3 below.

Q1: Distillation reproducibility and simulation ablations

(1) We add repeated experiments for motion distillation in Table 2, corresponding to configuration (d).

PSNR↑SSIM↑FID↓FVD↓LPIPS↓
Eval (seed=42)24.3100.85219.18100.270.0769
Eval (seed=2026)24.2400.85218.78101.570.0752
Repeated training24.5580.85419.51100.430.0713

The results from repeated training or inference are very close to the original results, indicating that the effects of motion distillation are reproducible.

(2) We add experiments to separate the effects of distillation from those of simulation initialization and additional data.

SettingPSNR↑SSIM↑FID↓FVD↓LPIPS↓
Wan2.2 (SFT)22.7510.83823.97168.710.0926
w/ simulation FM, w/ simulation init (original MD)24.3100.85219.18100.270.0769
w/o simulation FM, w/ simulation init23.5100.84521.06104.910.0728
w/ simulation FM, w/o simulation init23.7230.84722.30121.270.0809
w/o simulation FM, w/o simulation init23.3660.84422.73120.990.0855

The results show that both simulation FM and initialization from the simulation world model contribute to the final performance of MD training. Removing either component lowers PSNR and SSIM and worsens FID and FVD, although removing simulation FM improves LPIPS. We believe that simulation FM may promote shared representation learning across simulated and real videos, while initialization from the simulation world model provides a better starting point for real-video prediction. Removing both components yields the worst PSNR, SSIM, FID, and LPIPS among the four MD configurations, but still outperforms the SFT baseline on all metrics.

Q3: LIBERO pretraining and distillation ablations

We add comparisons of different pretrained initializations and motion distillation on LIBERO. The experiments differ only in the pretrained models used for initialization. All use action-only training, in which only the action expert is trained, while the video expert is frozen and used for visual feature extraction. Thus, the different pretraining configurations introduce no additional downstream training cost. All models use the same fine-tuning settings and training budget. We fine-tune on 40 LIBERO tasks using 1,712 demonstrations (32–50 per task) for 20,000 steps, with a batch size of 16 per GPU and AdamW with a learning rate of 1e-4, weight decay of 1e-2, and a cosine learning rate schedule.

Init.SpatialObjectGoalLongAvg.
Wan2.288.095.682.864.082.6
Joint-FM91.697.291.871.688.1
Joint-FM + MD93.098.890.673.689.0
Full89.495.692.479.289.2

Joint-FM substantially improves over the Wan2.2 baseline, supporting the effectiveness of joint video prediction pretraining. Adding MD increases the average success rate from 88.1% to 89.0%, demonstrating the effectiveness of motion distillation for downstream tasks. The full method, pretrained jointly with MD and SC, achieves the highest average success rate of 89.2%.

References:
[1] IRASim: Learning Interactive Real-Robot Action Simulators. ICCV 2025.
[2] Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling. ICLR 2026.
[3] Worldarena: A unified benchmark for evaluating perception and functional utility of embodied world models. 2026.

Reviewer_sTN1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimForcing is a simulation-based framework for action-conditioned robot video prediction. A simulation world model is trained using replayed and augmented robot trajectories, and is subsequently used to initialize and supervise a model trained on real-world data. Action-related knowledge is transferred by aligning the temporal differences between the latent variables of a "teacher" model and the target model; meanwhile, simulation-based latent variables are injected via multi-block conditioning, incorporating techniques such as corruption and dropout. During inference, the model generates its own predictions within the simulation domain and employs classifier-free guidance to combine predictions conditioned on simulation data with those that are not. Experiments include video prediction and component ablation studies on the Bridge and InternData-A1 datasets, as well as policy initialization on the LIBERO dataset.

Critical strengths
  1. A coherent approach that combines internalized prior information from simulations with explicit simulation-based guidance.

Transferring temporal latent differences rather than directly matching latent values proves more effective at reducing discrepancies in appearance. By integrating this supervised learning method with conditional corruption, dropout, and adjustable guidance, one can address a critical challenge: the fact that even imperfect simulation predictions can still provide useful behavioral information. Furthermore, through joint learning, the model can generate predictions for both the simulation and real-world domains during inference, without requiring a separate teacher model.

  1. Comparisons using the same video backbone, along with an analysis of preliminary downstream results, confirm a significant improvement in predictive performance.

The evaluation includes both an action-conditioned Wan2.2 baseline model and a re-implemented version of GeniWorld - both utilizing the same video backbone - to ensure that the observed benefits are not limited to the specific base model. Furthermore, by transferring a pre-trained model, the average success rate on the LIBERO benchmark (within an action-only training setting) increased from 82.6% to 89.2%. These results suggest that the full training pipeline learns useful representations that extend beyond mere initial weight initialization.

Critical weaknesses
  1. There is no direct testing of the fidelity of motions or interactions.

Video metrics measure only visual quality; they do not assess whether a robot follows commanded actions or whether objects respond correctly to contact. Since pre-training in simulation excludes object interactions, performance gains may simply stem from the appearance of the robot arm or the stability of the background. In LIBERO, performance improvements are most significant in motion-only settings, where no video is generated. The gains from incorporating video are marginal in the absence of uncertainty (98.1% → 98.4%).

At least one direct metric, such as end-effector trajectory error, consistency in object movement, or alignment with the intended task outcome, would be necessary for comprehensive results.

  1. The utility of joint simulation-real training is mixed.

Joint flow matching and augmentation already account for a significant portion of the performance gains. Motion distillation improves FVD but degrades PSNR and LPIPS, whereas simulation conditioning has the opposite effect. On InternData-A1, joint flow matching alone outperforms the full method in terms of LPIPS and FID. Latent differences capture aspects of appearance and object motion that are invisible to the teacher robot, and simpler temporal regularization techniques cannot be ruled out. Furthermore, the computational cost and sensitivity to mask errors in the SAM3-based inference pipeline have not been characterized.

Questions for authors
  1. How do the full method, Wan2.2 SFT, and joint flow matching compare on an action-sensitive measure?

  2. Does the LIBERO gain hold over a joint-flow-matching initialization under identical training, with uncertainty across seeds?

  3. How sensitive is performance to mask errors, and how does the full pipeline’s latency compare with a path that skips segmentation and simulation generation?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.

W1&Q1: Action-sensitive metrics and LIBERO conditioning

(1) Additional metrics sensitive to actions and motion
We add action-sensitive metrics from WorldArena [1] on the Bridge dataset, including Trajectory Accuracy (TA), Dynamic Time Warping Distance (DTW), Average Displacement Error (ADE), Final Displacement Error (FDE), Motion Smoothness, Interaction Quality, and Instruction Following.

ModelTA↑DTW↓ADE↓FDE↓Motion smoothness↑Interaction Quality↑Instruction Following↑
SFT (Baseline)60.70740.067950.085900.108251.760333.05262.7368
Joint-FM (Ours)79.13810.037120.051250.077871.756023.12282.7368
Full (Ours)88.02670.030920.044670.062141.772133.15792.8596

These results demonstrate the effectiveness of our full method on action-sensitive metrics, with substantially better scores than both Wan2.2 SFT and Joint-FM.

(2) Clarification of the role of video prediction in LIBERO
The interpretation that incorporating video yields only marginal improvements is incorrect. In our LIBERO experiments, both unconditional training and video+action training include a video generation loss. The difference is that, in the former setting, action prediction can access only features from the initial observation, while in the latter, it can access features from both the initial observation and future frames. Thus, incorporating video does not yield only marginal improvements: unconditional training already provides a 1% increase in success rate. As baseline performance on LIBERO increases, further improvements become harder to achieve. The recent ImageWAM [2], which focuses on improving VLA performance on top of FastWAM, also achieves only a 98.4% success rate. The results presented here reflect only the direct transfer of our world model to VLA.

W2: PSNR/SSIM/LPIPS as the primary metrics for action-to-video prediction

(1) Explanation of the importance of the evaluation metrics
PSNR, SSIM, and LPIPS compare each predicted video with its corresponding ground-truth (GT) video frame by frame. They measure pixel-level error, local structural similarity, and perceptual distance in deep feature space, respectively, and are averaged over frames and samples. FID and FVD compare feature distributions between generated and real data. Both compute the Fréchet distance using the means and covariances of the two feature distributions. They therefore require no one-to-one correspondence between predicted and GT videos and do not directly measure whether each prediction matches its corresponding real future.

IRASim [3] notes that action-to-video prediction requires strict adherence to the input actions, in contrast to text-to-video generation, and should therefore prioritize similarity metrics such as PSNR. It also reports that PSNR aligns better with human preferences, while FID/FVD do not align sufficiently well. LPWM [4] likewise uses only PSNR/SSIM/LPIPS for action-to-video tasks and only FVD for stochastic video generation. These findings indicate that PSNR/SSIM/LPIPS are the primary metrics for action-to-video prediction, while FID/FVD provide a complementary assessment of the overall distributional quality of generated videos.

To provide a comprehensive assessment of these components, we report results on the downstream VLA task.

Init.SpatialObjectGoalLongAvg.
Wan2.288.095.682.864.082.6
Joint-FM91.697.291.871.688.1
Joint-FM + MD93.098.890.673.689.0
Joint-FM + MD + SC (Full)89.495.692.479.289.2

(2) Effects of motion distillation
In Table 2, motion distillation (MD) in configuration (d) improves learning of the overall video distribution, particularly motion, yielding better FID and FVD. Using MD alone prioritizes motion consistency at the expense of visual detail, leading to lower PSNR and SSIM. The downstream VLA results above show that adding MD nevertheless improves task success rates.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

(3) Effects of simulation conditioning
For the Bridge dataset, we mistakenly reported the results with cfg=1 for configuration (e) in Table 2. For a fair comparison, we should report results with cfg=0.6, using the same setting as the full model. We provide results for different cfg values below.

Additional results for configuration (e)PSNR↑SSIM↑FID↓FVD↓LPIPS↓
cfg=022.0690.83633.73257.310.1050
cfg=0.323.5920.84637.74253.290.0925
cfg=0.624.0860.85026.07144.710.0762
cfg=1.023.7320.84821.83145.400.0772

Different cfg values yield different results. With cfg=1 (conditional generation), the predictions more closely match the real distribution, resulting in better FID. Mixing the predictions with cfg=0.6 better balances action following and distributional realism, yielding higher PSNR and SSIM. Simulation conditioning (SC) alone introduces errors from simulation guidance, which reduces performance. Joint training with MD and SC enables the model to adapt to this guidance, improving PSNR and SSIM. As shown in the VLA results above, adding SC further improves downstream task success rates.

For InternData-A1, we likewise reported the results with cfg=1 in the “With simulation conditioning” row of Table 4. This row should also report results with cfg=0.6, using the same setting as the full model. We provide results for additional cfg values below.

Additional results with simulation conditioningPSNR↑SSIM↑FID↓FVD↓LPIPS↓
Joint-FM25.3820.81330.30317.050.0742
+ cfg=025.2430.81132.11316.080.0784
+ cfg=0.325.4490.81332.29321.020.0772
+ cfg=0.625.4710.81332.30351.150.0769
+ cfg=1.025.0290.80832.06318.780.0806

Using cfg=0.6 achieves higher PSNR and the same SSIM as Joint-FM alone, demonstrating the effectiveness of simulation conditioning.

(4) Comparison with simple temporal regularization
We implemented a simple temporal regularization method that encourages changes in predicted video latents between adjacent time steps to match those in the GT latents. The results are shown below.

ModelPSNR↑SSIM↑FID↓FVD↓LPIPS↓
Wan2.2 (SFT)22.7510.83823.97168.710.0926
Temporal Reg.22.2710.83427.85182.600.0995
Our MD24.3100.85219.18100.270.0769

This simple temporal regularization adversely affects direct flow-matching training with SFT, leading to worse performance, while our MD method provides substantial improvements.

Mask sensitivity results are provided in our response to Q3 below.

Q2: LIBERO gains over Joint-FM initialization across seeds

We add a comparison of success rates under action-only training.

Init.SpatialObjectGoalLongAvg.
Wan2.288.095.682.864.082.6
Joint-FM (seed=2026)87.899.091.873.488.0
Joint-FM (seed=42)91.697.291.871.688.1
Full89.495.692.479.289.2

We provide additional Joint-FM results using different random seeds, with normal variation in performance. These results demonstrate that Joint-FM provides consistent, substantial improvements, and our full model further achieves the best performance.

Q3: Mask sensitivity and inference latency

(1) Mask sensitivity
We compare the full model using different masks.

MaskPSNR↑SSIM↑FID↓FVD↓LPIPS↓
GT mask25.1850.86025.11131.210.0651
SAM325.0770.85825.21137.490.0673
SAM224.9540.85725.92146.120.0688

More accurate masks yield better predictions. GT masks slightly outperform SAM3 masks, which in turn slightly outperform SAM2 masks, with modest differences across settings.

(2) Inference latency
The full pipeline takes 4.1 s per sample, including 0.3 s for mask segmentation and 3.8 s for model inference. When segmentation and simulation generation are skipped, inference takes 1.8 s, matching the SFT baseline.

References:
[1] Worldarena: A unified benchmark for evaluating perception and functional utility of embodied world models. 2026.
[2] ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing? 2026.
[3] IRASim: Learning Interactive Real-Robot Action Simulators. ICCV 2025.
[4] Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling. ICLR 2026.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their careful evaluation and constructive feedback, as well as their recognition of the paper’s strengths. Below, we briefly summarize this feedback and present the key experiments and clarifications added during the rebuttal.

Positive feedback

  • Complementary simulation priors: Reviewers sTN1 and B6u5 recognized the coherent design combining motion transfer with explicit simulation guidance, which allows adjustable reliance on imperfect simulation predictions and requires no separate teacher at inference.
  • Improved video prediction: Reviewers sTN1, B6u5, and SMbY acknowledged improvements in prediction performance, including comparisons against baselines using the same backbone.
  • Downstream transfer: All three reviewers recognized that the LIBERO results demonstrate the value of the learned representations beyond video generation.

Key additional experiments

1. Action-sensitive evaluation
We added seven action-sensitive metrics on Bridge, covering trajectory accuracy, displacement errors, motion smoothness, interaction quality, and instruction following. The full model outperforms both Wan2.2 SFT and Joint-FM on all seven metrics, providing evidence of improved motion accuracy and stability beyond visual similarity alone.

2. Primary metrics and component contributions
We clarified why PSNR/SSIM/LPIPS serve as the primary metrics for our action-to-video task: they compare predictions with their corresponding ground-truth futures, while FID/FVD provide complementary distribution-level assessments. Through motion distillation (MD) and simulation conditioning (SC), our full model achieves SOTA PSNR and SSIM on both Bridge and InternData-A1 among methods without embodied pretraining, with SSIM tied on InternData-A1 at the reported precision. We also added guidance-strength comparisons, repeated MD experiments, and further ablations, while acknowledging trade-offs across metrics.

3. Segmentation-mask sensitivity
We compared ground-truth, SAM3, and SAM2 masks to evaluate the sensitivity of simulation guidance to segmentation quality. More accurate masks improve predictions, while PSNR/SSIM differences remain modest across the tested mask sources.

4. LIBERO component comparisons
We added comparisons of real-only pretraining, Joint-FM, Joint-FM+MD, and the full model under the same downstream action-only protocol. Average success increases from 88.1% with Joint-FM to 89.0% with MD and 89.2% with the full model. Compared with real-only pretraining, the full model achieves a modest average improvement and a larger gain on Long (79.2% vs. 73.6%). These results provide complementary evidence of component effectiveness, without requiring additional simulation-video generation during downstream use. We also clarified the training and evaluation protocol.

Thank you for the opportunity to strengthen our work. We sincerely appreciate the valuable time and effort of the Area Chair and all reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.