acceptodds
Under review as a conference paper at ICLR 2027

Does the Future Follow the Action? Measuring Action Fidelity in World Models

Abstract

Action-conditioned video world models are used to plan, to evaluate policies, and to compare models; each use depends on whether the predicted video follows the supplied action, a property we call action fidelity. Existing evaluations measure visual quality, VLM-judged plausibility, or downstream policy performance; none isolates action fidelity, and reference-based metrics can point the wrong way: on a world model's own predictions, PSNR ranks the pixel average of the executed futures above every checkpoint. Learning the judgment directly invites a shortcut. A contrastive scorer trained to match futures with their histories reaches 99% accuracy on its training task, yet exactly chance (12.5%) when asked which of eight futures from the same state an action produced: what a contrastive score measures is decided by what it is contrasted against. Contrasting each future with futures of alternative actions executed from the same state removes the shortcut and yields LAFS (Learned Action-Fidelity Score), which rates a prediction from its history and action without a reference rollout. We introduce BranchPairs, which pairs an executed future that follows the queried action, clean or under an appearance change, with one that deviates from it, on LIBERO, RoboTwin, and Action-MNIST. Trained only on clean executions, LAFS prefers the future that follows the action in 89% of LIBERO pairs (chance 50%), versus 57% for PSNR and 75% for an inverse dynamics model trained on the same data, and keeps its lead on pairs matched in PSNR (93%, versus 80% and at most 60% for visual metrics). On a Cosmos world model fine-tuned on LIBERO-10, LAFS orders three checkpoints and three controls as an executed-future reference does, unlike PSNR and a VLM critic.

Primary area:
generative models
Keywords:
world models, action fidelity, evaluation metrics, action-conditioned video prediction, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gMS1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies action fidelity in action-conditioned video world models: whether a generated future actually follows the given action, rather than simply looking realistic or matching a reference video. The paper proposes LAFS, a learned reference-free metric trained with contrastive learning. The key idea is to use futures from different actions executed from the same state as negatives. This prevents the scorer from relying on scene-level shortcuts and forces it to focus on action-dependent changes. The authors also introduce BranchPairs for controlled evaluation and test the method on LIBERO, RoboTwin, Action-MNIST, and a fine-tuned Cosmos world model.

Critical strengths
  1. A clear and important problem formulation. The strongest aspect of the paper is the observation that the negative distribution determines the property learned by a contrastive metric. The distinction between visual fidelity and action fidelity is useful, and the paper gives a convincing explanation of why existing visual metrics can fail badly in this setting.

  2. Strong experimental design. BranchPairs establishes controlled comparisons where the preferred future is determined independently of the evaluated metric, and the authors additionally control for pixel similarity, intervene on the action input, analyze feature representations, study negative composition, compare against an inverse dynamics baseline trained on the same data, and evaluate checkpoints of an actual learned world model. These experiments provide strong evidence that LAFS is genuinely using action-future correspondence rather than simply learning visual similarity.

Critical weaknesses
  1. Generalization to real world-model outputs is still somewhat limited. Most experiments use controlled branches in simulation. The Cosmos experiment is useful, but it is still based on one main world-model setup. I would be more convinced by evaluation across several independently trained world models or architectures.

  2. The metric requires domain-specific branch data. Training LAFS requires executing multiple actions from the same state, which assumes access to a simulator or resettable environment. The paper also notes that the learned score depends on the branching distribution, so robustness to a different action distribution is an important open question.

Questions for authors

How sensitive is LAFS to the distribution used to generate alternative action branches? For example, if it is trained with one perturbation magnitude or policy distribution, how well does it generalize to another?

Could the authors evaluate the same LAFS scorer on outputs from multiple world-model architectures or independently trained models? This would help show that the metric generalizes beyond the current Cosmos experiment.

How much branch data is actually needed? A simple data-efficiency experiment would help clarify the practical cost of training the metric.

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer gMS1 [part 1]

Thank you for the careful and constructive review. We have run the experiments you suggested and revised the paper accordingly. Please find our response to each point below.

W1. … I would be more convinced by evaluation across several independently trained world models or architectures.

Q2. Could the authors evaluate the same LAFS scorer on outputs from multiple world-model architectures or independently trained models?

Following your suggestion, we evaluated the same LAFS scorer on six world models from three families, on the 40 LIBERO-10 conditions of Sec. 4.2: two world models trained by other teams (WoVR [1] and VLA-RFT [2]), two Cosmos3-Edge [3] models that we fine-tuned (the world model of Sec. 4.2 and a world–action model), and two public LIBERO-10 policies built on Cosmos3, which predict video along with their actions. Each model predicts the futures of the 8 executed actions of every condition. Action correspondence is the reference check of Fig. 6 (is a prediction's own executed future the nearest of the 8 by PSNR?), and D is the action-specific part of LAFS. LAFS orders the six models the way the reference check does; PSNR does not:

World modelFamily, sizeAction corr. (%, uses real futures) ↑LAFS D ↑PSNR (dB) ↑
WoVR world model†Wan2.2 diffusion, 5B64.77.2625.96
Fine-tuned Cosmos3-Edge world modelCosmos3-Edge, 4B58.85.7425.79
Fine-tuned Cosmos3-Edge world–action modelCosmos3-Edge, 4B42.53.6325.13
VLA-RFT world model†autoregressive, 138M17.81.6520.05
Cosmos3-Nano LIBERO policyCosmos3-Nano, 16B13.10.1723.10
Cosmos3-Edge LIBERO policyCosmos3-Edge, 4B13.10.1622.57
Reference: pixel average of the 8 futures (ignores the action)12.50.0028.91
Rank correlation with action corr. over the six models (Spearman ρ)0.990.81

† trained by other teams. Rows are sorted by action correspondence. The reference row (the branch mean of Fig. 6) ignores the action, yet has the highest PSNR. For independent training runs, we also fine-tuned two architectures with two recipes and two training seeds each: the two seeds differ by at most 0.08 in D, the recipes by 0.55 and 0.20 (Table 13). App. G.4 (Table 12) gives the intervals and each model's training data.

Q1. How sensitive is LAFS to the distribution used to generate alternative action branches? For example, if it is trained with one perturbation magnitude or policy distribution, how well does it generalize to another?

To test this, we scored branches from other distributions with the frozen LIBERO-10 scorer, which is trained on one kind of branch only: random perturbations of one size. From the 50 evaluation conditions we executed branches from six other sources: the same type at other sizes, smooth drifts, and other demonstrations, which come from a different policy distribution. Branches from 2/3 to twice the training size and from other demonstrations score close to the training distribution; smooth drifts fall below it:

Branch sourceGripper shift (median, cm)Pairwise (%) ↑8-way retrieval (%) ↑
Same type, training size1.5192.478.9
Same type, twice the size3.0395.184.4
Other demonstrations5.7491.573.8
Same type, 2/3 size0.9888.467.6
Same type, 1/3 size0.4978.746.7
Smooth drift, large3.8379.643.3
Smooth drift, small1.2764.723.4
Chance50.012.5

At the same gripper shift, branches from 2/3 to twice the training size score alike; smooth drifts stay harder (pairwise accuracy for pairs with a gripper shift of 1–2 cm):

Branch sourcePairwise (%) ↑
Same type, 2/3 size90.6
Same type, training size93.7
Same type, twice the size92.0
Smooth drift, small66.6
Chance50.0

If slow drifts matter for an application, they should be among the training branches; App. C now says so. App. G.6 (Table 15) gives the intervals and the comparisons at equal gripper shift.

References

[1] WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL. arXiv 26'

[2] VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators. arXiv 25'

[3] Cosmos 3: Omnimodal World Models for Physical AI. arXiv 26'

AuthorsOpenReview ↗

Rebuttal to Reviewer gMS1 [part 2]

W2. The metric requires domain-specific branch data. Training LAFS requires executing multiple actions from the same state, which assumes access to a simulator or resettable environment. The paper also notes that the learned score depends on the branching distribution, so robustness to a different action distribution is an important open question.

Branches are needed only once per domain, to train the scorer; evaluating a world model afterwards needs neither branches nor a reference rollout. In simulation the branches need no labels, and our answer to Q3 below gives how many are needed and what they cost. For real robots, we agree that this is an important setting. Within the short rebuttal period we did not have the time and resources to set up a real robot and scene corresponding to LIBERO or RoboTwin and to collect branches there, so we leave this to future work (App. C). The data-scaling curves of Figure 9 nevertheless give us reason to believe that this transfer is feasible when the scene does not change drastically: the scorer is pretrained in simulation and then trained on a small amount of real branch data. The dependence on the branching distribution is addressed in our answer to Q1 above.

Q3. How much branch data is actually needed? A simple data-efficiency experiment would help clarify the practical cost of training the metric.

We ran the data-efficiency experiment you asked for, on LIBERO-10: we trained the scorer on nested subsets of its branch groups, from scratch and starting from a scorer trained on LIBERO-Spatial/Goal (8-way retrieval on the 50 evaluation conditions; chance 12.5%). Accuracy keeps rising with more branch data, and with pretraining on related tasks 4,000 groups reach what 10,617 reach from scratch (78.6% and 78.9%):

LIBERO-10 branch groupsFrom scratch (%) ↑Pretrained on LIBERO-Spatial/Goal (%) ↑
015.926.9
75016.262.9
1,50027.670.5
2,50038.7not run
4,00058.278.6
6,50071.4not run
10,61778.986.4
21,24987.8not run

The 0 row is the scorer before any LIBERO-10 branches. Ordering the three checkpoints of Sec. 4.2 correctly needs less: 2,500 groups from scratch, and any size with pretraining.

Cost on LIBERO-10 (training times are for one seed):

StepDataTime
Branch collection (simulation, no labels)10,617 training and 1,180 validation groups2 h 16 min on one server with two 64-core AMD EPYC 9575F CPUs
Stage 1 (demonstrations only)360 demonstrations1 h 11 min on one AMD MI325X GPU
Branch stage10,617 groups1 h 45 min on the same GPU

For scale, Ctrl-World reports training its world model on 16 H100 GPUs for two to three days [1]. App. G.7 (Table 16 and Figure 9) gives the intervals and this comparison for further world models.

Changes made for this review: Sec. 4.2 (six world models); App. B (the training perturbations have a white spectrum; the submitted text called them smooth); App. C; and the new App. G.4, G.6 and G.7 (Tables 12, 13, 15 and 16, Figure 9).

Thank you again for the careful review; we are happy to answer any further questions.

References

[1] Ctrl-World: A Controllable Generative World Model for Robot Manipulation. ICLR 26'

Thanks for your detailed repland your updated manuscript. My concerns have been addressed. Overall, the research topic is interesting, and the paper’s contributions are substantial, so I will maintain my score.

Thank you for your reply. We are glad that our response addressed your concerns. The experiments you suggested are now part of the paper and will remain in the final version: the evaluation on six world models (App. G.4), the other branch sources (App. G.6) and the data-efficiency study (App. G.7).

Reviewer_2cmiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors start with the concept of action fidelity for action-conditioned video models, rather than visual fidelity. Given a set of action-conditioned videos, their scores can change considerably depending on the metric you use. In particular, visual fidelity and action fidelity can be misaligned. The authors then learn an action fidelity score contrastively, yielding a substantial increase in preference for futures that follow the actions that conditioned the video generation.

Critical strengths
  • The motivation of the paper is easy to follow, and the examples help the reader.
  • The score and its training procedure are simple and well explained.
  • The results are thorough, and the advantage of the new metric is clear.
Critical weaknesses
  • While I think it's interesting to understand if the score can select the video which best correlates with the actions, it isn't clear if those actions that are selected through the video can actually be the best actions for the robotics task.
  • There is no experiment that shows that the score which has been trained on simulation can transfer to videos generated from real observations.
Questions for authors
  • Have the authors seen a correlation between actions with high score fidelity and performance in the robotic tasks?
  • Something which I'm slightly missing is understanding what the final use of such a metric is that the authors envision for the robotic task. This is linked to the question before. I think it is clear that the score can be used to fine-tune word models, as shown in Section 4. However, I'd like to understand if it can also be used directly in the robotic task.
  • Do the authors think that there is a way to show the sim-to-real gap for the metric you have designed?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 2cmi [part 1]

Thank you for the positive and thoughtful review. Please find our response to each point below.

W1 + Q1. … it isn't clear if those actions that are selected through the video can actually be the best actions for the robotics task. … Have the authors seen a correlation between actions with high score fidelity and performance in the robotic tasks?

Yes: world models that LAFS rates higher choose the right action more often. LAFS rates a world model, not an action: it measures whether the video the model predicts follows the action it is given. To test whether this matters when a model is used to choose actions, we ran a new planning experiment with six world models (new App. G.4 and G.5 of the revised PDF). For each of the 40 LIBERO-10 conditions, we take the executed future of one action as the goal, and each world model picks, among the 8 actions, the one whose predicted future is closest to the goal. The three models LAFS rates highest pick the action that produced the goal for 56–68% of goals, the three it rates lowest for 17–23% (chance 12.5%), and the action they pick ends closer to the goal. The last column gives the distance between the executed future of the picked action and the goal, relative to a random pick:

World modelLAFS action-specific D ↑Goal action picked (top-1, %) ↑Distance to goal ↓
WoVR world model† (Wan2.2, 5B)7.2667.50.32
Fine-tuned Cosmos3-Edge world model5.7457.50.40
Fine-tuned Cosmos3-Edge world–action model3.6356.20.42
VLA-RFT world model† (autoregressive, 138M)1.6516.90.92
Cosmos3-Nano LIBERO policy (16B)0.1717.80.88
Cosmos3-Edge LIBERO policy (4B)0.1622.50.83
Reference: a random pick12.51.00

† trained by other teams (WoVR [1], VLA-RFT [2]). This is a one-step planning test (Table 14).

Q2. … I think it is clear that the score can be used to fine-tune [world] models, as shown in Section 4. However, I'd like to understand if it can also be used directly in the robotic task.

In a robotic task, LAFS is used to choose and check the world model that the robot relies on. It is meant for settings where the robot relies on the world model's response to actions: evaluating a policy in imagination [3], post-training it with reinforcement learning inside the world model [1, 2, 4, 5], generating action-conditioned training data [3], and planning [6], as in the experiment above. There, a world model that ignores the action gives confident but wrong answers [1], and LAFS detects this without a reference rollout.

One clarification on Section 4: it does not fine-tune world models with LAFS; it scores checkpoints of a world model trained without it. Sec. 4.2 now states how the world model is trained. Using LAFS to further improve a world model is an interesting direction; we are running preliminary training experiments on it and leave it to future work.

References

[1] WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL. arXiv 26'

[2] VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators. arXiv 25'

[3] Ctrl-World: A Controllable Generative World Model for Robot Manipulation. ICLR 26'

[4] WMPO: World Model-based Policy Optimization for Vision-Language-Action Models. ICLR 26'

[5] Learning Interactive Real-World Simulators. ICLR 24'

[6] DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning. ICML 25'

AuthorsOpenReview ↗

Rebuttal to Reviewer 2cmi [part 2]

W2 + Q3. There is no experiment that shows that the score which has been trained on simulation can transfer to videos generated from real observations. … Do the authors think that there is a way to show the sim-to-real gap for the metric you have designed?

Yes. The gap is measured on held-out branches of the target domain, as we already do between task suites. A scorer trained on LIBERO-Spatial/Goal retrieves 26.9% on LIBERO-10 before any LIBERO-10 branch, 70.5% after 1,500 branch groups and 86.4% after 10,617 (chance 12.5%; Figure 9 in the new App. G.7 of the revised PDF). The first number is the gap; the others show how it closes.

Sim-to-real is the same measurement with real branches: from a resettable start, the robot executes a few short alternative actions, without labels, and the scorer is tested on held-out branches before and after training on them. We agree that this is an important setting. Within the short rebuttal period we did not have the time and resources to set up a real robot and scene corresponding to LIBERO or RoboTwin and to collect branches there, so we leave this to future work (App. C). The curves of Figure 9 nevertheless give us reason to believe that the gap can be closed when the scene does not change drastically: the scorer is pretrained in simulation and then trained on a small amount of real branch data.

Changes made for this review: Sec. 4.2 (how the world model is trained), the new App. G.5 (Table 14) and G.7 (Figure 9), and App. C.

Thank you again for the thoughtful questions; we are happy to answer any further ones.

I thank the authors for their clarification. Being able to rank world models is indeed an interesting contribution. I would love to see the results on a real robot, but I appreciate that the short review period does not allow for that.

Thank you for your reply; we are glad that the clarification was helpful. We agree that results on a real robot are the natural next step, and App. C of the revised paper lists them as future work. The planning experiment added for your question is part of the paper (App. G.5) and will remain in the final version.

Reviewer_x6pEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a learned scorer that measures whether a world model's predicted video follows the supplied actions. It uses self-supervised contrastive learning, with futures produced by alternative actions from the same state serving as negatives. Experiments on BranchPairs and a fine-tuned Cosmos model evaluate whether the scorer distinguishes action errors from changes in visual appearance.

Critical strengths
  1. The paper identifies a concrete shortcut in contrastive training and supports it with theoretical analysis and experiments.
  2. Same-state action branches provide supervision without human annotation, and the trained scorer requires no reference rollout at evaluation time.
Critical weaknesses
  1. Sensitivity to subtle action differences is unclear. Can LAFS distinguish small changes in action magnitude, direction or timing. A systematic analysis would help establish whether it detects fine-grained errors beyond clearly different action outcomes.
  2. Domain-specific training required: the scorer requires alternative actions executed from the same state and training in the evaluated domain. Its applicability to real-world data and unseen environments remains unclear.
  3. In the LIBERO-10 experiment, later checkpoints may improve visual reconstruction. Also, the generated video experiment uses three checkpoints from one Cosmos fine-tuning run.
Questions for authors
  1. How well does LAFS transfer to unseen tasks or environments without additional branch training?
  2. It would be more convincing to show that ranking remains reliable across different world-model architectures and independent training runs. For example, model only see success episodes vs. model with failure case aware.
7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer x6pE [part 1]

Thank you for the careful and constructive review. We have run the experiments you suggested and revised the paper accordingly. Please find our response to each point below.

W1. Sensitivity to subtle action differences is unclear. Can LAFS distinguish small changes in action magnitude, direction or timing. A systematic analysis would help establish whether it detects fine-grained errors beyond clearly different action outcomes.

Following your suggestion, we ran a systematic analysis of small changes in action magnitude, direction and timing. We scaled, rotated or delayed the executed actions of the 100 LIBERO conditions of Table 1 at 18 levels and executed each changed sequence from the same state. Each metric then compares two videos for the original action, as in Table 1: the video that follows it, clean or degraded, and the video of the changed action. LAFS prefers the video that follows the action more often than PSNR at all 18 levels. The table shows the smallest change of each kind that puts the two grippers at least one pixel apart in the 128×128 input (% of pairs in which the video that follows the action is preferred):

Change to the executed actionsGripper shift (px)LAFS ↑PSNR ↑
Whole sequence delayed by 1 step (0.05 s)1.3787.746.2
Arm commands scaled up by 25%1.5476.041.0
Arm commands scaled down by 25%1.7075.551.0
Movement direction rotated by 5°1.3053.338.0

The scorer is the frozen one of Table 1 and was never trained on scaled or rotated actions. App. G.1 lists all 18 levels for both metrics (Table 6), and Sec. 4.1 now summarizes this analysis.

W2. Domain-specific training required: the scorer requires alternative actions executed from the same state and training in the evaluated domain. Its applicability to real-world data and unseen environments remains unclear.

Q1. How well does LAFS transfer to unseen tasks or environments without additional branch training?

To answer Q1, we tested the frozen Table 1 scorer without any new branches. We re-rendered the 100 conditions of Table 1 with changed cameras and colors, physics held identical, and scored two task suites that the scorer never saw. Without new branches, the frozen scorer carries over to small camera and color changes and stays above chance on unseen task suites; its accuracy falls as the change grows (rows sorted by accuracy):

Test setting, frozen scorerPairwise (%) ↑8-way retrieval (%) ↑
Trained tasks, new demonstrations91.372.8
Camera shifted 2 cm89.969.3
Table and walls recolored, mildest level88.766.0
Camera orbited 5°87.863.3
Camera shifted 16 cm76.143.5
Table and walls recolored, strongest level76.042.9
Camera orbited 20°74.538.5
Unseen suite: LIBERO-Object68.533.7
Unseen suite: LIBERO-1062.827.1
Chance50.012.5

On W2, we measured how much branch data unseen tasks and environments need. We trained LIBERO-10 scorers on nested subsets of its branch groups, from scratch and starting from a scorer trained on LIBERO-Spatial/Goal. Starting from the scorer trained on other tasks, 4,000 branch groups reach the accuracy that 10,617 reach from scratch (8-way retrieval in %; chance 12.5%):

LIBERO-10 branch groups7501,5004,00010,617
Starting from a scorer trained on LIBERO-Spatial/Goal ↑62.970.578.686.4
From scratch ↑16.227.658.278.9

This table is an excerpt. Figure 9 (App. G.7 of the revised PDF) shows the full data-scaling curves that we trained, from scratch at seven sizes up to 21,249 groups and with pretraining at four, to measure how much data an unseen environment needs to reach a given accuracy. Collecting the 10,617 groups and their validation set took 2 h 16 min of simulation, without labels.

For real robots, we agree that real-world scenes are an important setting. Within the short rebuttal period we did not have the time and resources to set up a real robot and scene corresponding to LIBERO or RoboTwin and to collect branches there, so we leave this to future work (App. C). The curves of Figure 9 nevertheless give us reason to believe that this transfer is feasible when the scene does not change drastically: the scorer is pretrained in simulation and then trained on a small amount of real branch data, as it is between task suites here.

App. G.2 lists all 14 conditions of the first table (Table 8).

AuthorsOpenReview ↗

Rebuttal to Reviewer x6pE [part 2]

W3. In the LIBERO-10 experiment, later checkpoints may improve visual reconstruction.

To test this, we lowered the visual quality of the 9k checkpoint's predictions with blur or pixel noise until their reconstruction was worse than that of the 1k checkpoint, and scored them again (LAFS is reported as its action-specific part: the score of a video with its own action minus its score with the other 7 actions of its condition). The degraded 9k predictions lose at most 0.25, against the 1.59 that separates the two checkpoints:

PredictionsVisual reconstruction: LPIPS to the real future ↓LAFS action-specific part, relative to the 9k checkpoint
9k checkpoint0.0550
9k checkpoint, blurred0.096−0.02
9k checkpoint, with pixel noise0.077−0.25
1k checkpoint0.071−1.59

The action-specific part itself removes what does not depend on the action: visual quality is the same whichever action a video is paired with. From the 1k to the 9k checkpoint, the score of the same videos rises by 2.05 with their own action and by 0.47 with the other actions, so 77% of the rise appears only when the action matches. Sec. 4.2 and Fig. 6b now report the action-specific part.

Videos of very different visual quality show the same thing. Paired with a wrong action, a real video scores as low as the models' predictions; the scores separate only when each video is paired with its own action (mean LAFS score; excerpt of Table 10):

VideoWith its own actionWith the other actions
Real future+3.91−4.57
Fine-tuned Cosmos3-Edge world model+1.05−4.69
Fine-tuned Cosmos3-Edge world–action model−0.78−4.40

App. G.3 (Tables 10 and 11) gives the intervals, further controls and the results under stronger degradation.

AuthorsOpenReview ↗

Rebuttal to Reviewer x6pE [part 3]

W3 (second part). Also, the generated video experiment uses three checkpoints from one Cosmos fine-tuning run.

Q2. It would be more convincing to show that ranking remains reliable across different world-model architectures and independent training runs. For example, model only see success episodes vs. model with failure case aware.

Following your suggestion, we ran both tests, and the ranking is reliable in both.

(a) Different architectures. We scored six world models from three families on the 40 LIBERO-10 conditions of Sec. 4.2: two world models trained by other teams (WoVR [1] and VLA-RFT [2]), two Cosmos3-Edge [3] models that we fine-tuned (the world model of Sec. 4.2 and a world–action model), and two public LIBERO-10 policies built on Cosmos3, which predict video along with their actions. Each model predicts the futures of the 8 executed actions of every condition. Action correspondence is the reference check of Fig. 6 (is a prediction's own executed future the nearest of the 8 by PSNR?), and D is the action-specific part of LAFS. LAFS orders the six models the way the reference check does; PSNR does not:

World modelFamily, sizeAction corr. (%, uses real futures) ↑LAFS D ↑PSNR (dB) ↑
WoVR world model†Wan2.2 diffusion, 5B64.77.2625.96
Fine-tuned Cosmos3-Edge world modelCosmos3-Edge, 4B58.85.7425.79
Fine-tuned Cosmos3-Edge world–action modelCosmos3-Edge, 4B42.53.6325.13
VLA-RFT world model†autoregressive, 138M17.81.6520.05
Cosmos3-Nano LIBERO policyCosmos3-Nano, 16B13.10.1723.10
Cosmos3-Edge LIBERO policyCosmos3-Edge, 4B13.10.1622.57
Reference: pixel average of the 8 futures (ignores the action)12.50.0028.91
Rank correlation with action corr. over the six models (Spearman ρ)0.990.81

† trained by other teams. Rows are sorted by action correspondence. The reference row (the branch mean of Fig. 6) ignores the action, yet has the highest PSNR. App. G.4 (Table 12) gives the intervals and each model's training data.

(b) Independent training runs. As you suggested, we compared models trained on successful episodes only with models that also saw failures. On two architectures (Cosmos3-Edge [3] and OpenWAM [4]), we fine-tuned the same start model with these two recipes and two training seeds each. Successes only uses expert demonstrations. Also failures adds episodes rolled out by the model itself, with the failed ones used only to train the world model. As a check that does not involve LAFS, we also command zero motion, under which the arm stays still in the simulator, and measure how much the predicted video still moves, as a percentage of its motion under the 8 candidate actions (lower is better; the stop test of Table 13). On both architectures, the model that also saw failures gets a higher LAFS and moves less under the zero-motion command:

ArchitectureMeasureSuccesses onlyAlso failuresChange
Cosmos3-Edge 4BLAFS D ↑3.784.34+0.55
Motion under a zero-motion command (%) ↓54.836.6−18.1
Action corr. (%, chance 12.5) ↑43.844.2+0.5
PSNR (dB) ↑25.3025.11−0.18
OpenWAM, Wan2.2 5BLAFS D ↑0.380.58+0.20
Motion under a zero-motion command (%) ↓88.853.5−35.3
Action corr. (%, chance 12.5) ↑18.124.5+6.4
PSNR (dB) ↑23.6623.90+0.24

Each value is the mean of two training seeds; changes are computed before rounding. Action correspondence rises on OpenWAM and is flat on Cosmos. The two seeds of each recipe differ by at most 0.08 in LAFS, and PSNR moves in opposite directions on the two architectures. LAFS is compared within an architecture, because OpenWAM is read at a different frame rate.

App. G.4 (Table 13) gives the standard deviations, the OpenWAM protocol and the comparison over all 17 OpenWAM checkpoints.

Changes made for this review: Sec. 4.1, Sec. 4.2 and Fig. 6b, App. C, App. F, and the new App. G.1–G.4 and G.7 (Tables 6–13 and 16, Figure 9). Sec. 1 now states that LAFS is trained for each domain.

Thank you again for the detailed review; we are happy to answer any further questions.

References

[1] WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL. arXiv 26'

[2] VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators. arXiv 25'

[3] Cosmos 3: Omnimodal World Models for Physical AI. arXiv 26'

[4] OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv 26'

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reviews and for proposing concrete tests.

We are encouraged that the reviewers found:

  • the observation that the negatives decide what a contrastive score measures to be clear and important (gMS1), and supported by theory and experiments (x6pE);
  • the score and its training simple and well explained (2cmi), with no human annotation and no reference rollout at evaluation time (x6pE);
  • the experiments thorough (2cmi), and strong evidence that LAFS uses action–future correspondence rather than visual similarity (gMS1).

The reviews raised five main points. We ran new experiments for each; the results are in the new Appendix G of the revised PDF, and the individual responses give the details.

  • Evaluation on more world models and independent training runs (x6pE, gMS1). We scored six world models from three families, two of them trained by other teams, and compared two training recipes on two architectures with two seeds each. LAFS orders the six models as the reference check of action following does (Spearman ρ = 0.99, against 0.81 for PSNR), and the training seeds agree (App. G.4).
  • Transfer to new domains and to real robots (all reviewers). We tested a frozen scorer under changed cameras and colors and on unseen task suites, and measured how many branches a new domain needs. The frozen scorer carries over to small visual changes, and a scorer pretrained on other tasks reaches with 4,000 branch groups what training from scratch reaches with 10,617 (App. G.2 and G.7). For real robots, which we could not set up within the rebuttal period, we leave the experiment to future work; the data-scaling curves give us reason to believe that simulation pretraining plus a small amount of real branch data suffices when the scene does not change drastically (App. C, Figure 9).
  • Cost of branch data and dependence on the branch distribution (gMS1, x6pE). We trained scorers on seven amounts of branch data and timed each step: the branches of LIBERO-10 are collected in 2 h 16 min of simulation, without labels (App. G.7). We also scored branches from other sources: the scorer carries over to other perturbation sizes and to other demonstrations (App. G.6).
  • Small action changes, and action following versus image quality (x6pE). We scaled, rotated or delayed the executed actions at 18 levels: LAFS prefers the video that follows the action more often than PSNR at every level (App. G.1). We also scored the same videos with other actions: 77% of the rise of LAFS across checkpoints is specific to the action (App. G.3).
  • Action fidelity and the choice of actions in a robotic task (2cmi). We ran a planning experiment with the six world models: those that LAFS rates higher choose the right action more often (App. G.5).

The revised PDF marks edits to existing text in blue. The main changes, with the reviews they respond to:

  • Sec. 1: what world models are used for, and that LAFS is trained for each domain. [2cmi, x6pE, gMS1]
  • Sec. 4.1: small action changes. [x6pE]
  • Sec. 4.2 and App. F: six world models; how the evaluated world model is trained; the scorer is now trained on LIBERO-10 branches alone, and the adapted scorer of the submitted version gives the same order in Fig. 6. [x6pE, gMS1, 2cmi]
  • App. C: limitations and future work. [all reviewers]
  • App. G (new): G.1 small action changes, G.2 transfer, G.3 image quality [x6pE]; G.4 more world models [x6pE, gMS1]; G.5 choosing actions [2cmi]; G.6 branch sources, G.7 branch data and its cost [gMS1, x6pE].

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.