- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies action fidelity in action-conditioned video world models: whether a generated future actually follows the given action, rather than simply looking realistic or matching a reference video. The paper proposes LAFS, a learned reference-free metric trained with contrastive learning. The key idea is to use futures from different actions executed from the same state as negatives. This prevents the scorer from relying on scene-level shortcuts and forces it to focus on action-dependent changes. The authors also introduce BranchPairs for controlled evaluation and test the method on LIBERO, RoboTwin, Action-MNIST, and a fine-tuned Cosmos world model.
Critical strengths
-
A clear and important problem formulation. The strongest aspect of the paper is the observation that the negative distribution determines the property learned by a contrastive metric. The distinction between visual fidelity and action fidelity is useful, and the paper gives a convincing explanation of why existing visual metrics can fail badly in this setting.
-
Strong experimental design. BranchPairs establishes controlled comparisons where the preferred future is determined independently of the evaluated metric, and the authors additionally control for pixel similarity, intervene on the action input, analyze feature representations, study negative composition, compare against an inverse dynamics baseline trained on the same data, and evaluate checkpoints of an actual learned world model. These experiments provide strong evidence that LAFS is genuinely using action-future correspondence rather than simply learning visual similarity.
Critical weaknesses
-
Generalization to real world-model outputs is still somewhat limited. Most experiments use controlled branches in simulation. The Cosmos experiment is useful, but it is still based on one main world-model setup. I would be more convinced by evaluation across several independently trained world models or architectures.
-
The metric requires domain-specific branch data. Training LAFS requires executing multiple actions from the same state, which assumes access to a simulator or resettable environment. The paper also notes that the learned score depends on the branching distribution, so robustness to a different action distribution is an important open question.
Questions for authors
How sensitive is LAFS to the distribution used to generate alternative action branches? For example, if it is trained with one perturbation magnitude or policy distribution, how well does it generalize to another?
Could the authors evaluate the same LAFS scorer on outputs from multiple world-model architectures or independently trained models? This would help show that the metric generalizes beyond the current Cosmos experiment.
How much branch data is actually needed? A simple data-efficiency experiment would help clarify the practical cost of training the metric.
4 comments
Rebuttal to Reviewer gMS1 [part 2]
W2. The metric requires domain-specific branch data. Training LAFS requires executing multiple actions from the same state, which assumes access to a simulator or resettable environment. The paper also notes that the learned score depends on the branching distribution, so robustness to a different action distribution is an important open question.
Branches are needed only once per domain, to train the scorer; evaluating a world model afterwards needs neither branches nor a reference rollout. In simulation the branches need no labels, and our answer to Q3 below gives how many are needed and what they cost. For real robots, we agree that this is an important setting. Within the short rebuttal period we did not have the time and resources to set up a real robot and scene corresponding to LIBERO or RoboTwin and to collect branches there, so we leave this to future work (App. C). The data-scaling curves of Figure 9 nevertheless give us reason to believe that this transfer is feasible when the scene does not change drastically: the scorer is pretrained in simulation and then trained on a small amount of real branch data. The dependence on the branching distribution is addressed in our answer to Q1 above.
Q3. How much branch data is actually needed? A simple data-efficiency experiment would help clarify the practical cost of training the metric.
We ran the data-efficiency experiment you asked for, on LIBERO-10: we trained the scorer on nested subsets of its branch groups, from scratch and starting from a scorer trained on LIBERO-Spatial/Goal (8-way retrieval on the 50 evaluation conditions; chance 12.5%). Accuracy keeps rising with more branch data, and with pretraining on related tasks 4,000 groups reach what 10,617 reach from scratch (78.6% and 78.9%):
| LIBERO-10 branch groups | From scratch (%) ↑ | Pretrained on LIBERO-Spatial/Goal (%) ↑ |
|---|---|---|
| 0 | 15.9 | 26.9 |
| 750 | 16.2 | 62.9 |
| 1,500 | 27.6 | 70.5 |
| 2,500 | 38.7 | not run |
| 4,000 | 58.2 | 78.6 |
| 6,500 | 71.4 | not run |
| 10,617 | 78.9 | 86.4 |
| 21,249 | 87.8 | not run |
The 0 row is the scorer before any LIBERO-10 branches. Ordering the three checkpoints of Sec. 4.2 correctly needs less: 2,500 groups from scratch, and any size with pretraining.
Cost on LIBERO-10 (training times are for one seed):
| Step | Data | Time |
|---|---|---|
| Branch collection (simulation, no labels) | 10,617 training and 1,180 validation groups | 2 h 16 min on one server with two 64-core AMD EPYC 9575F CPUs |
| Stage 1 (demonstrations only) | 360 demonstrations | 1 h 11 min on one AMD MI325X GPU |
| Branch stage | 10,617 groups | 1 h 45 min on the same GPU |
For scale, Ctrl-World reports training its world model on 16 H100 GPUs for two to three days [1]. App. G.7 (Table 16 and Figure 9) gives the intervals and this comparison for further world models.
Changes made for this review: Sec. 4.2 (six world models); App. B (the training perturbations have a white spectrum; the submitted text called them smooth); App. C; and the new App. G.4, G.6 and G.7 (Tables 12, 13, 15 and 16, Figure 9).
Thank you again for the careful review; we are happy to answer any further questions.
References
[1] Ctrl-World: A Controllable Generative World Model for Robot Manipulation. ICLR 26'
Thanks for your detailed repland your updated manuscript. My concerns have been addressed. Overall, the research topic is interesting, and the paper’s contributions are substantial, so I will maintain my score.
Thank you for your reply. We are glad that our response addressed your concerns. The experiments you suggested are now part of the paper and will remain in the final version: the evaluation on six world models (App. G.4), the other branch sources (App. G.6) and the data-efficiency study (App. G.7).
Rebuttal to Reviewer gMS1 [part 1]
Thank you for the careful and constructive review. We have run the experiments you suggested and revised the paper accordingly. Please find our response to each point below.
Following your suggestion, we evaluated the same LAFS scorer on six world models from three families, on the 40 LIBERO-10 conditions of Sec. 4.2: two world models trained by other teams (WoVR [1] and VLA-RFT [2]), two Cosmos3-Edge [3] models that we fine-tuned (the world model of Sec. 4.2 and a world–action model), and two public LIBERO-10 policies built on Cosmos3, which predict video along with their actions. Each model predicts the futures of the 8 executed actions of every condition. Action correspondence is the reference check of Fig. 6 (is a prediction's own executed future the nearest of the 8 by PSNR?), and D is the action-specific part of LAFS. LAFS orders the six models the way the reference check does; PSNR does not:
† trained by other teams. Rows are sorted by action correspondence. The reference row (the branch mean of Fig. 6) ignores the action, yet has the highest PSNR. For independent training runs, we also fine-tuned two architectures with two recipes and two training seeds each: the two seeds differ by at most 0.08 in D, the recipes by 0.55 and 0.20 (Table 13). App. G.4 (Table 12) gives the intervals and each model's training data.
To test this, we scored branches from other distributions with the frozen LIBERO-10 scorer, which is trained on one kind of branch only: random perturbations of one size. From the 50 evaluation conditions we executed branches from six other sources: the same type at other sizes, smooth drifts, and other demonstrations, which come from a different policy distribution. Branches from 2/3 to twice the training size and from other demonstrations score close to the training distribution; smooth drifts fall below it:
At the same gripper shift, branches from 2/3 to twice the training size score alike; smooth drifts stay harder (pairwise accuracy for pairs with a gripper shift of 1–2 cm):
If slow drifts matter for an application, they should be among the training branches; App. C now says so. App. G.6 (Table 15) gives the intervals and the comparisons at equal gripper shift.
References
[1] WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL. arXiv 26'
[2] VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators. arXiv 25'
[3] Cosmos 3: Omnimodal World Models for Physical AI. arXiv 26'