Does the Future Follow the Action? Measuring Action Fidelity in World Models

Action-conditioned video world models are used to plan, to evaluate policies, and to compare models; each use depends on whether the predicted video follows the supplied action, a property we call action fidelity. Existing evaluations measure visual quality, VLM-judged plausibility, or downstream policy performance; none isolates action fidelity, and reference-based metrics can point the wrong way: on a world model's own predictions, PSNR ranks the pixel average of the executed futures above every checkpoint. Learning the judgment directly invites a shortcut. A contrastive scorer trained to match futures with their histories reaches 99% accuracy on its training task, yet exactly chance (12.5%) when asked which of eight futures from the same state an action produced: what a contrastive score measures is decided by what it is contrasted against. Contrasting each future with futures of alternative actions executed from the same state removes the shortcut and yields LAFS (Learned Action-Fidelity Score), which rates a prediction from its history and action without a reference rollout. We introduce BranchPairs, which pairs an executed future that follows the queried action, clean or under an appearance change, with one that deviates from it, on LIBERO, RoboTwin, and Action-MNIST. Trained only on clean executions, LAFS prefers the future that follows the action in 89% of LIBERO pairs (chance 50%), versus 57% for PSNR and 75% for an inverse dynamics model trained on the same data, and keeps its lead on pairs matched in PSNR (93%, versus 80% and at most 60% for visual metrics). On a Cosmos world model fine-tuned on LIBERO-10, LAFS orders three checkpoints and three controls as an executed-future reference does, unlike PSNR and a VLM critic.