acceptodds
Under review as a conference paper at ICLR 2027

Auditing Misalignment in Multi-Modal World Modeling

26 views

Abstract

An omnimodal world model can predict one physical outcome while generating another. We distinguish internal alignment with the model’s own forecast from external alignment with a physical reference, using matched predictive contracts over events, magnitudes, and, when available, timing. Two intervention ladders progressively condition generation on either the model’s own contract (A) or the physical contract (B), ranging from scene-only prompts to event, magnitude, and keyframe instructions. In a matched Super push cohort, 39/40 jointly valid pairs have correct textual forecasts but incorrect measured video endpoints; other source-bound cases exhibit agreement on an incorrect physical observable. Increasingly explicit conditioning changes whether the prescribed motion occurs, but does not establish reliable quantitative repair. We further test whether these discrepancies matter for downstream learning under fixed-budget supervision. Screening actions using the model’s own textual forecasts increases push-planning success from 0.338 to 0.635. At matched retained action support, gated supervision also reduces mean label and fitted-grid errors, although additional task-success gains over the 0.615 control remain unresolved. Together, these results provide a framework for auditing physical world models across forecasting, generation, and downstream learning, while separating three claims that are often conflated: cross-modal agreement, successful physical repair, and downstream utility.

Primary area:
generative models
Keywords:
World Modeling, Multi-Modal, Video Generation, Misalignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qWq4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper studies the mismatch between physical predictions made by multimodal world models and the outcomes they generate in videos. The authors distinguish internal consistency from physical correctness and evaluate both using controlled physics experiments. They also investigate whether consistency-based filtering can improve downstream policy learning. The topic is relevant, and the experiments reveal some interesting failure cases. However, I have concerns about the novelty, the generalizability of the findings, and the interpretation of the downstream results

Critical strengths
  1. The distinction between internal consistency and physical correctness is useful. The finding that 39 out of 40 samples have correct text predictions but incorrect video outcomes is particularly interesting.
  2. The paper includes several meaningful controls, including incorrect conditioning instructions and action-matched filtering baselines. I also appreciate that the authors report negative results and acknowledge the limitations of their conclusions.
Critical weaknesses
  1. The paper mainly combines existing ideas in physical consistency evaluation and sample filtering. Related work such as GapEval, WorldModelBench, and WorldEcho/WorldSync has already explored similar problems. The proposed distance formulation and threshold-based filtering are fairly straightforward. I am not convinced that the paper offers a sufficiently distinct contribution beyond a careful empirical study.
  2. Most experiments use Cosmos3-Super and Cosmos3-Nano, which belong to the same model family. Furthermore, the main discrepancy is largely driven by generated videos that barely move. It is unclear whether these findings reflect a broader limitation of multimodal world models or a specific weakness of these checkpoints. Results from another model family would make the conclusions more convincing.
  3. Filtering improves push-planning success from 0.338 to 0.635. However, only two of the 60 generated samples are retained, both at the same action value. Once the retained action distribution is matched, the improvement over the control drops to 0.020, with a confidence interval that includes zero. This suggests that much of the gain may come from changing the training data distribution rather than selecting physically consistent outcomes. The reported reductions in label error are encouraging, but the additional benefit for task success remains unclear.
  4. The video reader passes only three of six magnitude validation checks, and some analytic references differ from simulator outcomes. These issues raise questions about the reliability of the quantitative alignment results, particularly for the T2/T3 experiments.
Questions for authors
  1. What specific insight does this framework provide that existing physical consistency benchmarks cannot capture? A more direct comparison with related work would help clarify the novelty.
  2. Can the authors validate the main findings on another world model and provide stronger evidence that consistency-based filtering improves downstream performance when the action distribution is controlled?
1 comment

Thank you for pointing out additional experiments that helped clarify details in our work. Your two suggestions, a second model family and a filtering test with the action distribution held fixed, produced the strongest new results in the revision (see appendix Q additions with highlighted comments).

Q1. What does the framework capture that existing benchmarks cannot? Physical benchmarks such as WorldModelBench, VideoPhy and Physics-IQ score a video for plausibility or against a reference. GapEval and RealUnify score understanding and generation against reference answers, on images. WorldEcho and WorldSync diagnose action following and align generation for policy improvement, but they do not separate the model’s own forecast from an external reference. None of these measures one physical outcome in both the text and the video of the same model against a simulator, and none tests what happens when agreement between the two is used to choose training data.

Three results show why that matters. Agreement alone cannot tell failures apart: Nano’s 31 disagreeing pairs are 14 video failures, 4 text failures and 13 where both are wrong (Q.11). A motion metric misses the problem: VBench’s dynamic-degree accepts 19 clips that are physically wrong (Q.10). And an agreement filter with a biased forecast admits only wrong examples (Q.9). We agree that the distance and the threshold are simple, and we claim no new algorithm; the contribution is what the study shows.

Q2a. Another family, and “barely moving”. We agree: in this setting, near-static video looks like a Cosmos trait. We ran LTX-2 and LTX-2.5, two models from another family, on the same prompts, frames and seeds. They move the cube in 78/96 and 85/94 clips, against 7/36 for Cosmos3-Super on the new commands. Moving is not the same as being right, though: 71–82% of their videos miss a forecast that is correct (Q.1, Q.14). Stating the distance in the prompt does not help (6/48 pass; Q.3). The two families also fail in opposite directions. Given a wrong “does not swing” prompt, Cosmos stops the pendulum in 11/16 clips, while LTX-2.5 swings it in all 16 (Q.8). And inside LTX-2, the text encoder that conditions the video answers all 96 prompts correctly, while the video misses on 80 (Q.4).

Q2b. Filtering with the action distribution controlled. You noted that the original gate kept only two rows, both at one action. On LTX-2.5 it keeps 26 rows across 15 commands, and 23 of them are correct, so the matched control has real alternatives to choose from. That control still takes its action profile from the gate. Against it, the gate wins by +0.099 [+0.069, +0.130]. When the forecasts are shuffled across commands the gain disappears (mean +0.007; p = .010). An independent reader gives +0.104 (Q.14). The gain holds on new goals and actions (+0.086; Q.13) and with five other learners (post hoc; Q.14), and 100 newly generated clips replicate it (+0.055 and +0.041 with the two readers; Q.16). With the original nearest-neighbour learner and little clean data, gated video also beats clean data alone (+0.335), while unfiltered video makes the planner worse. On Cosmos the same comparison is still unresolved (+0.006 [−0.029, +0.038]; Q.2).

W4. Reader reliability. For Cosmos, the conclusion does not depend on measuring distance well, because the cube does not move. A second reader and a blinded VLM judge, validated on 34 planted clips, both confirm this (Q.6, Q.7). The three-of-six magnitude checks do limit the T2/T3 magnitude comparisons, and those stay qualified as in the paper. Where analytic and simulator references differ we report them separately, and the main push test uses executed endpoints. For LTX, distance does matter. Our reader is within 25% on all 18 planted clips of known travel, tumbles included, and an independent reader re-scored every LTX clip (Q.14). The gate result holds with both (+0.099 and +0.104), and every robustness check passes with each.

Reviewer_su2cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on diagnosing the misalignment issue in existing unified multimodal world models. A unified world model may correctly predict physical phenomena through language but fail to realize the same physical processes in its generated videos. The authors propose a physics fidelity evaluation framework, which considers several aspects: internal alignment between textual predictions and generated videos, external alignment with physical references, and whether such misalignment can be reduced through explicit conditioning. They further investigate whether alignment-based filtering can improve downstream learning. Experiments on Cosmos3-Super and Cosmos3-Nano across several controlled physical scenarios support their main findings.

Critical strengths
  1. The motivation is clear. The authors identify the inconsistency between physical prediction and video generation in unified multimodal models, which is a meaningful problem to investigate.

  2. The evaluation framework is well structured. By distinguishing internal and external alignment, the authors provide a systematic way to analyze different types of inconsistencies in physical world modeling.

  3. The experimental analysis includes several useful controls, especially the action-and-count-matched comparison in Section 5.3, which helps better understand the actual contribution of the proposed filtering strategy.

Critical weaknesses
  1. The main motivation of this paper, the understanding-generation gap in unified multimodal models, has already been studied in works like GapEval and RealUnify. This paper mainly extends the evaluation to physical quantities and combines several evaluation protocols. The main findings are relatively intuitive, and the new insights beyond existing works seem limited.

  2. In Section 5.3, after controlling the retained action distribution, the improvement becomes very marginal (0.635 vs. 0.615), and is not statistically conclusive. Moreover, the own-text gate selects the same samples as the physics-reference gate. Therefore, the unique benefit of the proposed alignment-based filtering is not clearly demonstrated.

  3. The experiments are limited to support the general conclusions. The evaluation only covers two checkpoints from the same model family (Cosmos3), with a few simple physical scenarios. For example, the main Super push experiment has only 48 attempts, and the A/B intervention experiments typically use only 8 samples per condition. Such a limited evaluation makes it difficult to judge whether the observed findings are general or specific to the selected models and scenarios.

Questions for authors

Please see the weakness section.

4 comments

We thank the reviewer for their time. Your comments helped us identify additional experiments involving a second model family (LTX) and a filtering test with the action distribution held fixed. We found that these experiments produced some of the strongest new results in the revision (see the additions to Appendix Q, highlighted for reference). First, the reviewer asks what this work shows that GapEval and RealUnify do not. Here is the direct answer:

W1. Novelty. GapEval and RealUnify score understanding and generation against reference answers, on images, with a model judge. We do three things they do not:

  • One physical quantity, checked between two modalities, with a simulator as validation. Text and video are scored on the same observable (how far the cube slides) against Isaac rather than by a model judge, where we provide a framework to do the cross modality comparison of text and video.
  • A downstream test. We use text–video agreement to choose training data and compare against a control with the same actions in the same numbers. On LTX-2.5 the gate wins by +0.099 [+0.069, +0.130] under two readers, with all robustness checks passing, and replicates on a new bank (+0.055) of generated clips; additionally with the original nearest-neighbour learner and scarce clean data it also beats clean data alone (+0.335) (Q.3, Q.14, Q.16).
  • An intervention on the reference. A 75% forecast bias makes the agreement filter admit only wrong examples and lose −0.392 [−0.477, −0.304] against a physics filter (Q.9). On LTX-2.5 the sweep gives an accuracy band: gated video beats clean data for errors from −30% to +50% under both readers, and the benefit disappears once no correct example passes (Q.15; the preregistered harm at ±50% was not resolved). That tells a practitioner how good a forecast must be before agreement can stand in for physics.

Agreement alone also cannot say which side failed: Nano's 31 disagreeing pairs are 14 video, 4 text and 13 double failures (Q.11). VBench's dynamic-degree flags 20 of 72 clips as dynamic, 19 of them physically wrong (Q.10).

W2. The marginal gain in Sec. 5.3, and the overlap with the physics gate. On Cosmos the gain over the matched control is +0.006 [−0.029, +0.038], which as the reviewer pointed out was unresolved; Cosmos produced only 2 correct examples in 97, but when we tested low yield as the explanation it did not account for the null (Q.4). However, on LTX-2.5 the gate beats the same control by +0.099, transfers to unseen goals and actions (+0.086), suggesting that the gain comes from the content of the forecasts. On your overlap point: on LTX-2.5 the gate ties the physics filter (0.875 against 0.875), and across the bias sweep it never beats it (Q.15). The gate cannot beat a ground truth physics filter; our claim is that an accurate forecast can stand in for one where none is available.

W3. Limited experiments. The revision adds a 48-command Cosmos replication (34/35), Cosmos3-Nano, LTX-2 and LTX-2.5 on all 96 prompts (LTX-2 also on new seeds), a 132-clip LTX-2.5 known-action bank plus a new 100-clip one, and a 48-command prompt-repair test. We find that both event-level claims replicate on 16 new seeds per cell: incline 1/16 against 13/16 (p = 2e-5), pendulum 5/16 against 13/16 (p = .006) (Q.8). The scenarios (push, incline, pendulum) are simple by design, so the simulator gives an exact reference; extending to contact-rich or multi-object scenes is future work, and within-model evidence is one family (Cosmos3).

Dear Reviewer su2c,

Thank you again for your review. We are following up briefly with two pointers, in case they are useful when you are able to revisit the paper.

  • Where the new material is. Appendix Q is on pages 59–75 of the revised PDF, at the very end of the file, with all new text in blue.
  • More detail on the matched-control result (W2). The control keeps the same number of generated rows at each action as the gate, chosen without the gate, so the two arms differ only in which outcomes fill those slots. When the forecasts are shuffled across commands, the gain over this control disappears (mean +0.007+0.007 against +0.099+0.099; p=.010p = .010; Q.3). This indicates that the gain comes from the forecasts being correct.

We would be glad to clarify anything further before the discussion period closes.

Thanks for the response. I appreciate the authors' efforts in addressing my concerns listed as W2 and W3. However, my main concern about the overall novelty and significance of this paper still remains. The proposed framework mainly provides a more fine-grained analysis of the already known understanding-generation gap in unified multimodal models. In particular, while the experiments identify quantitative physical inconsistencies and show that explicit conditioning does not reliably resolve them, I would expect deeper insights into what makes the understanding-generation gap distinctive in physical video generation, or a more systematic investigation into the underlying causes of such misalignment. A methodological contribution, particularly from the training perspective, to effectively mitigate this gap would also strengthen the paper. I understand that a diagnostic study does not necessarily require a new training method, but in that case, I would expect more novel findings or deeper insights into the underlying problem. Overall, I will maintain my current rating.

Thank you for returning to the discussion and for acknowledging W2 and W3. We respect your assessment, and would like to briefly clarify where we see the paper's novelty.

Novelty. These models state the correct outcome, can be told the correct outcome, and render the right event, yet still do not render the right amount. Our framework makes this measurable: a model's stated forecast is checked against what it generates, and both against a simulator. GapEval and RealUnify establish that an understanding–generation gap exists, on images scored against reference answers by a model judge; the points below concern how the gap behaves when the outcome is a measurable physical quantity, which those settings do not measure. Each point below is something the framework revealed, backed by a test or analysis in the paper:

  • Shared text components can correctly state that expected rollout. The text forecast is right in 34/35 pairs where the video is wrong (Q.1), and LTX-2's own conditioning encoder answers 96/96 commands correctly while its video misses 80/96 (Q.4).
  • Prompts move the event, not the magnitude. On the Cosmos incline, 1/16 clips move with a scene-only prompt and 13/16 with keyframe instructions (Q.8), but richer conditioning does not reliably correct the magnitude (Section 4.3). Stating the simulator's stopping distance to LTX-2.5 gives 6/48 passing videos, against 10/46 without (Q.3; one prompt on one generator).
  • Magnitude varies more with the sample than with the prompt. LTX errors run in both directions (video travel is 0.04 to 2.40 times the simulator's; Q.1), and stating the distance centres the median without narrowing the spread (0.10 to 2.87; Q.3). The gap is dispersion, not a bias that rescaling could correct.
  • Language and physics compete differently per generator. Told a pendulum "does not swing", Cosmos stops it in 11/16 clips and its own text forecast agrees; LTX-2.5 swings it in 16/16 (Q.4, Q.8). Text and video can therefore agree on a wrong outcome because both follow the prompt.
  • The errors are not only quantified in distance. Cosmos almost never shows one intact cube moving (3 of 42 and 3 of 36 clips): the cube usually deforms, or a second cube appears. LTX keeps the cube intact but tumbles or rolls it in 44–67% of moving clips, where the simulated cube slides (Q.7; a descriptive analysis).
  • The gap matters for control, and the forecast improves downstream training. On LTX-2.5, a planner trained on forecast-filtered video succeeds on 0.875 of goals, against 0.776 for a control with the same actions and counts and 0.367 with all generated video (Q.3). The gain holds on new goals and unseen actions (Q.13).

This is what we see as distinctive in physical video: our results indicate that the gap is quantitative, not semantic, and that the correct quantities appear to be lost when the model is asked to render the motion.

As you note, a diagnostic study does not require a new training method, and we agree. We believe that developing a method based on these findings is a substantial contribution in its own right, and would need a paper of its own to develop and evaluate properly. This paper's role is to establish what such a method has to satisfy, and to provide the measurements that would show whether it succeeds.

Reviewer_RPjoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an evaluation framework for auditing misalignment in multimodal world models. It distinguishes internal alignment, namely whether generated video agrees with the model’s own textual or image-text prediction, from external alignment, namely whether the generated video agrees with an analytic or simulator-based physical reference. Both comparisons are performed using matched physical contracts defined over events, magnitudes, timing/keyframes, and scene invariants.

The paper evaluates two intervention ladders. Ladder A supplies progressively richer versions of the model’s own predicted contract, while Ladder B supplies the corresponding physical contract. In the matched Super push cohort, 39 of 40 jointly valid pairs have a passing text prediction but a failing measured video endpoint. Additional Nano examples show that agreement between text and video can also occur around an incorrect forecast. The authors further studies whether disagreement can be used to filter generated supervision. However, under action-and-count matching, the success difference is 0.020, so the additional task-success benefit is unresolved. Overall, the paper presents a careful audit of forecasting, generation, and supervision quality.

Critical strengths
  1. Clear separation of internal alignment and external physical correctness. The paper avoids treating agreement between text and video as evidence of physical validity. This distinction is consequential because a model can produce internally consistent but physically incorrect outputs, and the proposed contract-based formulation makes this failure mode measurable.
  2. Careful controlled design with explicit negative results. The A/B ladders, correct and incorrect directives, source-bound counterexamples, and action-matched controls provide a more nuanced evaluation than a single aggregate accuracy score. The authors explicitly report when richer conditioning changes motion occurrence but does not repair quantitative accuracy, and when downstream improvement remains unresolved.
Critical weaknesses
  1. Limited external validity of the main empirical claims. The evidence is concentrated on controlled simulator scenes, two checkpoints from one model family, and small cells such as eight seeds per condition. Therefore, the results establish the reported phenomena in this experimental regime, but do not yet show how frequent or consequential these failures are across current multimodal world models, real videos, or diverse physical environments.
  2. The downstream filtering result does not isolate a causal benefit of own-text screening. The increase from 0.338 to 0.635 is meaningful relative to ingesting all generated labels, but the action-and-count-matched comparison is 0.635 versus 0.615 with a confidence interval crossing zero. The action support, retained counts, and selected examples therefore remain important confounds. The lower label and fitted-grid errors do not by themselves demonstrate better held-out control or a causal mechanism.
Questions for authors
  1. Were the prompts, thresholds, seed lists, inclusion/exclusion rules, and analysis procedures fixed before generating or inspecting the relevant outputs?
  2. For the 39/40 result, how sensitive is the conclusion to the 25% endpoint tolerance, the fixed-scene scale calibration, and the treatment of unreadable clips? Please report a threshold-sensitivity analysis.
  3. Can the authors evaluate the own-text gate on held-out action conditions, held-out scenes, or fresh simulator executions? In particular, does the improvement over the action-and-count-matched control persist when the selected action distribution is not reused from the same bank?
  4. How does the proposed audit behave on additional model families or independently generated physical-video datasets? If such experiments are outside the paper’s scope, please make the regime-specific nature of the conclusions even more prominent.
  5. What is the practical computational cost of the gate, including additional model queries, measurement, and calibration? The paper notes this cost but does not provide an end-to-end comparison against alternative filtering or clean-data acquisition strategies.
2 comments

Thank you for a careful review. Your questions translated directly into experiments, so most answers below are new data.

Q1. Were prompts, thresholds, seeds and analyses fixed in advance? For each new experiment, we froze the experimental protocol before generating any results. This included the hypotheses, predicted outcomes, pass/fail criteria, execution commands, random seeds, prompts, and analysis scripts. We hashed these files beforehand and will release them to allow reviewers to verify that the experimental specifications were not changed after observing the results (Q.12).

There are three qualifications:

Post hoc analyses: Four analyses in Appendix Q—the sensitivity analysis, agreement-only ablation, learner check, and second-reader audit—were conducted after the initial results were available. These are explicitly identified as exploratory rather than preregistered analyses.

Fresh-bank follow-up (Q.16): The experimental files were hashed locally before execution, but no public timestamped record was created.

Public timestamps: The Bitcoin timestamps (tamper-resistant records used to independently verify when experimental protocols were finalized) for experiments A9–A11 precede their results, providing independent evidence that the protocols were frozen beforehand. For A12–A15, the public timestamps were recorded after the results were available. Consequently, the timing of these amendments is supported by calendar receipts rather than public Bitcoin timestamps. Importantly, the follow-up experiments were developed after receiving reviewer feedback. We therefore do not present all of them as independent, preregistered confirmations. Several also produced results that did not support our expectations. For the original study, the matched push cohort was frozen before any video clips were generated.

The only subsequent amendment was a forecast token limit (primarily due to the reasoner needing a larger parameter of max tokens), which was documented before any forecasts were scored.

Finally, the 60-row bank and neutral sweeps had already been inspected before the action-matched and label-error analyses were conducted. We therefore classify these analyses as post hoc, as already disclosed in the paper.

All deviations, their justifications, and the aspects of the study they affect are documented in Q.12. Q2. How sensitive is 39/40? Very little we found. It is 39/40 for every tolerance from 0.09 to 0.91; rescaling the calibration from 0.5× to 3× leaves 39 or 40; counting all six unreadable clips as passes still gives 39/46. A second reader and a blinded VLM judge agree (Q.5–Q.7).

Q3. Held-out conditions, fresh execution, and the action-matched control. We agree that our matched control takes its action profile from the gate's own selection; it rules out "different actions" and "fewer rows", not every selection effect. However, the transfer test then shows the advantage is not tied to the bank's own actions, by evaluating on goals and actions the gate never saw.

On a held-out Cosmos bank, scored against newly executed Isaac outcomes (Q.2), the gate beats ingesting everything (+0.310 [+0.206, +0.417]) with a quarter of the label error, but the matched contrast is +0.006 [−0.029, +0.038], unresolved. On LTX-2.5, which moves the cube but reaches the Isaac outcome in only 25/119 clips (Q.3), the gate beats the matched control by +0.099 [+0.069, +0.130]. It survives Bonferroni and leave-one-out, and shuffled forecasts average only +0.007 (p = .010). With the original nearest-neighbour learner and 9 clean examples, gated video beats clean data alone by +0.335, whereas unfiltered video is worse. An independent reader gives +0.104, with all checks passing again (Q.14). The gain holds on new goals and actions (+0.086; +0.098 with the second reader, post hoc; Q.13) and with five other learners (post hoc; Q.14), and a fresh 100-clip bank replicates it (+0.055 and +0.041; Q.16).

Q4. Other families and data. LTX-2 and LTX-2.5 move the cube by the wrong distance: 71–82% of their videos miss a correct forecast (Q.1, Q.14). Within LTX-2, the text encoder that conditions the video answers 96/96 correctly while the video misses 80/96 (80/87 on fresh seeds; Q.4). LTX-2 is a pipeline, so this complements the Cosmos3 result. We did not add real or independent video datasets; the simulator is what makes the physical reference exact.

Q5. Cost. On two RTX PRO 6000 GPUs: 140 s per clip, a median 81 s per forecast (cached per condition), 1.2 s per clip for the reader, about 0.13 s per Isaac episode (Q.12). We did not run an end-to-end cost comparison; Section 6 now states that the gate is for settings without reference execution.

Thank you for the detailed rebuttal. The response substantially addresses the questions about protocol freezing, threshold sensitivity, cross-model evaluation, and computational cost. The additional experiments show that the forecast–video discrepancy is not restricted to the original Cosmos cohort, and the LTX-2.5 results provide positive evidence that forecast-based filtering can improve downstream supervision under several held-out and fresh-bank settings. I therefore retain Rating 3 (Weak Accept).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers. The reviews mainly ask three questions. Does the gap between forecast and video hold beyond one Cosmos cohort? Does filtering by the forecast do anything beyond changing which actions are kept? And what is new relative to GapEval and RealUnify? We ran new experiments for each. Their designs and pass/fail rules were hashed before they ran, and the few post hoc analyses are labelled as such (Appendix Q). The abstract, discussion and conclusion have been updated to match.

1. Correct forecasts, wrong videos, across models. We reran the main test on 48 new commands. Cosmos3-Super’s text forecast was right and its video wrong in 34 of 35 pairs, in line with the original 39/40 (Q.1). We then tried two generators from another vendor, LTX-2 and LTX-2.5. Unlike Cosmos, they do move the cube, but by the wrong distance: 71–82% of their videos miss an outcome the text gets right, across two independent distance readers (Q.1, Q.14). The same thing happens inside LTX-2 itself. The text encoder that conditions its video answers correctly on all 96 prompts, yet the video misses on 80 of them (80/87 on fresh seeds; Q.4). Putting the correct distance in the LTX-2.5 prompt does not help: 6/48 videos pass, against 10/46 without it (Q.3).

2. Filtering by the forecast selects better data, and this replicates on fresh seeds. We built a bank of 132 LTX-2.5 clips with known commanded actions and compared the forecast gate with a control that keeps exactly the same actions in the same numbers. The gate wins by +0.099 [+0.069, +0.130] (Q.3). When we shuffle the forecasts across commands, the gain disappears (mean +0.007; p = .010), so it comes from the forecasts being right and not from filtering as such. A second, independent reader gives +0.104 [+0.055, +0.153], and every robustness check passes with it as well (Q.14). The gain also holds when we evaluate on 60 new goals with actions that appear in no bank (+0.086; Q.13). Finally, we generated 100 new clips for the same commands with new seeds, under a criterion fixed beforehand (hashed locally). The gain replicates with both readers (+0.055 and +0.041), and it is positive in all 32 combinations of learner, clean-data budget and reader on both banks (Q.16).

Gated video can also beat clean data alone, but only when clean data are scarce. With the original nearest-neighbour learner, unfiltered video is worse than clean data (0.367 against 0.540) and gated video is better by +0.335 (Q.3). A neural learner given two clean examples gains +0.23 and +0.12 on the two banks. Given four or more, it no longer gains and is sometimes slightly worse (Q.16).

3. What is new relative to GapEval and RealUnify. Both benchmarks score understanding and generation against reference answers, on images, with a model judge. We do something narrower. We take one physical quantity, measure it in both the text and the video of a video model, and check each against a simulator. Then we ask what happens if agreement between the two is used to choose training data. That second step shows a risk a benchmark score cannot. When we bias the forecast by 75%, the agreement filter admits 7 examples, none of them correct, and task success falls 0.392 below a physics filter’s (95% interval 0.304 to 0.477; Q.9). On the LTX-2.5 bank we swept the bias in both directions. Gated video stays clearly better than clean data for forecast errors between −30% and +50%, and the benefit is gone once no correct example gets through (Q.15). So agreement is a useful filter inside an accuracy band that can be measured, and it takes the external reference to measure it.

Disclosures. LTX was chosen under an institutional restriction on the models we may use.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.