acceptodds
Under review as a conference paper at ICLR 2027

World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models

Abstract

Video world models must generate physically plausible futures, not just realistic frames. Existing approaches typically use semantic, physical, or predictive information to guide the visual generation stream, leaving a fundamental question: where inside the generator should predictions of scene evolution live? Standard video tokens jointly encode appearance and dynamics, but provide no distinct state for learning and reading predictive dynamics. We answer this question with the Physical Registry: temporally grounded registry slots inside a video DiT that are initialized from an observation-conditioned forecast, co-evolve with video tokens, and receive future-representation supervision, making predictive dynamics an explicit, readable state of the generator. In a controlled comparison of predictive-state interfaces on Physics-IQ Verified, Physical Registry improves vanilla fine-tuning by 9.9/8.7 points on I2V/V2V, the largest gain among the interfaces we compare. Across all Cosmos and WAN backbones, Physical Registry improves Physics-IQ Verified scores. With only 0.45% additional parameters, it enables Cosmos 3 Nano 16B to surpass the published I2V score of Cosmos 3 Super 64B (43.56 vs. 42.7). On WorldModelBench, it also improves physical adherence and the total score for all four evaluated backbones. Because the registry becomes candidate-specific while remaining readable, the Physical Registry Selector (PRS) ranks emerging futures by how far their intermediate registry readouts depart from the shared forecast before full denoising is complete. Across four backbones, this selection improves I2V scores by 5.6–15.9% without completed-video external scoring. Together, these results support a broader principle: dynamics prediction should live as an explicit model state that co-evolves with the video representation.

Primary area:
generative models
Keywords:
World model, physical video generation, predictive representations, video diffusion models, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BjLoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper notes the difficulty that world models face in generating videos that adhere to complex physical laws. To address this, it proposes injecting temporally grounded physical register slots into DiTs. By learning within a representation space aligned with JEPA, the model is endowed with the ability to generate scene evolutions that conform to physical laws.

Critical strengths
  1. The paper presents a clear methodological design and extensive experiments, demonstrating performance improvements across multiple benchmarks and base models.
  2. The visualized video generation results are clear, showing that state prediction based on temporally grounded representations enables DiT-based video generation to better adhere to the laws of physical evolution.
Critical weaknesses
  1. Although the method proposed in the paper introduces only a small number of new parameters, it leads to a significant increase in token sequence length; however, the paper fails to provide clear comparative experiments conducted under similar computational cost constraints.
  2. Furthermore, when compared to "register" methods, the proposed approach incurs a substantially higher computational cost. While the authors claim that aligning with JEPA representations enhances adherence to physical laws, they offer no theoretical explanation for why this leads to performance improvements. This argument is further weakened by the disparity in computational costs. The paper lacks direct empirical evidence regarding how the model accesses or retrieves physical information, making it impossible to determine whether the observed improvement in adherence to physical laws stems from the method's design or simply from an increase in computational expenditure.
  3. Additionally, the selection strategy which chooses candidates that deviate the most within the JEPA representation space lacks a solid theoretical grounding in physical principles. Moreover, as shown in Table 12, the performance of the PRS selection strategy is not significantly better than that of the WMReward-based selection.
Questions for authors
  1. Could you provide a qualitative and quantitative comparison of the physical registers against the baseline, while keeping the computational cost constant?
  2. Could you offer a more detailed explanation as to why selecting candidates that deviate the most from the initial JEPA distribution leads to a better ability to adhere to physical laws?
  3. Could you provide additional zero-shot examples to demonstrate the method's generalizability, and include further experiments to show that video generation performance does not degrade in out-of-distribution scenarios?
4 comments
AuthorsOpenReview ↗

Response to Reviewer BjLo#1

We thank Reviewer BjLo for noting the clear methodological design and the extensive experiments across multiple benchmarks and base models. We also appreciate the detailed feedback on computational cost (W1, W2-1 & Q1), the theoretical basis and evidence of forecast use (W2-2, W2-3), the departure rule (W3 & Q2), and zero-shot generalization (Q3).

W1, W2-1 & Q1 — Comparisons at the same computational cost

To address the concern about "comparative experiments conducted under similar computational cost constraints," we compare at the same inference cost, quantitatively (new, Table R1) and qualitatively (Figures 8 and 9), and at the same training cost.

  • W1 & Q1: Same inference cost. We ran each backbone without the registry at the number of steps whose per-video cost matches Physical Registry at its default steps (Table 8), with shared inputs and seeds.
  • Results:
    1. Physical Registry has the highest score on every backbone and task at matched compute.
    2. Matched compute raises the backbone score slightly on Cosmos v2.5 2B V2V and lowers it slightly on Cosmos 3 Nano 16B V2V. All other changes stay within the default-step std.

Table R1. Matched inference compute (Physics-IQ Verified; two seeds). * Published score.

Backbone / taskSteps (default → matched)Backbone, own run (official published * )Backbone, own run (matched)Physical Registry
Cosmos 3 Nano 16B / I2V35 → 5430.2 ± 0.829.9 ± 0.936.0 ± 1.0
Cosmos 3 Nano 16B / V2V35 → 5438.8 ± 0.837.8 ± 0.745.1 ± 0.9
Cosmos 3 Nano 16B† / I2V50 → 7837.0 ± 0.8 (37.3 ± 0.9*)36.9 ± 0.343.6 ± 1.0
Cosmos 3 Nano 16B† / V2V50 → 7846.5 ± 0.846.0 ± 0.249.5 ± 0.6
Cosmos v2.5 2B / I2V35 → 5930.7 ± 0.430.7 ± 0.233.7 ± 1.1
Cosmos v2.5 2B / V2V35 → 5937.4 ± 0.538.3 ± 0.342.3 ± 1.9
WAN I2V-A14B / I2V40 → 5831.3 ± 1.1 (32.2 ± 0.6*)31.2 ± 1.038.4 ± 1.1
WAN TI2V-5B / I2V50 → 6926.4 ± 0.8 (27.7 ± 0.9*)27.0 ± 0.629.2 ± 1.3
  • W1 & Q1: Qualitative comparison. Qualitative comparisons at matched compute are in Figures 8 and 9 (Appendix A.2).
  • W2-1: Same training cost. Even with the same backbone, adaptation data, and training budget, Physical Registry scores highest among all interface variants on both I2V and V2V (Section 4.2, Table 1).
  • Interpretation: the results suggest that, at the same inference and training cost, the gain comes mainly from the design of the registry.

So that readers can weigh the added cost against the gain, we will report the matched-compute comparison.

  • Revision: Appendix A.2 — add Table R1 with the matched-compute protocol and the results (Table 11, Figures 8 and 9), referenced from Section 4.5 (L512–L513).

W2-2 & W2-3 — Theoretical basis and direct evidence of forecast use

W2-2: Theoretical basis. We explain why aligning with JEPA representations can improve physical adherence.

  • Motivation. Physical Registry combines three lines of work in dedicated tokens supervised toward the future representation, which keeps dynamics apart from appearance (L047–L049).
    1. Prediction in representation space: JEPA predicts the predictable parts of a scene and discards unpredictable detail [1, 2].
    2. Alignment with pretrained representations: aligning generators with pretrained representations improves their training [3, 4].
    3. Dedicated tokens: registers provide dedicated tokens for computation [5].
  • Why alignment can improve generation (Section 3.3, L168–L170). The alignment trains the registry slots toward the features of the realized future, so the registry carries a forecast of the dynamics. The registry slots join the video tokens in self-attention at every denoising step, as register tokens do in ViTs [5], so the video tokens can read the aligned forecast during generation.
AuthorsOpenReview ↗

Response to Reviewer BjLo#2

W2-3: Direct evidence (new, Table R2). To address the lack of "direct empirical evidence regarding how the model accesses or retrieves physical information," we changed only the forecast that conditions the registry of the trained models at inference.

  • Static context: copies the observed context feature into all seven future slots.

  • Shuffled forecast: randomly reorders the seven future slots so that none stays at its own time step (one fixed order per scene and seed, shared across backbones).

  • Takeaway: both score below the predicted forecast in all three settings, which suggests that generation uses the predicted dynamics and their temporal order.

Table R2. Forecast-content interventions (Physics-IQ Verified). Mean ± std over two seeds (Δ vs. Physical Registry).

Registry inputNano 16B I2VNano 16B V2VWAN I2V-A14B I2V
Physical Registry (predicted forecast)36.01 ± 0.9545.12 ± 0.9438.44 ± 1.06
Static context35.22 ± 0.52 (−0.79)44.67 ± 0.09 (−0.45)36.73 ± 1.31 (−1.71)
Shuffled forecast35.48 ± 0.38 (−0.53)44.81 ± 0.04 (−0.31)37.52 ± 1.28 (−0.92)
Zero mask (Table 4a)16.78 ± 0.01 (−19.23)24.15 ± 0.32 (−20.97)35.88 ± 0.22 (−2.56)

The following analyses in the paper point the same way:

  • Forecast ablation (Table 4a): zeroing or perturbing the forecast lowers the Nano score.
  • Motion transfer (Appendix G): another video's forecast changes the ball's upward return from 80.1 to 3.6 pixels (Table 18).
  • Registry activation (Figure 5): attention and activation concentrate on the region undergoing physical change.

To make the role of shared attention visible in the method section, we will state how the DiT receives the forecast and updates the forecast toward the realized future.

  • Revision: Section 3.4 (L184–L186) — add one sentence stating that the DiT receives the forecast together with the video tokens and, through the alignment, updates the forecast to match the features of the actual future.

W3 & Q2 — Why select the largest departure

PRS selects the candidate whose generation refined the shared forecast the most, read from the model's own registry during denoising (Q2).

  1. One shared forecast: all candidates start from the same forecast, a single estimate of several possible futures [1, 6].
  2. Candidate-specific update: trained toward the realized future (Eq. 11), the registry moves from the shared forecast toward each candidate's own video during denoising (readout cosine 0.764 → 0.974; Figure 12a, L1559–L1561).
  3. Largest departure: departure (Eq. 12) measures how far each candidate's registry has moved from the shared forecast toward its own video, and PRS selects the candidate updated the most. Selecting the largest departure is +8.3 above the reversed ranking and +3.6 above random selection on average over the six settings (Figure 12c; L1569–L1572).
  • W3: Position of PRS. PRS offers two properties.
    1. The registry is a readable internal state: selecting from the registry without an external reward model shows that the registry can serve as a readout of the generation in progress.
    2. Early selection: PRS selects before the videos are fully generated and re-encoded for scoring, at 2.84–4.07× lower cost than best-of-N with WMReward (Table 17; Appendix F.4).

We agree that the submitted wording of Section 4.4 compares PRS on Physical Registry with WMReward on the backbone, and that on the same model the two are comparable in quality (Table 12 in the submission, now Table 16; L1593–L1596). In the revision, we will state PRS as comparable to WMReward at lower cost, and add the three steps above to the method section. Thank you for suggesting the same-model comparison.

  • Revision: Section 3.6 (L222–L229) — add the three steps above as the rationale for selecting the largest departure.
  • Revision: Section 4.4 (L371–L376) — replace "PRS on Physical Registry exceeds WMReward on the corresponding backbone" with the same-model comparison: PRS is comparable to WMReward in quality and costs 2.84–4.07× less.
AuthorsOpenReview ↗

Response to Reviewer BjLo#3

Q3 — Zero-shot examples and out-of-distribution performance

To address the request to show that "video generation performance does not degrade in out-of-distribution scenarios," we evaluate on the WorldModelBench domains that never appear in our Enlarge fine-tuning data (driving, industrial, gaming, and animation; 200 prompts, Table 5), where the total score rises on all four backbones (Q3; Table R3).

  • Common sense: maintained or slightly higher on all four backbones in these domains, suggesting that generation quality is preserved outside the fine-tuning domains.

Table R3. WorldModelBench on the four domains absent from Enlarge. Each cell shows the Physical Registry (PR) score as mean ± std, with the difference from the backbone in parentheses. Bold marks the higher of the two: the PR score when PR is higher, and the difference when the backbone is higher.

ModelInstruction (0–3)Physical (0–5)Common sense (0–2)Total (0–10)Δ Total, 95% CI
Cosmos 3 Nano 16B + PR2.00 ± 0.05 (−0.11)4.86 ± 0.03 (+0.40)1.83 ± 0.03 (+0.54)8.69 ± 0.06 (+0.83)[+0.68, +0.98]
Cosmos v2.5 2B + PR2.23 ± 0.11 (−0.01)4.83 ± 0.06 (+0.08)1.78 ± 0.06 (+0.07)8.84 ± 0.19 (+0.15)[+0.02, +0.31]
WAN I2V-A14B + PR2.50 ± 0.07 (+0.07)4.83 ± 0.05 (+0.02)1.86 ± 0.07 (+0.04)9.19 ± 0.14 (+0.12)[+0.01, +0.26]
WAN TI2V-5B + PR2.38 ± 0.09 (+0.13)4.79 ± 0.09 (+0.02)1.76 ± 0.10 (+0.04)8.93 ± 0.16 (+0.19)[+0.09, +0.30]
  • Zero-shot examples (Q3): additional Physical Registry generations on WorldModelBench industrial, video-game, and nature prompts are in Figures 23–25 (Appendix I.2).

We sincerely appreciate the helpful review and hope that these results and revisions address your concerns. If you have any further questions or would like additional discussion, please feel free to leave a comment at any time.

References

[1] Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985, 2025.

[2] Mur-Labadia et al. V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. arXiv:2603.14482, 2026.

[3] Yu et al. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. ICLR 2025.

[4] Wang et al. Learning Diffusion Models with Flexible Representation Guidance. NeurIPS 2025.

[5] Darcet et al. Vision Transformers Need Registers. ICLR 2024.

[6] Denton and Fergus. Stochastic Video Generation with a Learned Prior. ICML 2018.

I thank the authors for their efforts during the rebuttal period. My concerns have been largely resolved, and I will maintain my original rating.

Reviewer_GKSwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Video world models must generate physically plausible futures, not just realistic frames, which raises the question of where inside the generator predictions of scene evolution should live. The paper answers with the Physical Registry: temporally grounded registry slots inside a video DiT that are initialized from an observation-conditioned forecast, co-evolve with video tokens, and receive future-representation supervision. This makes predictive dynamics an explicit, readable state of the generator. In a controlled comparison of predictive-state interfaces, the Physical Registry gives the largest gain among the interfaces compared. It improves Physics-IQ Verified scores across all Cosmos and WAN backbones, and improves WorldModelBench physical adherence and total score for all four evaluated backbones. With only a small number of additional parameters, it enables the smaller Cosmos 3 Nano to surpass the published I2V score of the larger Cosmos 3 Super. Because the registry becomes candidate-specific while remaining readable, the Physical Registry Selector (PRS) ranks emerging futures by how far their intermediate registry readouts depart from the shared forecast before full denoising is complete, improving I2V scores without completed-video external scoring. Together, the authors argue that dynamics prediction should live as an explicit model state that co-evolves with the video representation.

Critical strengths

S1. This is a well-motivated architectural primitive. The paper asks where inside the generator predictions of scene evolution should live, and answers by separating predictive dynamics from the video tokens that jointly encode appearance and dynamics. The Physical Registry gives predictive dynamics a dedicated state that is an explicit, readable state of the generator. The recipe is simple: temporally grounded registry slots that are initialized from an observation-conditioned forecast, co-evolve with video tokens, and receive future-representation supervision. The gain generalizes to Cosmos and WAN backbones.

S2. The readable state is actually put to use. Because the registry remains readable during denoising, PRS ranks emerging futures from intermediate registry readouts, without an external reward model or verifier and before full denoising is complete. Continuing only the selected trajectory reduces the serial denoising budget and saves the decodes and reward evaluations that completed-video selection needs. The paper’s own paired bootstrap shows that the differences from WMReward are not statistically distinguishable from zero. That is a useful result for a selector that decides from internal registry readouts before full candidate completion.

S3. Negative and nuanced results are also honestly reported. Instruction following decreases on Cosmos v2.5 and WAN TI2V-5B. PRS and completed-video WMReward are not statistically distinguishable. EWMBench diversity decreases. The registry lengthens the latent sequence, so a Physical Registry sampler step costs more than a backbone step. This makes the paper easier to trust than its abstract alone would suggest. Overall, the method and protocols are easy to follow.

Critical weaknesses

W1. The headline controlled comparison trains on the test scenarios. The controlled comparison of predictive-state interfaces, the registry ablation, and the abstract’s headline gains come from adapting each variant on the center-view Physics-IQ videos and evaluating on the held-out left and right views of the same scenarios. The objects, the physical event and its outcome are seen in training; only the camera moves. The paper’s own cross-view JEPA check shows that features of the same scenario are nearly identical across views, so interfaces that ingest JEPA features directly are best placed to exploit this.

W2. Gains on the Enlarge dataset are not separated from fine-tuning the whole backbone on physics-rich data. The backbone itself is fine-tuned on Enlarge, yet ∆ is taken against the pretrained or published backbone, not against vanilla fine-tuning on the same data. The only matched control is WAN TI2V-5B, which is also the backbone with the smallest gain. For WAN 2.2 I2V-A14B, zeroing the registry at inference still yields a score well above the published one with no forecast content at all. Only part of the reported gain can be attributed to registry content, and the rest is consistent with fine-tuning on Enlarge, which only a vanilla fine-tuning control can disentangle.

W3. Baselines of mixed provenance inside one ∆ column. Plain for the expanded-prompt Nano configuration, WAN 2.2 I2V-A14B, and WAN 2.2 TI2V-5B are published leaderboard values, while the “+ Physical Registry” rows are the authors’ runs. Although the authors’ own pretrained WAN TI2V-5B scores below the published value, I believe that comparisons should have been made against baselines properly fine-tuned on the authors' enlarged dataset, rather than relying on previously published values.

W4. The selection direction is justified only empirically, and the WMReward comparison is framed generously. PRS picks the candidate that departs most from the shared forecast. Because the IoU and MSE sub-scores of Physics-IQ reward motion in the right place, a plausible confound is that r_depart proxies for motion magnitude in the emerging video. Please report the correlation between r_depart and a simple motion-energy statistic of the completed candidate, and add a motion-energy best-of-N baseline. The statement that “PRS on Physical Registry exceeds WMReward on the corresponding backbone” compares a selector on the Physical Registry model with a selector on the weaker backbone. On the Physical Registry model itself, WMReward wins in several configurations (for example on Nano V2V), and the paired differences have intervals containing zero. The defensible claim is “comparable quality at substantially lower cost”, which is a good result and should be stated as such.

W5. The ablations do not isolate the predictive part of the forecast. Zero-masking and Gaussian perturbation measure dependence on the input, not whether predicted dynamics are what helps. A static-forecast control is needed to rule out that gains come from JEPA scene features acting as extra conditioning. It could replicate the context feature into all registry slots or drop the predictor. Zero-masking also costs far more on Nano than on WAN 2.2 I2V-A14B. Calling this “the same trend” hides a several-fold difference in registry dependence that deserves an explanation.

Questions for authors

Q1. Was the published Cosmos 3 Super 64B I2V score obtained under the same protocol as your Nano models? What does pretrained score in your experiment?

Q2. What happens when the predicted slots are replaced by the context feature, or when the predictor is removed?

Q3. What is the correlation between r_depart and a motion-energy statistic of the completed candidate, and how does a motion-energy best-of-N baseline perform?

Q4. Why does registry dependence differ so much between Nano and WAN 14B?

Q5. Could you report the Table 1 with vanilla fine-tuning on Enlarge dataset for the baseline models? Or, you can also report them with training data disjoint from Physics-IQ scenarios.

9 comments
AuthorsOpenReview ↗

Response to Reviewer GKSw #1

We are grateful to Reviewer GKSw for viewing the registry as a well-motivated architectural primitive whose readable state is actually put to use. The comments on training disjoint from the test scenarios (W1 & Q5), Enlarge gains versus fine-tuning the whole backbone (W2 & Q5), own-run and prompt protocol (W3 & Q1), r_depart-based selection and WMReward (W4 & Q3), and isolating the predictive part of the forecast (W5, Q2 & Q4) helped us strengthen the paper considerably.

W1 & Q5 — Training disjoint from the test scenarios

Thank you for suggesting a scenario-disjoint split. We agree that holding out whole scenarios gives a clearer test than holding out views. Following your suggestion (Q5), we re-ran the Table 1 interface comparison with training data disjoint from the test scenarios, and Physical Registry has the highest score on both tasks (Table R1).

1. Scenario-disjoint split (Table R1) supports the advantage of Physical Registry.

  • Data: Physics-IQ Solid Mechanics category videos, chosen because this category has the most scenarios (38 of 66) while the others are too small to split, divided 19/19 within each phenomenon group (e.g., collision, falling, dominoes), with all views of a scenario in one split.
  • Training: every interface re-adapted from the pretrained Cosmos v2.5 2B; same budget.
  • Evaluation: all test views (19 test scenarios × 3 views); two seeds.
  • Results: on unseen scenarios, Physical Registry scores above every other interface on both I2V and V2V.
  • Interpretation: The advantage of Physical Registry generalizes to unseen scenarios.

Table R1. Interface comparison on unseen scenarios (Physics-IQ Verified; 19 test scenarios × 3 views). Mean ± std over two seeds; Δ relative to vanilla fine-tuning.

InterfaceI2VΔ I2VV2VΔ V2V
Vanilla fine-tuning36.5 ± 0.1–44.6 ± 1.7–
REPA33.5 ± 2.9−3.039.0 ± 3.2−5.6
Cross-attention38.6 ± 0.6+2.145.2 ± 1.5+0.6
AdaLN35.3 ± 1.3−1.247.0 ± 1.7+2.4
Joint generation40.7 ± 0.3+4.246.8 ± 0.1+2.2
Registry only40.4 ± 1.6+3.948.9 ± 2.0+4.3
Physical Registry44.3 ± 1.2+7.850.8 ± 0.2+6.2

We will make the scenario-disjoint comparison the main controlled comparison (Table 1) and keep the held-view comparison, which covers all 66 scenarios, in the appendix as a complement.

2. Interfaces that ingest JEPA features directly. Joint generation and Registry only also take the JEPA features as a direct input path, yet Physical Registry scores above both on the scenario-disjoint split (Table R1) and in the held-view setting (Table 12, originally Table 1). Registry only differs from Physical Registry only in the alignment term, so the gap can be explained by aligning the registry tokens directly to the future JEPA features (Section 4.2, L314–L319).

3. Registry ablation on unseen scenarios (new, Table R2). We also re-ran the registry ablation of Table 4(b) on the scenario-disjoint split, adding ViT Needs Registers [1], which prepends input-independent learnable tokens without the forecast or the alignment.

  • Results: ViT Needs Registers scores below vanilla fine-tuning on both I2V and V2V, and Physical Registry scores above both.
  • Interpretation: as in the held-view setting, the gain does not come from added register capacity alone.

Table R2. Registry ablation on unseen scenarios (Physics-IQ Verified; 19 test scenarios × 3 views). Mean ± std over two seeds; Δ relative to vanilla fine-tuning.

VariantI2VΔ I2VV2VΔ V2V
Vanilla fine-tuning36.5 ± 0.1–44.6 ± 1.7–
ViT Needs Registers [1]34.1 ± 1.4−2.442.9 ± 1.1−1.7
Physical Registry44.3 ± 1.2+7.850.8 ± 0.2+6.2

We will report the registry ablation on unseen scenarios as Table 4(b).

Thanks to your suggestion, the interface comparison and the registry ablation can now be shown on scenarios unseen during adaptation.

  • Revision: Abstract (L021–L024) — add that on unseen Physics-IQ scenarios, Physical Registry improves vanilla fine-tuning by 7.8/6.2 points on I2V/V2V, the largest gain among the interfaces we compare.
  • Revision: Section 4.2 and Table 1 (L291–L297) — make the scenario-disjoint split the main controlled comparison.
  • Revision: Appendix A.3 (new, L1273–L1314) — move the held-view comparison here, with the cross-view JEPA check (Figure 10), as a complement covering all 66 scenarios.
  • Revision: Section 4.5 and Table 4(b) (L454–L460) — report Table 4(b) on the scenario-disjoint split instead of the held-view setting (Table R2).
AuthorsOpenReview ↗

Response to Reviewer GKSw #2

W2 & Q5 — Enlarge gains vs fine-tuning the whole backbone

Following the reviewer's suggestion, we fine-tuned Cosmos v2.5 2B on Enlarge without the registry (new, Table R3), and the paper already has the same control for WAN TI2V-5B (Appendix B). With the same data and training setup, Physical Registry stays above vanilla fine-tuning on both.

1. Cosmos v2.5 2B (new, Table R3). To separate the registry gain from Enlarge fine-tuning, we fine-tuned Cosmos v2.5 2B on Enlarge without the registry, with the same training setup and the same number of iterations.

  • Results: Enlarge fine-tuning alone stays near the pretrained level, while Physical Registry is above Enlarge fine-tuning on both tasks.

Table R3. Matched fine-tuning of Cosmos v2.5 2B on Enlarge (same iterations; Physics-IQ Verified).

TaskPretrainedFine-tuning with EnlargePhysical RegistryRegistry − Fine-tuning, 95% CI
I2V30.7 ± 0.430.5 ± 0.233.7 ± 1.1+3.2 [0.4, 6.0]
V2V37.4 ± 0.538.4 ± 1.442.3 ± 1.9+3.9 [0.6, 7.3]

2. WAN TI2V-5B (Appendix B). As the reviewer notes, the paper already includes a vanilla fine-tuning control on Enlarge for WAN TI2V-5B, and Physical Registry is ahead of the control at all eight checkpoints.

3. Cosmos 3 Nano 16B and WAN I2V-A14B. Fine-tuning these two large backbones on Enlarge with the same setting and iterations is not feasible with our GPU resources within the rebuttal period. We will run both controls and add them to Appendix B, together with Table R3, in the camera-ready version.

  • Zero mask on WAN I2V-A14B: we address the zero mask in W5 below.

Thank you for pointing out the need for matched fine-tuning controls.

W3 & Q1 — Own-run and prompt protocol

  • Prompt protocol (Q1): yes, the published Cosmos 3 Super 64B† and our Nano† runs share the same prompt-expansion protocol (L341, L349–L350), and our own run of pretrained Nano† scores 37.0 ± 0.8.
  • Choice of Plain (Table R4): our own runs scored below the published values in all three cases. We used the published values to stay consistent with the original reports and to avoid overstating the gain.

Table R4. Published and own-run pretrained scores (Physics-IQ Verified).

Backbone / taskPublishedOwn run
WAN TI2V-5B / I2V27.7 ± 0.926.4 ± 0.8
WAN I2V-A14B / I2V32.2 ± 0.631.3 ± 1.1
Nano† / I2V37.3 ± 0.937.0 ± 0.8

So that readers can see how each Plain value was chosen, we will document the rule and both baselines.

  • Revision: Table 2 note (L350) — add that Plain uses the published value where one exists, which was higher than our own run in all three cases, with own-run values and Δ in Appendix A.2.
  • Revision: Appendix A.2 — add the own-run Plain values and Δ as Table 10.
AuthorsOpenReview ↗

Response to Reviewer GKSw #3

W4 & Q3 — r_depart-based selection and WMReward

To address the concern that r_depart "proxies for motion magnitude in the emerging video," we report the correlation between r_depart and motion energy and a motion-energy best-of-N baseline on the existing candidate pools (Table R5). For WMReward, we restate the comparison as "comparable quality at substantially lower cost," as suggested.

  • Motion energy: the mean absolute frame difference, adapted from motion-energy images [2], ranks candidates by how much the candidates move. If departure tracked motion, selecting the highest-energy candidate would reproduce PRS.

E(v(n))=1(T−1)HWC∑t=1T−1∑x,c∣vt+1(n)(x,c)−vt(n)(x,c)∣E\big(v^{(n)}\big)=\frac{1}{(T-1)HWC}\sum_{t=1}^{T-1}\sum_{x,c}\big|v_{t+1}^{(n)}(x,c)-v_{t}^{(n)}(x,c)\big|

Here v(n)v^{(n)} is candidate nn over the scored segment, sampled at 16 FPS (TT frames), resized to H×W=180×320H\times W=180\times320 with C=3C=3 color channels and pixel values in [0, 255]. Max-energy BoN selects arg⁡max⁡nE(v(n))\arg\max_n E(v^{(n)}) and Min-energy BoN selects arg⁡min⁡nE(v(n))\arg\min_n E(v^{(n)}).

  • Results (Table R5): ρ(r_depart, energy) is negative in all eight settings.
  • Highest energy: the highest-energy picks move 1.40× the pool mean and fall below Plain in all eight settings. PRS − Max CIs stay above zero, also under nine other energy definitions.
  • Motion of PRS picks: near-average motion, 0.90× the pool mean (0.79–1.01×), whereas lowest-energy picks move 0.75×.
  • Interpretation: departure measures what generation added beyond the shared forecast (Section 3.6).
    Departure is distinct from motion magnitude in either direction.

Table R5. Departure versus motion energy (Physics-IQ Verified). All rows are the Physical Registry model of each backbone, and every policy selects from the same candidate pool. ρ is the Spearman correlation between r_depart and motion energy across the candidates of each input, averaged over inputs. Intervals: paired scenario bootstrap, 10,000 replicates (L1579–L1580).

Backbone / taskρ(r_depart, energy)PlainMax-energy BoNMin-energy BoNPRSPRS − Max, 95% CIPRS − Min, 95% CI
Cosmos 3 Nano / I2V−0.3936.0132.8237.9240.27+7.45 [3.31, 12.01]+2.35 [0.67, 3.97]
Cosmos 3 Nano / V2V−0.2745.1239.6746.2046.78+7.11 [3.75, 10.57]+0.58 [−1.29, 2.39]
Cosmos 3 Nano† / I2V−0.1243.5640.1645.1345.81+5.65 [3.03, 8.35]+0.68 [−1.16, 2.45]
Cosmos 3 Nano† / V2V−0.0349.5446.6450.4052.06+5.42 [3.34, 7.56]+1.66 [−0.10, 3.43]
Cosmos v2.5 / I2V−0.6533.7126.4436.7239.08+12.64 [8.84, 16.70]+2.36 [0.41, 4.33]
Cosmos v2.5 / V2V−0.5442.3333.5746.0647.67+14.10 [9.73, 18.83]+1.61 [0.17, 3.08]
WAN 14B / I2V−0.4438.4435.0639.3340.58+5.52 [2.87, 8.24]+1.25 [−0.58, 3.09]
WAN 5B / I2V−0.2429.2423.9434.1332.03+8.09 [5.62, 10.79]−2.10 [−3.99, −0.26]
  • Comparable quality of PRS and WMReward (L1593–L1596, Table 17): We agree that PRS and WMReward reach comparable quality. The position of PRS rests on two properties.
    1. The registry is a readable internal state: selecting from the registry without an external reward model shows that the registry can serve as a readout of the generation in progress.
    2. Early selection: PRS selects before the videos are fully generated and re-encoded for scoring, at 2.84–4.07× lower cost than best-of-N with WMReward (Table 17; Appendix F.4).

We will tone down the comparison in Section 4.4 accordingly, stating PRS as comparable to WMReward and presenting PRS by the two properties above. Thank you for pointing out the cross-model comparison.

  • Revision: Section 4.4 (L371–L376) — mark the comparison across models as system-level, and state that on the same model PRS is comparable to WMReward in quality at 2.84–4.07× lower cost.
Reviewer_R3pGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Physical Registry, a set of forecast-initialized tokens inside a video DiT for explicitly representing future dynamics. The registry co-evolves with video tokens and is aligned with future V-JEPA representations during training. The authors also propose PRS, which uses intermediate registry states to select candidates before denoising is complete. Experiments on Physics-IQ Verified, WorldModelBench, and EWMBench show consistent improvements across several Cosmos and WAN backbones.

Critical strengths
  1. The paper studies a clear and important question: where predictive dynamics should be represented inside a video generator.

  2. The method is evaluated through controlled interface comparisons, ablations, multiple backbones, and several physical-world benchmarks.

  3. The registry provides a readable internal state that can also support efficient candidate selection without requiring an external reward model or fully generated videos.

Critical weaknesses
  1. Generality. The paper asks where predictive dynamics should reside, but all comparisons use V-JEPA features. It is unclear whether the conclusion still holds with other physical or predictive representations.

  2. Generalization. The controlled experiment trains on center views and tests on other views of the same scenarios. This demonstrates cross-view generalization, but not generalization to unseen objects, scenes, or physical events.

  3. Physical meaning of the registry. The current ablations show that the model uses the forecast, but do not prove that it specifically encodes physical dynamics. Temporal shuffling, static forecasts, or forecasts with matched appearance but different dynamics would provide stronger evidence.

  4. Evaluation. Physical Registry performs best, while the differences among AdaLN, joint generation, and registry-only are relatively small and often within the reported variation. The lack of human evaluation and statistical significance testing makes it difficult to determine whether the generated videos are genuinely more physically plausible.

Questions for authors
  1. Does the advantage of the registry remain when using predictive representations other than V-JEPA?

  2. How well does the method generalize to unseen objects, scenes, and physical events, rather than new views of the same scenarios?

  3. What happens when the forecast is temporally shuffled, made static, or replaced by a forecast with different dynamics? Would the generated motion follow these changes?

  4. Can the authors provide human evaluation or paired significance tests to confirm that the improvements correspond to genuinely better physical plausibility?

3 comments
AuthorsOpenReview ↗

Response to Reviewer R3pG#1

We sincerely thank Reviewer R3pG for finding our question clear and important and for noting the controlled interface comparisons and the readable internal state. We also appreciate the feedback on generality beyond V-JEPA (W1 & Q1), unseen scenarios (W2 & Q2), the physical meaning of the registry (W3 & Q3), and statistical evaluation (W4 & Q4).

W1 & Q1 — Generality

We use V-JEPA because the model fits the task in two ways.

  1. Forecast and target in one space: the V-JEPA predictor predicts the features of future frames from the observed frames in the same space, so one model supplies both the forecast that initializes the registry and the encoded future frames that supervise the registry [1].
  2. Physics- and dynamics-aware content: V-JEPA features are sensitive to intuitive-physics violations and carry information about motion and physical plausibility (Section 2, L088–L089) [2, 3].
  • Other representations: image features such as DINOv2 [4] have no temporal model or predictor, and pixel-reconstruction video features such as VideoMAE [5] may also encode appearance details and have no feature-space predictor.
  • Possible extensions: DINOv2 or VideoMAE features could still feed the registry by (1) attaching a learned predictor, as DINO-WM does for DINOv2 features [6], or (2) training an encoder–predictor pair end to end on the target domain, as LeWorldModel does from pixels [7].

Other predictive representations. Task-specific predictors could add further benefits.

  • Robot manipulation: action-conditioned JEPAs [1, 8] forecast the future under a given action, so the registry could render the outcome of each candidate action.
  • Autonomous driving: latent world models [9, 10] forecast scene features under the ego trajectory or driving intention, so the registry could keep the generated scene consistent with the planned maneuver.

We will add the possible extensions and the task-specific predictors to Section 5, renamed Conclusion and Future Work. Thank you for raising the question of generality.

  • Revision: Section 3.2 (L159) and Appendix A.1 (L1136–L1145) — explain that the registry needs both a forecast and a target for the realized future, which V-JEPA provides in one feature space, and that other encoder–predictor pairs with these two properties could be used with the same adapter, slots, and alignment loss.
  • Revision: Section 5 (L523, L526–L532) — rename "Discussion and Conclusion" to "Conclusion and Future Work", and add the possible extensions and task-specific predictors above as future work.

W2 & Q2 — Generalization to unseen objects, scenes, and physical events

Thank you for pointing out the limits of the held-view split. We agree that held-out views of the same scenarios may not by themselves show generalization to unseen objects, scenes, or physical events. To test generalization to unseen scenarios, we re-ran the Table 1 interface comparison with training data disjoint from the test scenarios, and Physical Registry has the highest score on both tasks (Table R1).

1. Scenario-disjoint split (new, Table R1) supports the advantage of Physical Registry.

  • Data: Physics-IQ Solid Mechanics category videos, chosen because this category has the most scenarios (38 of 66) while the others are too small to split, divided 19/19 within each phenomenon group (e.g., collision, falling, dominoes), with all views of a scenario in one split.
  • Training: every interface re-adapted from the pretrained Cosmos v2.5 2B; same budget.
  • Evaluation: all test views (19 test scenarios × 3 views); two seeds.
  • Results: on unseen scenarios, Physical Registry scores above every other interface on both I2V and V2V.
  • Interpretation: The advantage of Physical Registry generalizes to unseen scenarios.

Table R1. Interface comparison on unseen scenarios (Physics-IQ Verified; 19 test scenarios × 3 views). Mean ± std over two seeds; Δ relative to vanilla fine-tuning.

InterfaceI2VΔ I2VV2VΔ V2V
Vanilla fine-tuning36.5 ± 0.1–44.6 ± 1.7–
REPA33.5 ± 2.9−3.039.0 ± 3.2−5.6
Cross-attention38.6 ± 0.6+2.145.2 ± 1.5+0.6
AdaLN35.3 ± 1.3−1.247.0 ± 1.7+2.4
Joint generation40.7 ± 0.3+4.246.8 ± 0.1+2.2
Registry only40.4 ± 1.6+3.948.9 ± 2.0+4.3
Physical Registry44.3 ± 1.2+7.850.8 ± 0.2+6.2

2. Additional evidence of the generalization of Physical Registry (Tables 2–3, Appendix E).

  • Physics-IQ (Table 2): Physical Registry improves Physics-IQ Verified on all four backbones.
  • WorldModelBench (Table 3): physical adherence and the total score improve on all four backbones.
  • EWMBench (Appendix E): on real robot manipulation episodes, Physical Registry improves Cosmos 3 Nano's scene consistency and all three motion metrics.
AuthorsOpenReview ↗

Response to Reviewer R3pG#2

W3 & Q3 — Shuffled, static, and different-dynamics forecasts

To address the concern that the current ablations "do not prove that it specifically encodes physical dynamics," we ran static and shuffled forecasts and also report forecasts with different dynamics (Appendix G) and registry activation (Figure 5).

1. Static and shuffled forecasts (new, Table R2). To test whether the registry relies on the predicted dynamics beyond scene content, we change only the forecast features that condition the registry of the trained models at inference.

  • Shuffled forecast: randomly reorders the seven future slots so that none stays at its own time step (one fixed order per scene and seed, shared across backbones).

  • Static context: copies the observed context feature into all seven future slots.

  • Results (Table R2): a static or shuffled forecast consistently scores lower than the original predicted forecast.

  • Interpretation: the results suggest that the forecast contributes predicted dynamics and their temporal order.

Table R2. Forecast-content interventions (Physics-IQ Verified). Mean ± std over two seeds (Δ vs. Physical Registry).

Registry inputNano 16B I2VNano 16B V2VWAN I2V-A14B I2V
Physical Registry (predicted forecast)36.01 ± 0.9545.12 ± 0.9438.44 ± 1.06
Static context35.22 ± 0.52 (−0.79)44.67 ± 0.09 (−0.45)36.73 ± 1.31 (−1.71)
Shuffled forecast35.48 ± 0.38 (−0.53)44.81 ± 0.04 (−0.31)37.52 ± 1.28 (−0.92)
Zero mask (Table 4a)16.78 ± 0.01 (−19.23)24.15 ± 0.32 (−20.97)35.88 ± 0.22 (−2.56)

2. Forecast with different dynamics: motion transfer (Appendix G, Figures 15–16). When the forecast is replaced by one with different dynamics, the generated motion follows the replaced forecast.

  • Setup: we replace the forecast with the JEPA features of a source video with different dynamics, keeping the image, text, initial noise, and sampler fixed.
  • Results: a near-contact source reduces the ball's upward return from 80.1 to 3.6 px (Table 18), and a slow-descent source increases descent time from 371 to 478 ms.

3. Registry activation concentrates around interacting or changing regions (Section 4.6, Figure 5; Appendix I.3, Figures 26–31).

  • Attention to the aligned registry slots and registry activation concentrate around the region undergoing physical change (L520–L521, Figure 5), and more Physics-IQ examples show the same tendency (L2342–L2344, Figures 26–31).
  • Across DiT layers, the alignment-supervised layer responds around the balloon and its expanding contour, while the other layers keep broad scene-level activation (L1966–L1970, Figure 17).

We will add the static and shuffled forecast results and the interpretation to the forecast ablation table (Table 4a). Thank you for suggesting the forecast controls.

  • Revision: Section 4.5 and Table 4(a) (L451–L454, L474–L476) — add static context and shuffled forecast to Table 4(a), with the interpretation that the registry uses the forecast the registry receives (protocol in Appendix A, L955–L956).
AuthorsOpenReview ↗

Response to Reviewer R3pG#3

W4 & Q4 — Paired significance tests

To address the concern about significance, we computed a scenario-level paired bootstrap 95% CI for every Physical Registry − variant difference, both in the held-view setting originally in Table 1 (now Table 12) and on the scenario-disjoint split added in this rebuttal (Table R1, now Table 1), shown in Table R3.

  • Scenario-disjoint (Table R1): every difference is positive, and seven of the twelve intervals exclude zero. With only 19 test scenarios the intervals are wider, and the other five lie mostly above zero.
  • Held-view (Table 12, originally Table 1): on all 66 scenarios, all twelve intervals exclude zero, and every gap is well above the seed standard deviations.

Table R3. Paired differences in the held-view setting (Table 12, originally Table 1) and on the scenario-disjoint split (Table R1, now Table 1) (Physics-IQ Verified):

Δ = Physical Registry − variant [95% CI].

Task, settingVanilla FTREPACross-attentionAdaLNJoint generationRegistry only
I2V, held-view+9.9 [7.2, 12.7]+9.4 [6.6, 12.3]+8.1 [5.4, 10.9]+6.1 [3.5, 8.8]+6.1 [3.3, 8.9]+6.2 [3.6, 8.9]
I2V, scenario-disjoint+7.8 [2.2, 14.1]+10.8 [5.2, 16.5]+5.7 [0.1, 11.3]+9.0 [3.6, 15.0]+3.6 [−0.7, 8.9]+3.9 [−0.3, 8.8]
V2V, held-view+8.7 [6.3, 11.2]+8.2 [5.7, 10.8]+5.9 [3.5, 8.4]+6.8 [4.3, 9.4]+7.1 [4.6, 9.7]+4.8 [2.4, 7.3]
V2V, scenario-disjoint+6.2 [0.03, 13.1]+11.8 [4.8, 20.3]+5.6 [−1.1, 14.0]+3.8 [−0.5, 8.7]+4.0 [0.3, 8.1]+1.9 [−1.8, 5.5]
  • Scope of Tables 1 and 12:
    1. Goal: to test whether the Physical Registry structure uses the same predictive information more effectively than the other interfaces, rather than to rank the variants against one another.
    2. Design: all variants share the backbone, data, budget, and JEPA features and differ only in whether and how the forecast enters.

To make the scope of Tables 1 and 12 clear in the main text, we will state the scope in Section 4.2. We hope the paired intervals address your concern about significance.

  • Revision: Section 4.2 (L290–L291) — introduce the controlled comparison as a test of whether the Physical Registry structure uses the same predictive information more effectively than the other interfaces.

We sincerely appreciate the helpful review and hope that these results and revisions address your concerns. If you have any further questions or would like additional discussion, please feel free to leave a comment at any time.

References

[1] Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985, 2025.

[2] Garrido et al. Intuitive Physics Understanding Emerges from Self-Supervised Pretraining on Natural Videos. arXiv:2502.11831, 2025.

[3] Joseph et al. Interpreting Physics in Video World Models. arXiv:2602.07050, 2026.

[4] Oquab et al. DINOv2: Learning Robust Visual Features without Supervision. TMLR 2024.

[5] Tong et al. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS 2022.

[6] Zhou et al. DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.

[7] Maes et al. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312, 2026.

[8] Fu et al. LeWAM: End-to-End World and Action Modeling with JEPAs. arXiv:2610.12818, 2026.

[9] Li et al. Enhancing End-to-End Autonomous Driving with Latent World Model. ICLR 2025.

[10] Zheng et al. World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model. ICCV 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their thoughtful comments and are glad the reviewers found the question important and the design clear (R3pG, GKSw, BjLo), the evaluation extensive (R3pG, BjLo), and the readable registry useful (R3pG, GKSw). Below we summarize the experiments added in this rebuttal and the changes in the revision.

  • Scenario-disjoint split: we re-ran the interface comparison with training data disjoint from the test scenarios (R3pG W2 & Q2; GKSw W1 & Q5).
  • Same computational cost: we compared at the same inference cost, and fine-tuned on Enlarge without the registry at the same training cost (GKSw W2 & Q5; BjLo W1, W2-1 & Q1).
  • Direct evidence of forecast use: we ran static and shuffled forecasts, and extended the zero mask to all four backbones (R3pG W3 & Q3; GKSw W5, Q2 & Q4; BjLo W2-3).
  • Out-of-distribution domains: we evaluated on the WorldModelBench domains absent from Enlarge (BjLo Q3).
  • Paired significance tests: we computed a scenario-level paired bootstrap 95% CI for every Physical Registry − variant difference (R3pG W4 & Q4).
  • Own-run backbones: we report our own runs of the pretrained backbones next to the published values (GKSw W3 & Q1).
  • Selection by departure: we compared departure with motion energy, and explain why PRS selects the largest departure (GKSw W4 & Q3; BjLo W3 & Q2).

Summary of revisions. New text is in blue in the revised PDF, and all line, table, and appendix numbers in our responses refer to it.

  • Abstract (L021–L024), Section 4.2 and Table 1 (L291–L297): report the unseen-scenario comparison as the main controlled comparison (R3pG W2, GKSw W1).
  • Section 3.2 (L159) and Appendix A.1 (L1136–L1145): explain why V-JEPA suits the registry (R3pG W1).
  • Section 3.4 (L184–L186): state how the DiT receives the forecast and updates the forecast toward the realized future (BjLo W2-2).
  • Section 3.6 (L222–L229): explain why PRS selects the largest departure (BjLo W3).
  • Table 2 note (L350): add the rule for choosing published or own-run Plain (GKSw W3).
  • Section 4.4 (L371–L376): state PRS as comparable to WMReward at lower cost (GKSw W4, BjLo W3).
  • Section 4.5 and Table 4(a) (L451–L483): add static and shuffled forecasts and the interpretation of the zero-mask results on WAN I2V-A14B and Nano (R3pG W3, GKSw W5).
  • Section 5 (L523–L538): rename the section Conclusion and Future Work, and add the unseen-scenario comparison and other predictive representations (R3pG W1, W2).
  • Appendix A (L954–L956): add the zero-mask and shuffled-forecast details (R3pG W3, GKSw W2, W5).
  • Appendix A.2 (new): add zero mask on four backbones, own-run Plain, and matched sampling compute (GKSw W3, W5; BjLo W1).
  • Appendix A.3 (new, L1273–L1314): move the held-view comparison and the cross-view JEPA check here (GKSw W1).
  • Appendix I.2 (L2095–L2096): add out-of-domain zero-shot examples (Figures 23–25) (BjLo Q3).

We hope the new results and revisions address the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.