acceptodds
Withdrawn from ICLR 2027

Learning Factorized World Transitions for Geometry-Aware Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video world models must preserve both physical structure and visual detail when predicting future observations. Pretrained geometry models provide valuable structural knowledge, but geometry alone does not specify the full visual prediction. This raises the question of how complementary representations should be organized for world-transition modeling. We introduce UPCAST, a framework that learns a geometry-anchored shared physical state together with a private appearance state. Cross-modal reconstruction relates transitions from different representations, while invertible coupling combines their shared and complementary information. A conditional teacher and selective knowledge transfer consolidate the learned representation into the original autoregressive video backbone, without auxiliary modules at inference. On camera-conditioned RealEstate10K, UPCAST reduces 64-frame FVD from 534.9 to 497.8 and JEDi from 4.40 to 3.88 relative to Geometry Forcing. Zero-shot ARKitScenes evaluation shows improved RGB fidelity with mixed sensor-referenced geometric outcomes under domain shift. An action-conditioned Minecraft evaluation further demonstrates applicability beyond camera control. Representation probes and training-stage analysis provide complementary evidence for the learned factorization and its transfer to standalone generation. Together, these results support organizing shared structure and complementary visual information within a predictive transition representation. Project page: https://upcast-anon.github.io/

Primary area:
generative models
Keywords:
World Models, Video Diffusion Models, Geometry-Aware Video Generation, Multimodal Representation Learning, Long-Horizon Video Generation, Factorized Representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1n7tOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes UPCAST, a framework to learn the factorized representation for geometry-aware video world models. The geometry codes and appearance codes are learned through cross-modal reconstruction with intermediate DFoT features, PAGE-4D features, and DINOv2 features. They are coupled to predict visual features, while the geometry decoder reads only geometry features. The learned representations are consolidated into the original autoregressive video backbone with a conditional teacher and selective knowledge transfer. Experiments are conducted on RealEstate10K and ARKitScenes, where results of UPCAST could outperform baselines.

Critical strengths
  1. This paper aims to solve a clear and useful question. It is worthwhile to ask how to combine geometric structure and appearance information within a predictive representation in video world models.

  2. The paper adopts an effective distillation framework. Most auxiliary modules like PAGE-4D and DINOv2 encoders can be removed in the inference stage, so the efficiency of the proposed method can be improved.

Critical weaknesses
  1. This proposed method has a very complex architecture (Fig. 2a) with many connections and components across multiple branches. Although the authors briefly mention the high-level motivation in the Introduction, the reason of such complicated design and the motivation/goal of each module are still very unclear.

  2. What makes the above problem worse is that the paper lacks ablation study about why and how to design each component of the architecture. As a result, the overall architecture design is to some extent confusing to readers.

  3. The paper writing should be significantly improved. Currently, it is hard for the readers to understand the methods. The authors use lots of symbols in Sec. 3, but some are not clearly defined when they are used for the first time. Fig. 2a is very dense but less clear. Some symbols (including subscripts) in Fig. 2 are not aligned with the main text, and some important symbols are missed in the figure.

  4. There are also some concerns in the comparison fairness. The computations for finetuning models on RealEstate10K are different for the proposed method and baselines. In addition, there is no controlled baseline that finetunes DFoT on RealEstate10K using the same computation without extra modules.

Questions for authors

Please consider replying to the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1n7t

Thank you for pinpointing where Figure 2a obscures the method. We explain the two reconstruction targets first, then give the plain-backbone comparison and the public release dates.

Q1 (W1–W2). What does each path in Figure 2a learn, and how should the symbols be read?

A1. Both paths reconstruct the same later observation, but both decoders receive an anchor and time span; only the visual decoder also reads the appearance state. Let i be the anchor frame, j the later frame and Δ their separation. The subscript identifies the source of evidence: v is video, g is geometry-semantic and f is fused.

PathInputs to the decoderTarget at j
GeometryInitial geometry anchor, Δ, shared code QGeometry features and point predictions
VisualInitial visual anchor, Δ, coupled state U from Q and appearance state AVisual features

This is the training computation. At generation time the sole predictor is the camera- or action-conditioned DFoT. In revising Section 3 and Fig. 2a, we would introduce these two rows before the branch-specific encoders, put the same subscripts on the arrows and in the equations, and draw the downstream teacher separately.

Q2 (W1). Does the trained decoder actually use both states?

A2. The visual decoder is substantially more sensitive to removing or replacing appearance than to the corresponding physical-code intervention. We keep the acquired decoder frozen and evaluate 64 clips; the replacement comes from a different temporal span of the same scene.

InterventionVisual reconstruction loss change
Remove A / replace A with another span+45.1% / +15.3%
Remove Q / replace Q with another span+4.7% / +0.086%

In the same frozen decoder, replacing A with another span raises visual loss by 15.3%, whereas replacing Q changes it by only 0.086%.

Q3 (W3). Does plain DFoT catch up with the same additional fine-tuning and rollout refinement?

A3. With the same downstream schedule, plain DFoT reaches FVD 592.44 versus 488.60 for the acquired backbone with the downstream distillation weight set to zero. All rows receive 250 continuation and 150 refinement updates (effective batch 40), evaluated on the same 100 clips.

Backbone after continuationFVD ↓JEDi ↓PSNR ↑
Plain DFoT592.444.88513.733
Acquired backbone, transfer off488.603.72513.810
Acquired backbone, transfer on488.913.78713.816

For transfer-on minus plain DFoT the paired differences are −103.53 FVD [−146.84, −81.56] and −1.10 JEDi [−1.58, −0.70]. The two models receive the same 400 downstream updates; UPCAST also incurs the earlier acquisition and teacher-training costs.

Q4. When did the website change and the source code become available?

A4. The project page was updated after the deadline, and the separate source repository was released on October 3. The repository history records the following changes:

Date (2026)Artifact recorded in commit history
September 24–25Initial project-page commit, followed by the interactive page
September 26Interaction and media adjustments
September 29–30Expanded walkthrough, animations and narrated overview
October 3Initial source-repository commit and code release; project page links to the code

The October code release was not part of the submission-time materials. The new controls reported in this reply were conducted for the discussion period.

Reviewer_RzhxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper propose UPCAST, a factorized transition framework used only during training, for autoregressive video world models that is aware of geometry. Building on the DFoT video diffusion backbone, the authors learns a factorized representation with two part: a shared physical state anchored to geometry, and continuous private state for appearance. They learn it through cross-modal reconstruction and invertible additive coupling. A multi-stage knowledge transfer pipeline then fold what was learned back into original DFoT backbone, so none of the extra modules is needed at inference. In the experiments, the long-horizon video metrics improves over baselines such as Geometry Forcing, REPA, and VideoREPA, although the geometric metrics are more mixed under domain shift.

Critical strengths
  1. The paper offer a sensible way to organize shared geometric information together with complementary visual feature for predictive video generation. Other methods aligns video features directly with geometric representations, but UPCAST instead learn the physical and appearance states jointly, through cross-modal reconstruction and factorized representations. I think this is interesting extension of geometry-aware video modeling. 2.Because the knowledge transfer only happen during training, the original architecture stay the same at inference, with no extra geometric module. The experiments shows gains on several generation metric. The representation probes and the analyses of each training stage also gives useful evidence about what the representations learns and how that carries over to standalone backbone.
Critical weaknesses

1.The paper doesn't really isolate what the factorized representation itself contribute. UPCAST put together several existing technique, but there are no ablations that tests each component on its own. On top of that, the baselines isn't trained with matched budgets, so it's hard to say whether the gains comes from proposed method or just from extra training. 2.The experiments doesn't yet give enough evidence. For the FVD and JEDi gains over Geometry Forcing, the 95% confidence intervals includes zero, and the 128- and 256-frame evaluations use only 12 video. The geometric results on ARKitScenes is mixed, and the Minecraft evaluation is small, so it say little about whether method generalizes to action-conditioned world modeling.

Questions for authors
  1. Can the authors provide controlled ablations for the shared RVQ codebook, private appearance state, invertible coupling, and cross-modal reconstruction? How much does each component contribute to the final video generation performance?

  2. Geometry Forcing uses a released checkpoint, while UPCAST undergoes additional multi-stage training. Can the authors provide comparisons under matched training compute and supervision to isolate the contribution of the proposed method?

  3. The reported confidence intervals for FVD and JEDi improvements over Geometry Forcing include zero. Can the authors evaluate more test sequences and training seeds to establish whether these improvements are statistically reliable?

  4. The representation probes show cross-modal correspondence, but do not directly establish that the shared state captures physical transitions. Can the authors provide more direct evaluations of geometric or motion prediction accuracy using the learned physical representations?

  5. How does the proposed factorization compare with simpler alternatives, such as direct geometry-feature distillation, a single fused latent representation, or a continuous shared representation without vector quantization?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Rzhx

Thank you for suggesting a direct test of the physical state. We evaluated the encoded transitions and the deployed generator separately.

Q1. What do Q and A encode beyond the camera controls?

Both states add information about PAGE-derived point changes. We split 300 RealEstate10K source sequences 180/60/60 for training, validation and test, keeping each sequence's spans together. B includes initial RGB statistics, camera inputs and time span. Each augmented probe adds 128 representation dimensions in total.

InputMotion R² ↑Color/contrast R² ↑PAGE point-change R² ↑
B.865.300.385
B + physical Q.877.263.512
B + appearance A.875.345.549
B + Q and A.884.274.568
B + coupled U.878.310.553
B + raw video difference.897.445.571
B + raw PAGE difference.902.290.785

Adding Q to B reduces normalized point-change MSE by 0.135 [0.066, 0.230]. At fixed capacity, Q+A improves on Q across five sequence splits; its smaller gain over A has an interval crossing zero on the primary split. A alone gives better color/contrast readout than Q+A. These probes read encoded frame pairs; the point target is teacher-derived and the motion target comes from the camera controls.

Q2. What happens with matched updates or simpler supervision?

Matching the final 250 continuation and 150 refinement updates gives UPCAST lower JEDi than continued Geometry Forcing (GF): −1.10 [−2.02, −0.24]. Separately, we tested direct PAGE/DINO alignment with video-feature regularization. Both new baselines below receive 1,150 backbone updates. The original UPCAST checkpoint is a historical reference, not a total-compute-matched rerun.

Model (100 clips, 64 frames)FVD ↓JEDi ↓PSNR ↑LPIPS ↓
DFoT + direct multi-teacher alignment737.6411.33013.043.4537
Continued GF objective516.874.87812.384.3791
Original UPCAST497.823.87513.938.3957

Q3. Which components improve final generation?

Switching transfer on has little measured effect when both runs start from the same post-teacher checkpoint.

Downstream transferFVD ↓JEDi ↓PSNR ↑
Off488.603.72513.810
On488.913.78713.816

The paired FVD difference is +0.31 [−7.30, 6.22]. The generation ablation varies selective transfer while holding RVQ, private appearance, coupling and cross-modal reconstruction fixed. The U probe in Q1 measures what can be read from the resulting coupled state.

Q4. Do the longer rollouts hold up on more sequences?

Across 50 clips for all five methods, the PSNR advantage persists; both FVD difference intervals cross zero. The 128-frame measurements use prefixes of the same 256-frame rollouts.

FramesGF / UPCAST FVD ↓GF / UPCAST PSNR ↑Paired PSNR difference, dB [95% CI]
128773.3 / 737.59.77 / 11.68+1.91 [1.57, 2.27]
256909.4 / 927.88.10 / 9.93+1.83 [1.56, 2.11]

On the 100-clip, 64-frame test, UPCAST minus GF PSNR is −0.97 dB at frames 2–16, +0.73 at 17–32 and +1.93 at 33–64. At frames 129–256 in the longer cohort, UPCAST retains a +1.79 dB advantage, while LPIPS and SSIM favor GF. The original 64-frame FVD and JEDi intervals against released GF also cross zero. UPCAST generation uses one training seed: the paired intervals measure test-clip variability, and the five probe splits vary the readout data partition.

Q5. What generalizes beyond RealEstate10K?

On 100 zero-shot ARKitScenes clips UPCAST gains 1.65 dB PSNR, while GF leads on Chamfer (0.110 vs. 0.120 m) and pose recall (0.318 vs. 0.255). Minecraft shows that the same interface accepts actions on a split of ten training and ten validation trajectories. The strongest generation evidence concerns camera-controlled scenes.

Thank you for the responses and revisions. The clarifications of the metrics generally make sense. I will maintain my original score.

Reviewer_Z931OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UPCAST, a training recipe for autoregressive video world models built on DFoT. The main idea is to learn, only during training, a factorized representation of the transition between two frames: a shared physical state (a discrete codebook anchored by geometry features) and a private appearance state. The two are joined by an additive coupling and trained to reconstruct the future frame. A conditional teacher then uses this representation, and its benefit is distilled back into the plain DFoT backbone, so the final model is just DFoT at inference. The paper compares with DFoT, REPA, VideoREPA and Geometry Forcing on RealEstate10K, ARKitScenes and Minecraft.

Critical strengths
  • Since all the extra modules are dropped at test time, the final model costs nothing extra at inference, and it does improve over the DFoT backbone, with less color and exposure drift in long rollouts. I think a training-only recipe like this is useful in practice.
  • I appreciate that the paper is honest about its results. It reports bootstrap intervals and failure cases, and the ARKitScenes evaluation with sensor depth is a nice idea.
Critical weaknesses
  • To be honest, I'm not convinced the gain comes from the proposed factorization. The method has a lot of parts (shared codebook, private state, coupling, teacher, gated distillation, rollout refinement), but there is no ablation on the final model that removes one part at a time. The stage study even suggests most of the gain comes from the first stage, and the teacher and distillation stage doesn't clearly help. On top of that, compared with Geometry Forcing, UPCAST uses a different geometry teacher, two extra teachers, rollout training and much longer training, so the gain could come from any of these.
  • The gains over Geometry Forcing are not significant, and the geometry results are mixed. By the paper's own bootstrap, the FVD and JEDi improvements are not separated from zero, and they come from a single training run. On ARKitScenes, Geometry Forcing is actually better on pose and Chamfer distance, which are the most direct checks of geometry. So I don't think the "geometry-aware" claim in the title is supported yet.
Questions for authors
  1. For a fair comparison, could the authors fine-tune DFoT and Geometry Forcing with the same training budget and rollout refinement as UPCAST?
  2. I would like to see one component removed at a time (private state, codebook, coupling, teacher) on the final model.
  3. Please report results over several seeds, and on more ARKitScenes clips.
  4. What exactly happens in the first stage (learning rate, loss weights, the rollout-state adapter)?
  5. Each component comes with a short reason, but the paper doesn't explain why this design was chosen over simpler ones, and none of these choices is tested. In particular:
    • Why PAGE-4D instead of VGGT (which Geometry Forcing uses), when RealEstate10K scenes are mostly static? Why three frozen teachers, and why a small DINOv2 but a large V-JEPA?
    • Why do the transition encoders use learned queries over both frames plus a local residual, rather than a simple feature difference? Which DFoT layer do they read, and why noisy intermediate features? Why separate fusion encoders for the physical and appearance states, and why does the "geometry-semantic" branch take DINOv2 as its appearance input?
    • Why is the shared state discrete but the private state continuous? How were the codebook size and the 8×8, 768-d token grid chosen?
    • Why invertible coupling, if the inverse doesn't seem to be used by any loss? How does it compare with simple concatenation?
    • Why train a conditional teacher and then distill it, instead of distilling the representation directly? Also, the teacher reads geometry features of the future frame. Doesn't that give it information the student can never have?

Overall: The direction makes sense, and the training-only design is practical. But the method is quite complex, the writing is dense and hard to follow, and the paper doesn't show which part really matters. The gains over Geometry Forcing are not significant, and the geometry results are mixed. Overall, I currently lean toward a weak rejection, but I'm open to raising my score if same-budget comparisons show a clear gain.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Z931

Thank you for asking where the generation gains arise. We tested continued training of the baselines separately from selective transfer.

Q1. Do DFoT and Geometry Forcing catch up with rollout refinement?

With downstream updates matched, UPCAST gains 1.30 dB PSNR and reduces JEDi by 1.10 relative to continued Geometry Forcing (GF). Each row receives 250 continuation and 150 refinement updates (effective batch 40) and uses the same 100 matched 64-frame RealEstate10K clips.

BackboneFVD ↓JEDi ↓PSNR ↑LPIPS ↓SSIM ↑
DFoT592.444.88513.733.4165.4840
GF523.884.88712.517.3832.5285
UPCAST, transfer off488.603.72513.810.3964.4978
UPCAST, transfer on488.913.78713.816.3964.4978

For transfer-on minus continued GF, paired 95% intervals are [−2.02, −0.24] for JEDi and [0.88, 1.72] dB for PSNR. The FVD interval crosses zero, and GF retains better LPIPS and SSIM. Against the original released GF checkpoint, both FVD and JEDi intervals cross zero. The new control matches downstream updates; UPCAST's earlier acquisition and teacher training incur additional compute.

Q2. What does the teacher contribute? Does it see future information?

The teacher uses future geometry during training, but switching transfer on changes final JEDi by only +0.062 [−0.082, +0.312]. Both UPCAST runs start from the same post-teacher checkpoint. Transfer off sets the distillation weight to zero in both downstream phases. With transfer on, a correction receives weight only when it improves on the student's flow prediction against the known target. Generation uses only the original DFoT inputs.

Teacher diagnosticResult
Correct condition beats shuffled, 64 held-out clips, three noise levels69–88% of frame pairs
Initial reduction in flow-matching error0.052–0.072%
Teacher beats student during continuation71/1,954 eligible queries
Last update with positive transfer weight23/250

Q3. How is the transition space acquired?

We learn the space before adapting selected backbone weights to it. Representation learning takes 1,000 updates (unit learning rate 8×10⁻⁵; visual, geometry and tail loss weights 1 each). We then train the rollout-state adapter for 200 updates at 10⁻⁵ to inject state tokens into DFoT. Selective joint training takes 250 updates (diffusion learning rate 2×10⁻⁶; corresponding loss weights 0.10, 0.25 and 1.0). We read the last block of downsampling level 2: a 576-channel, 32×32 noisy feature map used by the denoiser itself.

Q4. Why these encoders, teachers and coupling?

The geometry decoder needs a common structural code; visual reconstruction also needs appearance detail. PAGE-4D supplies point maps and geometry features, DINOv2 adds appearance evidence to the geometry-semantic branch, and V-JEPA supplies a video-level target. Teacher sizes and PAGE versus VGGT on static scenes were not separately ablated.

Learned queries can match features across positions as the camera moves; the local residual retains aligned changes. Separate fusion encoders map physical evidence into two 256-entry RVQ levels and preserve appearance in a continuous state. The 8×8 grid and width 768 are interface settings. Additive coupling keeps both inputs recoverable without an inverse loss. Removing appearance raises frozen visual-decoder loss by 45.1%, versus 4.7% for the physical input. This tests decoder dependence; among the requested removals, only the transfer switch has been tested through final video generation.

Q5. What changes with more ARKitScenes clips and training seeds?

On 100 zero-shot clips, UPCAST improves RGB PSNR while GF leads on the three geometry measures below.

MethodPSNR ↑Chamfer-L1 (m) ↓F@10 cm ↑Pose R@10 ↑
GF11.51.110.615.318
UPCAST13.16.120.590.255

We used one training seed for UPCAST and computed paired 95% intervals by resampling test clips.

Comments on the paper

1 comment
Area_Chair_EvBMOpenReview ↗

Please check the authors' rebuttal

Dear reviewers,

As the rebuttal period is coming to an end, I kindly remind you to check the authors' responses. If you have not yet responded, please indicate whether your concerns have been adequately addressed. Many thanks!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.