- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes UPCAST, a framework to learn the factorized representation for geometry-aware video world models. The geometry codes and appearance codes are learned through cross-modal reconstruction with intermediate DFoT features, PAGE-4D features, and DINOv2 features. They are coupled to predict visual features, while the geometry decoder reads only geometry features. The learned representations are consolidated into the original autoregressive video backbone with a conditional teacher and selective knowledge transfer. Experiments are conducted on RealEstate10K and ARKitScenes, where results of UPCAST could outperform baselines.
Critical strengths
-
This paper aims to solve a clear and useful question. It is worthwhile to ask how to combine geometric structure and appearance information within a predictive representation in video world models.
-
The paper adopts an effective distillation framework. Most auxiliary modules like PAGE-4D and DINOv2 encoders can be removed in the inference stage, so the efficiency of the proposed method can be improved.
Critical weaknesses
-
This proposed method has a very complex architecture (Fig. 2a) with many connections and components across multiple branches. Although the authors briefly mention the high-level motivation in the Introduction, the reason of such complicated design and the motivation/goal of each module are still very unclear.
-
What makes the above problem worse is that the paper lacks ablation study about why and how to design each component of the architecture. As a result, the overall architecture design is to some extent confusing to readers.
-
The paper writing should be significantly improved. Currently, it is hard for the readers to understand the methods. The authors use lots of symbols in Sec. 3, but some are not clearly defined when they are used for the first time. Fig. 2a is very dense but less clear. Some symbols (including subscripts) in Fig. 2 are not aligned with the main text, and some important symbols are missed in the figure.
-
There are also some concerns in the comparison fairness. The computations for finetuning models on RealEstate10K are different for the proposed method and baselines. In addition, there is no controlled baseline that finetunes DFoT on RealEstate10K using the same computation without extra modules.
Questions for authors
Please consider replying to the weaknesses.
Response to Reviewer 1n7t
Thank you for pinpointing where Figure 2a obscures the method. We explain the two reconstruction targets first, then give the plain-backbone comparison and the public release dates.
Q1 (W1–W2). What does each path in Figure 2a learn, and how should the symbols be read?
A1. Both paths reconstruct the same later observation, but both decoders receive an anchor and time span; only the visual decoder also reads the appearance state. Let i be the anchor frame, j the later frame and Δ their separation. The subscript identifies the source of evidence: v is video, g is geometry-semantic and f is fused.
This is the training computation. At generation time the sole predictor is the camera- or action-conditioned DFoT. In revising Section 3 and Fig. 2a, we would introduce these two rows before the branch-specific encoders, put the same subscripts on the arrows and in the equations, and draw the downstream teacher separately.
Q2 (W1). Does the trained decoder actually use both states?
A2. The visual decoder is substantially more sensitive to removing or replacing appearance than to the corresponding physical-code intervention. We keep the acquired decoder frozen and evaluate 64 clips; the replacement comes from a different temporal span of the same scene.
In the same frozen decoder, replacing A with another span raises visual loss by 15.3%, whereas replacing Q changes it by only 0.086%.
Q3 (W3). Does plain DFoT catch up with the same additional fine-tuning and rollout refinement?
A3. With the same downstream schedule, plain DFoT reaches FVD 592.44 versus 488.60 for the acquired backbone with the downstream distillation weight set to zero. All rows receive 250 continuation and 150 refinement updates (effective batch 40), evaluated on the same 100 clips.
For transfer-on minus plain DFoT the paired differences are −103.53 FVD [−146.84, −81.56] and −1.10 JEDi [−1.58, −0.70]. The two models receive the same 400 downstream updates; UPCAST also incurs the earlier acquisition and teacher-training costs.
Q4. When did the website change and the source code become available?
A4. The project page was updated after the deadline, and the separate source repository was released on October 3. The repository history records the following changes:
The October code release was not part of the submission-time materials. The new controls reported in this reply were conducted for the discussion period.