- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GeoScale, a training-free inference-time method for improving the geometric consistency of videos generated by a frozen image-to-video world model. The method is motivated by the observation that, for the same input, different random seeds produce rollouts of very different geometric quality. GeoScale scores candidate videos with a frozen feed-forward 4D reconstruction model 4RC. The reward combines three terms: cross-frame depth consistency of static points, smoothness of dynamic point tracks, and smoothness of the camera trajectory. The reward is used in two ways. A progressive Best-of-N search scores the candidates at two intermediate denoising checkpoints and discards half of them each time. A gradient-based guidance step back-propagates a differentiable version of the geometric loss through the VAE decoder and 4RC, and uses it to adjust the denoising velocity of the surviving candidates.
Critical strengths
- Training-free and applicable to existing generators. GeoScale keeps both the video generator and the 4D verifier frozen and only changes how samples are drawn and selected. It can therefore be added to an existing video world model without geometry-labeled data or retraining.
- The reward is carefully designed. The reward evaluates static structure, dynamic tracks and camera motion separately, so legitimate object motion is not counted as scene deformation. The static term uses a log-depth ratio, which is invariant to the overall reconstruction scale. The validity rules handle under-observed, motionless and teleporting reconstructions. A component without enough valid observations receives a neutral score rather than the maximum, so an empty reconstruction cannot win the selection.
- Evaluation extends to downstream use. Besides visual quality, the paper evaluates geometric accuracy and tests whether the generated videos are more useful for robot execution, which is the right question for this application.
Critical weaknesses
- The main comparisons are not compute-matched. Each your mehtod output is selected from 8 candidates and further refined by guidance, while each baseline output is a single sample. Without comparisons under the same compute budget, for example Best-of-N with a simpler reward or random selection, it is hard to tell how much of the improvement comes from the proposed reward and guidance, and how much comes from extra sampling.
- The inference cost is very high. Generating one video takes about 35 minutes on four A800 GPUs at the default setting. This makes the method impractical for closed-loop control, and the paper does not clarify whether it is intended for online planning or offline data generation.
- The evidence for robot manipulation is indirect. On RLBench, actions are obtained by running pose estimation and ik on the generated video and replaying them open-loop, so the results also depend on this conversion pipeline. The real-robot experiment is qualitative only, with no success rates or baseline comparison.
Questions for authors
- I'm not sure the static/dynamic split does what the static term needs. A pixel counts as dynamic once its 4RC track moves away from its starting position by more than 0.01 of the median first-frame depth. A background that slowly drifts or deforms, like the "background melts" case in Fig. 3, would cross this threshold. It would then fall into the dynamic set, where it only has to move smoothly. So a background that drifts smoothly might not be penalized at all. Have you checked how the reward behaves in this case? On RLBench the simulator provides ground-truth segmentation, so the split could at least be checked against it on the reference videos.
- How many trials are behind each RLBench number in Table 2, and how do they relate to the 50 RLBench sequences in Table 1? The actions are obtained with FoundationPose, waypoint selection and IK, and are then replayed, so part of the success rate reflects this pipeline rather than the video itself. Running the same pipeline on the ground-truth videos would give an upper bound and show how many failures come from the conversion.
- Geometric consistency is not the same as physically correct interaction. The reward checks whether static points stay in place and whether tracks and the camera move smoothly. A rollout in which an object slides along with the gripper without being grasped, or passes through another object, could satisfy all of these. The paper itself notes that the dynamic term is "a temporal regularizer rather than a dynamics model" and that legitimate accelerations are penalized too. Contact is exactly where such accelerations happen. So I'm not sure a higher R4D means a rollout is more executable, rather than just smoother. Why should this reward push rollouts toward correct interactions?