acceptodds
Withdrawn from ICLR 2027

GeoScale: Inference-Time Scaling of Geometry-Aware Video World Models for Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video world models provide a promising interface for robot manipulation by predicting future visual states that can be converted into actions. Geometric consistency and motion plausibility in these predictions are critical for recovering executable action trajectories, as robots must carry out these actions in the physical world. We observe that different random seeds at inference time produce rollouts with substantially different geometric quality, suggesting that better predictions can be obtained by searching over seeds. We therefore propose **GeoScale**, an inference-time scaling framework that searches for seeds yielding geometrically stronger rollouts. GeoScale uses feed-forward 4D reconstruction as geometric feedback for both candidate selection and guidance during denoising. It progressively scores and prunes candidate rollouts to concentrate computation on promising candidates, while applying differentiable geometric guidance to refine the surviving candidates at selected denoising steps. Experiments on real-world Droid and simulated RLBench show consistent gains in depth accuracy, 3D reconstruction, and point correspondence while preserving visual quality. GeoScale also improves human-rated task completion and simulator execution success, demonstrating more actionable rollouts for robot manipulation.

Primary area:
generative models
Keywords:
Video Diffusion Model, Geometry Consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1AqtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoScale, a training-free inference-time method for improving the geometric consistency of videos generated by a frozen image-to-video world model. The method is motivated by the observation that, for the same input, different random seeds produce rollouts of very different geometric quality. GeoScale scores candidate videos with a frozen feed-forward 4D reconstruction model 4RC. The reward combines three terms: cross-frame depth consistency of static points, smoothness of dynamic point tracks, and smoothness of the camera trajectory. The reward is used in two ways. A progressive Best-of-N search scores the candidates at two intermediate denoising checkpoints and discards half of them each time. A gradient-based guidance step back-propagates a differentiable version of the geometric loss through the VAE decoder and 4RC, and uses it to adjust the denoising velocity of the surviving candidates.

Critical strengths
  1. Training-free and applicable to existing generators. GeoScale keeps both the video generator and the 4D verifier frozen and only changes how samples are drawn and selected. It can therefore be added to an existing video world model without geometry-labeled data or retraining.
  2. The reward is carefully designed. The reward evaluates static structure, dynamic tracks and camera motion separately, so legitimate object motion is not counted as scene deformation. The static term uses a log-depth ratio, which is invariant to the overall reconstruction scale. The validity rules handle under-observed, motionless and teleporting reconstructions. A component without enough valid observations receives a neutral score rather than the maximum, so an empty reconstruction cannot win the selection.
  3. Evaluation extends to downstream use. Besides visual quality, the paper evaluates geometric accuracy and tests whether the generated videos are more useful for robot execution, which is the right question for this application.
Critical weaknesses
  1. The main comparisons are not compute-matched. Each your mehtod output is selected from 8 candidates and further refined by guidance, while each baseline output is a single sample. Without comparisons under the same compute budget, for example Best-of-N with a simpler reward or random selection, it is hard to tell how much of the improvement comes from the proposed reward and guidance, and how much comes from extra sampling.
  2. The inference cost is very high. Generating one video takes about 35 minutes on four A800 GPUs at the default setting. This makes the method impractical for closed-loop control, and the paper does not clarify whether it is intended for online planning or offline data generation.
  3. The evidence for robot manipulation is indirect. On RLBench, actions are obtained by running pose estimation and ik on the generated video and replaying them open-loop, so the results also depend on this conversion pipeline. The real-robot experiment is qualitative only, with no success rates or baseline comparison.
Questions for authors
  1. I'm not sure the static/dynamic split does what the static term needs. A pixel counts as dynamic once its 4RC track moves away from its starting position by more than 0.01 of the median first-frame depth. A background that slowly drifts or deforms, like the "background melts" case in Fig. 3, would cross this threshold. It would then fall into the dynamic set, where it only has to move smoothly. So a background that drifts smoothly might not be penalized at all. Have you checked how the reward behaves in this case? On RLBench the simulator provides ground-truth segmentation, so the split could at least be checked against it on the reference videos.
  2. How many trials are behind each RLBench number in Table 2, and how do they relate to the 50 RLBench sequences in Table 1? The actions are obtained with FoundationPose, waypoint selection and IK, and are then replayed, so part of the success rate reflects this pipeline rather than the video itself. Running the same pipeline on the ground-truth videos would give an upper bound and show how many failures come from the conversion.
  3. Geometric consistency is not the same as physically correct interaction. The reward checks whether static points stay in place and whether tracks and the camera move smoothly. A rollout in which an object slides along with the gripper without being grasped, or passes through another object, could satisfy all of these. The paper itself notes that the dynamic term is "a temporal regularizer rather than a dynamics model" and that legitimate accelerations are penalized too. Contact is exactly where such accelerations happen. So I'm not sure a higher R4D means a rollout is more executable, rather than just smoother. Why should this reward push rollouts toward correct interactions?
Reviewer_TnEDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoScale, a training-free inference-time scaling framework that enhances the 3D/4D geometric consistency of pretrained video world models (Wan2.2-I2V-14B, Cosmos-Predict2.5, and HunyuanVideo) using a frozen feed-forward 4D reconstructor (SpaTrackerV2). During 40-step flow-matching sampling, GeoScale combines a multi-term 4D consistency reward (4DC-Reward) with two-stage Progressive Best-of-N (PBoN, N=8, checkpoints at sigma in {0.83, 0.63}) pruning and normalized Geometric Optimization Guidance over sigma in (0.08, 0.83), evaluating video quality on Droid and trajectory execution on 10 RLBench tasks.

Critical strengths

(1) Practical training-free plug-and-play formulation: Using an off-the-shelf feed-forward 4D reconstructor to prune and guide frozen video diffusion rollouts at test time is a well-motivated approach to narrowing the gap between visual quality and 3D geometric consistency without expensive 4D post-training.

(2) End-to-end validation from video generation to robot manipulation: Beyond appearance and depth metrics on the real-world Droid benchmark, the paper closes the control loop on 10 RLBench tasks by recovering end-effector poses from generated 4D point clouds and executing them in simulation (raising average success from 49.1% to 67.9%, Table 2).

Critical weaknesses

(1) Axis relabeling, non-linear tick intervals, and data-provenance anomalies in Figures 6 and 13 alongside table-text mismatches: In Figures 6 (p. 9) and 13 (p. 21), the visible delta_1 y-axis places 0.56, 0.58, 0.62, 0.66, and 0.70 at nearly equal vertical grid intervals (so 0.56->0.58 spans +0.02 whereas 0.58->0.62 spans +0.04 over the same spacing), and the underlying PDF vector layers retain masked-over matplotlib tick labels (x = 1..8; original y-axes 0.4–1.1 / 0.2–1.1, 0.3–1.4, 0.1–0.9, 0.1–0.9) beneath white rectangles with overlaid labels (10^3, 10^4, 46–55). Moreover, each fixed-N curve in Figure 6 (N in {2, 4, 8, 16}) plots 8 points spanning 10^2 to 10^4 "Inference Compute" even though K = 40 steps and checkpoints sigma in {0.83, 0.63} are fixed and Table 5 reports a single runtime per N (1264s, 1432s, 2094s, 3698s), while several underlying curve points of HY-Video in Figure 13 coincide with those of Wan (n=4) in Figure 6 (the first 4 points in delta_2 and 5 of 8 points in FVD). Additional discrepancies include w/o PBoN runtime reported as 4188s in Section 4.3 (p. 9) vs. 4478s in Table 3, Figure 1 (middle) plotting baseline AbsRel as 0.449 (vs. 0.446 for Wan and 0.443 for Geometry-Forcing in Table 1), and Table 3 leaving unspecified how w/o 4DC-Reward (1973s) prunes 8 -> 4 -> 2 -> 1 without the reward.

(2) Unmatched N=8 test-time compute in main tables and incomplete reward/verifier ablations: Tables 1 and 2 compare GeoScale with N = 8 candidate rollouts (2094s ~ 34.9 min per video on 4xA800 GPUs, Table 5) against single-rollout (N = 1) baselines, without comparing against existing test-time reward verifiers (such as WMReward, cited in Section 2) under an equal N = 8 budget or reporting N = 1 with guidance alone. Furthermore, no ablation isolates the three reward terms (R_s, R_d, R_g with weights 0.5, 0.3, 0.2), and the motionless-scene gate in Appendix B.2 (p. 14) zeroes only the camera term R_g (lambda_g = 0.2) while leaving 80% of the reward (R_s and R_d) active and ungated.

(3) Per-frame alignment in geometry evaluation and under-specified guidance scaling in Equation (11): Appendix D.2 (p. 17) states that predicted depth is aligned to ground truth via per-frame median scaling (s_t) and point clouds via per-frame ICP before computing AbsRel, delta_1, delta_2, Chamfer, and F-score, which can absorb frame-to-frame scale and rigid-pose drift even though GeoScale targets multi-frame 4D consistency. Meanwhile, Equation (11) normalizes the reward gradient to match ||v_k||_2 and multiplies by eta_k = eta_0 * (1 - sigma_k^2) with eta_0 = 0.001 every f = 3 steps across sigma in (0.08, 0.83) (8–10 active updates out of K = 40 steps, Table 4), yielding a < 0.1% single-step relative velocity perturbation (and <= 0.054% single-step relative change in hat_z0 under orthogonal equal-norm latents) whose mechanism for driving the large visual and +2.41 pp SSIM gains in Figure 7 and Table 3 needs clarification.

Questions for authors
  1. Could you provide the raw data arrays and plotting scripts for Figures 6 and 13 to clarify the masked matplotlib axes (x = 1..8, y = 0.3..1.4, 0.1..0.9), the non-linear delta_1 tick spacing (0.56, 0.58, 0.62, 0.66, 0.70), what the 8 x-axis points represent for each fixed N, why HY-Video in Figure 13 shares several points with Wan (n=4) in Figure 6, and what criterion w/o 4DC-Reward uses to prune candidates in Table 3?

  2. How does GeoScale perform in Tables 1 and 2 when compared at equal N=8 test-time compute against an existing video reward verifier (e.g., WMReward), and what are the individual ablation contributions of R_s, R_d, and R_g?

  3. What are the depth and point-cloud metrics in Table 1 under a single sequence-level scale/SE(3) alignment across all T frames (rather than per-frame median scaling and per-frame ICP), and how does the eta_0 = 0.001 velocity update in Equation (11) produce the large structural improvements in Figure 7 and Table 3 (e.g., via nonlinear trajectory amplification, spatial masking, or an unstated scaling factor)?

Reviewer_AkqEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoScale, a training-free inference-time framework for improving the geometric consistency of video world-model rollouts. Given an initial image and language instruction, it samples multiple futures from a frozen video generator, scores them with a frozen 4D reconstruction model, progressively prunes candidates, and applies geometric guidance to the survivors. The paper evaluates the method on Droid and RLBench and reports improvements in geometric metrics, human-rated task completion, and simulator execution. The most interesting potential contribution is the observation that different random seeds of a frozen generator can produce futures with different geometric quality.

Critical strengths
  1. The problem motivation is clear. The paper identifies temporal geometry and correspondence as possible bottlenecks when generated videos are converted into robot actions.
  2. The inference-time design is coherent. Candidate search and geometric guidance provide complementary mechanisms for using geometric feedback without updating the video generator.
  3. The evaluation spans several levels. The paper reports geometric reconstruction and video metrics, human-rated Droid results, simulator-executed RLBench results, and qualitative physical-robot demonstrations.
Critical weaknesses
  1. The methodological contribution and generality are not sufficiently established. The paper’s key claim is that seed-dependent geometric variation in a frozen generator can be exploited at inference time. However, most results evaluate the complete 4RC + PBoN + guidance pipeline; the multi-backbone study also evaluates the integrated method, rather than testing whether this principle consistently holds against simpler, compute-matched inference strategies.
  2. The downstream application evidence is not sufficiently causal. Task results are reported for the full method, while the geometric/search ablations do not include downstream execution success. Since RLBench uses a multi-stage video-to-action pipeline and the physical-robot study is qualitative, the experiments do not directly show that the proposed geometric improvements cause more reliable manipulation.
  3. Reliability is bounded by the 4RC verifier. 4RC supplies both the ranking reward and guidance gradients, so verifier errors can affect candidate selection and refinement simultaneously. The paper acknowledges failures under difficult visual conditions but does not systematically evaluate their effect on the final decision.
Questions for authors
  1. Can the seed-dependent geometry principle be tested across multiple backbones against simpler, compute-matched inference strategies, separately from the full GeoScale pipeline?
  2. Under the same video-to-action pipeline, do the geometric and search ablations produce corresponding changes in downstream execution success?
  3. Under what conditions do 4RC failures change candidate selection or the final action decision?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.