- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GeoWAM adds training-time geometry supervision to a video-action world model without requiring geometry inputs at deployment. A frozen VGGT-Ω teacher processes multi-camera video windows, and a separately trained spatiotemporal GeoVAE compresses its features into latent sequences. An auxiliary geometry expert predicts these sequences through flow matching. Asymmetric attention allows geometry queries to read video/action keys and values, carrying the geometry loss back into the deployed experts while preventing those experts from depending on geometry tokens. The primary model generates actions from the current observation without future-video rollout; GeoWAM-Joint jointly denoises future video and actions. The reported LIBERO success rates are 98.30% and 98.65%, respectively, and LIBERO-Plus category averages are 65.20% and 76.49%. On LIBERO, the matched video-action baseline reaches 96.00%. Detachment, static-target, and decayed-anchor ablations examine the supervision pathway.
Critical strengths
-
The separation between training-time supervision and inference-time conditioning is clear and useful. Under the attention mask in Figure 3, video and action states never read geometry tokens, so removing the geometry branch leaves their forward computation unchanged. This is the strongest technical aspect of the paper: a relatively expensive geometry teacher can shape the policy without becoming an inference-time dependency, in either the no-rollout or joint-generation setting.
-
Table 3 includes a matched baseline and a targeted gradient intervention. GeoWAM improves average LIBERO success from 96.00% to 98.30%, and Long-suite success from 90.60% to 96.40%. Keeping the geometry branch and its forward connections but detaching the video/action keys and values it reads gives 95.90% average success. This provides direct evidence for the value of attention-mediated supervision transfer. The appendix also usefully states the limits of the controls: the static target retains full-window context, the anchor variant changes attention as well as noise assignment, and cross-paper results are not controlled interventions.
Critical weaknesses
- The robustness claim needs matched comparisons and uncertainty across training runs. Table 2 uses published baselines, whereas Appendix C.3 explains that the locally matched FastWAM differs from the original Fast-WAM in multi-camera encoding. LIBERO-Plus results are not reported for the matched baseline, Detach, or Static-Geometry controls. The 65.20% versus 51.36% comparison therefore cannot establish how much of the robustness gain comes from geometry gradient transfer. The margins over DreamWAM are 1.76 percentage points without rollout and 1.02 points for Joint, and several LIBERO margins over strong baselines are also small. No means and variation across independent training seeds are provided. The appendix acknowledges this limitation, but it remains consequential: many evaluation rollouts do not replace independent training repetitions.
The evaluation is also limited to LIBERO and LIBERO-Plus, with no real-robot experiments. Although the paper acknowledges this limitation, simulated perturbations do not establish that the gains persist under real camera artifacts, geometry-teacher errors, contact dynamics, and hardware execution uncertainty. This leaves the practical effectiveness of geometry supervision in physical manipulation unverified and limits the scope of the generalization evidence.
- The experiments do not sufficiently distinguish the benefit of predicting geometric evolution from that of auxiliary teacher-feature prediction more generally. Detach establishes that the gradient path helps, but not that geometry trajectories are the essential target. The static control uses the first latent from a full-window teacher and encoder; it removes variation across target frames but does not isolate geometry available from the current observation alone. Figure 5 provides one qualitative example, without a matched comparison to generic visual-feature prediction or a simpler geometry-regression head. The cost of teacher-feature extraction, GeoVAE pretraining, and the additional 882 geometry tokens is also not quantified. Since related work already uses training-time geometric supervision and removable branches, these missing comparisons make the specific contribution and its cost-benefit tradeoff difficult to assess. Together with the first concern, this leads to a weak-reject recommendation: the design is reasonable, but its distinctive benefit and robustness need stronger evidence.
Questions for authors
-
Experimental clarification and decisive additional evidence: How many independent training seeds and evaluation seeds per checkpoint underlie Tables 1–3? Can you provide per-seed results, means, and standard deviations for Matched FastWAM, GeoWAM, Detach, and Static-Geometry on both LIBERO and LIBERO-Plus under identical training/evaluation settings? In particular, does the robustness improvement persist after matching camera encoding? These results could materially change the recommendation.
-
Mechanism: Have you compared against generic visual-teacher feature prediction using the same auxiliary expert and a similar token/training budget, or a simpler head that directly regresses the same geometry latents? Do these alternatives retain most of the gain? If such comparisons are unavailable, what existing evidence separates the contribution of geometric evolution prediction from auxiliary representation supervision more generally? Full-window conditioning is not being treated here as evaluation-data leakage; the concern is what the static-target control can isolate.
-
Cost and reproducibility: What are the GPU-hours and peak memory for teacher-feature extraction, GeoVAE pretraining, and policy training, relative to the matched baseline? Please also specify the GeoVAE training-data scope, training steps/epochs, batch size, and learning rate, as well as the effective policy-training batch size. What are latency and memory for each deployment mode after actually removing the geometry modules? These details affect both reproducibility and the practical value of the training-time supervision.
5 comments
Q2. What distinguishes geometric evolution from generic auxiliary supervision?
Generic visual features and a simpler head. We explored DINOv3 as a generic visual teacher in preliminary experiments. The observed policy gains were limited, so we did not pursue that variant. We do not present this pilot as a token- and training-budget-matched comparison. We also have not tested a lightweight head that directly regresses the same GeoVAE latents. For context, DreamWAM (Table 2 of our submission) uses RAFT flow, targets derived from Depth Anything V3 depth maps, and DINOv2 semantic features. GeoWAM instead treats one additional signal, the VGGT-Ω feature trajectory, as a state to denoise. Its reported LIBERO-Plus averages are higher both without rollout (65.20% vs. 63.44%) and with rollout (76.49% vs. 75.47%). This shows what is achievable with one auxiliary target.
A controlled change of target. In DepthMap-GeoWAM we kept the frozen VGGT-Ω teacher and the auxiliary denoising pathway, but predicted teacher depth maps directly, without a geometry VAE. Splitting each 224 × 224 depth map into 32 × 32 patches yields the same 882 auxiliary tokens as GeoWAM (2 cameras × 9 frames × 49 patches). This matches token count, not necessarily wall-clock cost.
| No-rollout policy / auxiliary target | Cam. | Robot | Lang. | Light | Bg. | Noise | Layout | Plus avg. |
|---|---|---|---|---|---|---|---|---|
| Matched Fast-WAM / none | 27.77 | 43.81 | 69.81 | 83.27 | 53.25 | 49.84 | 65.11 | 56.12 |
| DepthMap-GeoWAM / depth maps | 24.02 | 47.81 | 78.14 | 90.02 | 51.67 | 47.72 | 65.44 | 57.83 |
| GeoWAM / implicit feature trajectories | 25.89 | 56.32 | 87.38 | 95.36 | 62.92 | 53.84 | 74.69 | 65.20 |
Depth prediction itself helps (+1.71 points over the matched baseline). Predicting the teacher's feature trajectory rather than its depth output yields a further 7.37 points, with higher success in every category. Both variants denoise an evolving target: the distinction tested here is which representation of that evolution supervises the policy.
What Static isolates. Our motivating question was whether the policy benefits more from recovering a scene representation or from predicting how it changes given the observation, instruction, and evolving action state. Static repeats the first geometry latent at every target time; GeoWAM predicts the sequence of distinct latents. Both keep the gradient path and use the same full-window teacher and encoder. Static therefore tests the value of temporal variation in the target, not whether current RGB alone suffices to recover geometry. The evolving target raises success from 97.20% to 98.30% on LIBERO and from 55.24% to 65.20% on LIBERO-Plus. We will make this distinction explicit in the revision.
Due to character limitations, subsequent replies will be in the next section
Q3. Training cost, GeoVAE training data, and deployment.
Policy training. We benchmarked on four NVIDIA RTX 6000D GPUs at an effective batch of 128 windows per optimizer step (4 GPUs × 4 windows × 8 accumulation steps). We measured 30 steps after 20 warm-up steps. Peak memory is the maximum allocated on any rank. The last two columns extrapolate to 21,700 optimizer steps (10 epochs).
| Policy | s/step | Samples/s | Peak GiB/GPU | Step overhead | Wall-hours (est.) | GPU-hours (est.) |
|---|---|---|---|---|---|---|
| Matched Fast-WAM | 8.61 | 14.87 | 45.49 | Reference | 51.9 | 207.6 |
| GeoWAM | 13.24 | 9.67 | 57.98 | +53.8% | 79.8 | 319.2 |
| Matched Fast-WAM-Joint | 8.33 | 15.37 | 45.49 | Reference | 50.2 | 200.8 |
| GeoWAM-Joint | 13.30 | 9.62 | 57.98 | +59.8% | 80.2 | 320.8 |
VGGT-Ω runs online to avoid storing dense teacher features for every overlapping video window. The GeoWAM rows include teacher extraction, frozen GeoVAE encoding, and the additional 882-token expert; there is no separate offline extraction run to add to these totals. The reusable LIBERO GeoVAE took approximately one additional day.
GeoVAE recipe. We trained the GeoVAE on demonstrations from the 4 LIBERO training suites only, with no large-scale GeoVAE pretraining and no LIBERO-Plus data. Each window contains nine frames per camera. The policy uses the GeoVAE checkpoint at 21,700 optimizer updates; its saved optimizer state records an initial learning rate of 5 × 10⁻⁵ and a final rate of 1 × 10⁻⁵. For reproducibility, the launcher defaults to 10 epochs, cosine decay, and an effective batch of 256 (4 GPUs × 16 windows × 4 accumulation steps). For the physical experiments, we initialized from the LIBERO GeoVAE and fine-tuned it on the two tasks' demonstration videos with a fresh optimizer and schedule, retaining the pretrained feature normalization.
Deployment after removing geometry. We measured all four LIBERO policies sequentially on the same NVIDIA H20 at batch size one in BF16. Neither GeoWAM policy loaded VGGT-Ω, the GeoVAE, or the geometry expert onto the GPU. Each replan used the same preprocessed two-camera observation and instruction and generated 32 actions in 10 denoising steps. Latency includes text encoding, image encoding, denoising, and return of the action chunk to CPU; it excludes model loading and simulation. Joint inference denoises future-video latents without decoding RGB frames.
| Policy | Mean (ms) | Median (ms) | P95 (ms) | Resident allocated (GiB) | Peak allocated (GiB) | Peak reserved (GiB) |
|---|---|---|---|---|---|---|
| Matched Fast-WAM | 385.88 | 385.25 | 389.03 | 23.1563 | 23.2323 | 23.4805 |
| GeoWAM | 391.99 | 392.41 | 393.73 | 23.1563 | 23.2323 | 23.4805 |
| Matched Fast-WAM-Joint | 650.32 | 649.94 | 653.89 | 23.1563 | 23.2324 | 23.4414 |
| GeoWAM-Joint | 643.66 | 643.86 | 648.44 | 23.1563 | 23.2324 | 23.4414 |
Thank you for recognizing the distinction between training-time supervision and inference-time conditioning, and for identifying the comparisons needed to assess robustness. We now report matched LIBERO-Plus results, a direct future-depth control, real world physical-robot experiments, and measured policy-training costs.
Q1. Does the robustness gain survive matched camera encoding? How many seeds were used?
Yes. With identical per-camera video encoding within each comparison, GeoWAM improves LIBERO-Plus success by 9.08 percentage points without future-video rollout and 9.96 points with joint rollout. For our own entries in Tables 1–3, we used one training seed per configuration and one evaluation seed per checkpoint(followed by previous work). Given the time and compute available, we prioritized evaluating the matched checkpoints on LIBERO-Plus, where there is more room for improvement than on LIBERO. We report the individual-run scores below. We would welcome the opportunity to add independent training runs in a future revision.
All values are success rates (%). Each LIBERO-Plus row covers the same 10,030 cases. GeoWAM improves on its matched baseline in six of seven categories (camera viewpoint is the exception). GeoWAM-Joint improves in all seven. Preserving the geometry forward pass but detaching its gradients reduces the no-rollout average from 65.20% to 54.51%.
Physical manipulation. We collected 100 demonstrations per task and trained Fast-WAM-Joint and GeoWAM-Joint on the same data. Both policies use joint rollout without a geometry teacher at deployment. The reference test condition already changes the camera viewpoint; the other conditions add a lighting or layout change to that reference. We ran 30 trials per task, condition, and policy (360 physical executions).
GeoWAM-Joint completed 39 more trials and had more successes in every setting. On block placement, a two-success difference under the camera change alone became a twelve-success difference when the layout also changed. The gains therefore extend beyond simulated perturbations to these two physical tasks, although changed lighting remains difficult for both policies.
Due to character limitations, subsequent replies will be in the next section