acceptodds
Under review as a conference paper at ICLR 2027

GeoWAM: Geometry Gradient Transfer For World Action Models

14 views

Abstract

World action models learn robot policies through video prediction and action generation, but predicting visual appearance does not explicitly encourage representations of how scene geometry evolves. For manipulation, the critical question is not only what a future scene will look like, but how the spatial relationships between the robot and surrounding objects will change. We argue that predicting geometry trajectories provides complementary supervision for understanding how scenes evolve during interaction and learning policies that generalize beyond visual appearance. We introduce GeoWAM, which augments video–action learning with generative modeling of geometric evolution. A frozen 3D foundation model and a spatiotemporal geometry VAE transform multi-camera video windows into structured geometry latent sequences, which a geometry expert learns to predict through flow matching. Our central mechanism, geometry gradient transfer, uses asymmetric attention: the geometry expert reads video and action representations, while neither expert reads geometry. The geometry objective thus backpropagates into the deployed experts, encouraging their representations to support geometric prediction without making action generation depend on geometry inputs. The geometry branch is omitted at deployment, supporting both joint video–action denoising and action generation without future-video rollout. On LIBERO, no-rollout GeoWAM achieves 98.3% success, improving over a matched baseline by 2.3 percentage points, and attains 65.2% on LIBERO-Plus. Our joint variant reaches 76.49% category-averaged success on LIBERO-Plus. These results support modeling geometric evolution as a training-time source of policy improvement without test-time geometry inference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Predictive Geometry Modeling, Geometry Representation, Flow Matching, Geometry Gradient Transfer, Visual Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JXc6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoWAM adds training-time geometry supervision to a video-action world model without requiring geometry inputs at deployment. A frozen VGGT-Ω teacher processes multi-camera video windows, and a separately trained spatiotemporal GeoVAE compresses its features into latent sequences. An auxiliary geometry expert predicts these sequences through flow matching. Asymmetric attention allows geometry queries to read video/action keys and values, carrying the geometry loss back into the deployed experts while preventing those experts from depending on geometry tokens. The primary model generates actions from the current observation without future-video rollout; GeoWAM-Joint jointly denoises future video and actions. The reported LIBERO success rates are 98.30% and 98.65%, respectively, and LIBERO-Plus category averages are 65.20% and 76.49%. On LIBERO, the matched video-action baseline reaches 96.00%. Detachment, static-target, and decayed-anchor ablations examine the supervision pathway.

Critical strengths
  1. The separation between training-time supervision and inference-time conditioning is clear and useful. Under the attention mask in Figure 3, video and action states never read geometry tokens, so removing the geometry branch leaves their forward computation unchanged. This is the strongest technical aspect of the paper: a relatively expensive geometry teacher can shape the policy without becoming an inference-time dependency, in either the no-rollout or joint-generation setting.

  2. Table 3 includes a matched baseline and a targeted gradient intervention. GeoWAM improves average LIBERO success from 96.00% to 98.30%, and Long-suite success from 90.60% to 96.40%. Keeping the geometry branch and its forward connections but detaching the video/action keys and values it reads gives 95.90% average success. This provides direct evidence for the value of attention-mediated supervision transfer. The appendix also usefully states the limits of the controls: the static target retains full-window context, the anchor variant changes attention as well as noise assignment, and cross-paper results are not controlled interventions.

Critical weaknesses
  1. The robustness claim needs matched comparisons and uncertainty across training runs. Table 2 uses published baselines, whereas Appendix C.3 explains that the locally matched FastWAM differs from the original Fast-WAM in multi-camera encoding. LIBERO-Plus results are not reported for the matched baseline, Detach, or Static-Geometry controls. The 65.20% versus 51.36% comparison therefore cannot establish how much of the robustness gain comes from geometry gradient transfer. The margins over DreamWAM are 1.76 percentage points without rollout and 1.02 points for Joint, and several LIBERO margins over strong baselines are also small. No means and variation across independent training seeds are provided. The appendix acknowledges this limitation, but it remains consequential: many evaluation rollouts do not replace independent training repetitions.

The evaluation is also limited to LIBERO and LIBERO-Plus, with no real-robot experiments. Although the paper acknowledges this limitation, simulated perturbations do not establish that the gains persist under real camera artifacts, geometry-teacher errors, contact dynamics, and hardware execution uncertainty. This leaves the practical effectiveness of geometry supervision in physical manipulation unverified and limits the scope of the generalization evidence.

  1. The experiments do not sufficiently distinguish the benefit of predicting geometric evolution from that of auxiliary teacher-feature prediction more generally. Detach establishes that the gradient path helps, but not that geometry trajectories are the essential target. The static control uses the first latent from a full-window teacher and encoder; it removes variation across target frames but does not isolate geometry available from the current observation alone. Figure 5 provides one qualitative example, without a matched comparison to generic visual-feature prediction or a simpler geometry-regression head. The cost of teacher-feature extraction, GeoVAE pretraining, and the additional 882 geometry tokens is also not quantified. Since related work already uses training-time geometric supervision and removable branches, these missing comparisons make the specific contribution and its cost-benefit tradeoff difficult to assess. Together with the first concern, this leads to a weak-reject recommendation: the design is reasonable, but its distinctive benefit and robustness need stronger evidence.
Questions for authors
  1. Experimental clarification and decisive additional evidence: How many independent training seeds and evaluation seeds per checkpoint underlie Tables 1–3? Can you provide per-seed results, means, and standard deviations for Matched FastWAM, GeoWAM, Detach, and Static-Geometry on both LIBERO and LIBERO-Plus under identical training/evaluation settings? In particular, does the robustness improvement persist after matching camera encoding? These results could materially change the recommendation.

  2. Mechanism: Have you compared against generic visual-teacher feature prediction using the same auxiliary expert and a similar token/training budget, or a simpler head that directly regresses the same geometry latents? Do these alternatives retain most of the gain? If such comparisons are unavailable, what existing evidence separates the contribution of geometric evolution prediction from auxiliary representation supervision more generally? Full-window conditioning is not being treated here as evaluation-data leakage; the concern is what the static-target control can isolate.

  3. Cost and reproducibility: What are the GPU-hours and peak memory for teacher-feature extraction, GeoVAE pretraining, and policy training, relative to the matched baseline? Please also specify the GeoVAE training-data scope, training steps/epochs, batch size, and learning rate, as well as the effective policy-training batch size. What are latency and memory for each deployment mode after actually removing the geometry modules? These details affect both reproducibility and the practical value of the training-time supervision.

5 comments

Thank you for recognizing the distinction between training-time supervision and inference-time conditioning, and for identifying the comparisons needed to assess robustness. We now report matched LIBERO-Plus results, a direct future-depth control, real world physical-robot experiments, and measured policy-training costs.

Q1. Does the robustness gain survive matched camera encoding? How many seeds were used?

Yes. With identical per-camera video encoding within each comparison, GeoWAM improves LIBERO-Plus success by 9.08 percentage points without future-video rollout and 9.96 points with joint rollout. For our own entries in Tables 1–3, we used one training seed per configuration and one evaluation seed per checkpoint(followed by previous work). Given the time and compute available, we prioritized evaluating the matched checkpoints on LIBERO-Plus, where there is more room for improvement than on LIBERO. We report the individual-run scores below. We would welcome the opportunity to add independent training runs in a future revision.

PolicyLIBERO avg.Cam.RobotLang.LightBg.NoiseLayoutPlus avg.
Matched Fast-WAM96.0027.7743.8169.8183.2753.2549.8465.1156.12
GeoWAM-Detach95.9021.7044.0666.3681.6156.5144.5366.8254.51
Static-Geometry Transfer97.2016.6447.4870.7989.5854.1841.6666.3655.24
GeoWAM98.3025.8956.3287.3895.3662.9253.8474.6965.20
Matched Fast-WAM-Joint—35.4667.8791.3589.8448.1454.0379.0266.53
GeoWAM-Joint98.6558.1674.0093.3696.2363.8568.0281.8476.49

All values are success rates (%). Each LIBERO-Plus row covers the same 10,030 cases. GeoWAM improves on its matched baseline in six of seven categories (camera viewpoint is the exception). GeoWAM-Joint improves in all seven. Preserving the geometry forward pass but detaching its gradients reduces the no-rollout average from 65.20% to 54.51%.

Physical manipulation. We collected 100 demonstrations per task and trained Fast-WAM-Joint and GeoWAM-Joint on the same data. Both policies use joint rollout without a geometry teacher at deployment. The reference test condition already changes the camera viewpoint; the other conditions add a lighting or layout change to that reference. We ran 30 trials per task, condition, and policy (360 physical executions).

TaskTest conditionFast-WAM-JointGeoWAM-Joint
Red block onto black boxCamera change23/30 (76.7%)25/30 (83.3%)
Red block onto black boxCamera + lighting change6/30 (20.0%)9/30 (30.0%)
Red block onto black boxCamera + layout change4/30 (13.3%)16/30 (53.3%)
Red radish onto plateCamera change0/30 (0.0%)14/30 (46.7%)
Red radish onto plateCamera + lighting change4/30 (13.3%)8/30 (26.7%)
Red radish onto plateCamera + layout change1/30 (3.3%)5/30 (16.7%)
OverallAll six settings38/180 (21.1%)77/180 (42.8%)

GeoWAM-Joint completed 39 more trials and had more successes in every setting. On block placement, a two-success difference under the camera change alone became a twelve-success difference when the layout also changed. The gains therefore extend beyond simulated perturbations to these two physical tasks, although changed lighting remains difficult for both policies.

Due to character limitations, subsequent replies will be in the next section

Q2. What distinguishes geometric evolution from generic auxiliary supervision?

Generic visual features and a simpler head. We explored DINOv3 as a generic visual teacher in preliminary experiments. The observed policy gains were limited, so we did not pursue that variant. We do not present this pilot as a token- and training-budget-matched comparison. We also have not tested a lightweight head that directly regresses the same GeoVAE latents. For context, DreamWAM (Table 2 of our submission) uses RAFT flow, targets derived from Depth Anything V3 depth maps, and DINOv2 semantic features. GeoWAM instead treats one additional signal, the VGGT-Ω feature trajectory, as a state to denoise. Its reported LIBERO-Plus averages are higher both without rollout (65.20% vs. 63.44%) and with rollout (76.49% vs. 75.47%). This shows what is achievable with one auxiliary target.

A controlled change of target. In DepthMap-GeoWAM we kept the frozen VGGT-Ω teacher and the auxiliary denoising pathway, but predicted teacher depth maps directly, without a geometry VAE. Splitting each 224 × 224 depth map into 32 × 32 patches yields the same 882 auxiliary tokens as GeoWAM (2 cameras × 9 frames × 49 patches). This matches token count, not necessarily wall-clock cost.

No-rollout policy / auxiliary targetCam.RobotLang.LightBg.NoiseLayoutPlus avg.
Matched Fast-WAM / none27.7743.8169.8183.2753.2549.8465.1156.12
DepthMap-GeoWAM / depth maps24.0247.8178.1490.0251.6747.7265.4457.83
GeoWAM / implicit feature trajectories25.8956.3287.3895.3662.9253.8474.6965.20

Depth prediction itself helps (+1.71 points over the matched baseline). Predicting the teacher's feature trajectory rather than its depth output yields a further 7.37 points, with higher success in every category. Both variants denoise an evolving target: the distinction tested here is which representation of that evolution supervises the policy.

What Static isolates. Our motivating question was whether the policy benefits more from recovering a scene representation or from predicting how it changes given the observation, instruction, and evolving action state. Static repeats the first geometry latent at every target time; GeoWAM predicts the sequence of distinct latents. Both keep the gradient path and use the same full-window teacher and encoder. Static therefore tests the value of temporal variation in the target, not whether current RGB alone suffices to recover geometry. The evolving target raises success from 97.20% to 98.30% on LIBERO and from 55.24% to 65.20% on LIBERO-Plus. We will make this distinction explicit in the revision.

Due to character limitations, subsequent replies will be in the next section

Q3. Training cost, GeoVAE training data, and deployment.

Policy training. We benchmarked on four NVIDIA RTX 6000D GPUs at an effective batch of 128 windows per optimizer step (4 GPUs × 4 windows × 8 accumulation steps). We measured 30 steps after 20 warm-up steps. Peak memory is the maximum allocated on any rank. The last two columns extrapolate to 21,700 optimizer steps (10 epochs).

Policys/stepSamples/sPeak GiB/GPUStep overheadWall-hours (est.)GPU-hours (est.)
Matched Fast-WAM8.6114.8745.49Reference51.9207.6
GeoWAM13.249.6757.98+53.8%79.8319.2
Matched Fast-WAM-Joint8.3315.3745.49Reference50.2200.8
GeoWAM-Joint13.309.6257.98+59.8%80.2320.8

VGGT-Ω runs online to avoid storing dense teacher features for every overlapping video window. The GeoWAM rows include teacher extraction, frozen GeoVAE encoding, and the additional 882-token expert; there is no separate offline extraction run to add to these totals. The reusable LIBERO GeoVAE took approximately one additional day.

GeoVAE recipe. We trained the GeoVAE on demonstrations from the 4 LIBERO training suites only, with no large-scale GeoVAE pretraining and no LIBERO-Plus data. Each window contains nine frames per camera. The policy uses the GeoVAE checkpoint at 21,700 optimizer updates; its saved optimizer state records an initial learning rate of 5 × 10⁻⁵ and a final rate of 1 × 10⁻⁵. For reproducibility, the launcher defaults to 10 epochs, cosine decay, and an effective batch of 256 (4 GPUs × 16 windows × 4 accumulation steps). For the physical experiments, we initialized from the LIBERO GeoVAE and fine-tuned it on the two tasks' demonstration videos with a fresh optimizer and schedule, retaining the pretrained feature normalization.

Deployment after removing geometry. We measured all four LIBERO policies sequentially on the same NVIDIA H20 at batch size one in BF16. Neither GeoWAM policy loaded VGGT-Ω, the GeoVAE, or the geometry expert onto the GPU. Each replan used the same preprocessed two-camera observation and instruction and generated 32 actions in 10 denoising steps. Latency includes text encoding, image encoding, denoising, and return of the action chunk to CPU; it excludes model loading and simulation. Joint inference denoises future-video latents without decoding RGB frames.

PolicyMean (ms)Median (ms)P95 (ms)Resident allocated (GiB)Peak allocated (GiB)Peak reserved (GiB)
Matched Fast-WAM385.88385.25389.0323.156323.232323.4805
GeoWAM391.99392.41393.7323.156323.232323.4805
Matched Fast-WAM-Joint650.32649.94653.8923.156323.232423.4414
GeoWAM-Joint643.66643.86648.4423.156323.232423.4414
Reviewer_azq4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoWAM augments a video-action world action model with a training-only geometry expert so that geometric supervision can shape a policy without geometry being required at inference. The backbone is the Fast-WAM mixture-of-transformers (Wan2.2-TI2V-5B video expert, action expert); the addition is a third expert that flow-matches sequences of implicit geometry latents rather than depth or point maps. Targets are produced by a frozen VGGT-Omega teacher applied to the multi-camera window, whose four dense feature levels are compressed roughly 85x by a separately pretrained spatiotemporal "GeoVAE" (Eqs. 4-5), so no depth, point-map, or camera supervision is used.

The central mechanism, geometry gradient transfer, is an asymmetric attention mask (Fig. 3): the geometry expert queries video and action keys/values, while video and action never query geometry. The geometry loss therefore backpropagates into the deployed experts through the shared keys and values (Eq. 7), yet no deployed query ever depends on a geometry token, so by induction over layers the geometry branch can be deleted at inference without changing the deployed forward computation. The paper also defines a gradient-rescaling operator R_alpha (Eq. 8) whose alpha = 0 setting blocks this pathway while leaving forward values and attention edges intact. Two variants are evaluated: GeoWAM, which acts from current-observation video representations with no future-video rollout, and GeoWAM-Joint, which restores action access to future-video tokens at deployment.

Validation is confined to LIBERO and LIBERO-Plus. On LIBERO, GeoWAM reaches 98.30 average vs. the authors’ locally trained Matched FastWAM at 96.00 (+2.30) and published Fast-WAM at 97.60; GeoWAM-Joint reaches 98.65. On LIBERO-Plus (10,030 variants, no adaptation), GeoWAM reaches 65.20 macro-average vs. reported Fast-WAM 51.36, and GeoWAM-Joint 76.49 vs. Fast-WAM-Joint 69.16. The mechanism is probed by three LIBERO ablations: GeoWAM-Detach (alpha = 0) 95.90, Static-Geometry Transfer 97.20, Decayed-Anchor 97.70, against GeoWAM 98.30. Appendix E adds a qualitative depth-decoding diagnostic, and the supplementary slide deck contains paired rollout and depth videos.

Critical strengths
  1. The detach ablation is a precise causal test of the claimed mechanism, and very few papers in this area attempt one. GeoWAM-Detach retains the geometry expert, its loss, its tokens, and every forward attention edge, and removes only the gradient flowing through the video/action keys and values (Eq. 8, alpha = 0). It lands at 95.90, essentially at Matched FastWAM’s 96.00 and 2.4 points below GeoWAM. This separates the claimed pathway — cross-expert gradients — from the confounds that would otherwise explain the gain: added parameters, extra tokens, longer effective training, or generic multi-task regularization. Static-Geometry Transfer then separates temporally varying targets from geometric structure per se. This is the right experimental design for a mechanism claim and is what makes the contribution legible rather than merely plausible.

  2. Geometry-free deployment is a structural guarantee rather than an empirical approximation. Because no video or action query attends to geometry at any layer, removing the geometry tokens and attention rows provably leaves the deployed computation unchanged (Section 3.4). This is a stronger and cleaner property than decayed read-masks (MECo-WAM) or retained geometry prediction heads, where the deployed model is only approximately geometry-independent. The idea is backbone-agnostic and transfers to any mixture-of-experts policy that wants auxiliary supervision without inference cost, which gives it value beyond this particular system.

Critical weaknesses
  1. The mechanism claims rest on single-run margins of 0.6-2.4 points, and the paper itself states these should not be read as established. Section 4.1 reports evaluation "once per seed" without ever naming a seed count; the Conclusion concedes that "consistency across training seeds remain to be established"; and Appendix C.3 states that "small differences in reported success rates should not be read as statistically established improvements without repeated-run uncertainty estimates." That disclaimer is admirable, but it applies directly to Table 3, which is the paper’s principal contribution: Detach −2.40, Static −1.10, Decayed-Anchor −0.60. The Decayed-Anchor conclusion ("a clean-anchor curriculum offers no overall advantage") is drawn from 0.60 points and is not distinguishable from run-to-run variation at all. LIBERO is also near-saturated here (controls at 96-98.5%), which compresses exactly the range the argument lives in, and Table 3 is never repeated on LIBERO-Plus, where there is genuine headroom. Three training seeds with intervals, plus an ablation table on LIBERO-Plus, would convert the paper’s best idea from suggestive to demonstrated — this is a tractable revision, not a fundamental flaw, but as submitted the central evidence does not carry the central claim.

  2. The matched control exists only where the margins are small, and the large margins exist only where the control is cross-paper. The abstract’s "+2.3 over a matched baseline" is measured against Matched FastWAM at 96.00, while published Fast-WAM in Table 1 stands at 97.60 — so against the published number the margin is 0.70, and GeoWAM sits below LingBot-VA and Fast-WAM-Joint (both 98.50). The 1.6-point shortfall of the authors’ own control relative to the published figure is never explained, yet more than half of the headline improvement lives inside that gap. Conversely, the results that would be genuinely persuasive — +13.84 on LIBERO-Plus for GeoWAM and +7.33 for GeoWAM-Joint, over 10,030 variants where binomial noise is negligible — are computed against reported Fast-WAM numbers, and no Matched FastWAM row appears for LIBERO-Plus at all. The evidence is thus structurally mismatched: the controlled comparisons are too small to interpret, and the interpretable comparisons are uncontrolled. Adding Matched FastWAM on LIBERO-Plus would address this with one training run the authors already have the pipeline for.

Questions for authors
  1. Seeds. How many training seeds and evaluation seeds underlie Tables 1, 2, and 3? Can you report per-seed values or intervals for Table 3? Given Appendix C.3, which of the three ablation conclusions do you consider established?
  2. Matched control gap. What accounts for the 1.6-point difference between Matched FastWAM (96.00) and published Fast-WAM (97.60) beyond per-camera video-VAE encoding? Does adding geometry supervision to the original Fast-WAM packing also exceed 97.60?
  3. Matched control on LIBERO-Plus. What does Matched FastWAM score on LIBERO-Plus? This single number would determine how much of the 13.84-point gain is attributable to geometry supervision rather than to encoding and training differences.
  4. Ablations on LIBERO-Plus. What do Detach, Static-Geometry, and Decayed-Anchor give on LIBERO-Plus, where the dynamic range is far larger than on LIBERO?
  5. Gradient strength. Is the effect monotone in alpha? Sweeping alpha over {0, 0.25, 0.5, 1} would test the gradient-transfer claim far more sharply than the binary detach, at low additional cost.
  6. Training overhead. What are the wall-clock time, peak memory, and parameter counts for GeoWAM vs. Matched FastWAM, and what fraction is frozen-teacher feature extraction? Can teacher features be cached across epochs?
  7. Primary vs. Joint. GeoWAM-Joint exceeds GeoWAM by 11.29 points on LIBERO-Plus, with camera-viewpoint success rising from 25.89 to 58.16. Since Joint restores the future-video rollout that the no-rollout line of work argues is unnecessary, does geometry supervision in fact pay off most when visual rollout is retained? How should a practitioner choose between the two?
  8. Quantitative geometry diagnostics. Appendix E is one qualitative example. Can you report a scalar measure of geometry prediction quality (teacher-feature cosine or decoded depth error vs. horizon) and correlate it with task success across the Table 3 variants? If better geometry prediction does not track better control, the mechanism story needs revisiting.
  9. Loss weighting. Equation 10 fixes lambda_v = lambda_a = lambda_g = 1 and lambda_cos = 0.2. How sensitive are the results to lambda_g, and were these values tuned on the evaluation suites?
6 comments

Thank you for recognizing the role of the detach control and for identifying where our evidence was incomplete. We agree that the most informative test is to put the matched baseline, the mechanism ablations, and the full model on the same benchmark with room for improvement. We have now done this on LIBERO-Plus for the primary no-rollout policy. GeoWAM scores 65.20% versus 56.12% for matched Fast-WAM; with geometry gradients detached, the score is 54.51%. Below we distinguish what these comparisons support from what remains uncertain across training runs.

Q1. Seeds and the strength of the Table 3 conclusions.

For our own runs in Tables 1–3, we used one training seed per configuration and one evaluation seed per checkpoint. Given our compute budget, we prioritized testing the controls on LIBERO-Plus rather than repeating training on near-saturated LIBERO.

Of the three conclusions, the gradient pathway and the use of temporally varying targets are now supported on both benchmarks: removing either reduces the observed score, with substantially wider gaps on LIBERO-Plus (Q3–Q4). The clean-anchor conclusion is weaker.

Q2. Why does matched Fast-WAM differ from the published baseline? Does GeoWAM work with the original packing?

We evaluated the official Fast-WAM checkpoint on LIBERO with our evaluator and obtained 96.75%, versus the published 97.60% and our locally trained matched baseline's 96.00%. Thus, 0.85 points of the 1.60-point discrepancy are already present when evaluating the released weights, before comparing separately trained models. We cannot attribute the remaining 0.75 points to camera encoding alone. More directly, yes: when we trained GeoWAM with Fast-WAM's original packing (concatenate the two RGB views before the video VAE), it reached 98.10% on LIBERO. The improvement over 97.60% therefore does not require per-camera encoding.

Interestingly, the baseline ordering reverses on LIBERO-Plus: our per-camera baseline scores 56.12%, above the published Fast-WAM result of 51.36%. One hypothesis is that encoding the wrist and external views separately helps preserve view-specific information when their visual dynamics differ.

Q3–Q4. Matched Fast-WAM and the mechanism ablations on LIBERO-Plus.

All variants were evaluated on the same 10,030 cases without future-video rollout.

MethodCam.RobotLang.LightBg.NoiseLayoutAvg.
Matched Fast-WAM27.7743.8169.8183.2753.2549.8465.1156.12
GeoWAM-Detach21.7044.0666.3681.6156.5144.5366.8254.51
Static-Geometry Transfer16.6447.4870.7989.5854.1841.6666.3655.24
Decayed-Anchor GeoWAM22.8949.1073.8587.6557.3442.5470.3657.67
GeoWAM25.8956.3287.3895.3662.9253.8474.6965.20

Replacing the published baseline with the matched one reduces the apparent gain from 13.84 to 9.08 percentage points. This remaining gain is measured with the same per-camera encoding on both sides. Detaching the gradient while preserving the forward computation lowers the score by 10.69 points relative to GeoWAM. Transferring gradients from a repeated initial-frame target rather than an evolving trajectory lowers it by 9.96 points.

Decayed-Anchor GeoWAM reaches 57.67%, 1.55 points above matched Fast-WAM but 7.53 points below standard GeoWAM. Standard GeoWAM outperforms the anchor variant in all seven perturbation categories.

Q5. Does performance vary with gradient-transfer strength?

We added a run at α=0.5\alpha=0.5, between detach (α=0\alpha=0) and full transfer (α=1\alpha=1). This operator scales the gradient from the geometry expert into the video and action experts without changing forward attention.

Transfer scaleCam.RobotLang.LightBg.NoiseLayoutAvg.
0 (Detach)21.7044.0666.3681.6156.5144.5366.8254.51
0.523.5849.8776.4588.7959.2949.4771.0859.79
1 (GeoWAM)25.8956.3287.3895.3662.9253.8474.6965.20

At half strength, success lies between detach and full transfer in all seven categories. The macro-average rises by 5.28 points from 0 to 0.5 and by a further 5.41 from 0.5 to 1. The observed trend is thus monotone and approximately linear at the three tested settings. it does not establish a general scaling law or show that 1 is optimal. Within the rebuttal budget we did not test the suggested 0.25 setting or scales above 1. We will do it in the future.

Due to character limitations, subsequent replies will be in the next section

Q6. Training cost, parameter counts, and caching.

We benchmarked all 4 policies on the same 4 NVIDIA RTX 6000D GPUs with a global batch of 128 (4 samples/GPU × 4 GPUs × 8 accumulation steps). We timed 30 optimizer steps after 20 warm-up steps. Memory is peak allocated GPU memory on the highest-usage rank.

Policys/optimizer stepSamples/sPeak GiB/GPUStep overhead10-epoch policy time (est.)
Matched Fast-WAM8.6114.8745.49Reference51.9 h
GeoWAM13.249.6757.98+53.8%79.8 h
Matched Fast-WAM-Joint8.3315.3745.49Reference50.2 h
GeoWAM-Joint13.309.6257.98+59.8%80.2 h

Counting policy experts only, the video and action experts total approximately 6.02B parameters (5.00B + 1.02B); the training-only geometry expert adds approximately 0.45B, for 6.47B. We keep the frozen teacher loaded and run it online on each multi-camera window, then compress its features with the frozen GeoVAE. The measured overhead includes this extraction and encoding as well as training the extra expert. We have not separately profiled the teacher's fraction and do not attribute the entire overhead to it. GeoVAE pretraining takes approximately one additional day and is reusable. With fixed windows and preprocessing, targets could be cached across epochs, preferably after GeoVAE compression. We chose online extraction to avoid storing a target for every overlapping training window.

Q7. Does geometry help only, or most, with future-video rollout?

Future-video rollout is not required for geometry supervision to improve control. On LIBERO-Plus cases, GeoWAM improves macro-average success over matched Fast-WAM from 56.12% to 65.20% without rollout (+9.08 percentage points). With rollout, GeoWAM-Joint improves over matched Fast-WAM-Joint from 66.53% to 76.49% (+9.96 points). Geometry supervision thus adds more than nine points in both deployment modes. For deployment, the primary policy avoids future-video denoising while retaining a substantial gain over its matched baseline. Joint offers the higher observed success rate when rollout latency is acceptable.

Q8. Does better future-geometry prediction track better control?

Yes, but not one-to-one. We fixed 140 LIBERO-Plus cases before evaluation (20 per perturbation category). From the first decision, we executed the action chunk sampled alongside the geometry forecast and compared the prediction with frozen VGGT-Ω outputs on the resulting observations. We measured control success in a separate rollout from the same initial state using standard 10-step replanning. Success counts use all 140 cases; geometry scores use the 139 common cases with a complete forecast window and matching diagnostic/deployment first images within every policy. Cosine compares decoded features with teacher features; AbsRel compares decoded depth with teacher depth, not simulator ground truth. Both average the two cameras equally.

PolicyControl successes / 140Feature cosine ↑ (+8)Depth AbsRel ↓ (+8)Depth AbsRel ↓ (+32)r(feature, success) (+8)r(depth error, success) (+8)
GeoWAM-Detach760.6510.2110.257+0.322−0.240
Static-Geometry Transfer640.6630.2940.462+0.173−0.050
Decayed-Anchor GeoWAM880.7010.1580.202+0.418−0.192
GeoWAM910.7030.1630.192+0.346−0.223
GeoWAM-Joint (extension)1010.7120.1600.180+0.278−0.184

Across the four Table 3 variants, higher feature cosine and lower depth error at +8 each broadly track higher success (descriptive Spearman ρ = 0.80 for cosine and −0.80 for AbsRel). The most informative comparison is GeoWAM versus Detach: restoring gradient transfer improves feature cosine by 0.052, reduces depth error by 0.048, and yields 15 more successful trials. The evolving target matters particularly at longer horizons: from +8 to +32, Static's depth error rises from 0.294 to 0.462, versus 0.163 to 0.192 for GeoWAM. Offset +8 falls within the deployment replan interval; +32 is an open-loop forecast diagnostic.

Within each policy, the last two columns give point-biserial correlations across cases. For GeoWAM, successful cases have higher feature cosine than failures (0.718 vs. 0.675) and lower depth error (0.148 vs. 0.192). Two qualifications matter. First, simply repeating current-observation features is a strong short-horizon baseline: at +8 it scores 0.721 cosine versus 0.703 for GeoWAM. Second, Decayed-Anchor has slightly lower +8 depth error than GeoWAM without higher control success. Thus geometry quality is informative, but no single reconstruction metric determines policy success. These associations are descriptive; the evidence for the gradient pathway comes from the detach intervention.

Due to character limitations, subsequent replies will be in the next section

Q9. Geometry-loss weighting.

We fixed λv=λa=λg=1\lambda_v=\lambda_a=\lambda_g=1 and λcos⁡=0.2\lambda_{\cos}=0.2 for the reported GeoWAM setting. We did not conduct a systematic search over these weights or an evaluation-suite sweep to optimize λg\lambda_g. We therefore cannot claim insensitivity to this choice. The α\alpha experiment above answers a different question: it changes only the gradient reaching video and action, whereas changing λg\lambda_g also changes the velocity-regression gradient that trains the geometry expert itself.

Additional evidence: does the advantage extend to a physical robot? To test whether the gains extend beyond simulated perturbations, we trained Fast-WAM-Joint and GeoWAM-Joint on the same 100 real-robot demonstrations per task. Both policies use future-video rollout; neither uses a geometry teacher at deployment. The reference condition already includes a camera perturbation. The other two conditions add either a lighting change or a scene-layout change to that reference. We ran 30 trials per policy in each of the six task–condition pairs (360 physical executions in total).

TaskTest conditionFast-WAM-JointGeoWAM-Joint
Red block onto black boxCamera change23/30 (76.7%)25/30 (83.3%)
Red block onto black boxCamera + lighting change6/30 (20.0%)9/30 (30.0%)
Red block onto black boxCamera + layout change4/30 (13.3%)16/30 (53.3%)
Red radish onto plateCamera change0/30 (0.0%)14/30 (46.7%)
Red radish onto plateCamera + lighting change4/30 (13.3%)8/30 (26.7%)
Red radish onto plateCamera + layout change1/30 (3.3%)5/30 (16.7%)
OverallAll six settings38/180 (21.1%)77/180 (42.8%)

GeoWAM-Joint completed 39 more trials (+21.7 percentage points) and had more successes in every setting. The block task illustrates why the test condition matters: with the camera change alone the policies differed by only two successes (25 vs. 23), but after the layout also changed, GeoWAM-Joint succeeded in 16 trials versus 4. These trials provide evidence that the Joint policy's advantage extends to physical manipulation on these two tasks. They complement the gradient intervention in Q3–Q4.

Reviewer_neTJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoWAM adds a training-only geometry expert to Fast-WAM's video-action backbone: frozen VGGT-Ω features, compressed by a geometry VAE, predicted with flow matching. Asymmetric attention lets the geometry loss shape the deployed experts while the forward pass stays geometry-free, so the branch is deleted at test time. LIBERO 98.3% vs 96.0% for a matched control; LIBERO-Plus 65.20%/76.49% against weaker published numbers.

Critical strengths
  1. The mechanism is simple, and the authors ablate the mechanism itself. With α=0\alpha = 0 in Eq. 8 the LIBERO score is 95.9% versus 98.3% (matched baseline 96.0%), so an extra loss by itself does nothing; the gradients have to reach the video and action branches. And the whole branch disappears at inference.
  2. The experimental honesty is better than most submissions. Matched vs quoted baselines are distinguished, the Joint variant's confounds are stated out loud, and the lack of repeated runs is acknowledged rather than hidden.
Critical weaknesses
  1. The LIBERO-Plus conclusion, which is the paper's main robustness claim, compares against published numbers from models with different video encodings and training recipes (65.20% vs 51.36%; 76.49% vs 69.16%/75.47%). The only matched FastWAM was run on LIBERO, and the per-camera encoding change alone could account for part of the gap. Running matched FastWAM on LIBERO-Plus is the one experiment I would insist on.
  2. LIBERO is nearly saturated (many methods above 98%); without repeated runs, differences between a few tenths of a point and 2.3 points say very little. There is no real-robot evidence either.
  3. The pipeline is heavy: multi-view VGGT-Ω, a separately trained VAE, fixed-statistics calibration. It is never compared against a plain future-depth loss in the style of WAM4D or QDepth-VLA, and zero inference cost does not mean zero training cost.
Questions for authors
  1. Can you run matched FastWAM on LIBERO-Plus under your exact protocol?
  2. What does a simple future-depth auxiliary loss give (e.g., Depth Anything targets, same token budget, no geometry VAE)?
  3. Is the teacher feature extraction offline? How much do total training time and per-step cost grow?
4 comments

We appreciate your insightful suggestions and valuable feedback. We agree that the most important missing comparison was matched Fast-WAM on LIBERO-Plus. We have now completed that comparison for the primary, no-rollout model: GeoWAM reaches 65.20% versus 56.12% for matched Fast-WAM (+9.08 percentage points). Below we report the new controls, real world physical-robot experiments, and training-cost measurements.

1. Matched baselines and the future-depth control on LIBERO-Plus (Q1 and Q2).

We evaluated matched Fast-WAM, matched Fast-WAM Joint, GeoWAM-Detach, Static-Geometry Transfer, and DepthMap-GeoWAM using the same protocol as GeoWAM.

MethodCam.RobotLang.LightBg.NoiseLayoutAvg.
Matched Fast-WAM27.7743.8169.8183.2753.2549.8465.1156.12
GeoWAM-Detach21.7044.0666.3681.6156.5144.5366.8254.51
Static-Geometry Transfer16.6447.4870.7989.5854.1841.6666.3655.24
DepthMap-GeoWAM24.0247.8178.1490.0251.6747.7265.4457.83
GeoWAM25.8956.3287.3895.3662.9253.8474.6965.20
Matched Fast-WAM-Joint35.4667.8791.3589.8448.1454.0379.0266.53
GeoWAM-Joint58.1674.0093.3696.2363.8568.0281.8476.49

What does matching change? As the reviewer anticipated, the choice of baseline matters. On LIBERO-Plus, matched Fast-WAM scores higher than published Fast-WAM (56.12% vs. 51.36%), whereas on standard LIBERO it does not (96.0% vs. 97.6%). This is consistent with the encoding or training recipe mattering more under distribution shift, but the cross-paper comparison does not isolate camera encoding. The controlled result is more informative: with identical per-camera encoding and no future-video rollout in either policy, GeoWAM improves six of seven categories. The largest gains are in language instructions (+17.57 points) and robot initial states (+12.51).

The matched comparison also holds with joint rollout: matched Fast-WAM-Joint achieved 66.53% macro-average success, compared with 76.49% for GeoWAM-Joint (+9.96 percentage points). GeoWAM-Joint improves on this baseline in all seven categories, with the largest gain on camera viewpoints (35.46% to 58.16%, +22.70 points).

What accounts for the remaining gain? Blocking geometry gradients from reaching the deployed experts (Detach), or keeping the gradient path but replacing the evolving target with repeated initial-frame geometry (Static), brings performance back to or below the matched baseline. For the reviewer's proposed simpler alternative, we kept the teacher fixed and supervised DepthMap-GeoWAM with VGGT-Ω depth maps directly, without a geometry VAE. Patching each 224×224 depth map into 32×32 patches gives the same auxiliary token budget as GeoWAM: 2 cameras × 9 frames × 49 tokens = 882. Direct depth supervision improves over matched Fast-WAM by 1.71 points, but falls 7.37 points short of GeoWAM. Together, these controls support predicting the evolution of 3D foundation-model features and transferring that supervision to the policy, rather than merely adding an auxiliary prediction loss.

Table 2 of our submission provides a complementary comparison with DreamWAM. DreamWAM combines RAFT optical flow, DINOv2 semantic features, and a geometry target constructed from depth maps predicted by a frozen Depth Anything V3 model. GeoWAM instead adds a single auxiliary target to video–action learning: a trajectory of implicit geometry features from VGGT-Ω. On LIBERO-Plus, GeoWAM achieves higher reported macro-average success in both the no-rollout setting (65.20% vs. 63.44%, +1.76 points) and the joint setting (76.49% vs. 75.47%, +1.02 points). These cross-paper results show what GeoWAM achieves with one auxiliary target; it is the matched DepthMap-GeoWAM ablation above that directly tests the choice of geometry target.

Due to character limitations, subsequent replies will be in the next section

2. Beyond saturated LIBERO: real world physical-robot experiments.

We agree that a small gap on near-saturated LIBERO is not decisive. In addition to the LIBERO-Plus controls above, we collected 100 demonstration trajectories for each of two physical manipulation tasks and trained Fast-WAM-Joint and GeoWAM-Joint on the same data. After training, in order to demonstrate our model's understanding of scene geometry, we chose to add camera perturbations as baseline testing conditions during testing, and separately added lighting perturbations and scene layout perturbations as additional real machine experiments. We ran 30 trials per task, condition, and policy .

TaskTest conditionFast-WAM-JointGeoWAM-Joint
Place the red block on the black boxbaseline23/30 (76.7%)25/30 (83.3%)
baseline + lighting change6/30 (20.0%)9/30 (30.0%)
baseline + layout change4/30 (13.3%)16/30 (53.3%)
Place the red radish on the platebaseline0/30 (0.0%)14/30 (46.7%)
baseline + lighting change4/30 (13.3%)8/30 (26.7%)
baseline + layout change1/30 (3.3%)5/30 (16.7%)
OverallAll six settings38/180 (21.1%)77/180 (42.8%)

GeoWAM-Joint completed 39 more trials and had a higher success rate in all six settings. For block placement, the policies were close under the camera change alone (25 vs. 23 successes), but the gap widened when the layout also changed (16 vs. 4). For radish placement, GeoWAM-Joint succeeded 14 times under the reference camera change, where Fast-WAM-Joint never succeeded. These results provide physical evidence for a benefit under visual and spatial shifts, not just on a saturated simulation benchmark.

3. Online teacher extraction and training cost (Q3).

We extract features online, not offline: storing dense multi-view features for every training window would require substantial disk space. We benchmarked all 4 policies on the same 4 NVIDIA RTX 6000D GPUs with global bs 128(4 GPU * 4 PER_GPU_BATCH_SIZE * 8 GRAD_ACCUM_STEPS).

Policys/optimizer stepSamples/sPeak GiB/GPUStep overhead10-epoch policy time (est.)
Matched Fast-WAM8.6114.8745.49Reference51.9 h
GeoWAM13.249.6757.98+53.8%79.8 h
Matched Fast-WAM-Joint8.3315.3745.49Reference50.2 h
GeoWAM-Joint13.309.6257.98+59.8%80.2 h

Extrapolating to 21,700 optimizer steps gives about 28 additional hours of policy training for GeoWAM and 30 for GeoWAM-Joint. Training the geometry VAE takes approximately one additional day, and the same VAE can be reused across policy runs. We agree with the distinction the reviewer raises: geometry adds additional training cost, but neither teacher extraction, VAE encoding of geometry, nor a geometry-expert forward pass is required for action inference.

Reviewer_neTJOpenReview ↗

Thanks the authors for response

The matched LIBERO-Plus controls, direct-depth comparison, and real-robot results resolve the main issues I raised, so I will raise my rating to weak accept.

Dear Reviewer neTJ,

We are glad that our responses have addressed your concerns. Thank you very much for your constructive feedback, time and effort, and for raising your score.

Best regards,

The Authors

Comments on the paper

2 comments

Dear Reviewers,

Please take time to review the responses from the authors and engage with them for further questions and comments if any.

Time left is not much and please do shortly.

Thanks

AC

We sincerely thank all reviewers for their careful reading and constructive feedback. We have provided detailed responses and additional experiments, and hope these new results and clarifications help resolve your concerns. If so, we would greatly appreciate your consideration of updating your rating. We welcome any further questions and would be happy to address them during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.