acceptodds
Withdrawn from ICLR 2027

Gae-WAM: Geometric Action Effect Modeling for World Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

World action models (WAMs) use future visual prediction to support robot action learning. Complete future observations contain object motion, backgrounds, and textures, but not all of this information is directly relevant to action selection.How can action prediction emphasize task-relevant information in visual predictions?We propose Gae-WAM, a WAM architecture that connects visual and action predictions through action-induced geometric changes in target objects. Modeling these changes as geometric action effects brings object-focused information from visual predictions into action-side supervision without substantial additional inference overhead. We find that geometric action effect modeling is effective for robot manipulation. We compare Gae-WAM with other methods on multiple simulation benchmarks and Astribot S1 real-world tasks. Gae-WAM achieves higher aggregate success, with gains under camera-viewpoint and spatial-configuration changes. Shared-representation and training-objective ablations support the contributions of geometric references and cross-branch consistency to manipulation performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action models, robot learning, world models, geometric representation learning, embodied AI, action prediction, 3D consistency, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x7LYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a geometric auxiliary training framework for world action models. This paper's method extends ImageWAM with two lightweight prediction heads, one head uses the predicted visual latent to predict 2D target object centroid and 3D target object displacement, while the other uses the predicted action chunk to predict the same things. Both heads' outputs are aligned with the 2D and 3D values actually computed from the real endpoint images, serving as auxiliary losses to help the model train its gemoetic and spatial capabilities.

Critical strengths
  1. Combines complementary 2D and 3D information. The per-view centroid/area changes retain image-space information, while the reconstructed 3D displacement provides a shared world-coordinate reference across views. The validity and confidence weighting is a sensible response to missing views and unreliable 3D reconstruction.

  2. No additional inference-time requirements. In this method, two projection heads are added only during training, and everything added extra at inference time is removed. Therefore, this method is essentially a model-agnostic training approach that can be conveniently applied to various WAM models as additional auxiliary supervision.

Critical weaknesses
  1. The overall improvement is small. The gains are +0.17 percentage points on standard LIBERO, +0.87 points on LIBERO-Plus, and +0.47 points on RoboTwin overall. RoboTwin Randomized performance decreases, and several LIBERO-Plus perturbation axes also decrease.

  2. The relationship between the projected visual effect and actual decoded-image geometry is unverified. e_edit is predicted from a latent difference through an auxiliary head. It is not computed by decoding predicted z to an image and measuring the target-object geometry in that image. Therefore, the current objective does not guarantee that the learned proxy corresponds to the geometric change actually depicted by the endpoint image.

  3. Due to the above two issues, along with the fact that the parameter counts of the two trainable projection heads do not appear to be explicitly stated in the paper, I have concerns about whether these auxiliary supervision losses change the backbone in a meaningful way.

  4. The method relies on extensive task-specific data processing. LIBERO uses SAM3; RoboTwin reads directly from the simulator; and the real-robot setup uses target masks, calibration, joint states, and forward kinematics. These processing pipelines do not appear to be very clearly presented in the paper. The paper also does not report these preprocessing times, cache size, GPU cost, or actual training overhead.

  5. The attention visualizations use an early GeoEffect V1 variant rather than the final Gae-WAM checkpoint and cover only a few qualitative examples. The paper does not show that the final action branch systematically attends more to target regions, nor that this attention is correlated with action accuracy or task success

Questions for authors
  1. Does the geometric loss actually change the WAM, or is it mostly absorbed by the two heads? Could you test whether the projection heads can still align well if you detach backbone and two project heads or freeze backbone?

  2. How close is e_edit to the geometry of the image the model actually predicts? e_edit is read from a latent difference by a learned head; it is not measured on the predicted endpoint. Could you fully denoise predicted z offline, decode it with the VAE, run the same SAM3 + visual-hull pipeline on the decoded image, and compare the resulting e_img with e_edit and e*?

  3. Could you give the exact architecture and parameter count of H_edit and H_action? The only detail I found is a hidden size of 256, and Table 19 covers RoboTwin only; I could not find the auxiliary loss weights, noise range, or head settings used for LIBERO. The approximately 2.18M parameters mentioned in Appendix E belong to FullAlign's state predictor, not to Gae-WAM.

  4. Is the method intended only for fine-tuning, and can it scale? It seems that Gae-WAM is applied only during downstream policy training, without robot-policy pretraining. The reference relies on per-task prompt maps and hand-written track-selection rules, and each Astribot task is trained as a separate model mentioned in table 8. How much compute and manual effort do SAM3 tracking and visual-hull construction require, for example for the LIBERO frames? How are the Astribot masks obtained? I could not find this described. Would the approach be applicable to large, heterogeneous pretraining data where the target object is not specified?

Reviewer_cW3SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Gae-WAM adds geometric auxiliary supervision to ImageWAM [1]. Target masks and camera calibration are used to estimate changes in object centroid, projected area, and 3D position between demonstration endpoints. Two heads predict these changes from the visual prediction and the predicted action chunk, conditioned on the current scene. Both receive reference supervision and a cross-branch consistency loss. The additional components are used only during training. Experiments cover LIBERO, LIBERO-Plus [2], RoboTwin 2.0, and four Astribot S1 tasks. The clearest gains are under camera-viewpoint perturbations.

Ref

[1] Yuyang Zhang, Wenyao Zhang, Zekun Qi, He Zhang, Haitao Lin, Jingbo Zhang, Yao Mu, Xiaokang Yang, Wenjun Zeng, and Xin Jin. “ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?” arXiv preprint arXiv:2606.19531, 2026. https://arxiv.org/abs/2606.19531.

[2] Senyu Fei et al. “LIBERO-Plus: A Progressive Robustness Benchmark for Visual-Language-Action Models.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 38574–38583, 2026. Publisher page.

Critical strengths
  1. The geometric supervision adds no deployment requirements. The two branches share an interpretable target, with validity and confidence weights accounting for missing or unreliable reference dimensions. Masks, calibration, reconstruction, and effect heads are confined to training; deployment follows ImageWAM [1]. This is a practical advantage for incorporating geometry into a policy that still operates from RGB observations, language, and proprioception.

  2. The camera-viewpoint improvement is supported by useful controls. On LIBERO-Plus [2], success under camera perturbations rises from 74.94% for the matched ImageWAM baseline to 78.82%. The full method also exceeds 2D-only by 4.19 percentage points and Grounding-only by 3.77 points (Tables 5–6). These comparisons support the contributions of 3D supervision and consistency in this setting. This is the paper's strongest result.

Ref

[1] Yuyang Zhang, Wenyao Zhang, Zekun Qi, He Zhang, Haitao Lin, Jingbo Zhang, Yao Mu, Xiaokang Yang, Wenjun Zeng, and Xin Jin. “ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?” arXiv preprint arXiv:2606.19531, 2026. https://arxiv.org/abs/2606.19531.

[2] Senyu Fei et al. “LIBERO-Plus: A Progressive Robustness Benchmark for Visual-Language-Action Models.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 38574–38583, 2026. Publisher page.

Critical weaknesses
  1. The aggregate gains are small, and their stability across training runs is unknown. Improvements over ImageWAM are 0.17 percentage points on LIBERO, 0.87 on LIBERO-Plus, and 0.47 on RoboTwin 2.0 (Tables 1–3). The LIBERO-Plus margins over DirectAlign and FullAlign are just 0.22 and 0.32 points (Table 5). Real-world success increases from 62/120 to 68/120, without uncertainty estimates (Table 4). RoboTwin Randomized and towel-folding performance decline. Given these small, uneven gains, I am not convinced that the experiments establish a sufficiently reliable improvement over the baseline and alternative auxiliary objectives.

  2. The effect estimates themselves are not directly evaluated. Both heads receive the current visual context in Equation (6). The paper does not report held-out effect-prediction errors or test how the estimates respond when the action or predicted visual change is varied with that context fixed. It therefore remains unclear how much the heads learn about the consequences of their respective predictions beyond information already available in the scene. The comparison with Grounding-only supports a downstream benefit from consistency, while Appendix K measures action sensitivity to camera changes; neither directly evaluates the correctness of the estimated object changes. This matters because the paper's main interpretation is that these estimates connect visual predictions to action consequences. The policy results support a useful auxiliary objective, but provide limited evidence for that interpretation.

Questions for authors
  1. Clarification of empirical uncertainty: For the existing evaluations, can you provide uncertainty estimates for the paired performance differences, accounting for the task/instance structure, and clarify whether the real-world trials were paired by initial condition? This would help assess the reliability of the small aggregate improvements and ablation margins.

  2. Evidence requested for the existing claim: How accurately do the heads predict geometric effects on held-out data, compared with a predictor using only the current visual context? With the scene fixed, do changes to the action chunk produce effect estimates that agree with the resulting object motion? A corresponding check of the visual head would help establish that both heads use their respective predictions. These checks would address the action-effect interpretation directly.

Reviewer_yD9kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Gae-WAM adds geometric action-effect supervision to ImageWAM. It constructs a reference from target-object centroid and area changes in each camera view, together with estimated 3D displacement. Two auxiliary heads predict this reference from the visual and action predictions, and a consistency loss aligns their outputs. The auxiliary components are used only during training. Experiments cover LIBERO, LIBERO-Plus, RoboTwin 2.0, and four Astribot S1 tasks, with the clearest improvement occurring under camera-viewpoint perturbations.

Critical strengths
  1. The method adds an object-focused training signal while retaining ImageWAM’s deployment inputs and action-generation procedure. This makes the approach practically appealing if its benefits can be established reliably.
  2. The matched ImageWAM comparison and component ablations are useful. In particular, the camera-viewpoint result on LIBERO-Plus improves from 74.94% to 78.82%, providing evidence that the geometric supervision can help in this setting. The paper also reports task-dependent regressions rather than presenting the gains as universal.
Critical weaknesses
  1. The motivation and the need for the proposed coupling are not sufficiently established. During approach or other non-contact motion, the target object can remain stationary across a 16-step chunk. Correct approach, an incorrect movement, and no movement may then receive essentially the same object-displacement target. This does not invalidate the base action supervision, but it raises a central question about how much useful action information the auxiliary target provides across manipulation stages. The paper does not report the distribution of near-zero effects or show which stages benefit. In addition, with a moving wrist camera, the 2D centroid and area changes in Eq. (3) can reflect camera motion even when the object is stationary. The interpretation of these quantities as object action effects therefore needs clearer support. The reason for predicting the same effect through two heads and then adding consistency also remains unclear. Different inputs can justify separate predictors, and consistency may help when their estimates are imperfect. However, both heads already receive the same geometric reference, so their agreement alone does not establish that the action prediction produces the visual outcome. The Grounding-only comparison is relevant, but the aggregate LIBERO-Plus gain is 0.55 percentage points, while standard LIBERO is slightly lower with consistency (98.10% versus 98.17%). The viewpoint gain is encouraging; it does not yet establish the broader need for cross-branch coupling over simpler one-branch or shared-reference supervision.
  2. The evaluation does not yet establish a reliable advantage over the most relevant alternatives. The tables list several policies, but the controlled comparisons are largely limited to ImageWAM and variants designed by the authors. Comparisons with representative object-centric supervision or visual–action alignment approaches discussed in Sec. 2, where applicable under a compatible protocol, would better test what the proposed mechanism adds. A simpler target-mask or relative gripper–object supervision baseline on the same backbone would also be informative. The issue is the lack of decisive comparisons, rather than the number of methods in the tables.
Questions for authors
  1. What evidence shows that the geometric target provides useful supervision during approach and non-contact motion? How are wrist-camera changes distinguished from physical object changes?
  2. Why are both effect heads and consistency needed beyond one-branch or shared-reference supervision? Do the heads respond to changed actions or visual predictions when the current scene is fixed?
  3. Can the gains be supported by independent training runs and a matched comparison with a representative object-focused or visual–action alignment baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.