acceptodds
Withdrawn from ICLR 2027

GeoFirst: Learning World-Consistent Geometry for Robust Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

World-action models (WAMs) have shown promise for embodied intelligence, but their core value lies in the latent representations learned through prediction rather than explicit future imagination. Existing WAMs rely on RGB video prediction, which lacks 3D geometric structure, while depth-based approaches treat depth as an auxiliary task from limited viewpoints—yielding representations that lack cross-view consistency and degrade under observation perturbations. We present GC-WAM (Geometry Consistent World-Action Model), which replaces RGB video prediction with metric depth prediction as the primary future-state supervision signal, encouraging the video foundation model to learn geometry-grounded latent representations rather than appearance-centric ones. GC-WAM introduces per-view depth normalization to preserve geometric detail across camera views, cross-view world consistency via an InfoNCE-based loss for viewpoint-invariant representations, and a geometry-first curriculum that first learns geometry then jointly learns geometry and action. Experiments on LIBERO Plus show that GC-WAM achieves 79.7% average success rate (+29.8 pp over RGB baseline), and achieves 85.4% on RobotTwin Clean2Clean. Different Views, Same World.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, World Action Model, 3D Representation Learning, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M2fUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GC-WAM, a world-action model that uses future-depth prediction as its main visual prediction objective, bidirectional contrastive alignment of geometrically corresponding local features across camera views, and a geometry-first training curriculum. Its central motivation is to learn geometric information that is accessible to the action policy and remains useful under viewpoint changes. Depth and camera calibration provide training supervision, while deployment retains RGB observations, language, and robot state as inputs without requiring future-depth generation. Evaluation includes frozen-feature geometry diagnostics, component ablations, LIBERO/LIBERO-Plus, and RobotTwin 2.0.

Critical strengths
  1. The problem is relevant to manipulation, and the proposed supervision addresses a meaningful distinction. Learning features from which geometry can be recovered is different from ensuring that descriptors of the same physical surface remain matchable across views. Future-depth supervision and calibration-guided local alignment are a coherent way to address these complementary requirements. Using geometry as training supervision while retaining RGB-based deployment is also practically appealing.

  2. The method shows substantial gains on tests that directly exercise observation robustness. On LIBERO-Plus, Camera success improves, and the method also improves Light, Background, and Sensor performance relative to that baseline (Tab.2). These are relevant interventions on the observations, rather than only improvements on unperturbed tasks. Tab.4 additionally reports an average success rate increase after adding geometric consistency, providing preliminary evidence that the component is useful. These results support the value of the overall training approach within the evaluated simulation conditions.

Critical weaknesses
  1. The targeted experiments do not yet establish the central link from improved geometry to cross-view consistency and robust control. E.g., Fig.1(b) is relevant, but unseen-view depth-probe AbsRel is 0.0355 for GC-WAM versus 0.0356 for RGB and 0.0368 for RGB-D. The difference from RGB is very small. At the control level, Tab.4 reports only average success, so it does not show whether the Geo component specifically improves Camera robustness. Tab.6's camera-augmentation baseline achieves 74.5% on Camera versus 67.7% for GC-WAM, although it uses more data exposure and longer training. Thus, this comparison does not establish the alignment objective's incremental advantage under matched conditions. Finally, the paper motivates geometry through shape, contact surfaces, and spatial relationships, but does not analyze whether gains occur in tasks or conditions where these particular capabilities determine success.

  2. The evidence for practically useful robustness is limited by near-failure under randomized conditions and the absence of real-robot validation. RobotTwin success falls from 85.4% on Clean2Clean to 4.7% on Clean2Random, compared with Fast-WAM's 70.2% to 1.2% (Table 3). The improvement therefore remains very limited in absolute terms under randomization. Moreover, all control evidence is simulated. Without physical-robot tests under controlled viewpoint or appearance changes, it remains unclear whether the benefits learned using simulated depth and calibrated view pairs translate to actual manipulation.

Questions for authors
  1. Please specify the data splits, probe architecture, feature/token selection, noise conditions, and training protocol for Fig.1, and define its variants and policy evaluation setting. Are the probed features obtained using only information available during RGB-based deployment? What repeated-evaluation or uncertainty estimates are available? Is there evidence that the improved readout corresponds to better handling of relative depth, spatial relations, or contact-relevant geometry in the evaluated tasks?

  2. What are the Camera results for the Tab.4 variants under the same view exposure and training budget? Please identify the training and test camera poses and, if available, separate camera distance, position, and orientation changes. Aggregate correspondence results on held-out scenes/views would help connect the Fig.3 observation to this control result.

Reviewer_WZqGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GC-WAM to improve visual representations for robotic manipulation policies. The model uses depth and camera calibration information during training and generates actions from RGB observations and other current inputs at deployment. The paper reports substantial improvements over Fast-WAM on LIBERO Plus and achieves a high success rate under the Clean2Clean protocol on RoboTwin.

Critical strengths

The paper distinguishes between “being able to extract geometric information from features” and “being able to match features corresponding to the same physical location across viewpoints.” This distinction is more targeted than simply adding a depth-prediction task and is directly relevant to policy stability under camera viewpoint changes. The paper is well written and generally clear.

Critical weaknesses

1.The paper builds on the basic framework and premise of Fast-WAM: using future prediction during training to improve representations for control while omitting future generation at inference time. Its main additions are using depth as the primary prediction target, introducing bidirectional InfoNCE based on reprojection correspondences, and adopting two-stage training. I consider these to be relatively established techniques, so the authors need to clearly demonstrate that this combination provides capabilities that are difficult to obtain with existing approaches. I think the current manuscript does not discuss this sufficiently. For example, does replacing the RGB prediction target entirely offer a consistent advantage over retaining RGB prediction and adding auxiliary depth supervision? What unique value do explicit geometric constraints provide compared with directly training on additional camera views? 2.The paper emphasizes a 29.8 percentage-point improvement over the RGB baseline, but the full method simultaneously changes the prediction target, training schedule, cross-view constraints, and direct supervision on perturbed views. In Table 4, the Base configuration already achieves 75.0%, increasing to 78.7% with geometric consistency and to 79.7% with additional supervision on perturbed views. This shows that geometric consistency provides a benefit, but also that most of the overall improvement occurs between the RGB baseline in the main results table and the Base configuration. Does this portion of the improvement come from depth prediction, the training strategy, or other training conditions? I suggest explicitly comparing RGB, RGB-D, and Depth under matched backbones, initialization, training data, camera augmentation, and training budgets, and then progressively introducing geometric consistency and two-stage training. 3.The method section states that depth and action losses are computed only on the original view, while the perturbed view is used for geometric alignment. However, the final configuration achieving 79.7% explicitly includes depth and action supervision on the perturbed view. The method section also states that the first stage jointly optimizes depth prediction and geometric consistency, whereas the ablation discussion repeatedly describes this stage as using depth prediction alone. I would appreciate clarification from the authors on these points. If the authors can provide reasonable explanations, I am willing to raise my score.

Questions for authors

Please see the specific concerns and questions raised in the Weaknesses section.

Reviewer_nDrVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

GC-WAM replaces future RGB prediction with depth prediction and adds cross-view geometric feature alignment and a geometry-first training curriculum. The method reports improved robustness on LIBERO Plus while retaining RGB-based, action-only inference.

Critical strengths
  • Relevant research question. The paper examines whether predictive supervision should prioritize geometry over appearance for robot control. Its distinction between per-view geometric accessibility and cross-view consistency provides a useful perspective on representation quality.
  • Coherent method design. Future-depth prediction, correspondence-based feature alignment, and geometry-first training address complementary aspects of geometric representation learning. Depth and camera calibration are required only during training, preserving RGB-based, action-only deployment.
  • Promising empirical results. GC-WAM reports a substantial improvement over the listed Fast-WAM baseline on LIBERO Plus while maintaining comparable performance on the original LIBERO benchmark. The component ablations provide initial evidence for the benefits of cross-view training and the two-stage curriculum.
Critical weaknesses
  1. The contribution of depth prediction is not sufficiently isolated.

    Despite emphasizing the replacement of RGB prediction, the paper does not report policy success rates for matched RGB, RGB-D, and depth-only variants. Figure 1(b) compares depth probes, but this does not establish the downstream benefit of the prediction target. Please provide these policy-level comparisons with matched initialization, camera supervision, training schedule, and budget.

    Relatedly, are the Fast-WAM scores cited, evaluated from released checkpoints, or obtained through retraining? Please specify the initialization of GC-WAM’s components and whether the baseline follows the same training protocol.

  2. The geometric consistency loss needs a cleaner ablation.

    Table 4 does not compare identical multi-view training configurations with and without Geo. Please use the same paired views, depth/action supervision on both views, and training budget, changing only the Geo loss. This would separate geometric alignment from the benefit of additional viewpoint supervision.

  3. The representation analysis is difficult to assess.

    Figure 1 contains the main quantitative geometric evidence, yet the v0–v4 configurations in Figure 1(c) are undefined, and no accompanying table provides their metrics or evaluation settings. The depth-probe protocol is also insufficiently described. Please define these variants, explain how the probes are trained and evaluated, and tabulate their geometric metrics and policy success. Otherwise, the claimed relationship between geometric quality and control performance is difficult to verify.

  4. The two-stage training budget is unclear.

    Does the joint-only baseline in Table 5 receive the same total training budget as both stages combined? Please report the per-stage budgets and, if necessary, add an equal-budget comparison before attributing the improvement to avoiding interference from action gradients.

  5. The final supervision configuration needs clarification.

    Section 3.3 applies depth/action supervision only to the original view, whereas Table 4’s final configuration includes perturbed-view supervision. Please reconcile these descriptions and specify which configuration produces the main results.

Questions for authors

Please see the Weaknesses section for the clarification requests and controlled comparisons that could materially affect my assessment, particularly regarding baseline initialization, training budgets, prediction-target and Geo-loss ablations, and the definitions and evaluation protocols of the Figure 1 variants. These concern the evidence supporting the current claims rather than suggestions for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.