acceptodds
Under review as a conference paper at ICLR 2027

Geometric Latent States for Spatially Aware Robot Manipulation

Abstract

Predicting physical scene dynamics under interaction is fundamental to reliable robotic manipulation. World Action Models (WAMs) have emerged as a promising paradigm by framing scene evolution as future video prediction. However, existing WAMs predominantly operate in appearance-oriented video latent spaces, lacking the 3D spatial understanding required for precise manipulation. While per-frame depth prediction constrains distances to visible surfaces without capturing spatial correspondences across views and time, emerging 3D foundation models offer rich spatial representations whose high dimensionality and redundancy make direct prediction computationally demanding. We present GeoAct, a world action model that grounds robot manipulation in latent 3D geometric prediction. GeoAct jointly models complementary future states: RGB latents to capture fine-grained visual dynamics and geometric latents to model evolving 3D scene structure. To obtain a compact spatial state, we design GeoAE to compress multi-layer VGGT-Omega features into a low-dimensional spatiotemporal latent space through joint feature reconstruction and explicit 3D geometric decoding. During training, GeoAct co-denoises future RGB, geometry, and actions, while retaining RGB-to-action inference without online geometry encoding at deployment. GeoAct achieves mean success rates of 84.6% on LIBERO-Plus and 92.6% on RoboTwin 2.0, with strong performance under camera viewpoint perturbations. Real-world experiments further demonstrate GeoAct's improved performance on manipulation tasks requiring precise spatial coordination.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, 3d computer vision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_khWFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GeoAct, which jointly predicts future RGB, geometric latent states, and robot actions to provide geometric supervision for manipulation policies. GeoAE compresses multi-layer VGGT-Omega features into spatiotemporal latents while preserving geometry through feature, depth, and camera reconstruction. Following multi-domain video pretraining and adaptation to robot demonstrations, deployment generates actions from current RGB, language, and proprioception without online geometry encoding or future video generation. Evaluations cover LIBERO-Plus, RoboTwin 2.0, and three real-world bimanual tasks, with ablations examining geometric supervision, prediction targets, and compression design.

Critical strengths
  1. Geometric supervision is supported by substantial ablations. Matched comparisons demonstrate its benefit for policy learning, while comparisons of prediction targets and compression methods further support GeoAE’s design.
  2. Geometric supervision preserves efficient deployment. Geometry prediction shapes representations during training, while deployment generates actions directly, avoiding online geometry encoding and future video generation. The inspected implementation is consistent with this design.
Critical weaknesses
  1. Robustness to background perturbations remains limited. GeoAct trails some baselines under background perturbations, and joint supervision does not consistently outperform RGB-only.
  2. Future geometry prediction is validated mainly qualitatively. Quantitative geometric metrics primarily assess GeoAE’s reconstruction of teacher geometry, while future forecasts are mainly visualized on training clips. Direct evidence of future geometric prediction accuracy on unseen clips therefore remains limited.
Questions for authors
  1. Based on existing failure cases, what primarily causes the weaker performance under background perturbations? What evidence supports the proposed explanation?
  2. Could the authors quantitatively evaluate future geometry prediction on unseen clips using the existing models, clarifying whether the reference is teacher predictions or simulator ground truth?
4 comments
AuthorsOpenReview ↗

How Future RGB Prediction Affects Background Robustness

We thank the reviewer for carefully examining our implementation and recognizing the efficiency of our closed-loop RGB deployment. In response to your two questions, we have added an analysis of performance under background perturbations and a quantitative evaluation of future geometry prediction on unseen clips, as detailed below. All revisions in the updated manuscript are highlighted in blue.

Corresponding to: W1, Question 1

GeoAct lags behind some baselines under background perturbations, and joint supervision does not consistently outperform RGB-only. Based on existing failure cases, what is the primary cause of this weaker performance under background perturbations, and what evidence supports this?

We attribute this to the dual role of future RGB prediction: identifying target objects from language instructions requires the appearance cues provided by RGB supervision, yet reconstructing full-frame future RGB latents also requires the representation to retain background textures. Removing the RGB branch (Geometry-only) renders the policy invariant to background texture replacement—achieving 87.73% under background perturbations (Table 3)—but deprives it of color and appearance cues, causing its performance under language perturbations to drop to 35.91% and yielding the lowest overall success rate (65.88% vs. 78.27% for joint prediction). Consequently, to ensure accurate instruction-conditioned target identification, GeoAct retains future RGB prediction, and the Action expert attends to the current RGB representation. Because reconstructing future video frames requires this representation to preserve background details, unseen background textures at test time propagate into the Action expert's input.

Video-prediction baselines in Table 1 exhibit the exact same pattern: although both lighting and background perturbations alter only visual appearance, LingBot-VA, Fast-WAM, and 4D-WAM achieve 78.2%–94.3% under lighting changes but drop to 53.1%–57.7% under background replacement, which introduces unseen textures. In contrast, VLA baselines that do not predict future video vary by no more than 4 percentage points between the two perturbations. The ablation variants in Table 3 are trained only on unaugmented LIBERO demonstrations, so unseen background textures at test time directly perturb their RGB representation. π₀.₅ mitigates this through large-scale multi-scene pretraining, whereas our main GeoAct model expands appearance coverage with image augmentation (Table A4) and reaches 75.2% under background perturbations (Table 1). We have incorporated this analysis into Section 4.4.1 of the revised manuscript.

AuthorsOpenReview ↗

Future Geometry Accuracy on Held-Out Clips

Corresponding to: W2, Question 2

Quantitative geometric metrics primarily evaluate GeoAE's reconstruction of teacher geometry, while future predictions are mainly visualized on training clips. Can you quantitatively evaluate future geometry on unseen clips using the existing model, and clarify whether the reference is teacher predictions or simulator ground truth?

Following your suggestion, we quantitatively evaluate future geometry prediction on held-out validation clips across all five domains, using both teacher estimates and simulator ground truth as references. Conditioned solely on the current observation and language instruction, GeoAct decodes predicted geometric latents into depth maps, which are compared against VGGT-Ω teacher estimates computed on the actual future frames; for LIBERO and RoboTwin 2.0, we additionally compare against simulator ground-truth depth.

Each entry below reports AbsRel (↓) / δ₁ (%, ↑); "—" denotes datasets without simulator ground-truth depth:

DatasetGeoAE Recon. vs. VGGT-ΩFuture Pred. vs. VGGT-ΩFuture Pred. vs. Sim. GTVGGT-Ω vs. Sim. GT
LIBERO0.042 / 96.50.066 / 94.60.177 / 75.00.139 / 70.8
RoboTwin 2.00.056 / 95.90.118 / 84.40.156 / 75.40.107 / 81.8
WorldArena 2.00.070 / 97.00.126 / 87.3——
BridgeData V20.071 / 96.60.330 / 62.2——
SSv20.021 / 99.10.107 / 87.6——

Against simulator ground truth, GeoAct's predictions reach accuracy on par with the teacher's direct estimates: on LIBERO, its δ₁ exceeds the teacher's (75.0% vs. 70.8%), while on RoboTwin 2.0 both metrics are slightly below the teacher's (see table above). This confirms that on unseen clips, GeoAct predicts 3D scene evolution comparable to teacher estimates computed directly from future frames. This evaluation has been added to Table 7 in Section 4.5 of the revised manuscript.

We hope these responses fully address your questions and welcome any further discussion.

Thank you for addressing both of my questions. The ablation results help explain the trade-off between retaining appearance cues for target identification and improving robustness to background changes. The new evaluation on held-out clips, including comparisons against simulator ground-truth depth where available, provides direct quantitative evidence for future geometry prediction. I appreciate these additions and maintain my rating of 3 (Weak acceptance).

Thank you very much for your follow-up and for confirming that both of your questions have been addressed. We are glad that the ablation analysis on the appearance-robustness trade-off and the quantitative evaluations on held-out clips have resolved your initial concerns regarding the soundness and completeness of our empirical validation. We truly appreciate your constructive engagement throughout the review process, and we hope these additions prove helpful in your final overall assessment of our work.

Reviewer_4c3KOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces GeoAct, a world action model that models the RGB, geometry latents, and actions jointly. The authors first design GeoAE to encode the geometry representation from VGGT-Omega into a compact spatiotemporal latent space. Then, GeoAct proposes different attention mechanisms for training and inference separately. During inference, GeoAct accepts the current RGB observation as input without geometry or future predictions, and directly predicts actions. Experiments in simulations (LIBERO-Plus, RoboTwin) and real-world (bimanual) tasks support the effectiveness of GeoAct.

Critical strengths

This work studies how the geometry information co-training in generative modeling would be helpful for world action model learning. The ablation is very helpful for understanding the design: it ablates the prediction target across RGB-only, with various prediction targets including DINO, depth, VGGT features, and GeoAE latent, and the experiments show that the GeoAE latent is the best choice for overall performance. It shows the effectiveness of the geometry latent and the joint training design.

Critical weaknesses
  1. GeoAct only slightly outperforms the strongest baselines in the benchmarks. For example, for LIBERO-Plus, GeoAct got 84.6, while pi-0.5 also got 84.4; and in RoboTwin 2.0, GeoAct got 92.6 and LingBot-VA got 92.2.
  2. The author shows the qualitative results of the future RGB and depth prediction, but lacks a quantitative comparison for the future predicted geometry.
Questions for authors

Can the author provide more details about the ablation design, and the efforts they made to ensure a fair comparison for ablations? This would be helpful to understand how the designed geometry branch is helpful for world action modeling.

5 comments
AuthorsOpenReview ↗

Performance and Pretraining Cost Relative to the Strongest Baselines

We thank the reviewer for appreciating our prediction-target ablations and for providing constructive suggestions. Below, we clarify our pretraining compute relative to baselines achieving similar success rates, present quantitative evaluations of future geometry prediction, and detail the motivation and experimental controls for our three ablation groups. All revisions in the updated manuscript are highlighted in blue.

Corresponding to: W1

GeoAct only slightly outperforms the strongest baselines on these benchmarks. For example, on LIBERO-Plus, GeoAct scores 84.6 while π₀.₅ reaches 84.4; on RoboTwin 2.0, GeoAct scores 92.6 while LingBot-VA reaches 92.2.

Your observation is accurate: on both benchmarks, GeoAct's mean success rate is close to that of the strongest baseline. However, GeoAct's primary advantage lies not in outperforming these baselines by a wide margin, but in reaching this performance level using action-free videos and orders of magnitude less pretraining compute. Our pretraining uses 156 hours of robot videos and 180 hours of human hand videos without action labels, totaling 3.31 × 10¹⁹ FLOPs—only 1/324 to 1/41 of the estimated pretraining compute of π₀.₅—while achieving 84.6% on LIBERO-Plus compared to 84.4% for π₀.₅. Similarly, on RoboTwin 2.0, GeoAct reaches 92.6% versus 92.2% for LingBot-VA, which is pretrained on approximately 16k hours of robot demonstrations. The pretraining compute and success rates are summarized below:

MethodPretraining FLOPsLIBERO-Plus Success (%) ↑
GeoAct3.31 × 10¹⁹84.6
π₀.₅(1.34–10.74) × 10²¹84.4
π₀(2.17–17.38) × 10²¹53.6
X-WAM(1.71–2.49) × 10²¹75.6
StarVLA7.53 × 10²¹74.1
Spatial Forcing7.53 × 10²¹72.7
LingBot-VA≈ 4.45 × 10²²69.5
Fast-WAM051.5
4D-WAM071.0

The table reports pretraining compute after foundation-model initialization and prior to LIBERO adaptation; baseline values are estimated from their published training configurations, with the calculation and a visual comparison (Figure A5) provided in Appendix B.6.

AuthorsOpenReview ↗

How Accurately GeoAct Forecasts Future Geometry

Corresponding to: W2

The paper shows qualitative results for future RGB and depth, but lacks a quantitative comparison for predicted geometry.

We have added a quantitative evaluation of future geometry prediction on held-out validation clips, reporting errors against both teacher geometry and simulator ground truth to distinguish between "how well the model predicts its training target" and "how closely the predictions match true 3D scene structure."

Conditioned solely on the current observation and language instruction, GeoAct decodes predicted geometric latents into depth maps, which are evaluated against VGGT-Ω estimates computed from the actual future frames; on LIBERO and RoboTwin 2.0, we additionally evaluate against simulator ground-truth depth. GeoAE's reconstruction accuracy on observed frames is included as a reference to separate compression error from forecasting error.

Each entry below reports AbsRel (↓) / δ₁ (%, ↑); "—" denotes datasets without simulator ground-truth depth:

DatasetGeoAE Recon. vs. VGGT-ΩFuture Pred. vs. VGGT-ΩFuture Pred. vs. Sim. GTVGGT-Ω vs. Sim. GT
LIBERO0.042 / 96.50.066 / 94.60.177 / 75.00.139 / 70.8
RoboTwin 2.00.056 / 95.90.118 / 84.40.156 / 75.40.107 / 81.8
WorldArena 2.00.070 / 97.00.126 / 87.3——
BridgeData V20.071 / 96.60.330 / 62.2——
SSv20.021 / 99.10.107 / 87.6——

Against simulator ground truth, GeoAct's forecasts reach accuracy on par with the teacher's direct estimates: on LIBERO, its δ₁ exceeds the teacher's (75.0% vs. 70.8%), while on RoboTwin 2.0 both metrics are slightly below the teacher's (see table above). This demonstrates that on unseen clips, GeoAct forecasts 3D scene structure comparable to teacher estimates obtained directly from future frames. These results have been added to Table 7 in Section 4.5 of the revised manuscript.

AuthorsOpenReview ↗

Ablation Design and Fair Comparison Controls

Corresponding to: Questions

Could you provide more details on the ablation design and the efforts made to ensure fair comparisons? This would help clarify how the geometry branch aids world-action modeling.

Our three ablation groups sequentially address three core design questions for the geometry branch: (1) whether a geometry branch is needed, (2) what representation it should predict, and (3) how to construct that prediction target. Within each group, we strictly fix the training budget and evaluation protocol while isolating the specific variables under study (inputs, prediction targets, or compression methods).

  • Controls for fair comparison: Across all variants, the Action expert is initialized from adapted Wan2.2 weights and trained on the same LIBERO demonstrations with a global batch size of 128 for 10 epochs in Stage 2. All ablations bypass Stage 1 pretraining and image augmentation, and are evaluated on the exact same 10,030 LIBERO-Plus episodes, initial states, and environment seeds, using 10 integration steps to generate 32-step action chunks and replanning every 8 steps.
  • Table 3: Whether a geometry branch is needed. Starting from our joint RGB, geometry, and action prediction model, we ablate either the geometry stream or the RGB stream while retaining action prediction across all three variants. The RGB-only baseline tests whether geometric supervision provides benefits beyond future RGB prediction; the Geometry-only baseline tests whether geometry can wholly replace RGB or if both streams are complementary. Their prediction targets, action conditioning inputs, and active inference modules are summarized below:
VariantWorld Prediction TargetVisual Condition of ActionsModules Run at Inference
RGB onlyFuture RGBCurrent RGBRGB and Action experts
Geometry onlyCurrent and future geometryCurrent geometry s₀, encoded online by VGGT-Ω and GeoAEVGGT-Ω, GeoAE, Geometry and Action experts
RGB + GeometryFuture RGB and geometryCurrent RGBRGB and Action experts

Joint prediction achieves the highest mean success rate (78.27%), confirming that the geometry branch provides complementary supervision beyond RGB modeling.

  • Table 4: What the geometry branch should predict. Having established the benefit of joint prediction, we compare alternative prediction targets for the auxiliary stream. An effective target should preserve 3D spatial structure while remaining compact enough for generative world modeling. Accordingly, we compare GeoAE latents against three baselines: Depth maps capture explicit geometry but only record visible-surface distances and require high-resolution pixel-wise prediction; VGGT-Ω features share the same source as GeoAE latents but remain uncompressed and high-dimensional; and DINO features serve as a semantic visual baseline without explicit 3D geometric supervision. Each target is added to joint RGB–action prediction:
Prediction TargetTarget ContentTarget Shape per ViewGeometry Readout
+ DINODINO visual features—None
+ DepthDepth maps[3, 1, 224, 224]Direct
+ VGGT featuresLayers 4, 11, 17, and 23[4, 3, 213, 2048]Frozen VGGT-Ω heads
+ GeoAE latentCompressed 4-layer features[3, 48, 14, 14]GeoAE decoder + frozen VGGT-Ω heads

GeoAE latents achieve the highest success rate, outperforming direct depth and raw VGGT feature prediction by 7.34 and 5.11 percentage points, respectively. As visualized in Figure 6, directly predicted depth progressively loses arm and cup boundaries over time, while decoding directly predicted high-dimensional VGGT features yields distorted object contours. GeoAE preserves sharper boundaries and their evolving spatial relationships, validating the use of a compact, geometry-preserving latent target.

Reviewer_ARshOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GeoAct, a geometry-aware World Action Model that jointly predicts future RGB images, geometric latents, and robot actions. It introduces GeoAE to compress VGGT-Omega features while preserving 3D spatial information. GeoAct uses joint flow-matching training but requires only RGB observations for action inference. Experiments on LIBERO-Plus, RoboTwin 2.0, and real-world bimanual tasks demonstrate improved manipulation performance, highlighting the benefits of geometric supervision for robot control.

Critical strengths
  1. Well-motivated integration of 3D geometry into world-action modeling. The paper addresses an important limitation of existing WAMs by introducing explicit geometric supervision beyond appearance-based video prediction. Jointly modeling RGB and geometric dynamics is a reasonable and effective approach.
  2. Effective geometric representation learning. GeoAE provides a practical solution for compressing high-dimensional 3D foundation model features while preserving geometric information. The comparisons against depth prediction, raw VGGT features, and feature-only autoencoders support the effectiveness of the proposed representation.
  3. Efficient inference design. Despite introducing additional geometric modeling components during training, GeoAct avoids online geometry extraction and future video generation during inference. This enables relatively low inference latency, making the approach practical for closed-loop robotic control.
  4. Good simulation evaluation. The paper evaluates GeoAct on LIBERO-Plus and RoboTwin 2.0, covering different manipulation tasks and environmental perturbations.
Critical weaknesses
  1. Insufficient evidence linking geometric learning to action improvement. While GeoAct demonstrates good geometric reconstruction and downstream manipulation performance, the paper does not clearly establish how the learned geometric representations contribute to action prediction. Since the Action Expert does not directly attend to geometric tokens, the mechanism by which geometric supervision improves policy behavior remains unclear. More direct evidence connecting geometric prediction accuracy with action quality would strengthen the central claim.
  2. Substantial training complexity with limited performance gains. GeoAct introduces an additional 1B-parameter Geometry Expert, a 330M-parameter GeoAE, and a VGGT-Omega teacher, requiring multiple stages of pretraining. Despite this considerable overhead, its improvement over the strong π0.5 baseline is marginal on LIBERO-Plus and relatively modest in real-world experiments. The absence of training-time or computational cost comparisons makes it difficult to assess whether these gains justify the additional complexity.
  3. Limited real-world validation. The real-world evaluation covers only three bimanual tasks, with 20 trials per task. GeoAct achieves only three additional successful trials compared to π0.5 across all 60 trials, without statistical significance analysis or confidence intervals. Moreover, the experiments lack systematic evaluations under camera viewpoint changes, object layout variations, and unseen configurations, which are particularly relevant to the claimed improvements in spatial understanding.
  4. Limited analysis of the proposed geometric representation. Although GeoAE outperforms several alternative prediction targets, the paper does not sufficiently isolate the contributions of its individual components, such as depth reconstruction, camera reconstruction, and multi-layer feature compression. It remains unclear whether a simpler geometric representation or a more lightweight supervision mechanism could achieve comparable performance.
Questions for authors

See weaknesses. I am especially curious about the evidence about how geometric representation improves the action learning. More visualization, interpretability or any other studies (not just SR results) would help. Also, increased training cost with limited improvement and weak real world experiments are also my concerns.

7 comments
AuthorsOpenReview ↗

The Role of Geometry Learning in Action Generation

Response to Reviewer ARsh

We thank the reviewer for the insightful and thorough review, as well as for recognizing the motivation of GeoAct, the GeoAE representation, and our inference efficiency. In response to your four comments regarding mechanism verification, training overhead, real-world generalization, and component ablations, we have conducted additional experiments and analyses, which we detail below. All revisions in the updated manuscript are highlighted in blue.

Corresponding to: W1, Questions

The Action expert does not directly attend to geometric tokens, leaving it unclear how geometric supervision improves policy behavior. Evidence directly linking geometric prediction accuracy to action quality, along with visualizations or interpretability analyses beyond success rates, would be helpful.

Excluding geometric tokens from the Action expert's attention is an intentional design: it eliminates online 3D encoding at deployment and ensures the low latency you noted. During training, geometry guides action learning implicitly through the shared current RGB representation: conditioned on the current RGB tokens, the Geometry expert predicts current and future 3D states, and its gradients backpropagate to the RGB expert. Because predicted geometry and executed actions describe the same 3D motion of the robot and objects, this joint supervision encourages the current RGB representation to encode relative robot–object 3D positions and their evolution, which the Action expert reads to generate actions.

To verify this correspondence, we compare two 3D gripper trajectories from the same model: one recovered from decoded geometric predictions and the other produced by executing the predicted actions. The table below reports the cosine similarity of 3D displacements across Stage 2 training:

Stage 2 training stepsCenter trajectory similarity ↑Per-point mean similarity ↑
5k0.8650.582
10k0.8920.616
15k0.9160.819
20k0.9650.838

From 5k to 20k steps, the center trajectory similarity increases from 0.865 to 0.965, and the per-point mean similarity improves from 0.582 to 0.838. This indicates that as joint training progresses, the 3D motion implied by geometric predictions aligns increasingly well with that produced by action execution. We have added this trajectory analysis and 3D visualizations to Appendix E.7 of the revised manuscript.

AuthorsOpenReview ↗

Training Cost of Action-Free Geometric Pretraining

Corresponding to: W2

GeoAct introduces a 1B Geometry expert, a 330M GeoAE, and a VGGT-Ω teacher, requiring multi-stage pretraining; however, the gains over π₀.₅ are small on LIBERO-Plus and limited on the real robot, and a comparison of training time or computational cost is missing.

GeoAct replaces the large-scale action-labeled pretraining required by mainstream baselines with geometric pretraining on action-free videos. All computation from the additional geometric modules is confined to the training phase: VGGT-Ω provides targets, GeoAE compresses them, and the Geometry expert learns their dynamics; all three are removed at deployment.

Our pretraining (comprising GeoAE pretraining and Stage 1) totals 3.31 × 10¹⁹ FLOPs, using only 156 hours of robot videos and 180 hours of human hand videos (SSv2) without action labels. The compute breakdown across stages is as follows:

StageTraining FLOPs
GeoAE pretraining1.656 × 10¹⁹
Stage 11.657 × 10¹⁹
Stage 26.498 × 10¹⁹

Compared with baselines, GeoAct achieves competitive or higher success rates with substantially less pretraining compute. On LIBERO-Plus, GeoAct reaches 84.6% versus 84.4% for π₀.₅, using only 1/324 to 1/41 of the estimated pretraining FLOPs of π₀.₅. The pretraining compute and success rates across methods are compared below:

MethodPretraining FLOPsLIBERO-Plus Success (%) ↑
GeoAct3.31 × 10¹⁹84.6
π₀.₅(1.34–10.74) × 10²¹84.4
π₀(2.17–17.38) × 10²¹53.6
X-WAM(1.71–2.49) × 10²¹75.6
StarVLA7.53 × 10²¹74.1
Spatial Forcing7.53 × 10²¹72.7
LingBot-VA≈ 4.45 × 10²²69.5
Fast-WAM051.5
4D-WAM071.0

The table measures pretraining compute after foundation-model initialization and prior to LIBERO adaptation; baseline values are estimated from their published training configurations. We have added detailed data and compute breakdowns to Appendix B.6.

AuthorsOpenReview ↗

Real-Robot Generalization across Viewpoints and Object Layouts

Corresponding to: W3

Real-robot evaluation only covers three bimanual tasks with 20 trials each; GeoAct succeeds in only 3 more trials out of 60 than π₀.₅, without significance analysis or confidence intervals; systematic evaluations under camera viewpoint changes, object layout variations, and unseen configurations—which directly test spatial understanding—are missing.

Our existing bimanual tasks evaluate coordinated 3D manipulation under a fixed camera; we agree that variations in camera viewpoint and object layout provide a more direct test of spatial understanding. Following your suggestion, we have conducted additional real-robot experiments specifically targeting these two factors.

We design an "Upright-and-Place" task: the right arm grasps a yellow cylinder and places it upright onto an orange block—a task whose success critically depends on estimating the 3D positions of both the cylinder and the block. GeoAct, π₀.₅, and Fast-WAM are fine-tuned on the same 100 demonstrations collected under a single fixed camera pose and within a single initial placement region. At test time, we perturb one factor at a time:

  1. Camera viewpoint: The front camera is moved across a grid of 3 heights (40, 50, 60 cm) × 3 pitch angles (30°, 45°, 60°), yielding 8 unseen test poses excluding the training pose (50 cm, 45°). Viewpoint changes alter the 2D projection of the scene while keeping the underlying 3D structure unchanged.
  2. Far-depth object shift: The cylinder is placed 40–50 cm and 50–60 cm from the robot base, beyond the 30–40 cm training region, along the table projection of the front camera's optical axis. Under this shift, the cylinder moves only slightly upward in the 2D image while its physical depth increases substantially, allowing us to distinguish whether the policy relies on 2D image coordinates or 3D spatial perception.

Each test condition is evaluated in 4 independent runs of 20 initial placements, with all three methods executed once per placement in each run. We report the mean ± SD across the 4 runs. The complete results are posted in the follow-up comment below and added to Appendix D.4 of the revised manuscript.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.