acceptodds
Under review as a conference paper at ICLR 2027

Geometric Action Model: Repurposing a Geometric Foundation Model for Robot Policy Learning

12 views

Abstract

Language-conditioned robot manipulation policies must follow user instructions while reasoning jointly about objects, camera viewpoints, and robot actions in the 3D physical world. Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic and temporal priors from large-scale foundation models. However, these models operate primarily in 2D image space, leaving the 3D geometry essential for contact-rich manipulation only implicitly represented. In this paper, we introduce the Geometric Action Model (GAM), which directly repurposes a pretrained geometric foundation model (GFM) for policy learning. We show that the GFM's rich visual representations and geometric priors provide an effective foundation for manipulation. However, a GFM alone does not natively support future prediction or action generation. GAM bridges this gap by splitting the GFM at an intermediate layer and inserting a causal future predictor between its shallow and deep blocks. The shallow blocks encode observations into geometric features, while the predictor integrates language instructions, proprioception, and action history to forecast future geometric features and action representations. These predictions are then processed by the remaining pretrained GFM blocks to reconstruct future geometry and decode robot actions, with future-feature and future-depth losses explicitly supervising geometric forecasting. This design unifies geometric perception, future prediction, and action decoding within a shared pretrained backbone. Ablations show complementary benefits from geometric initialization and future-prediction supervision for out-of-distribution robustness. Across diverse simulation and real-robot benchmarks, GAM achieves competitive success rates while requiring smaller model size and lower inference latency than representative foundation-model-scale baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Robotics, 3D World Model, Learning representations for robotic perception and control, Geometric Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wxJZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Geometric Action Model (GAM), which adapts a pretrained geometric foundation model for language-conditioned robot manipulation. The method inserts a causal future predictor between the shallow and deep layers of the geometric backbone. It predicts future geometric features and action representations, which are then processed by the remaining pretrained layers to produce actions and future depth predictions. Training combines action regression with future-feature and future-depth supervision. Experiments on LIBERO, LIBERO-Plus, RoboCasa-Kitchen, and real-robot tasks show competitive manipulation performance, particularly under camera perturbations, with relatively low inference latency.

Critical strengths

Using a geometric foundation model as the backbone for robot policy learning is an interesting direction. Rather than treating pretrained geometric features as an additional input, GAM integrates future prediction and action generation into the geometric backbone itself. This design could be useful for manipulation tasks where spatial relationships and camera changes matter. Table 2 provides evidence for the importance of geometric initialization and future prediction. On LIBERO-Plus, the DA3-pretrained model achieves 85.5% success with future supervision, compared with 79.3% using DINO initialization. Removing future supervision reduces the DA3-based result to 72.8%. The future-depth evaluation in Appendix B also provides some evidence that the model learns to predict geometric changes.

The robustness and efficiency results are another strength. GAM achieves 83.1% success under camera perturbations on LIBERO-Plus, compared with 73.4% for the strongest baseline. The real-robot experiments are relevant as well, although I have a concern about their evaluation protocol. In terms of efficiency, GAM takes 17.5 ms per action-chunk prediction under the matched runtime setting, compared with 29.2 ms for π0.5 and 70.1 ms for OpenVLA-OFT. The separately reported 6.9 ms optimized deployment latency is also useful. These results suggest that the approach offers a favorable balance between robustness and inference cost.

Critical weaknesses

My main concern is whether the contribution of the deep geometric decoding layers has been sufficiently isolated. One of the paper's architectural claims is that predicted action representations benefit from passing through the remaining pretrained GFM layers instead of being decoded directly from the causal predictor. Table 16 in Appendix C.1 provides a relevant comparison: the direct-action variant achieves 84.1% on LIBERO-Plus-Object, while the full model reaches 90.6%. This is a meaningful difference. However, both variants are initialized from a robot-pretrained GAM checkpoint, and the direct-action variant changes its output pathway only during downstream post-training. I am therefore not sure whether the improvement comes specifically from geometric processing in the deeper layers or partly from the advantage of retaining a decoding pathway already used during pretraining. The ablation supports the full architecture, but it does not completely separate these factors. A comparison in which the direct-action architecture is trained from the corresponding pretraining stage would be more convincing.

I also have concerns about the real-robot evaluation protocol. Appendix A indicates that GAM executes eight actions before receiving the next observation, whereas π0.5 and Spatial Forcing execute ten. Keeping the original configurations of the baselines is understandable for a system-level comparison, but this difference matters when evaluating robustness. A shorter execution horizon allows more frequent observation updates and potentially faster correction of action errors, especially during contact-rich manipulation or under camera perturbations. As a result, it is difficult to attribute the real-robot performance differences entirely to the geometric representations. I would like to see a comparison using the same execution horizon, or at least an analysis of how sensitive the results are to this setting. Since each real-robot task is evaluated with 20 trials per condition, reporting the underlying success counts and uncertainty estimates would also help establish the reliability of the observed improvements.

Questions for authors
  1. In Table 16, the direct-action variant is initialized from a robot-pretrained GAM checkpoint. Has the direct-action architecture also been evaluated when trained from the corresponding robot pretraining stage? If not, how can the contribution of the deep geometric decoder be separated from the advantage of the pretrained decoding pathway?

  2. How sensitive are the real-robot results to the action execution horizon? Would GAM maintain its advantage if all methods executed the same number of actions between observations?

  3. Could the authors provide the per-task success counts and explain how the uncertainty estimates in Figure 4 were computed? This would help assess the reliability of the real-robot comparisons.

4 comments
AuthorsOpenReview ↗

Response to the Reviewer wxJZ (1/2)

We thank the reviewer for the positive assessment of our work, for recognizing the value of integrating future prediction and action generation directly into a geometric backbone, and for the precise, constructive suggestions on isolating the deep geometric decoder and analyzing the real-robot evaluation protocol. We address each point below.

[W1, Q1] Isolating the contribution of the deep geometric decoding layers

We thank the reviewer for pointing out this important distinction. We agree that the suggested experiment more directly isolates the contribution of our architectural integration. As the reviewer suggested, we additionally trained the direct-action architecture with its own action pathway from the start of robot pretraining. In this variant, the action token is decoded directly but everything else matches GAM.

[Table R1. Action-decoding pathway, controlled from robot pretraining.]

VariantAction decoded fromParamsLIBERO-Orig.LIBERO-Plus
(a) (NEW) Direct-action, pretrained end-to-end Predictor output1.4B87.859.2
(b) GAM (ours)Deep GFM layers1.4B97.485.5

Variants (a) and (b) have the same parameter count, training data, pretraining budget, and future-feature and future-depth supervision. This makes variant (a) the matched baseline with an independent action decoder trained from the corresponding pretraining stage that the reviewer suggested.

GAM outperforms variant (a), so pretraining the direct-action pathway does not close the gap. This clearly shows the benefit of routing per-view action tokens alongside predicted future geometry tokens through the deep geometric decoding layers to fuse multi-view 3D structure.

We have added this end-to-end pretrained direct-action ablation to Appendix C.1 (Table 23) of the revised manuscript.

[W2, Q2] Sensitivity to the action execution horizon

We thank the reviewer for raising this thoughtful point regarding the real-robot evaluation protocol. As reviewer mentioned, we kept each method’s original trained configurations to not disadvantage the baselines. However, we agree that actions within an open-loop chunk may struggle to react to new observations, so a shorter execution horizon can in principle correct errors sooner.

Nevertheless, the difference is small in wall-clock terms. Our demonstrations are recorded at 15 Hz, so 8 and 10 actions correspond to 0.53 s and 0.67 s between observations, a difference of about 133 ms. We therefore expect a two-action difference to have little effect on error correction, which we verify below.

As the reviewer suggested, we additionally evaluated π0.5 executing the first 8 of its 10 predicted actions, following the practice of OpenVLA-OFT [A]. We conduct this comparison on two new, more challenging real-world tasks (see also R1 [W2-2, Q3]): Task 5 (opens a wooden drawer, places the yellow cube inside, and closes the drawer; 100 demonstrations) and Task 6 (unplugging a power cord and inserting the plug into another socket, a contact-rich task requiring precise alignment; 30 demonstrations). Because of the rebuttal timeline, we evaluate each condition with 12 trials and compare against π0.5, the stronger of the two real-world baselines in the paper. Table R7 reports π0.5 at both horizons on these two tasks.

[Table R7. Additional challenging real-world tasks with matched execution horizon.]

TaskSettingπ0.5 (10 actions)π0.5 (8 actions)GAM (8 actions)
Task 5: Drawer manipulationIn-domain9/129/1210/12
Out-of-domain6/127/128/12
Task 6: Plug relocationIn-domain2/122/126/12
Out-of-domain0/120/125/12

Executing 8 instead of 10 actions leaves π0.5’s pooled in-domain success unchanged and only slightly improves its success under the camera shift. GAM maintains its advantage at the matched horizon.

We have added this matched execution-horizon sensitivity analysis to §5.2 and Appendix A.5 (Table 13) of the revised manuscript.

[A] Kim, Moo Jin, Chelsea Finn, and Percy Liang. "Fine-tuning vision-language-action models: Optimizing speed and success." RSS’25.

AuthorsOpenReview ↗

Response to the Reviewer wxJZ (2/2)

[W2, Q3] Per-task evaluation.

Table R8 reports per-task success counts for all methods. The error bars in Figure 4 were mislabeled: they show the minimum and maximum success rates across two sessions of 10 trials each, not the standard deviation stated in the caption. We changed the bars to this range but did not update the caption, which still says standard deviation. We apologize for this error and have corrected the caption of Figure 4 and added the full per-task and per-session success counts to Appendix A.4 (Table 11) of the revised manuscript.

[Table R8. Per-task results.] Each task and camera condition is evaluated in two sessions of 10 trials. Entries report (S1+S2)/N and the pooled success rate.

Real-World TaskSettingπ0.5Spatial ForcingGAM (Ours)
Task 1: Pick & PlaceID(8+8)/20 (80%)(8+9)/20 (85%)(10+10)/20 (100%)
OOD(7+6)/20 (65%)(7+6)/20 (65%)(8+9)/20 (85%)
Task 2: Stack Milk & CubeID(7+6)/20 (65%)(3+4)/20 (35%)(8+7)/20 (75%)
OOD(1+2)/20 (15%)(1+1)/20 (10%)(6+5)/20 (55%)
Task 3: Pot & Pan on CooktopID(2+3)/20 (25%)(1+0)/20 (5%)(3+4)/20 (35%)
OOD(1+0)/20 (5%)(0+0)/20 (0%)(2+2)/20 (20%)
Task 4: Insert Cube in Covered PotID(5+3)/20 (40%)(2+4)/20 (30%)(5+6)/20 (55%)
OOD(3+3)/20 (30%)(1+2)/20 (15%)(4+3)/20 (35%)
All Four TasksID Total(22+20)/80 (52.5%)(14+17)/80 (38.75%)(26+27)/80 (66.25%)
OOD Total(12+11)/80 (28.75%)(9+9)/80 (22.5%)(20+19)/80 (48.75%)

Dear reviewer wxJZ,

We sincerely appreciate your time and thoughtful efforts in reviewing our manuscript. As the discussion period will end in three days, we would like to gently remind you in case you have any remaining comments. We believe that we have sincerely and successfully addressed your concerns, supported by the corresponding additional experimental results. If you have any questions or concerns, please do not hesitate to let us know. We remain fully available for further discussion and are prepared to conduct additional experiments to address any remaining concerns during the discussion period.

Best regards,

Authors

Dear Reviewer wxJZ,

Thank you very much for your reply and for confirming that our additional experiments and clarifications have resolved all of your concerns. We truly appreciate your time, constructive engagement, and positive support of our work, which have made our paper much stronger.

Best regards,

Authors

Reviewer_q2CnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geometric Action Model (GAM) for language-conditioned robot manipulation. The method inserts a causal predictor between the shallow and deep layers of a geometric backbone. Predicted future geometry and action representations then pass through the remaining deep layers. Training uses action, future-feature, and future-depth losses. The authors evaluate task performance, camera robustness, and inference efficiency in simulation and on real robots.

Critical strengths
  1. 3D geometry is useful for manipulation. Using pretrained geometric representations to support action learning is well motivated.
  2. Experiments show robustness to camera perturbations, competitive results across simulation and real robot tasks, and low model inference latency.
Critical weaknesses
  1. Evaluation with geometric backbones beyond DA3, such as VGGT, is missing.
  2. RoboCasa evaluation covers only 24 tasks. A matched comparison with π0.5 on the official RoboCasa365 multi-task benchmark, especially composite tasks, would better assess performance on complex tasks.
  3. DA3 provides geometric priors, while future prediction relies on an added predictor. The benefit of using DA3 as the policy backbone rather than a geometric feature encoder remains unclear.
Questions for authors
  1. It is unclear whether the gains extend to other geometric backbones, such as VGGT.
  2. A matched comparison with π0.5 on the full official RoboCasa365 multi-task benchmark would help assess performance on complex composite tasks.
  3. I would like to see a matched comparison with π0.5 using DA3 as a geometric encoder and the same future geometry supervision. This would clarify the benefit of using DA3 as the policy backbone.
4 comments
AuthorsOpenReview ↗

Response to the Reviewer q2Cn (1/2)

We thank the reviewer for the constructive feedback and for recognizing the clear motivation of using 3D geometric representations for manipulation, as well as GAM’s robustness to camera perturbations, competitive empirical performance, and low inference latency. We address the questions about backbone transfer, additional benchmark evaluation, and the distinction between a geometric backbone and an auxiliary geometric encoder below.

[W1, Q1] Evaluation with geometric backbones beyond DA3, such as VGGT, is missing.

We thank the reviewer for this suggestion. GAM’s recipe is not specific to DA3. To verify that the gains extend beyond DA3, we instantiated GAM with VGGT-Ω\Omega[A], a GFM from the VGGT family, using the same causal predictor, losses, data, and pretraining budget. We split VGGT-Ω\Omega at layer Ls=4L_s=4, following the same protocol as GAM-DA3, which is selecting LsL_s as the layer where the alternating attention begins.

[Table R5. Alternative Geometric Backbone.]

Backbone VariantLIBERO-Orig. (All)LIBERO-Plus (All)
GAM-DINO (Table 2)94.9%79.3%
GAM-VGGT$\Omega$97.3%83.8%
GAM-DA3 (Default Backbone)97.4%85.5%

GAM with VGGT-Ω\Omega improves over the DINO-initialized control on both benchmarks, showing that the benefits extend beyond DA3. While VGGT-Ω\Omega and DA3 perform similarly on the original LIBERO benchmark, DA3 remains stronger on LIBERO-Plus. These results support the applicability of our framework to another geometric backbone, while showing that backbone choice still affects robustness.

We have added this alternative geometric backbone evaluation and discussion to §5.4 (Table 8) of the revised manuscript.

[A] Wang, Jianyuan, et al. "VGGT-Ω\Omega." CVPR’26.

[W2, Q2] RoboCasa365 benchmark

Following the reviewer’s suggestion, we additionally evaluate GAM on the official RoboCasa365 multi-task benchmark in Table R6, focusing on 23 tasks that do not require mobile manipulation. Our focus is fixed-base manipulation, and mobile manipulation is beyond the scope of this work.

[Table R6. Additional Robocasa365 evaluation.]

Split# Tasksπ0.5GAM (ours)
Atomic1446.648.4
Composite-Seen518.819.2
Composite-Unseen42.01.0
Average2332.7833.83

This evaluation complements our RoboCasa-Kitchen results by extending the comparison to both atomic and composite manipulation tasks. GAM achieves a slightly higher average success rate than π0.5, showing that it remains competitive across these tasks while requiring a smaller model size and lower inference latency.

While we leave further gains on complex composite tasks via high-level reasoning, hierarchical system design, or RL post-training for future work, these results demonstrate that GAM is already competitive as a standalone learned policy.

We have added the additional RoboCasa365 evaluation to Appendix A.3 (Table 10) of the revised manuscript.

AuthorsOpenReview ↗

Response to the Reviewer q2Cn (2/2)

[W3, Q3] The benefit of using DA3 as the policy backbone rather than a geometric feature encoder remains unclear.

We appreciate this question. To clarify, GAM uses DA3 both as a geometric feature encoder and as the policy backbone, and we kindly refer the reviewer to the two variants in Table 2 that directly compare this against using DA3 as a geometric feature encoder alone.

DA3 as a geometric feature encoder. The last row of Table 2 uses DA3 as a feature encoder: an action head decodes actions from DA3 features, without the causal future predictor. It reaches 91.8% on LIBERO and 65.8% on LIBERO-Plus. With DA3 serving as the policy backbone and the predictor inside it (GAM), success rises to 97.4% and 85.5%. GAM with VGGT-Ω\Omega also outperforms the DINO-initialized control on both benchmarks and achieves comparable performance to DA3 on LIBERO, supporting the use of geometric backbones beyond DA3 (see Table R5 of W1/Q1).

DA3 as the backbone. With the same predictor and losses, replacing DA3 with DINOv2 lowers success to 94.9% and 79.3%. DA3 therefore matters as the backbone, not only as a source of features.

Comparison with π0.5 + DA3 (Q3). GAM decodes actions directly from the GFM, which serves as the policy backbone without a VLM. π0.5 with DA3 would instead be a VLM-based policy that uses DA3 as an auxiliary encoder, which is a different paradigm rather than a variant of GAM. Integrating DA3 into π0.5 would also require re-training its VLM and vision-language alignment from the pretraining stage, so a matched comparison is simply not possible.

The future-geometry supervision cannot be carried over either. When DA3 is used only as an encoder, nothing forecasts future geometry, so the same supervision cannot be applied. The closest alternative is current-frame depth supervision, which is what the last row of Table 2 uses. The closest existing instances of the VLM-plus-GFM paradigm are the geometry-aware VLAs in Table 1. These pair a VLM with a GFM, either through alignment or as an encoder with latent future prediction. GAM outperforms them on LIBERO and LIBERO-Plus.

Dear reviewer q2Cn,

We sincerely appreciate your time and thoughtful efforts in reviewing our manuscript. As the discussion period will end in three days, we would like to gently remind you in case you have any remaining comments. We believe that we have sincerely and successfully addressed your concerns, supported by the corresponding additional experimental results. If you have any questions or concerns, please do not hesitate to let us know. We remain fully available for further discussion and are prepared to conduct additional experiments to address any remaining concerns during the discussion period.

Best regards,

Authors

AuthorsOpenReview ↗

Follow-up with Reviewer q2Cn: Summary of Additional Experiments

Dear Reviewer q2Cn,

​As the discussion period is coming to a close very soon, we would greatly appreciate it if you could briefly revisit our responses and let us know whether they have adequately addressed your concerns. In our rebuttal, we performed the exact evaluations and analyses you suggested, including:

​1. Evaluating GAM with an alternative geometric backbone (VGGT-Ω\Omega) to demonstrate that our framework generalizes beyond DA3 (Table R5).

​2. A matched comparison with π0.5\pi_{0.5} on the official RoboCasa365 benchmark, covering both atomic and composite tasks (Table R6).

​3. Clarifying the role of DA3 as the policy backbone versus a standalone feature encoder. (Table 2 and R5).

​We would love to hear your thoughts on these results and whether they resolve your questions, or if there is anything else we can clarify before the deadline. Thank you again for your time and consideration.

​Best regards,

Authors

Reviewer_b72hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Geometric Action Model (GAM), a framework that repurposes a pretrained geometric foundation model (GFM) for robot policy learning. The key idea is to insert a causal future predictor between the shallow and deep layers of a GFM, enabling the model to jointly predict future geometric representations and robot actions. The predicted representations are subsequently processed by the remaining pretrained GFM layers for geometry and action decoding. The framework is trained with action regression, future-feature prediction, and future-depth supervision. The authors evaluate the proposed method on LIBERO, LIBERO-Plus, RoboCasa-Kitchen, and four real-world manipulation tasks. The results demonstrate competitive success rates, improved robustness to camera viewpoint perturbations, and relatively low inference latency compared with several existing VLA and world-action-model baselines.

Critical strengths
  1. Reasonable architectural design and effective integration of geometric representations.

The paper presents an interesting approach to incorporating pretrained geometric representations into robot policy learning. Rather than treating the GFM solely as a visual feature extractor, the proposed architecture reuses its deeper layers for action decoding. The overall framework is technically reasonable and clearly presented.

  1. Comprehensive simulation evaluations and promising empirical performance.

The authors conduct extensive experiments across multiple simulation benchmarks and include comparisons with representative VLA and world-action-model approaches. GAM achieves competitive success rates, particularly under camera viewpoint perturbations, while maintaining relatively low inference latency. The paper also includes several ablations to investigate its architectural components.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification of the proposed architecture.

My primary concern is the relatively incremental methodological contribution. The proposed framework mainly integrates an existing geometric foundation model with a causal future predictor and auxiliary geometric supervision. While the integration is reasonable, I am not fully convinced that it represents a sufficiently significant methodological advance over existing geometry-aware robot policies and latent world-modeling approaches.

More importantly, the empirical evidence does not fully isolate the benefits of the proposed architectural design. For example, although Table 16 demonstrates that propagating action representations through the deeper GFM layers improves performance compared with direct action supervision, both variants are initialized from the same GAM-pretrained checkpoint. This comparison does not fully establish whether the proposed architecture provides a substantial advantage over a separately trained, capacity-matched action decoder using the same pretrained geometric representations and supervision.

Given that the architectural integration is the central contribution of the paper, I would expect more convincing evidence demonstrating why this particular design is preferable to simpler alternatives.

  1. Limited experimental evidence on sufficiently challenging manipulation tasks.

Although the paper provides extensive simulation evaluations, I am less convinced by the practical significance of the reported improvements.

First, several key architectural ablations are conducted primarily on LIBERO-Object and LIBERO-Plus-Object, where strong existing methods already achieve relatively high success rates. For example, π0.5 and Cosmos-Policy achieve 89.9% and 88.3% on LIBERO-Plus-Object, respectively, compared with 90.6% for GAM. These results suggest that the benchmark may have limited discriminative power for evaluating incremental improvements among strong policies.

In contrast, the proposed method achieves substantially lower success rates on LIBERO-Plus-Goal (80.4%) and LIBERO-Plus-Long (78.0%). I would find the architectural claims more convincing if the key ablations were also conducted on these more challenging suites, where the benefits and limitations of the proposed design could be better assessed.

Second, the real-world evaluation is relatively limited. Although the authors evaluate four manipulation tasks, these tasks are mainly constrained tabletop object manipulation scenarios, involving picking, placing, stacking, and relatively simple multi-step interactions. The evaluation does not sufficiently demonstrate how the proposed framework generalizes to more diverse, challenging, or less structured real-world manipulation settings.

Overall, while the experimental evaluation is reasonably complete for a simulation-focused robotics paper, I do not find the current results sufficiently compelling to establish the broader practical significance of the proposed method.

Questions for authors
  1. Architectural contribution: Can the authors provide a comparison with a separately trained, capacity-matched architecture that uses the same GFM features and future-prediction supervision but employs an independent action decoder instead of propagating action representations through the deeper GFM layers? This would help isolate the contribution of the proposed architectural integration.

  2. Ablations on challenging benchmarks: Why are most architectural ablations conducted on LIBERO-Object and LIBERO-Plus-Object? Could the authors provide key ablations on LIBERO-Plus-Goal and LIBERO-Plus-Long to demonstrate that the proposed components remain beneficial on more challenging manipulation tasks?

  3. Real-world generalization: Can the authors provide additional evidence that the geometric representations contribute to robustness beyond the evaluated external-camera perturbation? For example, experiments involving more challenging manipulation tasks or ablations of the individual camera inputs would help clarify the practical benefits of the approach.

6 comments
AuthorsOpenReview ↗

Response to the Reviewer b72h (1/4)

We thank the reviewer for a careful review and for finding the framework technically reasonable and clearly presented. We agree that isolating why this design outperforms simpler alternatives is the key question, and we ran the controlled comparison the reviewer suggests.

[W1,Q1] Justification of the proposed architecture.

We agree that the suggested experiment more directly isolates the contribution of our architectural integration. Following the reviewer’s suggestion, we additionally trained the direct-action architecture with its own action pathway from the start of robot pretraining. In this variant, the action token is decoded directly from the causal predictor output, so the action decoder is independent of the deep GFM layers. All other components and training settings are identical to GAM.

[Table R1. Action-decoding pathway, controlled from robot pretraining.]

VariantAction decoded fromParamsLIBERO-Orig.LIBERO-Plus
(a) (NEW) Direct-action, pretrained end-to-end Predictor output1.4B87.859.2
(b) GAM (ours)Deep GFM layers1.4B97.485.5

Variants (a) and (b) have the same parameter count, training data, pretraining budget, and future-feature and future-depth supervision. This makes variant (a) the matched baseline with an independent action decoder that the reviewer suggested.

GAM outperforms variant (a), so pretraining the direct-action pathway does not close the gap. Because both variants are pretrained with their own action pathways and have the same capacity, the remaining gap comes from processing the action through the deep GFM layers, where DA3's alternating frame-wise and cross-view global attention begins. While variant (a) decodes actions before cross-view fusion, GAM routes per-view action tokens alongside predicted future geometry tokens through the remaining GFM blocks, allowing action representations to fuse multi-view 3D geometric features and attend directly to manipulated objects and contact regions (Figure 12) as the deep layers refine 3D structure.

We have added this controlled comparison to Appendix C.1 (Table 23) of the revised manuscript.

AuthorsOpenReview ↗

Response to the Reviewer b72h (2/4)

[W1] Methodological novelty

Existing geometry-aware policies use the GFM as an alignment target for a VLA (Spatial Forcing[A], ROCKET[B]) or as an encoder for a separate action head (GP3[C], VGGT-DP[D]), and concurrent work still decodes actions with a separate decoder (VGA[E]). Existing latent world-modeling approaches (e.g., FLARE[F], Fast-WAM[G], WorldVLA[H], and GWM-VLA[I] in Tables 1 and 3) either forecast future latents in 2D image/video spaces that lack viewpoint-consistent 3D structure, or rely on a separate world-model and action-decoding backbone outside the GFM. Just as VLAs inherit the priors of VLMs and WAMs inherit those of video diffusion models, GAM instead repurposes the GFM as the policy itself. Simply attaching an action decoder to a GFM is not enough: despite its strong geometric priors, this variant reaches only 91.8% on LIBERO and 65.8% on LIBERO-Plus, compared with 97.4% and 85.5% for GAM (Table 2, last row). This is because a GFM alone provides no priors for language grounding, action decoding, or temporal future prediction.

Our architectural design integrates these missing pieces into the GFM itself through three coupled mechanisms: (1) splitting the GFM at an intermediate layer between its shallow visual-encoding blocks and deep cross-view geometric blocks, (2) inserting a causal future predictor that fuses language, proprioception, and action history with shallow GFM features to jointly forecast future geometric and action tokens, and (3) routing the predicted geometric and per-view action tokens through the remaining deep GFM layers for joint 3D geometry and action decoding. Complementing this architecture is our future-prediction supervision (the second contribution in L103). Its future-feature and future-depth losses train the predictor to forecast geometry in the GFM’s own representation space. Without them, LIBERO-Plus success drops from 85.5% to 72.8%, with a similar drop under a DINOv2 backbone (79.3% to 68.1%; Table 2). They also make GAM’s predicted future depth more accurate (Table 22 in the revised manuscript). Together, these components are what enable a GFM to serve as a competitive, fast, and efficient policy. To our knowledge, GAM is among the first works to repurpose a GFM in this way, and Table R1 tests whether this specific design choice matters.

The result is a new direction for policy networks. With 1.4B parameters, GAM achieves highly competitive performance compared to the existing VLAs, WAMs and geometry-aware VLAs, with the lowest inference latency among the benchmarked policies. Like VLAs and WAMs when they were first introduced, GAM proposes a new way to build a policy from a pretrained foundation model, and we believe this conceptual contribution is valuable in its own right.

[A] Li, Fuhao, et al. "Spatial forcing: Implicit spatial representation alignment for vision-language-action model." ICLR’26.

[B] Sun, Guoheng, et al. "ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models." arXiv’26.

[C] Qian, Quanhao, et al. "Gp3: A 3d geometry-aware policy with multi-view images for robotic manipulation." ICRA’26.

[D] Ge, Shijia, et al. "Vggt-dp: Generalizable robot control via vision foundation models." AAAI’26.

[E] Song, Zijian, et al. "Robotic Manipulation is Vision-to-Geometry Mapping (f(v)→Gf (v)\rightarrow G): Vision-Geometry Backbones over Language and Video Models." ACMMM’26.

[F] Zheng, Ruijie, et al. "Flare: Robot learning with implicit world modeling." CORL’25.

[G] Yuan, Tianyuan, et al. "Fast-wam: Do world action models need test-time future imagination?." NeurIPS’26.

[H] Cen, Jun, et al. "Worldvla: Towards autoregressive action world model." NeurIPS’26.

[I] Zhao, Yanping, et al. "GWM-VLA: Geometry-aware latent world modeling for vision-language-action learning." arXiv’26.

AuthorsOpenReview ↗

Response to the Reviewer b72h (3/4)

[W2-1, Q2] Ablations on the harder suites

We thank the reviewer for this constructive suggestion. Our post-training ablations span 19 variants (Tables 4, 6, 7, 16, 17 of original manuscript), each requiring its own post-training run. To fit our compute budget, we ran them on a single suite. However, our main ablation (Table 2), was not limited to Libero-Object: it was evaluated on all four suites, and the paper reported only the averages.

We agree that Goal and Long, where every method has the most headroom, can be more informative suites for ablations. We additionally report the key ablations on Spatial, Goal, and Long in addition to Object. See Table R3.

For pretraining-level ablations, Table R2 breaks the same runs down by suite.

Table R2. Matched-budget pretraining ablations by LIBERO-Plus suite (success %).

BackboneCausal Future PredictorL_futureObjectSpatialGoalLongOverall
DINO-Initialized✓✓86.787.477.266.379.3
✓✗77.564.268.762.068.1
DA3-Pretrained✓✓90.693.480.478.085.5
✓✗82.266.971.570.472.8
✗✗85.778.451.748.465.8

Table R2 shows that both geometric initialization and future-prediction supervision remain beneficial on Goal and Long. Removing the causal future predictor together with future supervision causes particularly large performance drops on these suites, supporting the importance of the combined design on more challenging tasks.

For Post-training ablations, we additionally extend selected key architectural ablations to Spatial, Goal, and Long in Table R3. For this ablation, we use the same number of training steps and fixed post-training budget across all suites.

[Table R3. Key post-training component ablation on LIBERO-Orig. and LIBERO-Plus.]

VariantLIBERO-Orig.-ObjectLIBERO-Orig.-Spatial (new)LIBERO-Orig.-Goal (new)LIBERO-Orig.-Long (new)LIBERO-Plus-ObjectLIBERO-Plus-Spatial (new)LIBERO-Plus-Goal (new)LIBERO-Plus-Long (new)LIBERO-Orig.-Average (new)LIBERO-Plus-Average (new)
GAM (full)99.095.094.492.290.6078.3562.8769.9995.1575.45
GAM, post-training w/o L_future (Table 4)98.691.488.287.489.5078.1461.4063.5291.4073.14
GAM, split layer Ls=12L_s = 12 (Table 7)99.694.489.686.670.1072.8660.4861.7392.5566.29
GAM, split layer Ls=19L_s = 19 (Table 7)95.686.683.659.263.4068.4447.3635.0581.2553.56

While removing future-prediction supervision has little effect on Object, it causes noticeable drops on the more challenging suites, particularly on Goal and Long in LIBERO-Orig. and on Long in LIBERO-Plus. Under the split-layer ablation, moving the predictor to a later layer (Ls=19L_s = 19) causes even larger performance drops across all four suites on both LIBERO-Orig. and LIBERO-Plus, with the steepest degradation on Long. These results extend our architectural analysis beyond Object and support the selected design choices on the more challenging suites.

We have added these per-suite pretraining and extended post-training ablation results and discussions to Appendix A.6 (Tables 14 and 15) of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive feedback. The reviewers recognized several strengths of our work: that using pretrained geometric representations for action learning is well motivated (q2Cn) and an interesting direction (wxJZ), that the overall framework and architectural integration are technically reasonable and clearly presented (b72h, wxJZ), and that GAM achieves strong robustness to camera viewpoint perturbations with competitive task success rates and low inference latency across simulation and real-robot benchmarks (b72h, q2Cn, wxJZ).

We sincerely apologize for the delay in posting our response, as completing the additional large-scale robot pretraining, simulation benchmarks, and real-world robot experiments required substantial time.

Based on the reviewers' feedback, we have conducted additional experiments and revised the paper as follows with all changes highlighted in blue :

  • Alternative geometric backbone evaluation with VGGT-Ω (§5.4, Tab. 8)
  • Additional multi-task evaluation on the RoboCasa365 benchmark (§A.3, Tab. 10)
  • Clarified real-world task setup, and full per-task/per-session real-world success counts (§5.2, §5.3, §A.4, Fig. 4, Tab. 11)
  • Two additional challenging real-world tasks and an external-camera-only ablation (§5.2, §A.5, Fig. 9, Tab. 12)
  • Real-world action execution-horizon sensitivity analysis (§A.5, Tab. 13)
  • Per-suite LIBERO-Plus breakdowns for pretraining ablations and extended post-training ablations across Spatial, Goal, and Long (§A.6, Tab. 14, 15)
  • Controlled end-to-end pretraining ablation of the direct-action decoding pathway (§C.1, Tab. 23)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.