acceptodds
Withdrawn from ICLR 2027

GeoMem-VLA: Internalizing 3D Scene Understanding into Temporal Memory for Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Geometric structure provides a direct description of the 3D world and is essential for robotic manipulation and world understanding. Most VLA models primarily rely on 2D representations of the current observation, while geometry-aware approaches often require additional 3D geometric information. However, such geometric information, typically inferred independently from individual observations, is often discarded rather than accumulated into a persistent task representation. Our key insight is that action generation does not require a complete, renderable 3D reconstruction at every step; instead, a policy needs a compact, task-relevant geometric abstraction that can be continuously updated over time. Based on this insight, we introduce GeoMem-VLA, a VLA framework that internalizes explicit 3D supervision into dedicated geometric tokens, without directly aligning the VLM's native visual embeddings to the teacher feature space, and maintains them as temporal memory for action generation. During training, a frozen geometry foundation model supervises learnable GeoPlan tokens through a geometric flow-matching objective over the current scene point map and future end-effector tracks. These tokens are subsequently integrated with the memory of observation representations, where historical information is retrieved, adaptively fused with current observations, and consolidated to keep event frames as the task progresses. At inference, the policy retrieves historical representations and fuses them with the current working representation to predict actions, without invoking the 3D teacher or explicitly reconstructing the scene. After benchmark-specific downstream adaptation, GeoMem-VLA achieves 98.9% average success on LIBERO and 91.10% / 91.24% on the clean/randomized settings of the RoboTwin benchmark. On the real-world tasks, GeoMem-VLA achieves 78.0% success score. These results indicate that internalizing and remembering 3D scene understanding is a promising alternative to repeatedly reconstructing explicit geometry for robotic control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Reconstruction, VISION-LANGUAGE-ACTION MODELS

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kvfbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoMem-VLA, which combines geometrically supervised latent representations with causal episodic memory for robotic manipulation. The method uses an auxiliary flow matching objective, supervising learnable GeoPlan tokens via scene point maps generated by VGGT and future end-effector trajectories from demonstrations. The model stores these representations alongside observation features in memory, conditioning action generation through attention retrieval and gated fusion. At inference time, the geometric teacher model and reconstruction heads are no longer used. Experiments report an average success rate of 98.9% on LIBERO, 91.10%/91.24% on RoboTwin Clean/Randomized, and 78.0% on five real-world tasks.

Critical strengths

Dedicated GeoPlan tokens provide a clear interface between scene/motion supervision and temporal memory, while removing the teacher model and reconstruction heads from the deployment pipeline. This offers a valuable architectural path for policies to leverage geometric information without explicitly reconstructing scenes during action generation.

The experiments cover a broad scope and provide meaningful component ablations. The paper evaluates four LIBERO task suites, 50 RoboTwin tasks across two settings, and five real-robot tasks across two platforms. In Tables 3–4, the full model consistently outperforms variants without the geometric branch or temporal memory. In particular, removing memory drops the success rate on LIBERO-Long from 96.6% to 94.2%, supporting the value of historical information for actual control.

Critical weaknesses

The experiments do not sufficiently isolate the core contribution of "remembering geometric information." The geometric branch ablation in Section 4.3 simultaneously removes scene and end-effector trajectory supervision, GeoPlan tokens, geometric memory, and the fusion module; the memory ablation turns off historical retrieval across all representation streams. While these comparisons show that the combined module is effective, they cannot disentangle whether gains come from historical geometric information, current geometric conditioning, auxiliary motion prediction, or standard observation memory. The offline reconstruction and retrieval analyses in Section 4.4 indicate that decodable information exists within the representations, but they do not prove that historical geometric information actually influences subsequent action decisions. Real-robot experiments also lack component ablations that could address this attribution issue.

Several key implementation and evaluation details remain incomplete. Section 3.2 mentions transforming end-effector trajectories with physical scales into the VGGT point map coordinate system, but it does not explain the specific coordinate and scale alignment procedure. Appendix A provides configuration templates and explicitly states that checkpoint settings and command-line overrides take higher priority, yet it does not provide the final actual configurations. The frame sampling method for event-aware training and the task-adaptive execution horizon are also insufficiently detailed. These missing details hurt reproducibility and make it difficult for readers to judge whether performance gains stem from the proposed model under identical action execution setups.

Questions for authors
  1. Is it possible to keep the current GeoPlan tokens, corresponding supervision, and observation memory, while turning off only the retrieval of historical geometric information? Further disentangling the individual contributions of scene point map supervision versus future end-effector trajectory supervision would also help clarify whether performance gains truly depend on learning and remembering scene geometry.

  2. How exactly are the end-effector positions in demonstrations aligned with the VGGT point maps, including coordinate transformations, scale alignment, and joint normalization? What calibration or additional information is required for this process? When reporting end-effector trajectory errors, how are predictions converted back to coordinates with physical scale?

  3. Please provide the actual configuration used for the checkpoints that produced the reported results, especially the number of GeoPlan tokens, geometric loss weights, the event frame sampling method, and the task-adaptive execution horizon rules. How was the execution horizon chosen? Were identical rules and tuning protocols applied to the full model and all ablated variants?

2 comments

Thank you for the review. We added closed-loop ablations and implementation details.

Q1. Isolating geometric history and supervision sources.

We separately trained variants without GeoPlan history or observation history, disabling the corresponding retrieval during both training and evaluation. Both retain current observations, current GeoPlan tokens, and geometric fusion. The full model and both variants use aligned EEF supervision.

Table R1. LIBERO success rates (%), 50 trials per task.

ConfigurationSpatialObjectGoalLongAvg.
Full99.699.899.696.698.9
w/o GeoPlan history98.899.699.292.897.6
w/o observation history99.0100.099.093.297.8
EEF-only98.699.299.095.498.1
Scene-only98.299.699.495.898.3

Removing GeoPlan history lowers average success by 1.3 percentage points (pp) and Long by 3.8 pp, supporting its additional value when current geometry and observation history remain available.

Joint supervision also outperforms EEF-only and Scene-only. Relative to EEF-only, it improves Spatial by 1.0 pp and Long by 1.2 pp, supporting a contribution from scene supervision beyond future-motion prediction.

Offline reconstruction diagnoses decodable content; these ablations assess the contribution of historical pathways and supervision through closed-loop success.

Q2. Coordinate alignment, required information, and physical-scale errors.

Trajectory coordinate transformation is a well-established engineering method, so we haven't detailed it in the method section of this article. We will be releasing the code, at which point you can see the detailed process in the code; here I'll just give a brief overview. Thank you for your question; we will add more details in a future appendix.

(a) Metric EEF positions. We use recorded metric EEF states where available. For joint-state demonstrations, forward kinematics uses the robot model, required tool transform, and base-to-reference transform. Both arms are expressed in a common frame. For simulated demonstrations, replay recovers camera extrinsics and metric scene depth, with consistency checks against the recorded states.

(b) Current-camera frame and teacher scale. Let pt+kwp^w_{t+k} be a metric future EEF position. All future waypoints use the camera transform at the current observation:

pt+kc=Rtpt+kw+bt.p^c_{t+k}=R_t p^w_{t+k}+b_t.

Teacher depth and metric scene depth are paired at corresponding surface anchors. The teacher-units-per-meter scale is estimated by

st=median⁡j∈Vtzt,jvdt,jc.s_t=\operatorname{median}_{j\in\mathcal{V}_t}\frac{z^v_{t,j}}{d^c_{t,j}}.

Rotation and translation come from camera calibration; only the scalar scale is estimated from the teacher output.

(c) Shared normalization. Let μt\mu_t and rtr_t be the centroid and RMS radius of the pooled teacher point map. Scene and EEF targets share these parameters:

yt,iscene=qt,i−μtrt,yt,keef=stpt+kc−μtrt.y^{\mathrm{scene}}_{t,i}=\frac{q_{t,i}-\mu_t}{r_t},\qquad y^{\mathrm{eef}}_{t,k}=\frac{s_t p^c_{t+k}-\mu_t}{r_t}.

This preserves the trajectory's placement relative to the scene. Appendix A.1 specifies depth validity, anchor-count, and scale-consistency checks. Invalid samples receive zero EEF-loss weight.

(d) Recovering physical units. Inverting the normalization and camera transform gives

p^t+kw=Rt−1(rty^t,keef+μtst−bt).\hat p^w_{t+k}=R_t^{-1}\left(\frac{r_t\hat y^{\mathrm{eef}}_{t,k}+\mu_t}{s_t}-b_t\right).

EEF errors can then be computed in meters and converted to millimeters. Appendix D reports EEF RMSE in millimeters and scene RMSE in normalized coordinates.

Constructing these training targets requires camera calibration, metric depth, and robot states or kinematics. Deployment uses RGB, robot state, and latent memory, without these target-construction inputs or online teacher/decoder execution.

Q3. Configurations, event frames, and execution horizons.

Appendix A/Table 5 reports training updates, grouped learning rates, loss weights, memory capacity, sample group size, and prediction horizon. LIBERO/RoboTwin use 300000 updates, scene/EEF loss weights of 0.5/3.0, geometry coefficient 0.1, and memory capacities of 16/8. We use 16 GeoPlan tokens.

Sec. 3.3 specifies retrieval before writing, detached history, and episode-boundary resets. On overflow, the most similar temporally adjacent entries are merged to compress redundant states while retaining distinct stages.

Prediction horizon HH differs from execution length EtE_t: H=8H=8 for LIBERO and H=50H=50 for RoboTwin, with Et≤HE_t\leq H. Appendix C states that the full model and component ablations use task-adaptive execution horizons.

We will release training code, model checkpoints, and complete configurations, including event-frame selection, per-task execution horizons, and the tuning procedure.

Thanks for the authors' detailed rebuttal. However, it does not fully resolve my main concerns: the marginal performance differences in the isolated ablations make it difficult to justify the significance of the historical geometric components, and deferring key implementation details (like event-frame selection and execution horizons) to a future code release hurts the paper's self-contained completeness. Therefore, I'd like to maintain my original score.

Reviewer_mwNnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

GeoMem-VLA supervises GeoPlan tokens with VGGT-generated scene point maps and demonstration-derived future end-effector trajectories, then stores these representations in temporal memory for action generation. The geometry teacher and decoder are not required for action inference. Experiments cover LIBERO, RoboTwin, and five real-world tasks. Ablations show lower success when either the geometry branch or temporal memory is removed

Critical strengths
  1. Geometric supervision and policy memory are well integrated. GeoPlan tokens provide a clear interface between geometric supervision and policy memory, addressing the need to retain spatial context under partial observability. Keeping the teacher and decoder out of action inference also avoids their runtime cost.
  2. Experiments and ablations show performance gains. The full model exceeds the reported StarVLA-GR00T results by 2.4 percentage points on LIBERO and 2.92 on RoboTwin. Removing temporal memory lowers LIBERO-Long success from 96.6% to 94.2%, versus 0.2-point drops on each of the other three suites (Tables 3–4)
Critical weaknesses
  1. The benefit of geometric history is not isolated. The w/o Mem variant retains current geometric representations but disables historical retrieval from all banks, while w/o Geo removes the entire geometry branch (Tables 3–4). These comparisons do not isolate the added benefit of geometric history over observational history.
  2. The added benefit of scene supervision remains unclear. GeoPlan tokens are jointly supervised by scene point maps and demonstration-derived future end-effector trajectories (Eqs. 4–6), but the no-geometry ablation removes both objectives. Since future-trajectory prediction may itself improve control, the added contribution of scene supervision remains unclear
Questions for authors
  1. What is the effect of disabling geometric-history retrieval alone? Please retain current GeoPlan tokens, auxiliary supervision, observational memory, and fusion modules, and match total memory capacity and the number of history tokens retrieved per step across variants.
  2. What does scene point-map supervision add? Please compare success rates under scene-only, future-trajectory-only, and joint supervision, keeping token count, geometry-head architecture, memory capacity, and training settings fixed.
  3. Which training configurations were actually used? Table 5 reports templates that may be overridden. Please provide the configurations used for the main experiments and ablations, including the learning rate, effective batch size, modules trained, and number of training steps. Please also clarify whether the StarVLA baseline used the same training data and budget.
1 comment

Thank you for recognizing the integration of geometric supervision and policy memory. We added selective history and supervision ablations and report configurations in Appendix.

Q1. Disabling geometric-history retrieval alone.

We separately trained two variants, disabling the selected retrieval during both training and evaluation:

  • w/o GeoPlan history: retains current GeoPlan tokens and their supervision, observation history, and geometric fusion.
  • w/o observation history: disables retrieval from both observation banks while retaining current observations, current GeoPlan tokens, geometric-history retrieval, and fusion.

The full model and both variants use the aligned EEF supervision.

Table R1. LIBERO success rates (%), 50 trials per task.

ConfigurationSpatialObjectGoalLongAvg.
Full99.699.899.696.698.9
w/o GeoPlan history98.899.699.292.897.6
w/o observation history99.0100.099.093.297.8

Removing GeoPlan history lowers average success by 1.3 percentage points (pp) and Long by 3.8 pp. Since current geometry and observation history remain available, these results support additional control value from geometric history.

Q2. What does scene supervision add beyond future EEF prediction?

We added separate EEF-only, Scene-only, and joint-supervision evaluations.

Table R2. Supervision ablations: LIBERO success rates (%).

SupervisionSpatialObjectGoalLongAvg.
EEF-only98.699.299.095.498.1
Scene-only98.299.699.495.898.3
Joint (full)99.699.899.696.698.9

Joint supervision performs best across all four suites. Adding scene supervision to EEF-only improves Spatial by 1.0 pp and Long by 1.2 pp. Adding EEF supervision to Scene-only improves them by 1.4 pp and 0.8 pp, respectively.

The two targets describe current scene structure and future end-effector motion in a shared coordinate system and normalization. Joint learning encourages GeoPlan to represent both spatial structure and motion relative to that structure. These results support complementary benefits and address whether the gains arise solely from auxiliary trajectory prediction.

Q3. Actual configurations and the StarVLA comparison.

Appendix A/Table 5 reports the configurations used for the reported results. All platforms use Qwen3-VL-4B-Instruct and a frozen VGGT teacher.

Table R3. Training and inference configurations.

SettingLIBERORoboTwinAstribotAgileX
Training updates3000003000003000030000
Per-device batch (frames)16161616
Gradient accumulation4812
Training seed42424242
Backbone LR1e-51e-55e-55e-5
Action-expert LR1e-41e-45e-55e-5
Other trainable modules LR2.5e-52.5e-55e-55e-5
Scene loss weight0.50.50.50.5
EEF loss weight3.03.03.03.0
Geometry loss coefficient0.10.10.10.1
Memory capacity1681616
Sample group size32163232
Prediction horizon H8505050
Euler integration steps416416

Per-device batch and gradient accumulation are listed separately. We will release the training code, model checkpoints, and associated configurations so that the complete experimental settings can be inspected.

Reviewer_wqk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes GeoMem-VLA, a vision-language-action framework that couples geometry-supervised latent representations with causal episode-level memory. The method maintains perceptual, task-related, and geometric memory, with causal retrieval, adaptive fusion, similarity-based consolidation, and event-aware training that samples temporally separated observations from the same episode. The authors evaluate on LIBERO, RoboTwin 2.0, and five real-world manipulation tasks on two robot platforms. They report 98.9% average success on LIBERO and 91.10%/91.24% on RoboTwin Clean/Randomized.

Critical strengths
  1. The problem is well motivated for long-horizon spatial manipulation. The paper targets a real weakness of current VLAs: current observations may not reveal occluded objects or task progress, so geometry must be encoded and retained across decisions.

  2. The evaluation covers both simulation and real-world settings with tasks that directly stress spatial and temporal requirements. These are appropriate validations for the paper’s central claims.

Critical weaknesses
  1. The empirical gains are marginal and not statistically or protocol-wise established. On LIBERO, GeoMem-VLA’s 98.9% average is only 0.2 points above 3DThinkVLA (98.7%) and close to several other baselines; on RoboTwin, 91.17% is only 0.67 points above HyVLA-0.5 (90.50%).

  2. Real-world evaluation uses only 10 rollouts per task, with no variance, confidence intervals, or significance tests. Published baseline numbers are cited from different first-party sources and protocols, and the paper does not clearly establish comparable training budgets and tuning.

  3. The causal evidence for the core components is incomplete, and deployment costs are not analyzed. The diagnostics show that the auxiliary head can decode point maps and EEF tracks from GeoPlan tokens, but this does not establish that the deployed policy actually uses this geometric information or that memory retrieval causes the gains.

  4. The paper does not report closed-loop ablations isolating geometry supervision, geometric memory, event-aware training, and consolidation.

Questions for authors

Please refer to paper weakness.

1 comment

Thank you for the review. We added selective history-retrieval and supervision ablations.

W1. Small gains and comparability of published results.

Leaderboard margins alone do not capture the contribution on near-saturated benchmarks. LIBERO Spatial/Object/Goal are near saturation (99.6/99.8/99.6% for our model). Long better differentiates multi-stage execution. GeoMem-VLA reaches 96.6% on Long versus the reported 95.8% of 3DThinkVLA. Unlike 3DThinkVLA, our framework explicitly retains geometric history. We do not attribute this cross-paper difference directly to memory; the internal ablations below examine its contribution.

On RoboTwin, our average is 91.17%. HyVLA-0.5 uses large-scale pretraining, whereas our work investigates geometry supervision and historical representations during downstream adaptation. Achieving competitive performance while adding persistent geometric memory supports the effectiveness of this adaptation strategy. Published results are not comparisons under identical budgets.

W2. Does geometric history contribute to actual control?

We separately trained two history-ablation variants, disabling the corresponding retrieval during both training and evaluation. Removing GeoPlan history retains current GeoPlan tokens and their supervision, observation history, and geometric fusion. Removing observation history retains geometric history. The full model and both variants use the aligned EEF targets.

Table R1. LIBERO success rates (%), 50 trials per task.

ConfigurationSpatialObjectGoalLongAvg.
Full99.699.899.696.698.9
w/o geometry branch98.099.698.895.297.9
w/o all history retrieval99.499.699.494.298.2
w/o GeoPlan history98.899.699.292.897.6
w/o observation history99.0100.099.093.297.8

Removing GeoPlan history reduces average success by 1.3 percentage points (pp) and Long by 3.8 pp, despite retaining current geometry and observation history. This supports additional control value from historical GeoPlan representations. Removing observation history also lowers performance, supporting the value of both streams.

The original ablations show complementary effects: removing geometry lowers Spatial by 1.6 pp; removing all history lowers Long by 2.4 pp versus 0.2 pp on each other suite. Thus, our evidence includes closed-loop performance changes. Sec. 4.4 diagnoses representation content, while Tables 3-4 evaluate the contribution of the corresponding components and pathways.

W3. Finer supervision and memory ablations.

We added EEF-only, Scene-only, and joint supervision, obtaining 98.1%, 98.3%, and 98.9% average success. Joint supervision performs best among these variants on all four suites. Relative to EEF-only, it improves Spatial from 98.6% to 99.6% and Long from 95.4% to 96.6%, supporting a contribution from scene supervision beyond auxiliary trajectory prediction.

W4. Real-world trial counts and statistical precision.

Hardware evaluation covers precise manipulation, multi-stage execution, and progress tracking on two platforms. Each task uses 60 demonstrations and 10 independent rollouts with randomized initial conditions. Modest hardware evaluation budgets also appear in Spatial Forcing and MemoryVLA, although their exact trial allocations differ.

Appendix E.4 specifies shared task demonstrations, initial configurations, evaluation episodes, and success criteria across real-world baselines, with adaptation following each method's official recipe. This aligns data and evaluation conditions, without implying identical optimization budgets.

Ten trials limit estimation precision; we do not claim statistical significance for small differences. Hardware results provide deployment evidence, while component attribution primarily relies on simulation ablations: 50 trials per LIBERO task and 100 per RoboTwin task and setting.

W5. Deployment cost.

Sec. 3.3 clarifies that the VGGT teacher and geometry decoder are used only during training. Action inference requires neither module nor explicit point-map or EEF-track reconstruction. This removes their online computation while retaining latent geometry. GeoPlan tokens, retrieval, fusion, and consolidation remain active; the cost reduction concerns the omitted teacher/decoder computations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.