- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GeoMem-VLA, which combines geometrically supervised latent representations with causal episodic memory for robotic manipulation. The method uses an auxiliary flow matching objective, supervising learnable GeoPlan tokens via scene point maps generated by VGGT and future end-effector trajectories from demonstrations. The model stores these representations alongside observation features in memory, conditioning action generation through attention retrieval and gated fusion. At inference time, the geometric teacher model and reconstruction heads are no longer used. Experiments report an average success rate of 98.9% on LIBERO, 91.10%/91.24% on RoboTwin Clean/Randomized, and 78.0% on five real-world tasks.
Critical strengths
Dedicated GeoPlan tokens provide a clear interface between scene/motion supervision and temporal memory, while removing the teacher model and reconstruction heads from the deployment pipeline. This offers a valuable architectural path for policies to leverage geometric information without explicitly reconstructing scenes during action generation.
The experiments cover a broad scope and provide meaningful component ablations. The paper evaluates four LIBERO task suites, 50 RoboTwin tasks across two settings, and five real-robot tasks across two platforms. In Tables 3–4, the full model consistently outperforms variants without the geometric branch or temporal memory. In particular, removing memory drops the success rate on LIBERO-Long from 96.6% to 94.2%, supporting the value of historical information for actual control.
Critical weaknesses
The experiments do not sufficiently isolate the core contribution of "remembering geometric information." The geometric branch ablation in Section 4.3 simultaneously removes scene and end-effector trajectory supervision, GeoPlan tokens, geometric memory, and the fusion module; the memory ablation turns off historical retrieval across all representation streams. While these comparisons show that the combined module is effective, they cannot disentangle whether gains come from historical geometric information, current geometric conditioning, auxiliary motion prediction, or standard observation memory. The offline reconstruction and retrieval analyses in Section 4.4 indicate that decodable information exists within the representations, but they do not prove that historical geometric information actually influences subsequent action decisions. Real-robot experiments also lack component ablations that could address this attribution issue.
Several key implementation and evaluation details remain incomplete. Section 3.2 mentions transforming end-effector trajectories with physical scales into the VGGT point map coordinate system, but it does not explain the specific coordinate and scale alignment procedure. Appendix A provides configuration templates and explicitly states that checkpoint settings and command-line overrides take higher priority, yet it does not provide the final actual configurations. The frame sampling method for event-aware training and the task-adaptive execution horizon are also insufficiently detailed. These missing details hurt reproducibility and make it difficult for readers to judge whether performance gains stem from the proposed model under identical action execution setups.
Questions for authors
-
Is it possible to keep the current GeoPlan tokens, corresponding supervision, and observation memory, while turning off only the retrieval of historical geometric information? Further disentangling the individual contributions of scene point map supervision versus future end-effector trajectory supervision would also help clarify whether performance gains truly depend on learning and remembering scene geometry.
-
How exactly are the end-effector positions in demonstrations aligned with the VGGT point maps, including coordinate transformations, scale alignment, and joint normalization? What calibration or additional information is required for this process? When reporting end-effector trajectory errors, how are predictions converted back to coordinates with physical scale?
-
Please provide the actual configuration used for the checkpoints that produced the reported results, especially the number of GeoPlan tokens, geometric loss weights, the event frame sampling method, and the task-adaptive execution horizon rules. How was the execution horizon chosen? Were identical rules and tuning protocols applied to the full model and all ablated variants?
2 comments
Thanks for the authors' detailed rebuttal. However, it does not fully resolve my main concerns: the marginal performance differences in the isolated ablations make it difficult to justify the significance of the historical geometric components, and deferring key implementation details (like event-frame selection and execution horizons) to a future code release hurts the paper's self-contained completeness. Therefore, I'd like to maintain my original score.
Thank you for the review. We added closed-loop ablations and implementation details.
Q1. Isolating geometric history and supervision sources.
We separately trained variants without GeoPlan history or observation history, disabling the corresponding retrieval during both training and evaluation. Both retain current observations, current GeoPlan tokens, and geometric fusion. The full model and both variants use aligned EEF supervision.
Table R1. LIBERO success rates (%), 50 trials per task.
Removing GeoPlan history lowers average success by 1.3 percentage points (pp) and Long by 3.8 pp, supporting its additional value when current geometry and observation history remain available.
Joint supervision also outperforms EEF-only and Scene-only. Relative to EEF-only, it improves Spatial by 1.0 pp and Long by 1.2 pp, supporting a contribution from scene supervision beyond future-motion prediction.
Offline reconstruction diagnoses decodable content; these ablations assess the contribution of historical pathways and supervision through closed-loop success.
Q2. Coordinate alignment, required information, and physical-scale errors.
Trajectory coordinate transformation is a well-established engineering method, so we haven't detailed it in the method section of this article. We will be releasing the code, at which point you can see the detailed process in the code; here I'll just give a brief overview. Thank you for your question; we will add more details in a future appendix.
(a) Metric EEF positions. We use recorded metric EEF states where available. For joint-state demonstrations, forward kinematics uses the robot model, required tool transform, and base-to-reference transform. Both arms are expressed in a common frame. For simulated demonstrations, replay recovers camera extrinsics and metric scene depth, with consistency checks against the recorded states.
(b) Current-camera frame and teacher scale. Let pt+kw be a metric future EEF position. All future waypoints use the camera transform at the current observation:
pt+kc=Rtpt+kw+bt.
Teacher depth and metric scene depth are paired at corresponding surface anchors. The teacher-units-per-meter scale is estimated by
st=medianj∈Vtdt,jczt,jv.
Rotation and translation come from camera calibration; only the scalar scale is estimated from the teacher output.
(c) Shared normalization. Let μt and rt be the centroid and RMS radius of the pooled teacher point map. Scene and EEF targets share these parameters:
yt,iscene=rtqt,i−μt,yt,keef=rtstpt+kc−μt.
This preserves the trajectory's placement relative to the scene. Appendix A.1 specifies depth validity, anchor-count, and scale-consistency checks. Invalid samples receive zero EEF-loss weight.
(d) Recovering physical units. Inverting the normalization and camera transform gives
p^t+kw=Rt−1(strty^t,keef+μt−bt).
EEF errors can then be computed in meters and converted to millimeters. Appendix D reports EEF RMSE in millimeters and scene RMSE in normalized coordinates.
Constructing these training targets requires camera calibration, metric depth, and robot states or kinematics. Deployment uses RGB, robot state, and latent memory, without these target-construction inputs or online teacher/decoder execution.
Q3. Configurations, event frames, and execution horizons.
Appendix A/Table 5 reports training updates, grouped learning rates, loss weights, memory capacity, sample group size, and prediction horizon. LIBERO/RoboTwin use 300000 updates, scene/EEF loss weights of 0.5/3.0, geometry coefficient 0.1, and memory capacities of 16/8. We use 16 GeoPlan tokens.
Sec. 3.3 specifies retrieval before writing, detached history, and episode-boundary resets. On overflow, the most similar temporally adjacent entries are merged to compress redundant states while retaining distinct stages.
Prediction horizon H differs from execution length Et: H=8 for LIBERO and H=50 for RoboTwin, with Et≤H. Appendix C states that the full model and component ablations use task-adaptive execution horizons.
We will release training code, model checkpoints, and complete configurations, including event-frame selection, per-task execution horizons, and the tuning procedure.