- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Spatial Grafting, a plug-in module that delivers geometric information to pretrained flow-matching robot policies (VLAs and WAMs) without modifying their perception pathway. For each camera view, frozen multi-layer DA3 features are sampled on an 18×24 grid. Each grid token is bound, via FiLM and concatenation. The action expert reads these banks by cross-attention in its final blocks. It reads the primary view first; the auxiliary views are then fused through a bias-free merge. Experiments cover four hosts (π0.5, X-VLA, Fast-WAM, LingBot-VA), four benchmarks (RoboTwin 2.0, LIBERO, RoboPRO, BEHAVIOR-1K) and six real-robot tasks on three platforms.
Critical strengths
-
The graft improves every host. All four hosts improve on RoboTwin, in both the Clean and Randomized settings. On real robots, pooled over six tasks on three embodiments, success rises from 23/60 to 45/60, these results show that the graft itself adds value, beyond retraining the host. This is the main reason I lean towards acceptance.
-
A practical recipe for injecting geometry into a pretrained action expert, a route that prior evidence suggested might fail. Delivering geometry directly into the action expert is not obviously viable. This paper makes that route work across four hosts. Its retrained ablations (Fig. 3b) identify the obstacles involved, these are non-trivial, transferable findings, and practitioners are likely to adopt the interface. The supplementary material is well prepared: per-host training code with pinned checkpoints and data revisions, the calibration tools, the RoboPRO wrappers and the VGGT-Ω variant, and a narrated walkthrough with rollouts.
Critical weaknesses
The defining contribution, metric and end-effector-relative grounding, is not isolated. Prior work already covers much of the design. Routing tokens from a reconstruction model to the action head instead of the VLM, in order to protect VLM semantics, was proposed in FALCON (ICLR 2026), which does not appear to be cited. Attending from action queries to depth-lifted features relative to the gripper is the core of Act3D and 3D Diffuser Actor. What remains specific to this paper is binding frozen latents to metric, end-effector-relative coordinates, together with the late-graft recipe.
That binding is not tested. Bank-off and bank-shuffle are test-time interventions on one trained checkpoint. They show that the policy relies on the banks, but the banks also carry DA3 appearance features, so the interventions do not separate coordinates from features. Learned grid and view embeddings are added to every bank token and survive per-sample centering, so even with uninformative geometry the action states can attend to a set of learned keys and values. This is added capacity.
Questions for authors
A. Requests that could change my rating:
-
What does π0.5 achieve when fully fine-tuned without the graft, using your identical recipe, warmup, number of denoising steps and evaluation seeds?
-
Please retrain three variants under the reference recipe: the graft with DA3 latents but no coordinate terms; a capacity control whose banks carry only the learned grid and view embeddings; and world coordinates without the end-effector offsets. If the first two are clearly worse than the full model.
-
The reference π0.5 model uses ground-truth simulator depth, while the paper describes its inputs as unprivileged and compares against RGB-only methods (WAM4D, Spatial Forcing). Can you report results with DA3-predicted depth, or else qualify these claims?
-
Figure 1 needs substantial polishing. I have no objection to stylized or generated illustrations, but the overview figure should be internally consistent and informative. The top-left panel is meant to depict a single observation, yet the views do not match. The wrist views show an orange on the plate, while the third-person view of the same scene shows an empty plate. In the "Metric Grounded 3D Features" panel, the highlighted points do not clearly correspond to the objects in the photo. For example, the yellow lemon is drawn as a red sphere like the others, so it is unclear whether these points depict objects or sampled query points. The four small panels in the second row (depth, world-frame position + ray, per-camera relative geometry, latent spatial feature) add little beyond the text. A single panel showing how one grid token is bound to its feature, its metric position and its offset to the gripper would convey the core idea better. If any panel was generated or edited with AI tools, please state this in the AI-use statement, which currently covers only Fig. 4. For a venue like ICLR, I expect the main overview figure to meet this bar, and the revised figure will factor into my final assessment.
B. Clarifications:
-
How does the method differ from FALCON? Since 3D-Mix found that cross-attention inside the action expert fails, which ingredient makes it work here: the warmup, late injection, or the coordinate binding?
-
For the real robots, which host, how many demonstrations and which depth sensor were used? How were the per-step wrist-camera extrinsics obtained, given that Sec. 3.2 says no kinematic model is needed?
-
Each token carries a single nearest-neighbour depth sample on an 18×24 grid. Thin objects such as the marker or the candy cane may occupy less than one cell, or receive background depth at their boundaries, yet they are the showcase cases. Do their gains come from the coordinates or from the DA3 features?
6 comments
4. Predicted depth and input requirements
We agree that the simulator-depth reference did not establish performance without that input, and our wording needed qualification. Following your suggestion, we retrained the reference using depth predicted from RGB by frozen DA3NESTED-GIANT-LARGE-1.1 in both training and evaluation, without correction or rescaling against simulator depth. Reconstruction features, calibrated cameras, recipe, decoder, and seeds otherwise match the reference.
| Depth source | Clean SR (%) | Randomized SR (%) |
|---|---|---|
| Simulator depth | 94.0 | 92.4 |
| DA3-predicted depth | 90.1 | 89.8 |
| Matched ungrafted host | 83.1 | 78.5 |
As expected, simulator ground-truth depth performs best in this comparison. Predicted depth introduces estimation errors and noise, and the 3.9/2.6-point reduction is consistent with sensitivity to those errors. Nevertheless, predicted depth still delivers substantial gains of 7.0/11.3 points over the matched host. These results show that externally supplied depth can be replaced by DA3-predicted depth in the tested RoboTwin setting, while camera calibration remains required (Section 4.3, Figure 3(c)). They do not establish a general noise or calibration-error tolerance range.
5. Warmup, placement, and the relation to 3D-Mix
We agree that 3D-Mix’s poor action-expert-fusion result on SIMPLER makes it important to test which ingredients support our recipe. We have also corrected our earlier single-host description of that work. For context, its fusion arm scores 3.13 versus 57.81 for the base on SIMPLER and 97.40 versus 96.5 on LIBERO, suggesting that the outcome depends on the setting. 3D-Mix, Table 1.
To address your question, we added equal-module placement and explicit-geometry ablation study alongside the submitted warmup ablation. Figure 3(a) reports the geometry controls, Figure 3(b) the placement controls, and Figure 3(c) the warmup comparison. Section 4.3 describes the experiments and their interpretation.
Removing warmup loses 16.1/14.3 points. With six modules held fixed, middle placement loses 5.3/5.0, and early placement 12.8/9.9. Removing explicit geometry loses 5.3/8.1. The placement controls share the same warmup schedule and subsequent full-host fine-tuning. Their conclusion confirms the attachment depth under this schedule, including its interaction with which host blocks train during warmup. The warmup result is the submitted ablation, not a new experiment. In the development of Spatial Grafting, siginficant amount of work was put in to develop the exact training recipe for Spatial Grafting to work and the specific recipe and injection method ultimately differs from 3D-Mix.
Your question also highlights an important limit: these findings support the ingredients within our recipe, but do not identify which ingredient explains a difference from 3D-Mix: the hosts, data, and benchmarks differ, and we did not reproduce its AE-Fusion setup or evaluate on SIMPLER. We therefore do not make that cross-paper causal attribution.
6. Revised Figure 1 and AI disclosure
We agree that the original overview needed clearer, internally consistent illustrations. Thank you for the detailed suggestions. We have revised Figure 1 around the correspondence of one sampled spatial token to its reconstruction feature, depth/ray, metric position, and offset from the gripper. The observation views now depict the same plate, lemon, and strawberries. The previous ambiguous spheres are replaced by a sampled-token marker, its projected location on the lemon, and a gripper-relative offset. The linked illustration makes clear that the marker denotes a sampled spatial location, not an object identity. The enlarged graft layer shows primary-view attention followed by auxiliary-view attention and fusion.
We have also updated the AI-use statement. Only the lower-center metric-grounding illustration in Figure 1 contains AI-generated edits: the R1Pro scene and task were collected by us, and AI was used to illustrate how the spatial tokens are grounded. This is a schematic explanation, not a measured reconstruction. The existing disclosure of the illustrative, AI-enhanced Figure 4 panels is retained. We appreciate the reviewer’s comments, which helped make the overview more informative and internally consistent.
7. Hardware training, sensing, and wrist-camera poses
We agree that the hardware setup and wrist-camera poses needed clearer documentation. Table 4 and Appendix E report the results and protocol. All hardware tasks use π0.5, with 100 collected demonstrations per task, pooled within each embodiment and trained jointly for four epochs using the corresponding simulation recipe and host/graft learning-rate scaling. Checkpoints are in the anonymous repository linked from the supplementary materials.
All policy views provide RGB and measured depth from D455 external/head cameras and D405 wrist cameras. Cameras acquire at 30 FPS, while policy inference and action-chunk execution are synchronous. SDK/driver processing aligns depth to RGB and pairs frames within each camera. Cross-camera observations and joint samples are associated by nearest timestamp without interpolation.
Per-step wrist-camera poses combine joint-position forward kinematics with fixed hand–eye camera-to-gripper transforms. UR5e/Piper use fixed external-camera-to-base calibration. R1Pro composes its head-camera mounting transform with the relevant kinematic chain. Geometry is expressed in the external/head-camera frame. Thus our earlier “no kinematic model is needed” wording was too broad. Section 3.2 states the pose requirement. Reconstruction-estimated poses are also compatible with the interface when depth, poses and anchors share a consistent metric frame, but the reported hardware experiments use calibration.
Base and graft use paired starts and fixed calibration, with ten trials per condition per task. Success requires completing all subtasks. These trials support feasibility in the tested setups rather than precise effect-size estimates.
8. Thin objects and coarse depth sampling
We agree that current end-to-end experiments do not quantitatively isolate reconstruction features’ and depth’s contributions to thin-object grasping. Coarse depth samples can miss thin surfaces or capture background depth. We believe that Spatial Grafting's method gives reconstruction latent tokens an explicit metric reference. Their learned feature magnitudes can vary across layers and observations and do not themselves represent calibrated physical distances. We therefore bind each token to a metric position and end-effector-relative offsets, making its scene information directly accessible in absolute and robot-relative coordinates. The action expert can then use both what a feature represents and where it lies relative to the robot. Section 4.3, Figure 3(a) supports this combination, without establishing its separate contributions to individual thin-object grasps or claiming exact sub-cell geometry.
The revised supplementary submission contains the training/evaluation code and configuration mappings, with the anonymous checkpoint link provided there. Results are reported in the revised paper and this response. Updated walkthrough and hardware material are included. The original walkthrough predated the final experimental updates near the submission deadline.
Thank you for your careful reading and constructive review. We appreciate your detailed suggestions on the experimental controls, relationship to prior work, and presentation of the method. We address each concern below, with references to the revised paper.
1. Contribution relative to FALCON, Act3D, and 3D Diffuser Actor
We agree that FALCON already motivates action-side access to reconstruction features, while Act3D and 3D Diffuser Actor use depth-lifted features and gripper-relative geometry. We have made these relationships explicit in Section 2 (Related Work) and Appendix A, Table 5.
Our contribution is the combination described there: binding frozen reconstruction latent tokens at their grid locations to metric positions and current end-effector-relative offsets, then injecting the location-indexed banks directly inside pretrained VLA and WAM flow-matching action experts. FALCON pools spatial tokens into a global vector for action-token fusion. We retain token-level spatial structure for cross-attention during action generation. Unlike spatial action queries in earlier 3D policies, our latent action tokens have no explicit 3D query position, so the grounding resides in the bank. We do not claim the first action-side spatial route or superiority to these methods under matched conditions. Section 2 also distinguishes this interface from upstream robot-centric pointmap conditioning.
2. Matched π0.5 baseline and retrained ingredient controls
We agree that a matched ungrafted baseline is needed to assess the graft’s contribution. We retrained π0.5 without the graft using matched initialization, demonstrations, staged training recipe, denoising steps and evaluation seeds. The full graft improves over this baseline by 10.9/13.9 percentage points on RoboTwin Clean/Randomized (Section 4.1, Table 1; Appendix B, Table 7). Appendices D and E give the training and evaluation details. The earlier denoising-step count was a typo: these runs use 20 steps.
We also retrained feature-only, geometry-only, no-offset and embeddings-only controls. Explicit geometry adds 5.3/8.1 points beyond the same reconstruction features, and end-effector offsets add 4.1/4.0 points beyond absolute geometry. These controls support combining reconstruction features with explicit grounding. Bank-off/shuffle indicate reliance on bank content, without isolating geometry from features or capacity. Section 4.3, Figure 3(a) provides the control definitions, results and interpretation. The supplementary materials provide code, configurations and checkpoint mappings.
3. Learned embeddings and the capacity
We agree that centering does not remove learned embedding capacity, and our earlier categorical capacity claim was too strong. The active π0.5 branch has an independent 1×432×1024 learned token array for each view, added with unit coefficient before per-view token-mean centering. With scene inputs removed, the bank is the learned array minus its token mean, which remains token-dependent.
We checked the released implementation and trained embedding arrays: the centered arrays are nonconstant, and loaded attention weights produce varying keys/values and query-dependent outputs. The separately defined 2D-position MLP and view lookup are not used by this spatial-vector branch; their configured 0.25 scale does not apply to its bank-token arrays. Appendix D now describes the active parameterization.
The embeddings-only control reaches 82.1/76.9, 1.0/1.6 points below the matched host (83.1/78.5). These learned tokens contain no observation-specific information. The near-baseline performance is consistent with the policy learning to make little use of them, although success rates alone cannot establish that they are completely ignored. With the same added capacity, the full grounded bank reaches 94.0/92.4, exceeding embeddings-only by 11.9/15.5 points. Thus added capacity alone does not reproduce the improvement in this control. Together with the feature-only and geometry-only controls, this supports a gain from scene-conditioned reconstruction features and their explicit grounding, rather than merely adding trainable tokens.