acceptodds
Under review as a conference paper at ICLR 2027

Spatial Grafting: Grounding 3D Features for Flow-Matching Robot Policies

Abstract

Pretrained robot manipulator policies such as vision-language-action models (VLAs) or world-action models (WAMs) leave interaction-relevant metric geometry implicit. Recent breakthroughs in spatial reconstruction can supply the necessary geometry reliably, but their features only describe local shape without stating where it lies with respect to the robot and the most effective method to deliver this information is unsettled. We propose **Spatial Grafting**, a versatile, lightweight spatial module, which binds frozen reconstruction features to metric, robot-relative geometry. Spatial Grafting supplies the flow-matching-based action model with metric-grounded spatial tokens, and grafts the resulting spatial tokens by cross-attention into the flow-matching action expert, without modifying the host's perceptual pathway, so the host retains the full benefit of its pretraining. We evaluate it more broadly than any geometry-aware policy we compare against: one graft architecture, with no per-host redesign, on two VLAs and two WAMs, across four simulation benchmarks that span short-horizon manipulation, visual robustness, clutter and long-horizon mobile manipulation, and on three real-robot platforms with single- and dual-arm configurations. On RoboTwin 2.0, a dual-arm manipulation benchmark, the graft improves every host across VLAs and WAMs, gaining 11.3% and 15.6% on clean and randomized scenes. It reaches 94.0% and 92.4%, above the strongest published 3D-conditioned policy, WAM4D (93.8% and 89.9%). The margin widens as the horizon lengthens: on tasks from BEHAVIOR-1K, a dual-arm mobile manipulation challenge scored by average task progress, it surpasses the 2025 challenge winner on five of six tasks, by up to 0.47 Q-score, and exceeds a map-conditioned spatial policy on average across the three tasks both report.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic manipulation, vision-language-action models, world models, flow matching, 3D foundation models, spatial grounding, metric geometry, policy adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZvFCOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spatial Grafting, a plug-in module that delivers geometric information to pretrained flow-matching robot policies (VLAs and WAMs) without modifying their perception pathway. For each camera view, frozen multi-layer DA3 features are sampled on an 18×24 grid. Each grid token is bound, via FiLM and concatenation. The action expert reads these banks by cross-attention in its final blocks. It reads the primary view first; the auxiliary views are then fused through a bias-free merge. Experiments cover four hosts (π0.5, X-VLA, Fast-WAM, LingBot-VA), four benchmarks (RoboTwin 2.0, LIBERO, RoboPRO, BEHAVIOR-1K) and six real-robot tasks on three platforms.

Critical strengths
  1. The graft improves every host. All four hosts improve on RoboTwin, in both the Clean and Randomized settings. On real robots, pooled over six tasks on three embodiments, success rises from 23/60 to 45/60, these results show that the graft itself adds value, beyond retraining the host. This is the main reason I lean towards acceptance.

  2. A practical recipe for injecting geometry into a pretrained action expert, a route that prior evidence suggested might fail. Delivering geometry directly into the action expert is not obviously viable. This paper makes that route work across four hosts. Its retrained ablations (Fig. 3b) identify the obstacles involved, these are non-trivial, transferable findings, and practitioners are likely to adopt the interface. The supplementary material is well prepared: per-host training code with pinned checkpoints and data revisions, the calibration tools, the RoboPRO wrappers and the VGGT-Ω variant, and a narrated walkthrough with rollouts.

Critical weaknesses

The defining contribution, metric and end-effector-relative grounding, is not isolated. Prior work already covers much of the design. Routing tokens from a reconstruction model to the action head instead of the VLM, in order to protect VLM semantics, was proposed in FALCON (ICLR 2026), which does not appear to be cited. Attending from action queries to depth-lifted features relative to the gripper is the core of Act3D and 3D Diffuser Actor. What remains specific to this paper is binding frozen latents to metric, end-effector-relative coordinates, together with the late-graft recipe.

That binding is not tested. Bank-off and bank-shuffle are test-time interventions on one trained checkpoint. They show that the policy relies on the banks, but the banks also carry DA3 appearance features, so the interventions do not separate coordinates from features. Learned grid and view embeddings are added to every bank token and survive per-sample centering, so even with uninformative geometry the action states can attend to a set of learned keys and values. This is added capacity.

Questions for authors

A. Requests that could change my rating:

  1. What does π0.5 achieve when fully fine-tuned without the graft, using your identical recipe, warmup, number of denoising steps and evaluation seeds?

  2. Please retrain three variants under the reference recipe: the graft with DA3 latents but no coordinate terms; a capacity control whose banks carry only the learned grid and view embeddings; and world coordinates without the end-effector offsets. If the first two are clearly worse than the full model.

  3. The reference π0.5 model uses ground-truth simulator depth, while the paper describes its inputs as unprivileged and compares against RGB-only methods (WAM4D, Spatial Forcing). Can you report results with DA3-predicted depth, or else qualify these claims?

  4. Figure 1 needs substantial polishing. I have no objection to stylized or generated illustrations, but the overview figure should be internally consistent and informative. The top-left panel is meant to depict a single observation, yet the views do not match. The wrist views show an orange on the plate, while the third-person view of the same scene shows an empty plate. In the "Metric Grounded 3D Features" panel, the highlighted points do not clearly correspond to the objects in the photo. For example, the yellow lemon is drawn as a red sphere like the others, so it is unclear whether these points depict objects or sampled query points. The four small panels in the second row (depth, world-frame position + ray, per-camera relative geometry, latent spatial feature) add little beyond the text. A single panel showing how one grid token is bound to its feature, its metric position and its offset to the gripper would convey the core idea better. If any panel was generated or edited with AI tools, please state this in the AI-use statement, which currently covers only Fig. 4. For a venue like ICLR, I expect the main overview figure to meet this bar, and the revised figure will factor into my final assessment.

B. Clarifications:

  1. How does the method differ from FALCON? Since 3D-Mix found that cross-attention inside the action expert fails, which ingredient makes it work here: the warmup, late injection, or the coordinate binding?

  2. For the real robots, which host, how many demonstrations and which depth sensor were used? How were the per-step wrist-camera extrinsics obtained, given that Sec. 3.2 says no kinematic model is needed?

  3. Each token carries a single nearest-neighbour depth sample on an 18×24 grid. Thin objects such as the marker or the candy cane may occupy less than one cell, or receive background depth at their boundaries, yet they are the showcase cases. Do their gains come from the coordinates or from the DA3 features?

6 comments

Thank you for your careful reading and constructive review. We appreciate your detailed suggestions on the experimental controls, relationship to prior work, and presentation of the method. We address each concern below, with references to the revised paper.

1. Contribution relative to FALCON, Act3D, and 3D Diffuser Actor

We agree that FALCON already motivates action-side access to reconstruction features, while Act3D and 3D Diffuser Actor use depth-lifted features and gripper-relative geometry. We have made these relationships explicit in Section 2 (Related Work) and Appendix A, Table 5.

Our contribution is the combination described there: binding frozen reconstruction latent tokens at their grid locations to metric positions and current end-effector-relative offsets, then injecting the location-indexed banks directly inside pretrained VLA and WAM flow-matching action experts. FALCON pools spatial tokens into a global vector for action-token fusion. We retain token-level spatial structure for cross-attention during action generation. Unlike spatial action queries in earlier 3D policies, our latent action tokens have no explicit 3D query position, so the grounding resides in the bank. We do not claim the first action-side spatial route or superiority to these methods under matched conditions. Section 2 also distinguishes this interface from upstream robot-centric pointmap conditioning.

2. Matched π0.5 baseline and retrained ingredient controls

We agree that a matched ungrafted baseline is needed to assess the graft’s contribution. We retrained π0.5 without the graft using matched initialization, demonstrations, staged training recipe, denoising steps and evaluation seeds. The full graft improves over this baseline by 10.9/13.9 percentage points on RoboTwin Clean/Randomized (Section 4.1, Table 1; Appendix B, Table 7). Appendices D and E give the training and evaluation details. The earlier denoising-step count was a typo: these runs use 20 steps.

We also retrained feature-only, geometry-only, no-offset and embeddings-only controls. Explicit geometry adds 5.3/8.1 points beyond the same reconstruction features, and end-effector offsets add 4.1/4.0 points beyond absolute geometry. These controls support combining reconstruction features with explicit grounding. Bank-off/shuffle indicate reliance on bank content, without isolating geometry from features or capacity. Section 4.3, Figure 3(a) provides the control definitions, results and interpretation. The supplementary materials provide code, configurations and checkpoint mappings.

3. Learned embeddings and the capacity

We agree that centering does not remove learned embedding capacity, and our earlier categorical capacity claim was too strong. The active π0.5 branch has an independent 1×432×1024 learned token array for each view, added with unit coefficient before per-view token-mean centering. With scene inputs removed, the bank is the learned array minus its token mean, which remains token-dependent.

We checked the released implementation and trained embedding arrays: the centered arrays are nonconstant, and loaded attention weights produce varying keys/values and query-dependent outputs. The separately defined 2D-position MLP and view lookup are not used by this spatial-vector branch; their configured 0.25 scale does not apply to its bank-token arrays. Appendix D now describes the active parameterization.

The embeddings-only control reaches 82.1/76.9, 1.0/1.6 points below the matched host (83.1/78.5). These learned tokens contain no observation-specific information. The near-baseline performance is consistent with the policy learning to make little use of them, although success rates alone cannot establish that they are completely ignored. With the same added capacity, the full grounded bank reaches 94.0/92.4, exceeding embeddings-only by 11.9/15.5 points. Thus added capacity alone does not reproduce the improvement in this control. Together with the feature-only and geometry-only controls, this supports a gain from scene-conditioned reconstruction features and their explicit grounding, rather than merely adding trainable tokens.

4. Predicted depth and input requirements

We agree that the simulator-depth reference did not establish performance without that input, and our wording needed qualification. Following your suggestion, we retrained the reference using depth predicted from RGB by frozen DA3NESTED-GIANT-LARGE-1.1 in both training and evaluation, without correction or rescaling against simulator depth. Reconstruction features, calibrated cameras, recipe, decoder, and seeds otherwise match the reference.

Depth sourceClean SR (%)Randomized SR (%)
Simulator depth94.092.4
DA3-predicted depth90.189.8
Matched ungrafted host83.178.5

As expected, simulator ground-truth depth performs best in this comparison. Predicted depth introduces estimation errors and noise, and the 3.9/2.6-point reduction is consistent with sensitivity to those errors. Nevertheless, predicted depth still delivers substantial gains of 7.0/11.3 points over the matched host. These results show that externally supplied depth can be replaced by DA3-predicted depth in the tested RoboTwin setting, while camera calibration remains required (Section 4.3, Figure 3(c)). They do not establish a general noise or calibration-error tolerance range.

5. Warmup, placement, and the relation to 3D-Mix

We agree that 3D-Mix’s poor action-expert-fusion result on SIMPLER makes it important to test which ingredients support our recipe. We have also corrected our earlier single-host description of that work. For context, its fusion arm scores 3.13 versus 57.81 for the base on SIMPLER and 97.40 versus 96.5 on LIBERO, suggesting that the outcome depends on the setting. 3D-Mix, Table 1.

To address your question, we added equal-module placement and explicit-geometry ablation study alongside the submitted warmup ablation. Figure 3(a) reports the geometry controls, Figure 3(b) the placement controls, and Figure 3(c) the warmup comparison. Section 4.3 describes the experiments and their interpretation.

Removing warmup loses 16.1/14.3 points. With six modules held fixed, middle placement loses 5.3/5.0, and early placement 12.8/9.9. Removing explicit geometry loses 5.3/8.1. The placement controls share the same warmup schedule and subsequent full-host fine-tuning. Their conclusion confirms the attachment depth under this schedule, including its interaction with which host blocks train during warmup. The warmup result is the submitted ablation, not a new experiment. In the development of Spatial Grafting, siginficant amount of work was put in to develop the exact training recipe for Spatial Grafting to work and the specific recipe and injection method ultimately differs from 3D-Mix.

Your question also highlights an important limit: these findings support the ingredients within our recipe, but do not identify which ingredient explains a difference from 3D-Mix: the hosts, data, and benchmarks differ, and we did not reproduce its AE-Fusion setup or evaluate on SIMPLER. We therefore do not make that cross-paper causal attribution.

6. Revised Figure 1 and AI disclosure

We agree that the original overview needed clearer, internally consistent illustrations. Thank you for the detailed suggestions. We have revised Figure 1 around the correspondence of one sampled spatial token to its reconstruction feature, depth/ray, metric position, and offset from the gripper. The observation views now depict the same plate, lemon, and strawberries. The previous ambiguous spheres are replaced by a sampled-token marker, its projected location on the lemon, and a gripper-relative offset. The linked illustration makes clear that the marker denotes a sampled spatial location, not an object identity. The enlarged graft layer shows primary-view attention followed by auxiliary-view attention and fusion.

We have also updated the AI-use statement. Only the lower-center metric-grounding illustration in Figure 1 contains AI-generated edits: the R1Pro scene and task were collected by us, and AI was used to illustrate how the spatial tokens are grounded. This is a schematic explanation, not a measured reconstruction. The existing disclosure of the illustrative, AI-enhanced Figure 4 panels is retained. We appreciate the reviewer’s comments, which helped make the overview more informative and internally consistent.

7. Hardware training, sensing, and wrist-camera poses

We agree that the hardware setup and wrist-camera poses needed clearer documentation. Table 4 and Appendix E report the results and protocol. All hardware tasks use π0.5, with 100 collected demonstrations per task, pooled within each embodiment and trained jointly for four epochs using the corresponding simulation recipe and host/graft learning-rate scaling. Checkpoints are in the anonymous repository linked from the supplementary materials.

All policy views provide RGB and measured depth from D455 external/head cameras and D405 wrist cameras. Cameras acquire at 30 FPS, while policy inference and action-chunk execution are synchronous. SDK/driver processing aligns depth to RGB and pairs frames within each camera. Cross-camera observations and joint samples are associated by nearest timestamp without interpolation.

Per-step wrist-camera poses combine joint-position forward kinematics with fixed hand–eye camera-to-gripper transforms. UR5e/Piper use fixed external-camera-to-base calibration. R1Pro composes its head-camera mounting transform with the relevant kinematic chain. Geometry is expressed in the external/head-camera frame. Thus our earlier “no kinematic model is needed” wording was too broad. Section 3.2 states the pose requirement. Reconstruction-estimated poses are also compatible with the interface when depth, poses and anchors share a consistent metric frame, but the reported hardware experiments use calibration.

Base and graft use paired starts and fixed calibration, with ten trials per condition per task. Success requires completing all subtasks. These trials support feasibility in the tested setups rather than precise effect-size estimates.

8. Thin objects and coarse depth sampling

We agree that current end-to-end experiments do not quantitatively isolate reconstruction features’ and depth’s contributions to thin-object grasping. Coarse depth samples can miss thin surfaces or capture background depth. We believe that Spatial Grafting's method gives reconstruction latent tokens an explicit metric reference. Their learned feature magnitudes can vary across layers and observations and do not themselves represent calibrated physical distances. We therefore bind each token to a metric position and end-effector-relative offsets, making its scene information directly accessible in absolute and robot-relative coordinates. The action expert can then use both what a feature represents and where it lies relative to the robot. Section 4.3, Figure 3(a) supports this combination, without establishing its separate contributions to individual thin-object grasps or claiming exact sub-cell geometry.

The revised supplementary submission contains the training/evaluation code and configuration mappings, with the anonymous checkpoint link provided there. Results are reported in the revised paper and this response. Updated walkthrough and hardware material are included. The original walkthrough predated the final experimental updates near the submission deadline.

Reviewer_JCF8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a common failure in robot manipulation. A policy recognizes the right object and knows what to do, but grasps too high or places the object incorrectly. To improve spatial precision, it proposes Spatial Grafting. A frozen reconstruction model extracts features, while depth maps and camera calibration associate those features with 3D positions and offsets relative to the robot’s hands. The action model then reads these features through cross-attention in its final layers to refine its actions. The main contribution is a way to add this spatial information to both VLA and WAM policies without changing their perception architecture. The authors evaluate four policy hosts across four simulation benchmarks and three real-robot platforms. Results improve for all four hosts on RoboTwin, both tested hosts on RoboPRO, five of six BEHAVIOR-1K tasks, and all six real-robot tasks. Ablations show that the spatial features, their placement in the action model, and the training schedule all matter.

Critical strengths
  1. The method connects reconstruction features to physical positions relative to the robot’s hands, making them useful for correcting grasp height and placement errors.

  2. The method works with both VLAs and WAMs. It improves all four tested models on RoboTwin, suggesting that its benefits extend across architectures, though each host still needs some configuration and training adjustments.

Critical weaknesses

None

Questions for authors

I have no further questions.

1 comment

Thank you for your careful reading of our paper and your positive assessment. We appreciate the time you devoted to reviewing the method, experiments and ablations, and to providing a detailed summary of the work. Your summary captures the central motivation: helping policies translate object recognition and task understanding into more spatially precise actions.

We particularly appreciate your recognition of the value of grounding reconstruction features in metric, hand-relative coordinates and of the method’s applicability to both VLA and WAM policies. Thank you again for your thoughtful review and encouraging feedback.

Reviewer_D1Z8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Spatial Grafting uses frozen reconstruction features grounded in metric and end-effector-relative coordinates. It adds spatial banks to late flow-matching action-expert layers with cross-attention. The host's perceptual pathway is structurally unchanged. There are four simulation benchmarks plus six real-robot tasks on three platforms. Grafted π0.5 gets 94.0%/92.4% success on RoboTwin Clean/Randomized.

Critical strengths
  1. The same interface handles π0.5, X-VLA, Fast-WAM and LingBot-VA. It has potential beyond one host.

  2. Several task lengths and platforms are covered. The per-task results and retrained ablations help. Bank removal/shuffling shows that the added pathway matters. WAM regressions on LIBERO are noted in the appendix. So are trained-model limits of Proposition 1.

Critical weaknesses
  1. Matched retrained feature-only and no-end-effector-relative-coordinate controls are needed here. Fig. 3 changes reconstruction features along with metric coordinates. So it doesn't tell how much comes from metric grounding and how much from extra features/capacity. The video notes this but Sec. 4.3 makes a stronger capacity claim. The all-18-block versus last-six test also changes module count. That doesn't isolate placement.

Proposition 1 holds under its assumptions. Trained π0.5 may violate them (Appendices B.2–B.3). Sec. 3.3 needs that limit on its zero-bank/zero-delta claim. RoboTwin uses ground-truth depth (Appendix C). Depth/calibration sensitivity and end-to-end latency/memory including reconstruction aren't reported. Deployment/lightweight claims need evidence or narrower scope. Sec. 2's single-host comparison overlooks 3D-Mix's GR00T-style and π-style tests.

  1. Which WAM setup produced these results? That is still unclear from the supplied materials.

For Fast-WAM the main recipe freezes the host. There is a depth mismatch though. In fastwam_da3.py dataset depth stays in build_inputs (230–234). Then live-RGB infer_action overwrites da3_depth with predicted depth (258–266). Appendices B.3/C say measured depth. Precomputed features could bypass the branch but the evaluated wrapper could not be located. This doesn't prove that the rollouts took that path.

LingBot-VA starts by training the graft. The main recipe then trains the host too. There is also a graft-only continuation. Table 1 doesn't say whether 93.3/91.4 came from phase 0 or joint phase 1 or that continuation. The base in Appendix D is the released checkpoint. If training was joint it needs matched extra fine-tuning. If the table used graft-only training naming the run would settle this concern.

Appendix E promises a 20-seed bank and evaluation pipeline. The WAM rollout adapters and seed list could not be located. Nor could the table-to-run/checkpoint/configuration mapping. Generic settings don't identify the tested preprocessing or action normalization.

For RoboTwin π0.5/X-VLA and RoboPRO the bases in Appendix D are published results. Using shared demonstrations doesn't ensure matched tests. LIBERO has a 130-task protocol. How does that match Table 7's four suites and published bases? This asks about comparability. It doesn't establish leakage. BEHAVIOR combines public-instance and challenge results. Appendix D says 10 instances and also 20. That weakens comparison with the challenge winner and SERF.

At around 2:30 the video's RoboTwin/RoboPRO scores differ from Tables 1/3. Hardware results are called pending later on despite Table 4. The released π0.5 workspace center/scale don't match Appendix C. The README says zero initialization. Yet spatial_init_std=0.01 is in the recipe and nonzero output-projection initialization is in Appendix C. This is distinct from zero-bank behavior. Maybe these are old files. Which ones match the runs?

Ten trials per condition are used for hardware. That alone doesn't invalidate the results. The missing clarity is on host/checkpoint and training-budget details plus sensor/calibration and paired-start details. There is value in the design and broad tests. The soundness concerns are major though. They may need corrected controlled evaluations. This supports weak rejection.

Questions for authors
  1. What do matched retrained feature-only and no-end-effector-relative-coordinate controls show with training/capacity and decoder fixed? Do they separate binding from access to reconstruction features? Given Appendix B.2's trained-model caveat, which main-text claims need exact zero-bank behavior? Will you restrict them accordingly?

  2. For Tables 1/8, which Fast-WAM inference adapter preserves measured depth despite the overwrite in FastWAMDA3.infer_action? Give evaluated code/configuration and checkpoint, depth source, normalization-statistics artifact and shared 20-seed bank. Did LingBot-VA's 93.3/91.4 use phase 0, joint phase 1 or graft-only continuation? If joint, give matched additional host-only fine-tuning results. Map supplied to evaluated paths if different. If the overwrite path was used, provide corrected paired results. These clarify the existing comparison, not a future benchmark.

  3. Link each PDF table/video result to checkpoint, configuration and evaluation split. Resolve BEHAVIOR's 10/20 count. Provide a same-instance comparison or limit challenge/SERF to contextual scores. Supply promised evaluation scripts and seed/instance lists. Reconcile LIBERO's 130-task protocol with the four-suite aggregate and published base protocols. Explain workspace center/scale and zero-versus-nonzero-initialization differences between the release and Appendix C.

  4. For each hardware task, give host/checkpoint, demonstrations/training budget, depth sensor, calibration procedure and whether base/graft starts were paired. What are end-to-end latency/memory costs including reconstruction? What evidence supports the operating range under depth/calibration errors, or how should that claim be limited? These affect portability/lightweight assessment.

4 comments

Thank you for the detailed review. We ran the requested controls and reruns and revised the affected claims and protocol descriptions. References are to the revised manuscript.

1. Separating grounding from reconstruction features and added capacity

The new controls support a gain from scene-conditioned reconstruction features and their explicit grounding, beyond added capacity. We retrained feature-only, geometry-only, no-offset, embeddings-only and ungrafted-host arms (Figure 3(a), Section 4.3, Appendix D). Bank-off and bank-shuffle are now interpreted only as evidence of reliance on bank content.

ArmClean SR (%)Randomized SR (%)
Full graft94.092.4
Feature-only88.784.3
Geometry-only81.279.3
No end-effector offsets89.988.4
Embeddings-only82.176.9
Ungrafted host83.178.5

All arms share the pi05_base initialization, demonstrations from all 50 RoboTwin tasks, optimizer and schedule, batch size, two-epoch budget, 20-step decoder and 20 seeds per task (1,000 episodes per condition). During warmup the host backbone and action-expert blocks 0–11 are frozen while blocks 12–17 and the graft train; afterwards all VLA host and graft parameters train. The reconstruction model stays frozen, and the host-only control follows the same schedule. Each arm is one run. The earlier denoising-step count was a typo.

Ingredient arms keep the same modules, dimensions and learned view/grid-token embeddings, with 121.9M added trainable graft parameters (bank builder plus the six used injection blocks; the π0.5 export allocates arrays for all 18 blocks, 323.5M). Feature-only removes all explicit geometry: absolute/relative coordinate encodings, rays, depth validity and geometry-conditioned FiLM. Geometry-only removes the DA3 features, no-offset removes only the end-effector-relative channels, and embeddings-only removes both scene inputs while keeping a nonconstant learned token memory.

Explicit geometry adds 5.3/8.1 points over the same reconstruction features, and end-effector offsets add 4.1/4.0. Embeddings-only tokens carry no observation-specific information and stay 1.0/1.6 points below the host, while the full graft is 10.9/13.9 above it, so added capacity alone does not reproduce the gain. Geometry-only is 0.9 points below embeddings-only on Clean and 2.4 above on Randomized, so coordinates help in combination with reconstruction features rather than alone. These controls do not establish that our particular FiLM operator is uniquely necessary.

2. Placement at fixed module count

We moved exactly six graft modules to the first, middle or last action blocks, keeping the budget, bank, decoder and staged schedule of Section 1 (Figure 3(b), Section 4.3). Last-six (94.0/92.4) exceeds middle-six (88.7/87.4) and first-six (81.2/82.5). Since warmup trains host blocks 12–17 in every arm, this result holds under that schedule. All-18 is reported as a separate configuration, not a placement control.

3. Proposition 1 and zero-bank behavior

We retain Proposition 1 in Appendix C.2 as a conditional architectural result: it requires bias-free bank-dependent projections and scale-only key/value normalization. The evaluated π0.5 implementation retains trainable biases in its primary attention output projection, key/value projections and key/value LayerNorm, so the proposition is not guaranteed after training. We removed the unconditional exact-zero and cannot-help-without-geometry claims from the introduction and the standalone bank-only-routing paragraph from Section 3.3. Section 4.3 therefore interprets bank-off as evidence of dependence on bank content, not proof of a zero residual or exclusion of capacity effects.

4. Depth source and reconstruction-inclusive cost

The reference RoboTwin experiment uses simulator depth and calibrated cameras (Appendix D). The introduction now limits “unprivileged” to the absence of segmentation labels, object state and persistent maps; it does not mean RGB-only input. Comparisons with RGB-only spatial policies are contextual.

We also retrained with frozen DA3-predicted depth from RGB in both training and evaluation, without rescaling it to simulator depth; all other settings match the reference (Figure 3(c)).

Depth sourceClean SR (%)Randomized SR (%)
Simulator depth94.092.4
Predicted depth90.189.8
Matched ungrafted host83.178.5

Predicted depth loses 3.9/2.6 points but keeps gains of 7.0/11.3 over the matched host. This tests an alternative depth source, not a quantified operating range under depth or calibration errors (Section 4.3). Hardware results demonstrate feasibility with calibrated RGB-D inputs. Robustness depends jointly on reconstruction features, depth/calibration quality and the policy.

Added trainable graft parameters are 3.7–5.4% of host parameter counts, excluding the frozen reconstruction model. Relative to ungrafted π0.5, reconstruction-inclusive training time per step is 1.52×, inference time per action chunk is 1.84×, and peak inference GPU memory is 1.32× with simulator depth or 1.89× with predicted depth (manuscript Table 10(b)). This is an accuracy–cost trade-off. Inference time was not measured separately with predicted depth.

5. Related work and 3D-Mix

Section 2 and Table 5 now describe 3D-Mix as evaluating both GR00T-style and π-style hosts, including action-expert fusion, so we do not claim action-side access to reconstruction features as new. Our distinction is binding these features to explicit metric and end-effector-relative coordinates and supplying location-indexed banks inside released VLA and WAM flow-matching experts. Our controls test this grounding within our implementation, not against 3D-Mix under matched conditions.

6. The evaluated Fast-WAM depth path

We realized the supplied copy of fastwam_da3.py contained an unconditional depth overwrite and differed from the evaluated version. We have included the corrected version in the new supplementary materials, which preserves supplied depth and falls back to predicted depth only when da3_depth is None. The reported RoboTwin runs used simulator depth.

The adapter used is spatial-grafted-fastwam/evaluation/policy/fastwam_da3/. It quantizes simulator depth to integer millimetres as in training, samples it to the 18×24 grid by nearest neighbour, and passes it with RGB and calibrated cameras. Features are built live, not loaded from a cache. The reported checkpoint da3_fastwam_step10624.pt is released as robotwin/fastwam-graft.pt, with spatial-grafted-fastwam/evaluation/reference/training_config.yaml and the normalization file dataset_stats.json in the same folder. Evaluation uses shift 5.0, 10 flow-matching steps, 32 predicted/24 executed actions and no action smoothing. Seed lists are in seedbank/.

7. WAM training phases

Both RoboTwin WAM results use graft-only continuation from public RoboTwin checkpoints, so the gains come from training the added interface on unchanged hosts. Only the bank builder, graft modules and required bridges train; the whole host, including the action expert and video backbone, and the reconstruction model stay frozen. Base and graft use the same evaluation pipeline and seeds.

HostReleased base, Clean/Randomized (%)Frozen host + trained graft (%)Change (points)
Fast-WAM82.6 / 81.886.0 / 83.3+3.4 / +1.5
LingBot-VA88.9 / 85.793.3 / 91.4+4.4 / +5.7

The full-host fine-tuning launchers in the supplied code, including video-backbone updates, were not used for these results because of resource limits; Appendix D separates them from the executed recipe. The supplementary identifies the training configurations and checkpoints, including robotwin/lingbotva-graft.pt.

8. Matched RoboTwin and RoboPRO baselines

We trained and tested the π0.5 and X-VLA baselines on RoboTwin and RoboPRO in-house under the corresponding training and evaluation protocols for fair comparison. The π0.5 and X-VLA runs use the grafted model’s host initialization, demonstrations, trainable host components, update schedule, training budget, action representation, decoder and evaluation seeds. RoboTwin arms share normalization. On RoboPRO, each π0.5 arm uses its own uploaded checkpoint statistics (Appendix E). Results are in Tables 1 and 3, with published-versus-rerun comparisons in Appendix B, Tables 7 and 8. These VLA comparisons use matched training and evaluation protocols. The detailed run configurations, code and checkpoints can be found in the updated supplementary materials list and the results are updated to the main paper.

9. LIBERO training and evaluation scope

All four base/graft pairs in the main four-suite LIBERO table are our own runs, trained on all available demonstrations across 130 tasks and evaluated on the same 40 tasks with 50 seeds per task. We have further extended our evaluation to LIBERO-90 for all four hosts, using models trained on the full 130-task dataset and evaluating each task over 50 random seeds:

HostBase SR (%)Graft SR (%)Change (points)
π0.598.899.1+0.3
X-VLA96.796.3−0.4
Fast-WAM100.099.8−0.2
LingBot-VA97.899.3+1.5

Base rates of 96.7–100.0% indicate near saturation, and we report both small regressions. The main table keeps the standard four-suite aggregate for comparability with prior work.

10. Paired BEHAVIOR-1K evaluation

We have further evaluated the released RLC base and the graft on the same 20 public test instances per task (IDs 301–320) under the 2026 setup, with identical seeds, simulator, timeout and rollout protocol: one rollout per instance, 120 episodes per policy over six tasks. Results have been updated to the revised paper.

Mean Q-score rises from 0.249 to 0.619 (mean change +0.371, computed before rounding; largest task gain +0.623), and all six tasks improve. Published 2025 RLC scores appear separately in Appendix B, Table 9. Comet and SERF are contextual comparisons in Table 2. None of these published scores is used to compute the paired gains.

11. Workspace normalization and initialization

Thank you for identifying these inconsistencies. The reported π0.5 runs use workspace center c = (−0.0555, 0.0544, 0.8701) and scale λ = 0.5548; the submitted appendix listed an earlier fit. The README’s “zero initialization” description was not rigorous: π0.5, X-VLA and Fast-WAM use small nonzero initialization with standard deviation 0.01 for the output projections and FiLM. This gives the new branch a small random initial contribution and allows gradients to reach upstream graft parameters from the first update. It is intended to encourage the pretrained policy to use the new module while limiting perturbations to its existing behavior. Appendix D and the supplementary files specify these settings.

12. Hardware protocol

All hardware tasks use π0.5 with spatial grafting, trained for four epochs per embodiment on 100 demonstrations per task collected with D455 external/head cameras and D405 wrist cameras. UR5e/Piper use fixed external-camera-to-base calibration; R1Pro composes the head-camera mounting transform with the relevant kinematic chain. Geometry is expressed in the external/head-camera frame.

The SDK/driver aligns depth to RGB and pairs frames within each camera; cross-camera observations and joint samples are matched by nearest timestamp without interpolation. Cameras acquire at 30 FPS, while inference and action-chunk execution are synchronous. Base and graft use paired starts and fixed calibration, with ten trials per condition per task, and success requires completing all subtasks. Ten trials support feasibility in these setups, not precise effect sizes. Results are in Table 4, protocol details in Appendix E, and checkpoints in the anonymous repository linked from the supplementary materials.

13. Supplementary organization and video

We appreciate your correction on the submitted video. The replacement walkthrough video has been rerecorded and uploaded to the supplementary materials folder.

The supplementary folder contains the training/evaluation code, configurations, seed/instance lists and a link to the anonymous checkpoint repository with the HuggingFace access key is included in the supplementary materials. All checkpoints and deployments details are included in the README file.

We hope these controls and corrections address your concerns.

I thank the authors for their response and clarification. I remain concerned regarding the novelty. Therefore, I would retain my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.