acceptodds
Under review as a conference paper at ICLR 2027

Acting in Meters: Learning Metric Interactions for Precise Robotic Manipulation

Abstract

Vision-Language-Action (VLA) models and World-Action Models (WAMs) have increasingly advanced general language-conditioned robotic manipulation, yet often leave metric relations among actions, manipulated objects, and scene geometry implicit. Human manipulation combines semantic understanding of task-relevant objects with spatial feedback that guides hand motion relative to objects and their surroundings. Inspired by this, we introduce a metric interaction framework that models object-level and scene-level interactions in physical Cartesian space at a shared metric scale. At the object level, Interaction-Centric Tokens (ICTs) explicitly represent end-effector pose trajectories relative to manipulated objects and are jointly denoised with actions, providing physically grounded interaction supervision. At the scene level, the Metric Action Interaction Field (MAIF) uses action and ICT queries to attend to metric scene point-cloud features and learns geometry-conditioned action corrections. Through two-stage adaptation, our framework improves diverse VLA and WAM baselines with a small number of additional parameters and training steps. Experiments demonstrate average success-rate gains of 0.80 and 3.59 percentage points on LIBERO and RoboTwin 2.0, respectively, alongside gains of 6.80 percentage points on real-world tasks and 7.45 percentage points on their out-of-distribution variants.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Manipulation, World-Action Model, Vision-Language-Action, Robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XekYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how explicit metric relations can improve VLA and world-action policies. It first jointly predicts actions and object-relative end-effector trajectories using Interaction-Centric Tokens (ICTs). It then freezes the adapted policy and trains a Metric Action Interaction Field (MAIF), which uses point-cloud features to correct action predictions. The framework is evaluated on RoboTwin 2.0, LIBERO, and five real-robot tasks, with reported average gains of 3.59, 0.80, and 6.80 percentage points.

Critical strengths

The improvements extend to tasks where geometric errors directly affect success, including USB insertion and bimanual retrieval. For π0.5, average real-world success rises from 69.4% to 77.0%; Dual-USB Insertion improves from 47% to 59%.These execution results make a stronger case for the approach than the offline trajectory-error plots alone.

The evaluation covers five backbones on 50 RoboTwin tasks, including the geometry-aware X-WAM, whose average success increases from 78.41% to 82.01%. The ICT-only and full-method comparisons also suggest that the second stage contributes beyond ICT adaptation.

Critical weaknesses

The full method adds simulator depth for several RGB-based baselines, an approximately 200M-parameter adapter, and physical supervision. Table 3 includes ICT-only, flow-only, and extra-training controls, but does not compare object-relative supervision with an equivalent absolute end-effector target, or MAIF with a simpler adapter using the same geometry, capacity, and waypoint loss.

The real-robot success rates support practical usefulness, but the precision claim is assessed mainly through training/offline ADE and a small barrier-height analysis. Lower error relative to a demonstration does not necessarily imply more accurate object alignment or safer obstacle clearance during execution.

Equation (1) requires a full object pose at each time step, but the real-world annotation procedure is described only as 2D annotation followed by depth-based lifting. It remains unclear how object orientations are recovered, how reference objects are assigned during object switches or simultaneous manipulation of different objects, and how occluded frames are handled. These choices directly determine the ICT targets.

Appendix A.2 states that the Fast-WAM baselines on RoboTwin are taken from the public leaderboard, while the proposed variants are evaluated locally. The authors report that their local reproductions were close, but do not provide those numbers. The main results also lack variability across training seeds. This makes the exact improvement harder to assess, particularly on LIBERO, where the average gains are 0.55 and 1.05 percentage points.

Questions for authors

MAIF processes the action and ICT streams separately, and only its action branch is used at inference. How much does Stage 2 ICT supervision contribute beyond the action-position loss? An ablation with λICT = 0, while keeping the Stage 1 ICT-adapted backbone and all other settings unchanged, would help clarify its independent contribution.

How is the reference object selected during object switches or when the arms manipulate different objects? How do real-world 2D annotations and depth yield full object orientations, and how are occlusions handled?

Beyond binary task success, could you report execution-level measurements such as alignment error before USB insertion or minimum obstacle clearance during barrier crossing? These would help establish whether the lower ADE translates into more precise interaction with the actual objects.

Since local reproductions of Fast-WAM was already conducted, could you report their exact RoboTwin results alongside the proposed variants under matched evaluation conditions?

2 comments
AuthorsOpenReview ↗

Response 3

We thank you and address concerns below.

W1 & Q1: Matched controls and Stage 2 ICT supervision.

We added Absolute End-effector Tokens (AET), replacing object-relative targets while retaining the token architecture and supervision objectives. Stage 1 results on RoboTwin 2.0:

BackboneTargetRandomized (%)Avg. (%)
π0.5\pi_{0.5}AET45.4058.13
π0.5\pi_{0.5}ICT48.9859.95
Fast-WAMAET2.2039.82
Fast-WAMICT4.1041.25

ICT improves average success over AET by 1.82/1.43 percentage points for π0.5\pi_{0.5}/Fast-WAM, supporting object-relative supervision beyond absolute end-effector targets.

We also replace only MAIF with a parameter-matched MLP adapter, keeping identical geometry inputs, frozen Stage 1 checkpoint, losses (including waypoint supervision), and training settings. Separately, we set λICT=0\lambda_{\mathrm{ICT}}=0 only in Stage 2, retaining action-flow and -position supervision and all other settings.

Fast-WAM variantClean (%)Randomized (%)Avg. (%)
Ours (MLP adapter)79.962.6041.28
Ours w/o Stage 2 ICT loss78.668.7043.68
Ours (Full)80.0811.4845.78

With geometry, capacity, and losses matched, MAIF exceeds the MLP by 4.50/8.88 average/randomized percentage points. Stage 2 ICT supervision adds 2.10/2.78 points through the shared MAIF trunk, without an auxiliary MAIF ICT branch at inference.

W2 & Q3: Execution-level precision.

We agree that offline ADE alone does not establish execution-level precision. We compare π0.5\pi_{0.5} with π0.5\pi_{0.5} + Ours using planar USB alignment errors relative to the end-effector position for correct socket alignment. Each method uses five trials per task: five insertions for Task 2 and ten for Task 3.

Taskπ0.5\pi_{0.5} mean error (mm)π0.5\pi_{0.5} + Ours (mm)
Task 24.8104.544
Task 35.2855.176

These small-sample results provide limited evidence of improved alignment; protocols and per-trial results are in the appendix. Mean peak barrier-crossing height decreases from 307.151 to 282.621 mm at 100% success for both methods. This indicates reduced lifting; minimum obstacle clearance was not measured, so improved safety is not established.

W3 & Q2: Orientation, reference assignment, and occlusion.

The expanded ICT Annotation appendix details task-defined frames. For non-collinear keypoint direction d=p2−p1\mathbf d=\mathbf p_2-\mathbf p_1 and consistently oriented surface normal n\mathbf n in the same frame,

\mathbf u=\mathbf d-(\mathbf z^\top\mathbf d)\mathbf z,\quad \mathbf x=\frac{\mathbf u}{\|\mathbf u\|},\quad R=[\mathbf x\;\mathbf z\times\mathbf x\;\mathbf z]\in SO(3).$$ Keypoint ordering and normal signs fix axis directions. Plugs use their longitudinal direction and table normal; ball axes are base-aligned without spin estimation. Corn roll is not independently recovered. References are assigned per arm and phase, switching from grasp target to receiving container/port at visually reviewed boundaries, without interpolation across objects. In microwave manipulation, the left arm uses the handle/box/placement target by phase, while the right uses the door. During occlusion, stationary poses are retained only within stationary intervals. Moving references require tracking; ball-centre interpolation is limited to valid observations at most 20 frames apart. Unsupported intervals and specified uncertain transitions are excluded. **W4 & Q4: Local reproductions and training-seed variability.** We added exact local RoboTwin results alongside official results. Both augmented variants build on the locally trained baselines and use matched evaluation conditions across all 50 tasks: Average gains over local baselines are 1.61 and 4.98 percentage points. | Backbone | Method | Clean (%) | Randomized (%) | Avg. (%) | | --- | --- | ---: | ---: | ---: | | $\pi_{0.5}$ | Local baseline | 70.84 | 48.38 | 59.61 | | $\pi_{0.5}$ | + Ours | **71.26** | **51.18** | **61.22** | | Fast-WAM | Local baseline | 78.28 | 3.32 | 40.80 | | Fast-WAM | + Ours | **80.08** | **11.48** | **45.78** | LIBERO is an established benchmark on which our matched local baselines already exceed 96% success, leaving limited headroom. Our method improves success from 96.85% to 97.40% for $\pi_{0.5}$ and from 96.75% to 97.80% for Fast-WAM. Although modest in absolute terms, these 0.55/1.05-point gains represent 17.5%/32.3% reductions in observed failure rates, supporting the method's benefit even with near-saturated baseline performance. Stage 2 uses seed 42. The 2,000 LIBERO evaluation episodes per model do not quantify training-seed variability; we acknowledge this limitation and do not claim statistical significance across training seeds. We also polish the manuscript. Thank you for your constructive feedback.
AuthorsOpenReview ↗

Gentle follow-up before the discussion deadline

Dear Reviewer XekY,

As the discussion period closes on Nov 18 (AoE), we would like to briefly follow up. Following your suggestions, we added the absolute end-effector target control, a parameter-matched MLP adapter (MAIF +4.50 points), the λ_ICT = 0 ablation, USB alignment error measurements, exact local RoboTwin reproductions, and expanded annotation details. We would greatly appreciate it if you could let us know whether these address your concerns, and if so, whether you would consider updating your assessment. Thank you!

Reviewer_1hBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a metric interaction framework that models object-level and scene-level interactions in physical Cartesian space at a shared metric scale. The method first encodes future end-effector poses in object coordinates as ICT and jointly denoises them with native actions. It then freezes the adapted policy and uses MAIF to combine decoded end-effector trajectories with metric scene point-cloud features, producing residual action corrections while preserving the original action representation and sampling procedure. Experiments cover five policy backbones, two simulation benchmarks (RoboTwin 2.0 and LIBERO), and five real-world tasks.

Critical strengths
  1. Precise manipulation requires understanding the spatial relationships between the end effector, objects, and the surrounding environment. Representing interactions through object-relative poses and connecting candidate actions to scene geometry through metric end-effector trajectories is an intuitive design that directly addresses these spatial requirements.
  2. Object poses are used only for training supervision, while ICTs are generated jointly with actions at inference. This allows the policy to benefit from explicit interaction geometry during learning without making real-time object-pose estimation a prerequisite for deployment.
Critical weaknesses
  1. The method relies on accurate depth and camera calibration, but provides no controlled evaluation of depth noise or calibration errors.
  2. End-to-end inference latency is not reported, so the additional overhead of ICT and MAIF remains unclear.
  3. Although Table 3 includes ICT, Flow Only, and scale ablations, it does not isolate the contributions of end-effector position supervision and MAIF’s auxiliary ICT supervision. Controlled comparisons with prior geometry-based adaptation methods are also missing.
  4. ICT relies on object-pose supervision during training, but the paper does not evaluate how translation and rotation errors in these annotations affect manipulation performance.
Questions for authors
  1. Reporting task success under realistic depth noise and calibration errors would help establish the method’s robustness.
  2. Reporting the end-to-end latency of the baseline and full method under identical hardware and sampling settings, including point-cloud preprocessing and geometry encoding, would clarify the additional inference overhead.
  3. Separate ablations of end-effector position supervision and MAIF’s auxiliary ICT supervision, starting from the same ICT-adapted checkpoint, would clarify their individual contributions.
  4. A controlled comparison with a geometry-aware VLA method such as GeoPredict, using the same backbone, training data, and evaluation protocol, would help establish the relative benefit.
  5. Evaluating how translation and rotation errors in the object-pose training annotations affect manipulation performance would clarify the method’s sensitivity to noisy supervision.
2 comments
AuthorsOpenReview ↗

Response 2

We thank the reviewer for the constructive feedback. We address overlapping concerns together below.

W1, W4 & Q1, Q5: Depth, calibration, and annotation noise.

Our original real-world gains already reflect imperfect depth sensing and calibration: held-out calibration self-consistency residuals are 6.83 mm/0.61∘0.61^\circ, with the same calibration used during training and evaluation. Sonata features achieve mean cosine similarity 0.9493 across natural depth fluctuations (mean temporal standard deviation: 0.48 mm), and 0.8826 relative to the original observed input under added 1 mm Gaussian noise. We have added detailed analyses to the appendix. To assess downstream sensitivity, we additionally evaluate controlled perturbations across five real-world tasks.

Depth and extrinsic perturbations are applied separately at evaluation without retraining: Gaussian depth noise has a 2 mm standard deviation, with geometry recomputed; extrinsic bias is 20 mm/2∘2^\circ and remains fixed within each rollout. For annotation noise, we instead retrain with jointly perturbed object poses (nominal p95: 10 mm/10∘10^\circ), recompute ICT targets, and evaluate without additional sensing perturbations. Annotation errors remain fixed per reference segment across training; observations, action labels, and other settings are unchanged.

ConditionFast-WAM + Oursπ0.5\pi_{0.5} + Ours
Original46.8%77.0%
Depth noise22.4%38.6%
Extrinsic bias23.0%42.0%
Training-annotation noise27.0%45.2%

These average success rates reveal substantial sensitivity to additional geometric corruption. We therefore qualify our claims: improvements under the original sensing and annotation conditions do not establish robustness to stronger input or supervision errors.

These are controlled stress tests, not calibrated sensor-noise models. Annotation perturbations jointly test translation and rotation at one noise level.

W2 & Q2: End-to-end computational latency.

We measured both methods on the same RTX 5090 GPU and Ryzen 9 9950X3D CPU, using batch size 1, 10 denoising steps, and a 50-step prediction horizon. Timing spans in-memory RGB-D inputs to CPU action outputs, including point-cloud preprocessing, geometry encoding, and action generation/postprocessing. Sensor acquisition, disk I/O, communication, and actuator execution are excluded.

Latency per action chunk (ms), measured over 100 paired runs after warm-up:

BackboneMethodMedianP95SonataMAIF correction
π0.5\pi_{0.5}Baseline109.8113.0——
π0.5\pi_{0.5}+ Ours207.1212.1≈31≈62
Fast-WAMBaseline181.2185.6——
Fast-WAM+ Ours291.6297.5≈34≈70

Sonata and MAIF costs are approximate measurements from separate profiling runs; MAIF includes all 10 corrections per chunk. These component timings are not an exact additive breakdown of total latency. The combined ICT + MAIF median overhead is 97.3 ms for π0.5\pi_{0.5} and 110.4 ms for Fast-WAM.

W3 & Q3: Separate supervision ablations.

Both ablations use the same frozen Stage 1 ICT-adapted Fast-WAM checkpoint, retain action-flow supervision, and change only the indicated Stage 2 loss.

MethodClean (%)Randomized (%)Avg. (%)
Stage 1 ICT only78.404.1041.25
Ours w/o action-position loss78.223.7640.99
Ours w/o Stage 2 ICT loss78.668.7043.68
Ours (Full)80.0811.4845.78

Removing action-position or Stage 2 ICT supervision lowers average success by 4.79 or 2.10 percentage points, respectively. Action-position supervision has the larger ablation effect in this setting; without it, Stage 2 does not improve the observed average success over ICT-only adaptation. ICT provides a complementary signal through shared MAIF parameters.

W3 & Q4: Controlled GeoPredict comparison.

We apologize for overlooking GeoPredict and have added it to Related Work and the RoboTwin 2.0 experiments. GeoPredict and its augmented variant use the same backbone, training data, and evaluation protocol.

MethodClean (%)Randomized (%)Avg. (%)
GeoPredict53.5623.7638.66
GeoPredict + Ours56.7426.9841.86

Our framework adds 3.18/3.22/3.20 percentage points in clean/randomized/average success, supporting its complementarity to an existing strong geometry-aware VLA method.

We have also polished the manuscript to improve clarity, precision, and readability.

Thank you again for your careful review and constructive suggestions.

AuthorsOpenReview ↗

Gentle follow-up before the discussion deadline

Dear Reviewer 1hBC,

As the discussion period closes on Nov 18 (AoE), we would like to briefly follow up. Following your suggestions, we added a controlled GeoPredict comparison (+3.20 points average), separate Stage 2 supervision ablations from the same checkpoint, end-to-end latency measurements, and depth/extrinsic/annotation noise stress tests. We would greatly appreciate it if you could let us know whether these address your concerns, and if so, whether you would consider updating your assessment. Thank you!

Reviewer_5JoWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new metric interaction framework to improve the 3D spatial precision of VLA and WAM policies. It introduces Interaction-Centric Tokens inspired by HumanEgo to model object-relative end-effector poses and a Metric Action Interaction Field to refine actions using 3D scene geometry. Experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks demonstrate consistent improvements over several baselines.

Critical strengths

The paper learning precise spatial relationships between actions, objects, and surrounding geometry, it is an important problem in robotic manipulation:

The proposed framework works on VLA and WAM architectures, with add on improvement, demonstrating good generality.

The experiments are relatively comprehensive, including simulation benchmarks, real-world tasks, and OOD evaluations. The improvements on precision-sensitive tasks like plug in are promissing.

Critical weaknesses

The ICT representation is adapted from HumanEgo, while MAIF mainly combines pretrained point-cloud features, cross-attention, and residual action correction. The individual components are largely based on existing techniques, and the main novelty appears to be their integration.

The proposed method need additional depth information and a pretrained 3D encoder, while several baselines primarily rely on RGB observations. The RoboTwin experiment even use the simulator oracle depth information. It is unclear how much improvement comes from the proposed metric. Stronger RGB-D baselines with comparable model would be helpful.

The paper lacks sufficient analysis of inference latency and robustness to depth noise and calibration errors. Moreover, real-world training requires perfect object pose, which may limit scalability. For precision tasks such as USB insertion, geometric information alone may also be insufficient without modeling contact forces and friction.

Questions for authors

Since ICT is adapted from HumanEgo, what are the main technical differences? Also, how much does ICT contribute to MAIF performance?

How sensitive is the method to depth noise, camera calibration errors, and inaccurate object pose annotations?

2 comments

We thank the reviewer for the careful review and insightful feedback. We address related concerns together below.

W1 & Q1: Technical novelty and ICT's contribution.

Unlike HumanEgo's ICT inputs describing observed entities, we jointly predict future object-relative interactions and native actions through shared blocks with bidirectional attention. Our compact 9D pose representation per arm and timestep outperforms a HumanEgo-style 29D prediction control in Stage 1 ICT-only ablations, improving randomized success on RoboTwin 2.0 by 2.78 and 2.28 percentage points for π0.5\pi_{0.5} and Fast-WAM, respectively.

In Stage 2, ICT supervision benefits action refinement through shared MAIF parameters. With the same frozen Stage 1 checkpoint and otherwise identical settings, removing only Stage 2 ICT supervision reduces Fast-WAM's performance:

MethodRandomized (%)Avg. (%)
Ours w/o Stage 2 ICT loss8.7043.68
Ours (Full)11.4845.78

Thus, ICT benefits both Stage 1 adaptation and Stage 2 refinement, without requiring an additional MAIF branch at inference.

W2: RGB-D baseline and metric interaction.

We acknowledge that RoboTwin uses simulator depth and most baselines receive RGB at inference. We therefore added PointmapVLA, an RGB-D baseline using metric 3D pointmaps. Adding our mechanism improves all five real-world tasks:

MethodAverage success (%)
PointmapVLA56.2
PointmapVLA + Ours61.0
Gain+4.8 percentage points

Table 3 further varies metric scale and coordinate systems while keeping depth inputs and the pretrained 3D encoder unchanged. These controlled ablations support the contribution of correct metric scale and coordinate alignment beyond access to depth alone.

W3 & Q2: Latency, robustness, and practical limitations.

We measured computational latency on the same RTX 5090 GPU and Ryzen 9 9950X3D CPU, with batch size 1, 10 denoising steps, and a 50-step prediction horizon. Timing spans in-memory RGB-D inputs to CPU action outputs, including point-cloud preprocessing, geometry encoding, and action generation/postprocessing. Sensor acquisition, disk I/O, communication, and actuator execution are excluded.

Latency per action chunk (ms), over 100 paired measurements after warm-up:

BackboneMethodMedianP95SonataMAIF correction
π0.5\pi_{0.5}Baseline109.8113.0——
π0.5\pi_{0.5}+ Ours207.1212.1≈31≈62
Fast-WAMBaseline181.2185.6——
Fast-WAM+ Ours291.6297.5≈34≈70

Sonata and MAIF costs are approximate results from separate profiling runs, not an exact additive breakdown. MAIF costs include all 10 corrections. Action-chunk execution takes 1.67/1.40 s for π0.5\pi_{0.5}/Fast-WAM, exceeding their P95 inference latencies of 0.212/0.298 s and leaving headroom for overlapped inference. Supplementary videos show execution visually as smooth as the baselines.

We added calibration and depth-noise analyses to the appendix. Our real-world gains already reflect imperfect sensing and calibration: held-out calibration self-consistency residuals are 6.83 mm/0.61∘0.61^\circ, with the same calibration used during training and evaluation. Sonata features achieve cosine similarities of 0.9493 under natural depth fluctuations (mean temporal standard deviation: 0.48 mm) and 0.8826 under added 1 mm Gaussian noise, supporting feature-level tolerance to modest depth noise.

We also evaluate three separate perturbations across five real-world tasks. Depth noise and extrinsic bias are applied at evaluation without retraining; annotation noise is applied during retraining, followed by evaluation without additional sensing perturbations.

ConditionFast-WAM + Ours: Avg. (%)π0.5\pi_{0.5} + Ours: Avg. (%)
Original46.877.0
Depth noise: 2 mm std.22.438.6
Extrinsic bias: 20 mm/2∘2^\circ23.042.0
Annotation noise: nominal p95 10 mm/10∘10^\circ27.045.2

These substantial drops reveal sensitivity to additional geometric corruption. These are controlled stress tests, not calibrated models of the full sensor-error distribution; benefits under natural conditions do not imply robustness to stronger perturbations.

Human annotation limits scalability. Our semi-automated pipeline, detailed in the appendix, keeps annotation effort manageable for task-specific post-training. Object poses are needed only for training targets, with no annotation required at inference. We also agree that precision tasks may require force/tactile feedback; incorporating these signals and explicitly modeling contact forces and friction remain beyond this work's scope.

We have polished the manuscript to improve clarity, precision, and readability.

Thank you again for your careful review and constructive feedback.

AuthorsOpenReview ↗

Gentle follow-up before the discussion deadline

Dear Reviewer 5JoW,

As the discussion period closes on Nov 18 (AoE), we would like to briefly follow up. In response to your comments, we added an RGB-D baseline (PointmapVLA + Ours: +4.8 points on five real-world tasks), a Stage 2 ICT ablation, end-to-end latency measurements, and depth/calibration/annotation noise analyses. We would greatly appreciate it if you could let us know whether these address your concerns, and if so, whether you would consider updating your assessment. Thank you!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response(1/2)

Dear Reviewers, ACs, and SACs,

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate the recognition of our motivation, method, and experiments. Reviewers highlighted:

  • Addressing an important problem, precise spatial relationships among actions, objects, and scene geometry, through an intuitive object-relative design (5JoW, 1hBC).
  • Good generality as an add-on to both VLA and WAM architectures, evaluated across five backbones on 50 RoboTwin tasks (5JoW, XekY).
  • Promising gains on precision-sensitive real-robot tasks such as USB insertion (5JoW, XekY).
  • Practical deployment: object poses serve only as training supervision, with no pose estimation required at inference (1hBC).
  • Comprehensive evaluation across simulation benchmarks, real-world tasks, and OOD settings (5JoW, XekY).

We have incorporated the reviewers' suggestions and added new experiments and analyses to strengthen the submission.


Summary of Key Revisions and Additions

  • Two new baselines requested by reviewers: (i) PointmapVLA [1], an RGB-D VLA using metric robot-centric pointmaps, where adding our mechanism improves real-world average success from 56.2% to 61.0% (Appendix X) (5JoW); (ii) GeoPredict [2], a geometry-aware VLA, where adding our framework improves average success on RoboTwin 2.0 by +3.20 points under the same backbone, training data, and protocol (Table X) (1hBC).
  • Matched controls: object-relative ICT vs. absolute end-effector targets (+1.82 / +1.43), and MAIF vs. a parameter-matched MLP adapter (+4.50) (Appendix X). (XekY, 5JoW)
  • Supervision ablations from the same frozen Stage 1 checkpoint, isolating action-position (−4.79) and Stage 2 ICT (−2.10) supervision (Appendix X). (5JoW, 1hBC, XekY)
  • Quantified end-to-end latency on a single RTX 5090, with median overhead of 97.3 / 110.4 ms, well below action-chunk execution time (Appendix X). (5JoW, 1hBC)
  • Robustness analyses: calibration residuals, depth-noise feature similarity, and controlled stress tests for depth noise, extrinsic bias, and annotation noise, with claims qualified accordingly (Appendix X). (5JoW, 1hBC)
  • Execution-level precision measurements for USB alignment and barrier crossing (Appendix X). (XekY)
  • Exact local RoboTwin reproductions reported alongside official results (Table X). (XekY)
  • Expanded ICT annotation details (orientation, reference assignment, occlusion) and expanded limitations (annotation scalability, contact forces, training-seed variability) (Appendix X). (XekY, 5JoW)

The following section summarizes our key responses to each reviewer.

References

[1] D. Hwang, D. Kim, B. Lee, K. Lee, D. Lee, H. Lee, M. Park, and J. Choo. PointmapVLA: Scaling Robot Learning with Robot-Centric Pointmaps. 2026. https://github.com/DAVIAN-Robotics/PointmapVLA

[2] J. Qian, B. Han, C. Shi, L. Xiao, L. Yang, S. Shi, and L. Jiang. GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation. In CVPR, pp. 13529–13539, 2026.

AuthorsOpenReview ↗

General Response (2/2)

A Summary of Our Rebuttal

Reviewer 5JoW

  • Main concerns: (1) technical differences from HumanEgo and ICT's contribution to MAIF; (2) depth access relative to RGB baselines, calling for a stronger RGB-D baseline; (3) latency, robustness to depth/calibration/annotation errors, and annotation scalability.
  • Our response: (1) ICT jointly predicts object-relative interactions with native actions; our compact 9D representation outperforms a HumanEgo-style 29D control (+2.78 / +2.28 randomized), and removing Stage 2 ICT supervision lowers Fast-WAM's average from 45.78% to 43.68%; (2) on PointmapVLA [1], an RGB-D baseline with metric pointmaps, our mechanism improves all five real-world tasks (56.2% → 61.0%), while Table 3 isolates metric scale and coordinate alignment with depth inputs fixed; (3) measured latency (207.1 / 291.6 ms median per chunk, within the 1.67 / 1.40 s chunk execution time), added noise analyses and stress tests with qualified claims, and clarified that object poses are needed only for training.

Reviewer 1hBC

  • Main concerns: (1) robustness to depth noise, calibration errors, and annotation errors; (2) end-to-end latency; (3) separate contributions of action-position and Stage 2 ICT supervision; (4) a controlled comparison with GeoPredict [2].
  • Our response: (1) controlled stress tests (2 mm depth noise, 20 mm / 2° extrinsic bias, 10 mm / 10° annotation noise) and feature-level analyses, with claims explicitly qualified; (2) median latency of 207.1 ms (π0.5\pi_{0.5}) and 291.6 ms (Fast-WAM), including point-cloud preprocessing and geometry encoding; (3) from the same frozen Stage 1 checkpoint, removing action-position or Stage 2 ICT supervision lowers average success by 4.79 and 2.10 points; (4) GeoPredict [2] + Ours improves GeoPredict from 38.66% to 41.86% with the same backbone, data, and protocol.

Reviewer XekY

  • Main concerns: (1) matched controls for object-relative targets and the MAIF adapter, and the contribution of Stage 2 ICT; (2) execution-level precision beyond offline ADE; (3) orientation recovery, reference assignment, and occlusion handling in annotation; (4) local reproductions and training-seed variability.
  • Our response: (1) ICT outperforms absolute end-effector targets (+1.82 / +1.43), MAIF outperforms a parameter-matched MLP adapter (+4.50 average, +8.88 randomized), and λICT=0\lambda_{\mathrm{ICT}}=0 lowers average success by 2.10; (2) lower USB alignment errors (4.810 → 4.544 mm; 5.285 → 5.176 mm) and reduced barrier-crossing height (307.2 → 282.6 mm), reported as limited evidence; (3) an expanded annotation appendix covering frame construction, per-arm and per-phase reference assignment, and occlusion handling; (4) exact local reproductions showing +1.61 (π0.5\pi_{0.5}) and +4.98 (Fast-WAM) over local baselines, with training-seed variability stated as a limitation.

Should there be any questions about the details of our real-robot experiments or the human-inspired motivation of our method, we sincerely invite the reviewers and ACs to view the videos in our supplementary material, which provide extensive visualizations of our real-robot experiments.

As the discussion period closes, we hope this summary helps the reviewers and ACs assess our responses. We thank the reviewers, ACs, and SACs for their time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.