acceptodds
Under review as a conference paper at ICLR 2027

PointmapVLA: Scaling Robot Learning with Robot-Centric Pointmaps

Abstract

Vision-language-action models (VLAs) often struggle with precise manipulation because image observations do not directly provide the metric 3D geometry of the physical environment. We introduce PointmapVLA, the first VLA pretrained at scale with robot-centric pointmaps for joint 3D spatial understanding and robot control. Because pretrained VLM backbones lack a prior for interpreting pointmaps, we jointly pretrain on pointmap-augmented robot demonstrations and 3D spatial VQA, supervising 3D grounding, metric localization, spatial relation reasoning alongside action prediction. To support this pretraining, we construct a pointmap-paired corpus of 76.3M robot demonstration frames and 10.7M 3D spatial VQA examples. Across three simulation benchmarks and three real-world tasks, PointmapVLA delivers strong manipulation performance, establishing a new state of the art with a 92.4% success rate on LIBERO-Plus. In real-world manipulation under unseen target heights, lighting, and camera viewpoints, it achieves 1.8 the success rate of the strongest geometry-aware baseline. We will release the training code, model weights, and pointmap-paired training corpus.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Robot Foundation Models, Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_m4CAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PointmapVLA, which represents multi-view pointmaps in the robot-base coordinate frame and uses them for VLApretraining. The model is jointly trained on robot action prediction and 3D spatial question answering, followed by a second pretraining stage that introduces continuous action generation. The authors construct a corpus of approximately 76.3 million pointmap-paired robot frames and 10.7 million spatial QA examples. They report strong results on LIBERO, LIBERO-Plus, RoboCasa, and three real-robot tasks.

Critical strengths
  1. The robot-base coordinate frame is a well-motivated design choice. Expressing pointmaps from different cameras in a shared frame aligns multi-view geometry with the coordinates in which robot actions are executed. Although robot-centric pointmaps have appeared in prior work, the paper applies this idea consistently across its training pipeline.

  2. The paper covers standard simulation benchmarks, robustness to perturbations, real-robot manipulation, and metric 3D understanding. Its pretraining ablation shows that adding spatial QA to robot demonstrations improves success from 76.4% to 78.2% on RoboCasa and from 88.2% to 92.4% on LIBERO-Plus. This is useful evidence for the claimed connection between explicit spatial learning and action prediction.

Critical weaknesses
  1. My main concern is the degree of novelty and the limited insight provided by scaling. Depth, point clouds, and pointmaps have already been explored as inputs to robot policies. Prior work has also used 3D supervision to improve robot control. The main contribution here therefore appears to be combining and scaling existing ideas, rather than introducing a substantially new representation or model mechanism. If large-scale pretraining is the central contribution, the paper would be stronger with data-scaling curves, an analysis of the marginal value of different data sources, or other findings that explain what changes at this scale. Without such insights, the contribution reads primarily as an engineering-scale exploration.

  2. The current ablations do not fully isolate the contributions of the proposed training recipe. The comparison between robot-base and camera-frame pointmaps is conducted without the proposed pretraining, so it does not directly establish the benefit of that coordinate choice under the full training setting. Controlled ablations of these choices would better support the paper’s central claims.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer m4CA (1/4) - W1 (b) (data scale)

We appreciate the thoughtful review and constructive feedback. We hope our responses address the reviewer's concerns.

To strengthen our central claim on large-scale pretraining with robot-centric pointmaps, we conducted the two analyses the reviewer suggested, namely how performance scales with the amount of pretraining data and how much each data source contributes.

Data scaling (W1)

Since rerunning pretraining on several subsets of the full mixture was beyond our compute budget during the rebuttal period, we pretrained on 25%, 50%, and 100% of the DROID episodes (one epoch each) with either RGB or RGB + robot-centric pointmaps, and fine-tuned each model with either input. We also report our full mixture of DROID, InternData-M1, and spatial VQA as the largest data point.

The table below reports RoboCasa success rates (%). PM denotes robot-centric pointmaps. These results are reported in Section 4.4 and Figure 7, with full results in Appendix E.3 and Table 22 of the revised paper.

Pretraining inputFine-tuning inputNo pretraining25% DROID50% DROID100% DROIDFull (DROID + InternData-M1 + VQA)
ARGBRGB67.9 ± 0.168.2 ± 0.868.5 ± 0.969.6 ± 0.771.5 ± 0.3
BRGBRGB + PM73.6 ± 0.371.1 ± 0.372.5 ± 0.771.7 ± 0.374.5 ± 0.4
CRGB + PMRGB67.9 ± 0.166.9 ± 1.466.1 ± 1.069.3 ± 0.671.6 ± 0.2
DRGB + PMRGB + PM73.6 ± 0.373.2 ± 0.874.4 ± 1.175.3 ± 1.178.2 ± 0.9

Without pretraining, a model depends only on its fine-tuning input, so A and C share one value, as do B and D

We derive three observations from these results. First, robot-centric pointmaps bring larger gains from additional pretraining data than RGB alone. Pretraining on 25% of DROID alone brings no measurable change for either input. However, from 25% of DROID to the full mixture, the model pretrained and fine-tuned with PM (i.e., row D in the table above) improves by 5.0 points (73.2% to 78.2%), whereas the model pretrained and fine-tuned with RGB only (A) improves by 3.3 points (68.2% to 71.5%). D outperforms A at every scale, by 5.0, 5.9, and 5.7 points at 25%, 50%, and 100% of DROID and by 6.7 points with the full mixture.

Second, adding pointmaps only at fine-tuning helps, but learning them during pretraining helps further, and increasingly so with more data. The model pretrained with RGB and fine-tuned with PM (B) outperforms A by 2.9, 4.0, 2.1, and 3.0 points at 25%, 50%, and 100% of DROID and the full mixture. D outperforms B by a further 2.1 and 1.9 points at 25% and 50% of DROID, and by 3.6 and 3.7 points at 100% of DROID and the full mixture.

Third, pretraining with PM brings no benefit when the model is fine-tuned with RGB alone. The model pretrained with PM and fine-tuned with RGB only (C) shows no improvement over A and achieves similar performance across all scales (66.9% vs. 68.2%, 66.1% vs. 68.5%, 69.3% vs. 69.6%, and 71.6% vs. 71.5%). We believe this is because PM pretraining teaches the model to exploit 3D geometry when RGB and pointmaps are provided together, so the pointmaps must also be given at fine-tuning.

Overall, adding robot-centric pointmaps to RGB turns additional pretraining data into larger gains than RGB alone, provided that the pointmaps are also used at fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer m4CA (2/4) - W1 (b) (data sources)

Marginal value of each data source (W1)

We also examined how much each data source contributes by starting from DROID and adding InternData-M1 and then spatial VQA one at a time, for both RGB-only and RGB + robot-centric pointmap pretraining.

The table below reports success rates (%), with sources added cumulatively. DROID contains 33.7K real-robot episodes, InternData-M1 176.6K simulated episodes, and spatial VQA 14.8M samples (10.7M pointmap-paired 3D spatial VQA and 4.1M RGB-only VQA). These results are reported in Section 4.4 and Table 6 of the revised paper.

Pretraining dataRoboCasaLIBERO-PlusReal world (seen / unseen)
RGB only
None67.9 ± 0.177.7 ± 0.4- / -
DROID69.6 ± 0.778.5 ± 0.940 / 17
+ InternData-M170.2 ± 1.379.9 ± 1.047 / 18
+ Spatial VQA71.5 ± 0.380.4 ± 0.152 / 22
RGB + robot-centric pointmap
None73.6 ± 0.382.6 ± 0.1- / -
DROID75.3 ± 1.186.9 ± 0.762 / 33
+ InternData-M176.4 ± 0.188.2 ± 0.970 / 38
+ Spatial VQA (ours)78.2 ± 0.992.4 ± 0.377 / 48

With robot-centric pointmaps, pretraining on the 33.7K real demonstrations from DROID substantially improves over no pretraining (+1.7 on RoboCasa and +4.3 on LIBERO-Plus) and reaches 62 / 33 on the real robot under seen / unseen conditions. Scaling such real data further is difficult, however, since pointmap-paired demonstrations require calibrated cameras and metric depth and are therefore harder to collect than RGB-only data. Simulation offers a practical alternative, as simulated demonstrations come with exact pointmaps by construction and scale easily, so we added the 176.6K InternData-M1 episodes, about five times as many as DROID. This brings further, though smaller, gains on the simulation benchmarks (+1.1 and +1.3) and improves real-robot performance (+8 / +5). Adding spatial VQA, which directly supervises 3D spatial understanding, then yields another large gain (+1.8, +4.2, and +7 / +10). Each data source thus consistently improves every benchmark and the real robot.

These gains, however, depend strongly on robot-centric pointmaps. The same data sources added under RGB-only pretraining reveal three differences. First, spatial VQA benefits the pointmap model far more (+4.2 vs. +0.5 on LIBERO-Plus and +7 / +10 vs. +5 / +4 on the real robot), suggesting that its 3D grounding questions are most useful when the model can anchor them in metric geometry. Second, the full mixture improves zero-shot robustness on LIBERO-Plus by 9.8 points over no pretraining with pointmaps (82.6% to 92.4%), but by only 2.7 points with RGB (77.7% to 80.4%). Third, on the real robot, the gap between the two widens under unseen conditions as data sources are added (16, 20, and 26 points), as the pointmap model improves from 33% to 48% while the RGB model improves only from 17% to 22%. Overall, robot-centric pointmaps allow the model to benefit substantially more from each additional data source, especially in robustness to unseen conditions.

AuthorsOpenReview ↗

Response to Reviewer m4CA (3/4) - W2

Robot-centric vs. camera-frame pointmaps under full pretraining (W2)

We agree that the coordinate-frame pointmap comparison should be conducted under the full pretraining recipe in order to establish the benefit of robot-centric pointmaps. We therefore pretrained a camera-frame pointmap variant identical to PointmapVLA except that its pointmaps and spatial VQA coordinates are expressed in each camera's frame. We also pretrained an RGB-only variant as a reference, and fine-tuned each model with its own pretraining input.

The table below reports success rates (%). LIBERO-Plus (camera) denotes the camera-viewpoint perturbation category of LIBERO-Plus, and each real-world row lists the condition changed in the unseen setting. These results are reported in Section 4.4 and Table 5 of the revised paper.

RGBRGB + camera-frame pointmapRGB + robot-centric pointmap (ours)
Simulation
RoboCasa71.5 ± 0.376.0 ± 0.378.2 ± 0.9
LIBERO98.7 ± 0.398.8 ± 0.199.0 ± 0.1
LIBERO-Plus (total)80.4 ± 0.186.6 ± 0.392.4 ± 0.3
LIBERO-Plus (camera)51.1 ± 1.161.1 ± 0.688.0 ± 0.5
Real world (seen / unseen)
Peg in hole (unseen lighting)55 / 2060 / 3075 / 45
Stack blocks (unseen target height)40 / 1555 / 2575 / 50
Pick-and-place (unseen camera viewpoint)60 / 3070 / 3580 / 50
Average52 / 2262 / 3077 / 48

Under matched pretraining, robot-centric pointmaps outperform camera-frame pointmaps on every benchmark (+2.2 on RoboCasa, +5.8 on LIBERO-Plus, and +15 / +18 on the real robot under seen / unseen conditions). The advantage observed without pretraining (Table 6 of the submission) thus also holds under full pretraining, and it is largest when the camera viewpoint changes. LIBERO-Plus evaluates the same LIBERO-fine-tuned models under perturbations, without further fine-tuning. On unperturbed LIBERO, both variants perform alike at about 99%. Under the camera perturbations of LIBERO-Plus, however, the camera-frame variant drops to 61.1% while PointmapVLA retains 88.0%. On the real robot under the unseen camera viewpoint in Pick-and-place, PointmapVLA likewise reaches 50%, compared with 35% for the camera-frame variant. Camera-frame pointmaps express each view in its own coordinate frame, so the same scene has different coordinates across cameras and these coordinates change whenever a camera moves, whereas robot-centric pointmaps place all views in the shared frame in which robot actions are executed. Camera-frame pointmaps still improve over RGB-only pretraining (+4.5, +6.2, and +10 / +8), showing that metric geometry itself is useful, but expressing it in the robot-centric frame brings a substantial further gain. These results indicate that the robot-centric frame remains a key design choice under full pretraining, especially for robustness to camera viewpoint changes.

Reviewer_hUrPOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PointmapVLA extends a pretrained vision-language model with robot-centric pointmaps to support spatial understanding and robot control. RGB images and aligned pointmaps are processed by separate visual encoders and fused at the patch level. A two-stage training procedure combines spatial VQA and discrete action prediction before introducing a continuous action expert. To support this training, the authors construct a corpus containing 76.3M pointmap-paired robot demonstration frames and 10.7M spatial VQA examples from existing real-world, simulated, and spatial-understanding datasets. The evaluation covers three simulation benchmarks, three real-world manipulation tasks, and a spatial reasoning benchmark. The reported results show substantial gains from the overall pretraining recipe, particularly on LIBERO-Plus, where success improves from 82.6% to 92.4%. The model also achieves 78.2% on RoboCasa and, after task-specific fine-tuning, average real-world success rates of 77% and 48% under seen and unseen conditions, respectively.

Critical strengths
  1. Substantial data construction. The authors assemble a relatively large pointmap-paired corpus from heterogeneous robot demonstrations and spatial-understanding datasets. Establishing compatible geometric observations and supervision across these sources requires substantial technical work. The resulting data and training pipeline could provide a useful foundation for further research on geometric VLA pretraining.

  2. Effective pretraining. Table 7 shows consistent improvements from the overall pretraining recipe on RoboCasa and LIBERO-Plus, with additional gains from incorporating VQA supervision. The improvement on LIBERO-Plus is particularly encouraging, suggesting that the training recipe benefits robustness to the evaluated perturbations. These ablations provide meaningful support for the practical value of the approach.

  3. Broad experimental evaluation. The experiments cover simulation, real-world manipulation, spatial understanding, and several architectural and training ablations. The real-world comparisons use the same downstream demonstrations and fine-tuning schedule across methods. Together, these evaluations provide a reasonably comprehensive assessment of the system within the tested settings.

Critical weaknesses
  1. Limited conceptual novelty. Robot-centric pointmaps and spatial supervision for action learning have clear precedents, and the encoder initialization and feature-fusion mechanisms are relatively straightforward. The main novelty appears to lie in extending these ideas to a larger joint pretraining setup through substantial data construction and integration. I find this practically valuable, but the conceptual contribution of the core method remains limited.
  2. Unmatched RoboCasa comparisons. PointmapVLA uses 300 MimicGen demonstrations per task, whereas SeeR-VLA uses 50 human demonstrations and SA-WAM reports 50 demonstrations. The fine-tuning settings also differ in the number of training steps and batch size. These comparisons therefore do not establish superiority under matched training conditions. This is particularly important when interpreting the gap to SeeR-VLA as evidence that introducing pointmaps during pretraining is more effective than introducing them during downstream fine-tuning.
  3. The contribution of real-world pretraining data is unclear. Table 7 examines the overall pretraining recipe and the addition of VQA, but does not separate the contributions of real and simulated robot demonstrations. Consequently, it remains unclear how much the real-world action data contributes to the real-robot results. A more focused analysis would help explain the value of the chosen data mixture.
Questions for authors
  1. RoboCasa comparison protocol. Could the authors clarify the demonstration sources, counts, and training settings used by the key baselines, including whether the results were reproduced or taken from prior publications? A comparison with at least one key baseline under matched demonstration sources, counts, and evaluation conditions would substantially strengthen the empirical conclusions.

  2. Pointmaps during pretraining. Have the authors compared RGB-only pretraining with RGB + pointmap pretraining using the same action/VQA examples, supervision, and comparable training budgets, followed by identical RGB + pointmap fine-tuning and evaluation? This would isolate the value of introducing pointmaps during pretraining. If such a comparison is unavailable, please qualify the corresponding interpretation of the SeeR-VLA performance gap.

  3. Real-world action data. How much does real-world action data contribute beyond simulated action data? An ablation replacing real action samples with simulated samples, while keeping the VQA mixture, training budget, and downstream fine-tuning fixed, would help quantify its incremental value.

8 comments
AuthorsOpenReview ↗

Response to Reviewer hUrP (1/4) - W2/Q1

We appreciate the detailed review and constructive questions. We hope our responses address the reviewer's concerns.

Comparison under matched demonstrations and fine-tuning budgets (W2, Q1)

All baseline results on RoboCasa are taken from the original papers rather than reproduced. Our paper setting fine-tunes PointmapVLA on 300 MimicGen demonstrations per task with a batch size of 128 for 20K updates, while each baseline uses its own configuration. Following the reviewer's suggestion, we fine-tuned PointmapVLA with the same 50 human demonstrations per task and the fine-tuning budget of each baseline. For SA-WAM, whose batch size of 800 was too costly for us, we kept its 45K updates but reduced the batch size to 128.

The table below reports RoboCasa success rates (%). These results are reported in Appendix C.3.1 and Table 13 of the revised paper.

MethodDemos / taskFine-tuning budget (batch × steps)RoboCasa
SeeR-VLA (reported)50 human64 × 20K61.5
Ours50 human64 × 20K78.1 ± 0.9
SA-WAM (reported)50 human800 × 45K76.6
Ours50 human128 × 45K78.9 ± 0.6
Ours50 human128 × 20K78.1 ± 0.7
Ours (paper setting)300 MimicGen128 × 20K78.2 ± 0.9

PointmapVLA outperforms SeeR-VLA under its fine-tuning setting (78.1% vs. 61.5%) and SA-WAM even with a 6.25× smaller batch (78.9% vs. 76.6%). With 50 human demonstrations, it also matches the paper setting with 300 MimicGen demonstrations (78.1% vs. 78.2%). These results indicate that the advantage over both baselines does not come from differences in demonstrations or fine-tuning budgets, but from the stronger backbone that PointmapVLA builds through large-scale 3D pretraining on robot-base pointmaps.

However, we acknowledge that the gap between SeeR-VLA and PointmapVLA should not be attributed solely to pointmap pretraining. The two models also differ in pretraining data and initial backbone, so this comparison cannot isolate the benefit of introducing robot-base pointmaps during pretraining rather than only at fine-tuning. We isolate this factor under the same pretraining data and budget in our response to Q2.

AuthorsOpenReview ↗

Response to Reviewer hUrP (2/4) - Q2

RGB-only vs. RGB + robot-centric pointmap pretraining under matched conditions (Q2)

In order to isolate the benefit of introducing robot-base pointmaps during pretraining rather than only at fine-tuning, we pretrained an RGB-only variant with the same robot demonstrations, spatial VQA data, and pretraining schedule as PointmapVLA. We then fine-tuned this variant and PointmapVLA under the same protocol, both with RGB and robot-base pointmaps as input, so the two models differ only in whether pointmaps are used during pretraining.

The table below reports success rates (%). These results are reported in Appendix E.2 and Table 21 of the revised paper.

Pretraining inputFine-tuning inputRoboCasaLIBERO-PlusReal world (seen / unseen)
RGBRGB71.5 ± 0.380.4 ± 0.152 / 22
RGBRGB + robot-centric pointmap74.5 ± 0.482.3 ± 1.063 / 32
RGB + robot-centric pointmap (ours)RGB + robot-base pointmap78.2 ± 0.992.4 ± 0.377 / 48

Adding pointmaps only at fine-tuning improves over the RGB-only variant (+3.0 on RoboCasa, +1.9 on LIBERO-Plus, and +11 / +10 on the real robot under seen / unseen conditions), consistent with the finding of SeeR-VLA that robot-centric pointmaps benefit VLA fine-tuning. However, introducing robot-centric pointmaps during pretraining brings a clear further benefit (78.2% vs. 74.5% on RoboCasa and 77 / 48 vs. 63 / 32 on the real robot), which is especially large on LIBERO-Plus (92.4% vs. 82.3%). These results indicate that pointmaps are most effective when introduced during pretraining rather than only at fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer hUrP (3/4) - W3/Q3

Contribution of real-world robot data (W3, Q3)

We agree that replacing real action samples with simulated ones is the most direct way to quantify how much real-world robot data contributes to the real-robot results beyond simulated robot data. Since this ablation (InternData-M1 + VQA vs. DROID + InternData-M1 + VQA, with the VQA mixture, training budget, and downstream fine-tuning fixed) requires a full pretraining run, it is still in progress, and we sincerely apologize that we cannot report its results yet. We will share them in this thread as soon as they are available.

During the rebuttal period, we were able to extend our data-source ablation, which offers a partial answer to the same question by showing how much each pretraining data source contributes to the performance gains. Starting from real demonstrations from DROID alone, we added InternData-M1 and spatial VQA cumulatively, with the same downstream fine-tuning.

The table below reports success rates (%). All models use RGB and robot-centric pointmaps as input, following PointmapVLA. DROID contains 33.7K real-robot episodes, InternData-M1 176.6K simulated episodes, and spatial VQA 14.8M samples (10.7M pointmap-paired 3D spatial VQA and 4.1M RGB-only VQA). These results are reported in Section 4.4 and Table 6 of the revised paper.

Pretraining dataRoboCasaLIBERO-PlusReal world (seen / unseen)
None73.6 ± 0.382.6 ± 0.1- / -
DROID75.3 ± 1.186.9 ± 0.762 / 33
+ InternData-M176.4 ± 0.188.2 ± 0.970 / 38
+ Spatial VQA (ours)78.2 ± 0.992.4 ± 0.377 / 48

Pretraining on the 33.7K real demonstrations from DROID substantially improves over no pretraining (+1.7 on RoboCasa and +4.3 on LIBERO-Plus) and reaches 62 / 33 on the real robot under seen / unseen conditions. However, real demonstrations paired with pointmaps are harder to collect than RGB-only data, since they require calibrated cameras and metric depth. Simulation offers a practical alternative to collecting more real pointmap-paired demonstrations, since simulated demonstrations come with exact pointmaps by construction and scale easily. We therefore added the 176.6K InternData-M1 episodes, which is about five times as many as DROID. The additional simulation data brings further, though smaller, gains on the simulation benchmarks (+1.1 and +1.3) and improves real-robot performance (+8 / +5). Adding spatial VQA then yields another large gain (+1.8, +4.2, and +7 / +10). These results indicate that each data source contributes to PointmapVLA, with simulated demonstrations serving as a scalable complement to limited real pointmap data and spatial VQA further improving performance on top of both.

Reviewer_HbpXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PointmapVLA, which is pretrained with pointmaps and 3D spatial VQA for 3D spatial understanding. VLAs often struggle with precise manipulation, and PointmapVLA tries to solve this problem by pretraining on pointmap-augmented robot demonstrations and 3D spatial VQA. The authors evaluate the model on both simulation benchmarks and real-world tasks.

Critical strengths
  1. This paper pretrains PointmapVLA with pointmaps and spatial VQA, while constructing the corresponding training data and procedure.

  2. The experimental results of PointmapVLA are competitive on both simulation and real-world tasks. The ablation study also clarifies the contribution of spatial VQA.

Critical weaknesses
  1. The paper needs to isolate the contributions of robot-centric pointmaps, camera-frame pointmaps, and RGB-only inputs under the same pretraining setting.

  2. The authors need to clarify the coverage of the pretraining data. The training schedule described in the paper suggests that a large portion of the pretraining data was not covered during training.

  3. The contribution of robot-centric geometry to real-world task performance is unclear. It would be better to test missing, mismatched, or noise-perturbed pointmaps under controlled experimental conditions.

Questions for authors
  1. Please compare robot-centric pointmaps, camera-frame pointmaps, and RGB-only inputs under the same pretraining setting.

  2. Please clarify the sampling procedure and coverage of the pretraining data.

  3. Please clarify the contribution of robot-centric geometry to real-world task performance. It is necessary to test missing, mismatched, or noise-perturbed pointmaps under controlled experimental conditions.

6 comments
AuthorsOpenReview ↗

Response to Reviewer HbpX (1/3) - W1/Q1, W2/Q3

We appreciate the insightful comments and positive support with constructive feedback. We hope our responses address the reviewer's concerns.

Matched pretraining comparison with RGB-only and camera-frame pointmaps (W1, Q1, W2)

We agree that a matched pretraining comparison would strengthen the evidence for how much of the performance gain comes from robot-base pointmaps. We therefore pretrained an RGB-only variant and a camera-frame pointmap variant with the same robot data, spatial VQA, and schedule as PointmapVLA. Each model is fine-tuned with the same input as in its pretraining and evaluated in simulation and on the real robot under the same seen and unseen conditions.

The table below reports success rates (%). LIBERO-Plus (camera) denotes the camera-viewpoint perturbation category of LIBERO-Plus, and each real-world row lists the condition changed in the unseen setting. All models share the same robot demonstrations, spatial VQA data, and pretraining schedule, and each model is fine-tuned with the same input type it was pretrained with. These results are reported in Section 4.4 and Table 5 of the revised paper.

RGBRGB + camera-frame pointmapRGB + robot-centric pointmap (ours)
Simulation
RoboCasa71.5 ± 0.376.0 ± 0.378.2 ± 0.9
LIBERO98.7 ± 0.398.8 ± 0.199.0 ± 0.1
LIBERO-Plus (total)80.4 ± 0.186.6 ± 0.392.4 ± 0.3
LIBERO-Plus (camera)51.1 ± 1.161.1 ± 0.688.0 ± 0.5
Real world (seen / unseen)
Peg in hole (unseen lighting)55 / 2060 / 3075 / 45
Stack blocks (unseen target height)40 / 1555 / 2575 / 50
Pick-and-place (unseen camera viewpoint)60 / 3070 / 3580 / 50
Average52 / 2262 / 3077 / 48

Adding camera-frame pointmaps improves over RGB-only pretraining (+4.5 on RoboCasa, +6.2 on LIBERO-Plus, and +10 / +8 on the real robot under seen / unseen conditions), and expressing the same pointmaps in the robot-base frame improves performance further (+2.2, +5.8, and +15 / +18). The largest gap appears under camera-viewpoint changes. LIBERO-Plus evaluates the same LIBERO-fine-tuned models under perturbations, without further fine-tuning. On unperturbed LIBERO, all three variants perform alike at about 99%. Under the camera perturbations of LIBERO-Plus, however, the RGB-only and camera-frame variants drop to 51.1% and 61.1%, while PointmapVLA retains 88.0%. On the real robot, PointmapVLA likewise reaches 50% under the unseen camera viewpoint in Pick-and-place, compared with 35% and 30% for the camera-frame and RGB-only variants. Camera-frame pointmaps express each view in its own coordinate frame, so the same scene has different coordinates across cameras and changes whenever a camera moves. Robot-base pointmaps instead place all views in the shared frame in which robot actions are executed, keeping the scene geometry consistent across views and viewpoints. These results indicate that using robot-base pointmaps from the pretraining stage is a key reason for the performance and robustness of PointmapVLA, both in simulation and on the real robot.

AuthorsOpenReview ↗

Response to Reviewer HbpX (2/3) - W1/Q2

Pretraining sampling procedure and data coverage (W1, Q2)

We thank the reviewer for pointing out that the amount of pretraining data used was unclear. Our training uses all episodes from both datasets, but only a subset of the frames within each episode. Specifically, each update samples robot data from InternData-M1 and DROID at a fixed 70/30 ratio (Appendix A.4 and Figure 8 of the revised paper). Within each dataset, we first sample an episode uniformly and then a random timestep within it. The original schedule thus covers all 33.7K DROID episodes and all 176.6K InternData-M1 episodes, while sampling 5.5M frames from DROID (0.62 epochs over 8.8M frames) and 12.9M frames from InternData-M1 (0.19 epochs over 67.5M frames). Since consecutive frames within an episode are highly correlated, episode coverage helps characterize the diversity of scenes, objects, and tasks encountered during pretraining, separately from the number of frames sampled. To better reflect the data used during pretraining, we have revised the Abstract, Figure 1, Section 1, Figure3, and Section5 to describe the robot datasets by the number of episodes used rather than their total frame counts. We also report the sampling procedure and the number of frames sampled during training in Appendix A.4 (Table 7), clarifying that using all episodes does not mean using every frame.

To test whether sampling more frames from these episodes improves performance, we extended Stage 2, the continuous action co-training stage in which most robot frames are sampled, from 50K to 150K updates ("Longer"). This doubles the total schedule from the original 100K updates (Stage 1 50K + Stage 2 50K) to 200K updates (Stage 1 50K + Stage 2 150K) and increases the frames sampled to 13.2M on DROID (1.49 epochs) and 30.8M on InternData-M1 (0.46 epochs). The table below reports the data sampled during training and the resulting success rates (%) for downstream tasks.

Ours (Stage 1 50K + Stage 2 50K)Longer (Stage 1 50K + Stage 2 150K)
Data sampled during training
DROID (33.7K episodes, 8.8M frames)All episodes, 5.5M frames (0.62 epochs)All episodes, 13.2M frames (1.49 epochs)
InternData-M1 (176.6K episodes, 67.5M frames)All episodes, 12.9M frames (0.19 epochs)All episodes, 30.8M frames (0.46 epochs)
Spatial VQA (14.8M samples)18.4M samples (1.25 epochs)29.6M samples (2.01 epochs)
Success rate (%)
RoboCasa78.2 ± 0.979.4 ± 0.7
LIBERO99.0 ± 0.199.2 ± 0.1
LIBERO-Plus92.4 ± 0.391.0 ± 0.3
Real world (S / U)77 / 4880 / 55

Longer training improves performance on RoboCasa (+1.2), LIBERO (+0.2), and the real robot (+3 / +7 under seen / unseen conditions), while success on LIBERO-Plus decreases (−1.4). For LIBERO-Plus, we observed that the longer-pretrained checkpoint reaches a lower training loss during LIBERO fine-tuning. Since LIBERO-Plus evaluates the LIBERO-fine-tuned model under perturbations without further training, fitting the LIBERO data more closely can reduce robustness to these perturbations. A fine-tuning setup adjusted to this faster adaptation may therefore suit the longer-pretrained checkpoint better.

Overall, sampling more frames brings further gains on most benchmarks, but they are smaller than the gain from robot-base pointmap pretraining itself (+4.6 on RoboCasa and +9.8 on LIBERO-Plus over no pretraining, Table 6 of the revised paper).

To support further research on 3D robot foundation model pretraining, we plan to publicly release the pointmap annotations we constructed for all 176.6K InternData-M1 episodes, covering 67.5M frames. The original InternData-M1 provides only RGB observations, so we re-rendered its episodes from simulation metadata to obtain metric depth and construct robot-base pointmaps. Although depth can be rendered in simulation, large-scale robot datasets paired with metric depth remain limited. We hope that releasing these annotations will make 3D robot data more accessible and provide a valuable resource for the research community.

AuthorsOpenReview ↗

Response to Reviewer HbpX (3/3) - W2/Q3

Contribution of pointmaps and robustness to realistic errors (W2, Q3)

We appreciate the reviewer’s suggestion of these tests, which measure both how much the pointmaps contribute to PointmapVLA and how robust it is to realistic errors in them. We evaluated PointmapVLA on the real robot while perturbing only the pointmap input at test time, with 20 trials per task and condition. Each cell reports success rates (%) under seen / unseen conditions. Bold rows show PointmapVLA with unperturbed inputs and are repeated in each group for comparison. These results are reported in Section 4.3 and Figure 6, with per-task results in Appendix D.5 and Table 19 of the revised paper.

ConditionPegStackPnPAvg.
Pointmap input
Matched75 / 4575 / 5080 / 5077 / 48
Shuffled across views25 / 1520 / 545 / 3030 / 17
Removed25 / 105 / 035 / 2522 / 12
Depth source
Fast-FoundationStereo75 / 4575 / 5080 / 5077 / 48
Raw sensor depth75 / 4555 / 3080 / 5070 / 42
Calibration error
0°, 0 cm75 / 4575 / 5080 / 5077 / 48
1°, 1 cm75 / 4570 / 4080 / 5575 / 47
3°, 3 cm70 / 4060 / 3075 / 4568 / 38
5°, 5 cm50 / 2535 / 1570 / 4052 / 27
7°, 7 cm25 / 1015 / 545 / 3528 / 17

Contribution of pointmaps

We shuffled the pointmaps across views so that each RGB view is paired with the pointmap of another view, or removed the pointmaps entirely. Such perturbations lowered the average success from 77 / 48 to 30 / 17 and 22 / 12, respectively. With shuffling, every view still receives a pointmap, but it describes a different view, and success drops almost as much as with no pointmap at all. Having a pointmap as an extra input therefore does not help unless its geometry matches the view. These results indicate that PointmapVLA actively uses the pointmap geometry, rather than RGB alone, to understand the scene.

Robustness to raw sensor depth

Replacing the Fast-FoundationStereo depth with raw sensor depth leaves Peg in hole and Pick-and-place unchanged and keeps the average close to the unperturbed model (70 / 42 vs. 77 / 48). The drop is concentrated on Stack blocks (55 / 30 vs. 75 / 50), which relies most on close-range geometry, where raw sensor depth has more holes, mostly in the wrist camera. Fast-FoundationStereo removes these holes, running at about 15 Hz for all three views with 1.8 GB of peak GPU memory on the same RTX 4090 that runs the policy. PointmapVLA thus remains robust to raw sensor depth, while refined stereo depth adds a further gain at little extra cost.

Robustness to camera calibration error

We add rotation and translation errors of increasing magnitude to the camera extrinsics, from 1° and 1 cm up to 7° and 7 cm. Performance stays close to the unperturbed model at 1° and 1 cm (75 / 47 vs. 77 / 48) and decreases moderately at 3° and 3 cm (68 / 38). The drop becomes larger at 5° and 5 cm (52 / 27), and performance falls sharply at 7° and 7 cm (28 / 17). These results indicate that PointmapVLA tolerates calibration errors of up to 3° and 3 cm, and degrades substantially only under larger errors, such as when a camera is moved without recalibration.

We thank the reviewer again for the insightful suggestions. We hope these responses address the reviewer's concerns and would appreciate it if the reviewer could reconsider the assessment in light of them. We are happy to answer any further questions.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We are grateful to the reviewers for their thoughtful comments and their careful reading of our work. Reviewer HbpX noted that PointmapVLA goes beyond adding pointmaps as an input by building a training corpus and a training process that connect spatial understanding with robot actions. Reviewer m4CA found the robot-base frame a well-motivated design choice and noted that the paper applies it consistently across its training pipeline. Reviewer hUrP regarded the data construction as substantial and as a useful foundation for further research on geometric VLA pretraining, and the evaluation was seen as broad, spanning simulation, real-robot manipulation, and metric 3D understanding (hUrP, m4CA). The reviewers also acknowledged the strong results in simulation and on the real robot (HbpX), the gains from the pretraining recipe, particularly on LIBERO-Plus (hUrP), and the evidence that spatial VQA pretraining improves downstream control (HbpX, hUrP, m4CA).

In response to the reviews, we ran additional experiments and added clarifications, which fall into four groups:

  • Matched comparisons: We pretrained RGB-only and camera-frame pointmap variants with the same robot data, spatial VQA, and schedule as PointmapVLA, and compared all three in simulation and on the real robot (Section 4.4, Table 5). We also tested whether pointmaps need to be used during pretraining, by comparing PointmapVLA with a model that is pretrained with RGB only and receives pointmaps only at fine-tuning (Appendix E.2). For the RoboCasa baselines, we fine-tuned PointmapVLA on the same 50 human demonstrations per task as SeeR-VLA and SA-WAM, with fine-tuning budgets no larger than theirs (Appendix C.3.1).
  • Pretraining data scale and sources: We conducted a data-scaling experiment to examine how performance changes with the amount of pretraining data (Section 4.4, Figure 7; Appendix E.3), and a data-source ablation to measure the contribution of each pretraining data source (Section 4.4, Table 6).
  • Reliance on pointmaps and robustness to realistic errors: On the real robot, we tested how PointmapVLA responds when its pointmaps are perturbed by raw sensor depth or camera-calibration errors (Section 4.3, Figure 6), and when they are removed or paired with the wrong view (Appendix D.5).
  • Pretraining data coverage: We report the pretraining sampling procedure and the coverage of each dataset (Appendix A.4, Table 7), and tested whether longer pretraining further improves performance by doubling the number of pretraining updates (Appendix E.4). We also clarified the description of the robot data in the main text (the Abstract, Figure 1, Section 1, Figure3, and Section5).

In each reviewer's thread, we show the main results directly in our responses and cite the corresponding sections and tables of the revised manuscript.

We have uploaded the revised manuscript. Revised text is shown in red, and each revision is tagged in the margin with the ID of the reviewer who raised the point. Unless noted otherwise, the section, table, and figure numbers in our responses follow the revised manuscript.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of Discussion and Revisions

Dear AC and Reviewers,

Thank you for overseeing our submission and for the constructive feedback. We appreciate the reviewers' recognition of the well-motivated robot-base frame (m4CA), the substantial pointmap-paired data construction (HbpX, hUrP), the strong results in simulation and on the real robot (HbpX, hUrP), and the benefit of spatial VQA for downstream control (HbpX, hUrP, m4CA). The concerns we have addressed are briefly summarized below (revisions in red, tagged with reviewer IDs).

  1. Matched pretraining comparison with RGB-only and camera-frame pointmaps (HbpX W1/Q1, m4CA W2): Under matched pretraining, robot-centric pointmaps outperform both variants on every benchmark, most clearly under camera viewpoint changes (Section 4.4, Table 5).
  2. Pointmaps during pretraining vs. only at fine-tuning (hUrP Q2): With identical pretraining data and budget, introducing pointmaps from pretraining outperforms adding them only at fine-tuning on every benchmark (Appendix E.2, Table 21).
  3. Matched RoboCasa comparison with baselines (hUrP W2/Q1): With the same 50 demonstrations per task and fine-tuning budgets no larger than the baselines', PointmapVLA outperforms both SeeR-VLA and SA-WAM, even with a 6.25× smaller batch than SA-WAM (Appendix C.3.1, Table 13).
  4. Insight from data scaling and data sources (m4CA W1): Robot-centric pointmap pretraining outperforms RGB at every data scale, and each data source yields much larger gains with pointmaps than with RGB (Figure 7, Tables 6 and 22).
  5. Contribution of real-world robot data (hUrP W3/Q3): Removing DROID while keeping everything else fixed lowers real-robot success below even DROID-only pretraining, showing that real demonstrations are essential.
  6. Pretraining sampling procedure and data coverage (HbpX W1/Q2): We report the sampling procedure and coverage of each dataset, showing that pretraining covers all DROID and InternData-M1 episodes, and that longer pretraining brings further gains on most benchmarks (Appendices A.4 and E.4, Table 7).
  7. Contribution of robot-centric geometry on the real robot (HbpX W2/Q3): Shuffling or removing pointmaps sharply lowers real-robot success, while PointmapVLA remains robust to raw sensor depth and calibration errors up to 3° and 3 cm (Section 4.3, Figure 6).
  8. Novelty (hUrP W1, m4CA W1): We establish how robot-centric 3D representations should be used and scaled in VLA pretraining, and our controlled experiments show that the shared robot-base frame, its introduction from pretraining, and its interaction with data scale drive the gains.

We sincerely appreciate the reviewers' constructive feedback, which has substantially strengthened our empirical evidence and clarified the scope of our contribution. We have done our best to address every concern raised in the reviews, and we hope that the additional results and revisions resolve them. We kindly ask that these be considered during the final discussion.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.