- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PointmapVLA, which represents multi-view pointmaps in the robot-base coordinate frame and uses them for VLApretraining. The model is jointly trained on robot action prediction and 3D spatial question answering, followed by a second pretraining stage that introduces continuous action generation. The authors construct a corpus of approximately 76.3 million pointmap-paired robot frames and 10.7 million spatial QA examples. They report strong results on LIBERO, LIBERO-Plus, RoboCasa, and three real-robot tasks.
Critical strengths
-
The robot-base coordinate frame is a well-motivated design choice. Expressing pointmaps from different cameras in a shared frame aligns multi-view geometry with the coordinates in which robot actions are executed. Although robot-centric pointmaps have appeared in prior work, the paper applies this idea consistently across its training pipeline.
-
The paper covers standard simulation benchmarks, robustness to perturbations, real-robot manipulation, and metric 3D understanding. Its pretraining ablation shows that adding spatial QA to robot demonstrations improves success from 76.4% to 78.2% on RoboCasa and from 88.2% to 92.4% on LIBERO-Plus. This is useful evidence for the claimed connection between explicit spatial learning and action prediction.
Critical weaknesses
-
My main concern is the degree of novelty and the limited insight provided by scaling. Depth, point clouds, and pointmaps have already been explored as inputs to robot policies. Prior work has also used 3D supervision to improve robot control. The main contribution here therefore appears to be combining and scaling existing ideas, rather than introducing a substantially new representation or model mechanism. If large-scale pretraining is the central contribution, the paper would be stronger with data-scaling curves, an analysis of the marginal value of different data sources, or other findings that explain what changes at this scale. Without such insights, the contribution reads primarily as an engineering-scale exploration.
-
The current ablations do not fully isolate the contributions of the proposed training recipe. The comparison between robot-base and camera-frame pointmaps is conducted without the proposed pretraining, so it does not directly establish the benefit of that coordinate choice under the full training setting. Controlled ablations of these choices would better support the paper’s central claims.
Questions for authors
See Weaknesses.
5 comments
Response to Reviewer m4CA (2/4) - W1 (b) (data sources)
Marginal value of each data source (W1)
We also examined how much each data source contributes by starting from DROID and adding InternData-M1 and then spatial VQA one at a time, for both RGB-only and RGB + robot-centric pointmap pretraining.
The table below reports success rates (%), with sources added cumulatively. DROID contains 33.7K real-robot episodes, InternData-M1 176.6K simulated episodes, and spatial VQA 14.8M samples (10.7M pointmap-paired 3D spatial VQA and 4.1M RGB-only VQA). These results are reported in Section 4.4 and Table 6 of the revised paper.
| Pretraining data | RoboCasa | LIBERO-Plus | Real world (seen / unseen) |
|---|---|---|---|
| RGB only | |||
| None | 67.9 ± 0.1 | 77.7 ± 0.4 | - / - |
| DROID | 69.6 ± 0.7 | 78.5 ± 0.9 | 40 / 17 |
| + InternData-M1 | 70.2 ± 1.3 | 79.9 ± 1.0 | 47 / 18 |
| + Spatial VQA | 71.5 ± 0.3 | 80.4 ± 0.1 | 52 / 22 |
| RGB + robot-centric pointmap | |||
| None | 73.6 ± 0.3 | 82.6 ± 0.1 | - / - |
| DROID | 75.3 ± 1.1 | 86.9 ± 0.7 | 62 / 33 |
| + InternData-M1 | 76.4 ± 0.1 | 88.2 ± 0.9 | 70 / 38 |
| + Spatial VQA (ours) | 78.2 ± 0.9 | 92.4 ± 0.3 | 77 / 48 |
With robot-centric pointmaps, pretraining on the 33.7K real demonstrations from DROID substantially improves over no pretraining (+1.7 on RoboCasa and +4.3 on LIBERO-Plus) and reaches 62 / 33 on the real robot under seen / unseen conditions. Scaling such real data further is difficult, however, since pointmap-paired demonstrations require calibrated cameras and metric depth and are therefore harder to collect than RGB-only data. Simulation offers a practical alternative, as simulated demonstrations come with exact pointmaps by construction and scale easily, so we added the 176.6K InternData-M1 episodes, about five times as many as DROID. This brings further, though smaller, gains on the simulation benchmarks (+1.1 and +1.3) and improves real-robot performance (+8 / +5). Adding spatial VQA, which directly supervises 3D spatial understanding, then yields another large gain (+1.8, +4.2, and +7 / +10). Each data source thus consistently improves every benchmark and the real robot.
These gains, however, depend strongly on robot-centric pointmaps. The same data sources added under RGB-only pretraining reveal three differences. First, spatial VQA benefits the pointmap model far more (+4.2 vs. +0.5 on LIBERO-Plus and +7 / +10 vs. +5 / +4 on the real robot), suggesting that its 3D grounding questions are most useful when the model can anchor them in metric geometry. Second, the full mixture improves zero-shot robustness on LIBERO-Plus by 9.8 points over no pretraining with pointmaps (82.6% to 92.4%), but by only 2.7 points with RGB (77.7% to 80.4%). Third, on the real robot, the gap between the two widens under unseen conditions as data sources are added (16, 20, and 26 points), as the pointmap model improves from 33% to 48% while the RGB model improves only from 17% to 22%. Overall, robot-centric pointmaps allow the model to benefit substantially more from each additional data source, especially in robustness to unseen conditions.
Response to Reviewer m4CA (3/4) - W2
Robot-centric vs. camera-frame pointmaps under full pretraining (W2)
We agree that the coordinate-frame pointmap comparison should be conducted under the full pretraining recipe in order to establish the benefit of robot-centric pointmaps. We therefore pretrained a camera-frame pointmap variant identical to PointmapVLA except that its pointmaps and spatial VQA coordinates are expressed in each camera's frame. We also pretrained an RGB-only variant as a reference, and fine-tuned each model with its own pretraining input.
The table below reports success rates (%). LIBERO-Plus (camera) denotes the camera-viewpoint perturbation category of LIBERO-Plus, and each real-world row lists the condition changed in the unseen setting. These results are reported in Section 4.4 and Table 5 of the revised paper.
| RGB | RGB + camera-frame pointmap | RGB + robot-centric pointmap (ours) | |
|---|---|---|---|
| Simulation | |||
| RoboCasa | 71.5 ± 0.3 | 76.0 ± 0.3 | 78.2 ± 0.9 |
| LIBERO | 98.7 ± 0.3 | 98.8 ± 0.1 | 99.0 ± 0.1 |
| LIBERO-Plus (total) | 80.4 ± 0.1 | 86.6 ± 0.3 | 92.4 ± 0.3 |
| LIBERO-Plus (camera) | 51.1 ± 1.1 | 61.1 ± 0.6 | 88.0 ± 0.5 |
| Real world (seen / unseen) | |||
| Peg in hole (unseen lighting) | 55 / 20 | 60 / 30 | 75 / 45 |
| Stack blocks (unseen target height) | 40 / 15 | 55 / 25 | 75 / 50 |
| Pick-and-place (unseen camera viewpoint) | 60 / 30 | 70 / 35 | 80 / 50 |
| Average | 52 / 22 | 62 / 30 | 77 / 48 |
Under matched pretraining, robot-centric pointmaps outperform camera-frame pointmaps on every benchmark (+2.2 on RoboCasa, +5.8 on LIBERO-Plus, and +15 / +18 on the real robot under seen / unseen conditions). The advantage observed without pretraining (Table 6 of the submission) thus also holds under full pretraining, and it is largest when the camera viewpoint changes. LIBERO-Plus evaluates the same LIBERO-fine-tuned models under perturbations, without further fine-tuning. On unperturbed LIBERO, both variants perform alike at about 99%. Under the camera perturbations of LIBERO-Plus, however, the camera-frame variant drops to 61.1% while PointmapVLA retains 88.0%. On the real robot under the unseen camera viewpoint in Pick-and-place, PointmapVLA likewise reaches 50%, compared with 35% for the camera-frame variant. Camera-frame pointmaps express each view in its own coordinate frame, so the same scene has different coordinates across cameras and these coordinates change whenever a camera moves, whereas robot-centric pointmaps place all views in the shared frame in which robot actions are executed. Camera-frame pointmaps still improve over RGB-only pretraining (+4.5, +6.2, and +10 / +8), showing that metric geometry itself is useful, but expressing it in the robot-centric frame brings a substantial further gain. These results indicate that the robot-centric frame remains a key design choice under full pretraining, especially for robustness to camera viewpoint changes.
Response to Reviewer m4CA (1/4) - W1 (b) (data scale)
We appreciate the thoughtful review and constructive feedback. We hope our responses address the reviewer's concerns.
To strengthen our central claim on large-scale pretraining with robot-centric pointmaps, we conducted the two analyses the reviewer suggested, namely how performance scales with the amount of pretraining data and how much each data source contributes.
Data scaling (W1)
Since rerunning pretraining on several subsets of the full mixture was beyond our compute budget during the rebuttal period, we pretrained on 25%, 50%, and 100% of the DROID episodes (one epoch each) with either RGB or RGB + robot-centric pointmaps, and fine-tuned each model with either input. We also report our full mixture of DROID, InternData-M1, and spatial VQA as the largest data point.
The table below reports RoboCasa success rates (%). PM denotes robot-centric pointmaps. These results are reported in Section 4.4 and Figure 7, with full results in Appendix E.3 and Table 22 of the revised paper.
Without pretraining, a model depends only on its fine-tuning input, so A and C share one value, as do B and D
We derive three observations from these results. First, robot-centric pointmaps bring larger gains from additional pretraining data than RGB alone. Pretraining on 25% of DROID alone brings no measurable change for either input. However, from 25% of DROID to the full mixture, the model pretrained and fine-tuned with PM (i.e., row D in the table above) improves by 5.0 points (73.2% to 78.2%), whereas the model pretrained and fine-tuned with RGB only (A) improves by 3.3 points (68.2% to 71.5%). D outperforms A at every scale, by 5.0, 5.9, and 5.7 points at 25%, 50%, and 100% of DROID and by 6.7 points with the full mixture.
Second, adding pointmaps only at fine-tuning helps, but learning them during pretraining helps further, and increasingly so with more data. The model pretrained with RGB and fine-tuned with PM (B) outperforms A by 2.9, 4.0, 2.1, and 3.0 points at 25%, 50%, and 100% of DROID and the full mixture. D outperforms B by a further 2.1 and 1.9 points at 25% and 50% of DROID, and by 3.6 and 3.7 points at 100% of DROID and the full mixture.
Third, pretraining with PM brings no benefit when the model is fine-tuned with RGB alone. The model pretrained with PM and fine-tuned with RGB only (C) shows no improvement over A and achieves similar performance across all scales (66.9% vs. 68.2%, 66.1% vs. 68.5%, 69.3% vs. 69.6%, and 71.6% vs. 71.5%). We believe this is because PM pretraining teaches the model to exploit 3D geometry when RGB and pointmaps are provided together, so the pointmaps must also be given at fine-tuning.
Overall, adding robot-centric pointmaps to RGB turns additional pretraining data into larger gains than RGB alone, provided that the pointmaps are also used at fine-tuning.