- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether a foundation model specialized in precise visual grounding provides a stronger perceptual basis for action learning than general-purpose vision-language or video-generation backbones. GroundingPI combines a visual encoder, a Qwen3-4B language backbone, quantized coordinate tokens, multimodal and spatial training, and GRPO post-training. The authors evaluate perception across 34 benchmarks and transfer the representations to a common action architecture for manipulation and open-loop driving prediction.
Critical strengths
- The downstream comparison controls important confounders. The common action expert, conditioning budget, demonstrations, optimization schedule, and inference recipe make the study more informative than comparing unrelated complete robot systems.
- The strongest results address meaningful practical settings. GroundingPI leads all four OOD settings. These results motivate reusable perceptual pretraining.
Critical weaknesses
- The results do not fully isolate grounding supervision from other pretraining differences.
- Several consequential downstream conclusions rely on small differences without repeated-training uncertainty.
- The supervision and architectural analyses support useful hypotheses, but not all of the proposed mechanisms are experimentally identified.
Questions for authors
- Can the authors provide existing repeated-training results, or repeat the most important OOD and reduced demonstration settings? Per-task differences would also help distinguish broad gains from improvements concentrated in a few tasks.
- Does the headline perception ranking survive a shared-coverage, protocol-compatible analysis?
- What is the closest same-backbone control for attributing transfer gains to grounding specialization?
2 comments
Response to Reviewer Tacd (2/2)
[W2, Q1] Broad gains across settings (continued)
The expanded comparisons appear in the Rebuttal Revision, Appendices L.4–L.5 (Tables S42–S43).
- GroundingPI ranks first against all 12 baseline foundations across six robot settings and all nine available baselines across three reduced-data fractions, covering bimanual and humanoid embodiments, clean and shifted scenes, and 25% to 100% of the demonstrations.
- It beats the same-base control in 9 of 9 comparisons.
- In Fig. 7, four policies trained separately on checkpoints of increasing pretraining scale improve OOD success at every step.
At matched RoboCasa-GR1 ID demonstration fractions:
| Backbone | 25% data | 50% data | 75% data | 100% data |
|---|---|---|---|---|
| Qwen3-VL-4B | 11.94 | 18.28 | 26.94 | 34.94 |
| RoboRefer-2B-SFT | 15.03 | 27.11 | 32.61 | 38.03 |
| GroundingPI-BaseVLM | 15.72 | 28.81 | 34.31 | 39.61 |
| GroundingPI | 17.53 | 31.31 | 35.92 | 41.42 |
Mean success rates (%) across three seeds; all evaluated backbones and variants are in Table S43.
GroundingPI improves over BaseVLM by +1.81, +2.50, +1.61, and +1.81 pp across these fractions (Appendix L.5, Fig. S17, Table S43).
Each backbone uses one training run with repeated evaluation. These results support breadth across settings and demonstration fractions; independent retraining and per-task paired comparisons would further quantify variability and whether gains concentrate in particular tasks. We appreciate these useful directions for additional validation.
[W3] Which mechanisms are identified
Concern: Some proposed mechanisms are hypotheses rather than experimentally identified findings.
We agree and have made the distinction explicit. Measured results are presented with direct evidence in Section 5.3 and Appendix L, and mechanism explanations are introduced with "we hypothesize" in Section 6. The mapping is:
| Claim | Status | Evidence |
|---|---|---|
| Additional grounding specialization improves action learning on a fixed base | Observed in controlled comparisons | App. L.2, Fig. S16; Table S43, 9/9 comparisons |
| Transfer improves with grounding pretraining scale | Observed at the evaluated scales | Fig. 7 |
| Dense grounding is the strongest single supervision group | Observed under this recipe (leave-one-out) | Fig. 8 |
| OCR acts as a catalyst | Mechanistic hypothesis | Fig. 8, Sec. D.3.4; OCR-only control in App. L.3 |
| Feature-access effects of DeepStack / GatedDeltaNet | Hypothesis | Sec. D.4 |
| System-1 / System-2 split | Outlook | Section 6; App. D.4.4 |
The OCR-only control directly measures a transfer effect; the proposed feature-learning explanation remains a hypothesis. Detailed validation of these mechanisms is an important direction for subsequent research.
[Q2] Ranking under shared, protocol-compatible coverage
Concern: The headline average may depend on which baselines have results on which benchmarks.
Tables 1–3 cover box detection (COCO, LVIS, VisDrone), referring (HumanRef, RefCOCO family), robot spatial pointing (RefSpatial, RoboSpatial), GUI grounding (ScreenSpot, OSWorld-G), OCR, and document layout. On the same 20 evaluation entries where all six models below have protocol-compatible scores, incorporating the updated results in Tables S39–S40:
| GroundingPI | GPT-6 Astra | Qwen3.7-Max | Rex-Omni | LocateAnything | Qwen3-VL-4B | |
|---|---|---|---|---|---|---|
| Mean | 70.86 | 68.98 | 56.58 | 55.20 | 52.50 | 49.06 |
| Benchmarks won | 12/20 | 6/20 | 2/20 | 0/20 | 0/20 | 0/20 |
The ranking holds under shared coverage: GroundingPI has the highest mean and wins the most benchmarks, ahead of GPT-6 Astra. Against the four embodied models, all evaluated locally on the full 34-benchmark suite, GroundingPI averages 74.57 vs. at most 35.91 (App. L.4, Tables S36–S41).
We have completed explicit split-level and image-overlap checks across our training sources and evaluation sets, confirming their separation with no training–evaluation leakage detected. If accepted, we will publicly release the detailed audit report after the camera-ready stage, while preserving anonymity throughout review. This complements the shared-coverage analysis by addressing data separation.
Response to Reviewer Tacd (1/2)
Thank you for the review and the focused questions.
[W1, Q3] Closest same-backbone control
Concern: The results do not isolate grounding supervision from other pretraining differences.
Thank you for suggesting this important control. Same-base control. GroundingPI-BaseVLM shares the encoder, LLM, projector, and full base pretraining, but stops before dedicated spatial specialization and GRPO. Its roughly 500B-token base training already contains about 300B grounding/OCR-related tokens for alignment. Thus, this is the closest same-backbone control for the additional specialization, not a model without grounding exposure. Every row below uses the same action model, demonstrations, budget, and evaluation protocol (Rebuttal Revision, App. L.2, Fig. S16; Tables S36 and S42).
Other grounding results are in Appendix L.2, Tables S36–S41.
What the benchmarks test:
What the results show:
Qwen3-VL-4B provides a complementary comparison with the same language-model family and size, while BaseVLM controls the full multimodal backbone. BaseVLM already exceeds Qwen3-VL-4B by 3.1–10.5 pp. This also motivates the hypothesis that early grounding/OCR-based visual alignment affects embodied transfer; the same-base comparison isolates subsequent specialization.
[W2, Q1] Repeated results and breadth of gains
Concern: Downstream conclusions rest on small differences without repeated-run uncertainty.
How evaluation works. OOD settings are evaluation-only: each policy is trained once on in-distribution demonstrations and tested on the shifted scenes. Every backbone is evaluated three times with different evaluation seeds, each run covering 5,000 RoboTwin rollouts or 1,200 ID and 3,200 OOD RoboCasa-GR1 rollouts.
The intervals use pooled rollout counts under an independent-binomial approximation (Tables S42–S43). Every approximate interval lies above zero, supporting the observed ranking under repeated evaluation.