acceptodds
Under review as a conference paper at ICLR 2027

GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

Abstract

Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings. We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. Across RoboTwin 2.0 and RoboCasa-GR1, it outperforms all evaluated backbones in all four out-of-distribution settings, with relative gains of up to 24.8% over the strongest baseline. On RoboCasa-GR1, GroundingPI trained with 50% of the demonstrations outperforms those baselines trained with 75%. On nuScenes, used as the visual backbone, GroundingPI attains an average open-loop L2 error of 0.296 m. We analyze GroundingPI's pretraining in scale and data composition. Grounding quality and downstream manipulation improve with pretraining scale, with generalization tracking perceptual gains most closely. Analyzing the data recipe across these 11 perceptual capabilities shows dense grounding's substantial benefits for both, and OCR's potential as a catalyst for perceptual learning. These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Models, Vision-Language Models, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TacdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether a foundation model specialized in precise visual grounding provides a stronger perceptual basis for action learning than general-purpose vision-language or video-generation backbones. GroundingPI combines a visual encoder, a Qwen3-4B language backbone, quantized coordinate tokens, multimodal and spatial training, and GRPO post-training. The authors evaluate perception across 34 benchmarks and transfer the representations to a common action architecture for manipulation and open-loop driving prediction.

Critical strengths
  1. The downstream comparison controls important confounders. The common action expert, conditioning budget, demonstrations, optimization schedule, and inference recipe make the study more informative than comparing unrelated complete robot systems.
  2. The strongest results address meaningful practical settings. GroundingPI leads all four OOD settings. These results motivate reusable perceptual pretraining.
Critical weaknesses
  1. The results do not fully isolate grounding supervision from other pretraining differences.
  2. Several consequential downstream conclusions rely on small differences without repeated-training uncertainty.
  3. The supervision and architectural analyses support useful hypotheses, but not all of the proposed mechanisms are experimentally identified.
Questions for authors
  1. Can the authors provide existing repeated-training results, or repeat the most important OOD and reduced demonstration settings? Per-task differences would also help distinguish broad gains from improvements concentrated in a few tasks.
  2. Does the headline perception ranking survive a shared-coverage, protocol-compatible analysis?
  3. What is the closest same-backbone control for attributing transfer gains to grounding specialization?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Tacd (1/2)

Thank you for the review and the focused questions.

[W1, Q3] Closest same-backbone control

Concern: The results do not isolate grounding supervision from other pretraining differences.

Thank you for suggesting this important control. Same-base control. GroundingPI-BaseVLM shares the encoder, LLM, projector, and full base pretraining, but stops before dedicated spatial specialization and GRPO. Its roughly 500B-token base training already contains about 300B grounding/OCR-related tokens for alignment. Thus, this is the closest same-backbone control for the additional specialization, not a model without grounding exposure. Every row below uses the same action model, demonstrations, budget, and evaluation protocol (Rebuttal Revision, App. L.2, Fig. S16; Tables S36 and S42).

BackboneRoboTwin FullClean2RandomRoboCasa OOD
Qwen3-VL-4B59.0712.3729.97
GroundingPI-BaseVLM66.2118.5534.44
+ spatial training, textual coordinates66.2618.9934.64
+ quantized coordinates (w/o GRPO)67.4319.7835.16
+ GRPO (GroundingPI)68.7720.6536.10

Other grounding results are in Appendix L.2, Tables S36–S41.

What the benchmarks test:

  • RoboTwin 2.0: 50 bimanual tasks. Clean2Random trains on clean tables and tests on cluttered, re-textured ones.
  • RoboCasa-GR1: humanoid dexterous-hand tasks with shifted object appearance, type, or container.

What the results show:

  • Further grounding specialization improves action learning on a fixed base. It improves all 9 comparisons (6 settings and 3 reduced-data fractions), by +2.11 pp pooled (approx. rollout-level 95% CI ±0.58; reduced-data results in App. L.5, Table S43).
  • The gain grows with grounding pretraining scale. Scaling from 88.4B to 221B tokens on the same base adds +11.99 pp grounding and +6.78 pp RoboCasa OOD success (Fig. 7).
  • Each component contributes. Quantized coordinates improve transfer over textual coordinates, and GRPO adds a further gain.

Qwen3-VL-4B provides a complementary comparison with the same language-model family and size, while BaseVLM controls the full multimodal backbone. BaseVLM already exceeds Qwen3-VL-4B by 3.1–10.5 pp. This also motivates the hypothesis that early grounding/OCR-based visual alignment affects embodied transfer; the same-base comparison isolates subsequent specialization.

[W2, Q1] Repeated results and breadth of gains

Concern: Downstream conclusions rest on small differences without repeated-run uncertainty.

How evaluation works. OOD settings are evaluation-only: each policy is trained once on in-distribution demonstrations and tested on the shifted scenes. Every backbone is evaluated three times with different evaluation seeds, each run covering 5,000 RoboTwin rollouts or 1,200 ID and 3,200 OOD RoboCasa-GR1 rollouts.

GroundingPI vs.Gap (pp)Approx. 95% CI half-width
Strongest baseline (RoboBrain2.5), RoboTwin Full+3.69±1.06
Strongest baseline (RoboBrain2.5), Clean2Random+2.94±0.89
Strongest baseline (RoboBrain2.5), RoboCasa OOD+2.27±1.35
GroundingPI-BaseVLM (same base), all nine comparisons pooled+2.11±0.58

The intervals use pooled rollout counts under an independent-binomial approximation (Tables S42–S43). Every approximate interval lies above zero, supporting the observed ranking under repeated evaluation.

AuthorsOpenReview ↗

Response to Reviewer Tacd (2/2)

[W2, Q1] Broad gains across settings (continued)

The expanded comparisons appear in the Rebuttal Revision, Appendices L.4–L.5 (Tables S42–S43).

  • GroundingPI ranks first against all 12 baseline foundations across six robot settings and all nine available baselines across three reduced-data fractions, covering bimanual and humanoid embodiments, clean and shifted scenes, and 25% to 100% of the demonstrations.
  • It beats the same-base control in 9 of 9 comparisons.
  • In Fig. 7, four policies trained separately on checkpoints of increasing pretraining scale improve OOD success at every step.

At matched RoboCasa-GR1 ID demonstration fractions:

Backbone25% data50% data75% data100% data
Qwen3-VL-4B11.9418.2826.9434.94
RoboRefer-2B-SFT15.0327.1132.6138.03
GroundingPI-BaseVLM15.7228.8134.3139.61
GroundingPI17.5331.3135.9241.42

Mean success rates (%) across three seeds; all evaluated backbones and variants are in Table S43.

GroundingPI improves over BaseVLM by +1.81, +2.50, +1.61, and +1.81 pp across these fractions (Appendix L.5, Fig. S17, Table S43).

Each backbone uses one training run with repeated evaluation. These results support breadth across settings and demonstration fractions; independent retraining and per-task paired comparisons would further quantify variability and whether gains concentrate in particular tasks. We appreciate these useful directions for additional validation.

[W3] Which mechanisms are identified

Concern: Some proposed mechanisms are hypotheses rather than experimentally identified findings.

We agree and have made the distinction explicit. Measured results are presented with direct evidence in Section 5.3 and Appendix L, and mechanism explanations are introduced with "we hypothesize" in Section 6. The mapping is:

ClaimStatusEvidence
Additional grounding specialization improves action learning on a fixed baseObserved in controlled comparisonsApp. L.2, Fig. S16; Table S43, 9/9 comparisons
Transfer improves with grounding pretraining scaleObserved at the evaluated scalesFig. 7
Dense grounding is the strongest single supervision groupObserved under this recipe (leave-one-out)Fig. 8
OCR acts as a catalystMechanistic hypothesisFig. 8, Sec. D.3.4; OCR-only control in App. L.3
Feature-access effects of DeepStack / GatedDeltaNetHypothesisSec. D.4
System-1 / System-2 splitOutlookSection 6; App. D.4.4

The OCR-only control directly measures a transfer effect; the proposed feature-learning explanation remains a hypothesis. Detailed validation of these mechanisms is an important direction for subsequent research.

[Q2] Ranking under shared, protocol-compatible coverage

Concern: The headline average may depend on which baselines have results on which benchmarks.

Tables 1–3 cover box detection (COCO, LVIS, VisDrone), referring (HumanRef, RefCOCO family), robot spatial pointing (RefSpatial, RoboSpatial), GUI grounding (ScreenSpot, OSWorld-G), OCR, and document layout. On the same 20 evaluation entries where all six models below have protocol-compatible scores, incorporating the updated results in Tables S39–S40:

GroundingPIGPT-6 AstraQwen3.7-MaxRex-OmniLocateAnythingQwen3-VL-4B
Mean70.8668.9856.5855.2052.5049.06
Benchmarks won12/206/202/200/200/200/20

The ranking holds under shared coverage: GroundingPI has the highest mean and wins the most benchmarks, ahead of GPT-6 Astra. Against the four embodied models, all evaluated locally on the full 34-benchmark suite, GroundingPI averages 74.57 vs. at most 35.91 (App. L.4, Tables S36–S41).

We have completed explicit split-level and image-overlap checks across our training sources and evaluation sets, confirming their separation with no training–evaluation leakage detected. If accepted, we will publicly release the detailed audit report after the camera-ready stage, while preserving anonymity throughout review. This complements the shared-coverage analysis by addressing data separation.

Reviewer_XeWLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GroundingPI, a 4B grounding model that outputs points and boxes as quantized coordinate tokens. Training combines staged pretraining, supervised fine-tuning, and GRPO using public data and a multi-teacher data engine. Evaluation covers 34 grounding benchmarks and transfer to manipulation and open-loop driving, with manipulation backbones compared under a fixed action model and training budget. Further studies examine training scale and data composition, highlighting dense grounding and auxiliary OCR, layout, and GUI supervision.

Critical strengths
  • Evaluation spans 34 benchmarks against general VLMs, grounding specialists, embodied models, and frontier closed models. GroundingPI performs well on dense-scene box grounding (Dense200 F1mIoU: 74.53 versus 68.13 for SenseNova-Vision and 65.04 for GPT-6 Astra), while the paper acknowledges remaining gaps. These results suggest its potential as a useful grounding foundation model.
  • Controlled transfer evaluation with informative ablations. Manipulation comparisons fix the action model, robot data, and training budget across backbones. Scale and supervision-group ablations suggest that dense grounding and complementary OCR, layout, and GUI supervision benefit downstream transfer, providing useful evidence for embodied foundation-model design.
Critical weaknesses
  • Incomplete attribution of downstream gains. Matching the action model and training budget strengthens the transfer comparison, but backbones still differ in size, architecture, and pretraining resources. Coordinate-token and GRPO ablations evaluate grounding rather than manipulation, leaving their contributions to action transfer unclear. Missing training-seed uncertainty and the small open-loop driving advantage further limit broader conclusions about physical intelligence.
  • Insufficient detail on data construction and evaluation separation. The multi-teacher data engine lacks operational details such as teacher configurations, acceptance thresholds, sampling proportions, and output volumes. Explicit split-level and image-overlap checks would help establish separation between training sources and evaluation benchmarks. These gaps limit reproducibility and confidence in benchmark generalization, without establishing that contamination occurred.
Questions for authors
  • Do you have manipulation results without GRPO, with textual coordinates, or using the same base model without grounding pretraining?
  • How stable are the manipulation gains across training seeds? Can you report uncertainty estimates for comparisons with the strongest backbones?
  • Can you specify the data-engine teachers, acceptance thresholds, and per-source data volumes? How was training–evaluation overlap checked, and what does “zero-shot” mean in the tables?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XeWL (1/2)

Thank you for the constructive review and the precise questions.

[W1, Q1] Attribution: manipulation results without GRPO, with textual coordinates, and with the same base before dedicated grounding specialization

Concern: The backbones differ in size, architecture, and pretraining, and the coordinate-token and GRPO ablations were evaluated only on grounding.

All three variants you asked for are now run through the same manipulation pipeline. GroundingPI-BaseVLM shares the encoder, LLM, projector, and full base pretraining, but stops before dedicated spatial specialization and GRPO. Importantly, its roughly 500B-token base training already includes about 300B grounding/OCR-related tokens for alignment; this is not a model without prior grounding exposure. Every row serves as the backbone of the same action model, trained on the same demonstrations and budget and evaluated under the same protocol (Rebuttal Revision, App. L.2, Fig. S16; Tables S36 and S42).

BackboneRoboTwin FullClean2RandomRoboCasa OOD
Qwen3-VL-4B59.0712.3729.97
Same base, before dedicated specialization (BaseVLM)66.2118.5534.44
+ spatial training, textual coordinates66.2618.9934.64
+ quantized coordinates (w/o GRPO)67.4319.7835.16
+ GRPO (GroundingPI)68.7720.6536.10

Other grounding results are in Appendix L.2, Tables S36–S41.

What the benchmarks test:

  • RoboTwin 2.0: 50 bimanual tasks. Clean2Random trains on clean tables and tests on cluttered, re-textured ones.
  • RoboCasa-GR1: humanoid dexterous-hand tasks with shifted object appearance, type, or container.

What the results show:

  • Further grounding specialization on a fixed base helps. It improves all 9 comparisons (6 settings and 3 reduced-data fractions), by +2.11 pp pooled (approx. rollout-level 95% CI ±0.58; reduced-data results in App. L.5, Table S43).
  • The coordinate representation transfers. Textual coordinates achieve strong grounding but limited additional transfer; quantized coordinates improve it further.
  • GRPO adds to the gain in every setting.
  • The base is already strong. It is 3.1–10.5 pp above Qwen3-VL-4B, so the additional grounding gain is measured on top of a strong base.

Qwen3-VL-4B provides a complementary comparison with the same language-model family and size; BaseVLM is the exact multimodal control. The base's strong transfer also motivates the hypothesis that early grounding/OCR-based visual alignment affects embodied learning.

[W1, Q2] Uncertainty of the manipulation gains

Concern: The gains may be small relative to run-to-run variation.

How evaluation works. OOD settings are evaluation-only: each policy is trained once on in-distribution demonstrations and tested on the shifted scenes. Every backbone is evaluated three times with different evaluation seeds. Each run covers 5,000 RoboTwin rollouts (50 tasks × 100 episodes), or 1,200 in-distribution and 3,200 OOD RoboCasa-GR1 rollouts.

GroundingPI vs.Gap (pp)Approx. 95% CI half-width
Strongest baseline (RoboBrain2.5), RoboTwin Full+3.69±1.06
Strongest baseline (RoboBrain2.5), Clean2Random+2.94±0.89
Strongest baseline (RoboBrain2.5), RoboCasa OOD+2.27±1.35
GroundingPI-BaseVLM (same base), all nine comparisons pooled+2.11±0.58

The intervals use pooled rollout counts under an independent-binomial approximation; the underlying results are in Tables S42–S43. Every approximate interval lies above zero.

AuthorsOpenReview ↗

Response to Reviewer XeWL (2/2)

[W1, Q2] Stability across training seeds (continued)

Three results support consistent gains across settings (Rebuttal Revision, Apps. L.2, L.4–L.5; Tables S42–S43):

  1. Across baselines: GroundingPI ranks first against all 12 baseline foundations across six robot settings and all nine available baselines across three reduced-data fractions.
  2. Across same-base comparisons: it beats the same-base control in 9 of 9 comparisons.
  3. Across separately trained policies: in Fig. 7, four policies trained separately on checkpoints of increasing pretraining scale improve OOD success at every step.

[W1] Driving margin

Concern: The open-loop driving advantage is small.

On nuScenes, the model predicts the next 3 seconds of ego trajectory from the front camera and ego history; L2 is the distance to the logged trajectory. GroundingPI has the lowest L2 at every horizon (Fig. 4; App. G.1). In Fig. 8, changing only the grounding data composition on the same backbone moves average L2 from 0.391 to 0.296 m, consistent with sensitivity to perceptual training.

We acknowledge the modest absolute margin and the limitation of open-loop evaluation. We thank you for identifying a valuable future direction and plan a dedicated study of driving transfer, including closed-loop evaluation.

[W2, Q3] Data engine, evaluation, and "zero-shot"

Concern: The data engine is under-specified; it is unclear how evaluation is separated from training and what "zero-shot" means.

Data engine. Teachers combine multiple frontier models and our domain-specialized RL models at 4B, 9B, and 27B. Restricting each specialist to a concrete annotation domain concentrates its capabilities on that task. Acceptance criteria are case- and task-dependent, with three intended-use/cost tiers: alignment, pretraining, and RL. Engine-produced supervision and public data contribute approximately 4:6 overall; this is an aggregate ratio, not per-source counts.

Acceptance is decided per field. Semantic and geometric fields are checked for geometry, syntax, semantic correspondence, and consistency with evidence, then marked accepted, rejected, or unresolved. An item is kept only when all required fields are accepted; all-instance queries also require coverage checks. The pipeline is in Fig. S2, and the procedure from App. B.1 is now in Section 3.2.

Evaluation. All 34 benchmarks use official held-out splits and compatible evaluation protocols; public training data uses official training splits. We have completed explicit split-level and image-overlap checks across our training sources and evaluation sets, confirming their separation with no training–evaluation leakage detected. If accepted, we will publicly release the detailed audit report after the camera-ready stage, while preserving anonymity throughout review.

"Zero-shot." The paper makes no blanket zero-shot claim. The column in appendix Tables S15/S16 (COCO, LVIS) follows Rex-Omni's open-set detection convention. YES means no training on that benchmark's detection training set; NO marks detectors trained on it; UNK marks undisclosed training status. This does not mean absence of all grounding supervision.

Generalization to unseen data sources. The four benchmarks below are representative examples only, not an exhaustive list of held-out evaluations. Training–evaluation separation applies to the full evaluation suite. The following evaluations are held out. None of benchmarks has a corresponding training split in our training data, except RefSpatial, whose Unseen split excludes object–relation combinations seen in its training split:

BenchmarkWhat it testsGroundingPIBest ≤10BGPT-6 Astra
HumanRefPeople described by several attributes88.5679.8783.01
RoboSpatialSpatial-relation pointing in robot scenes74.0464.7565.69
RefSpatial UnseenSpatial referring with unseen combinations75.3237.0181.93
M6DocLayout of diverse modern documents74.8165.9460.59

Scores follow Tables 1–3, updated where available by Tables S39–S40. GroundingPI leads similarly sized models on all four and surpasses GPT-6 Astra on three. Other grounding results are in Appendix L. RoboTwin/RoboCasa scenes and nuScenes also never appear in grounding training, extending the evidence to downstream transfer.

Reviewer_XeWLOpenReview ↗

Maintain my positive rating

Thank you for the thorough response (1/2 and 2/2, highly appreciated) and the new same-base experiments. The fixed-backbone ablations address my main attribution concern, and the clarifications on “zero-shot” and held-out splits resolve my evaluation questions. Two minor limitations remain: uncertainty reflects evaluation rather than training seeds, and the data-engine description is still high-level. Please include these new ablations and details in the revision. I will maintain my positive rating.

AuthorsOpenReview ↗

Thank You for Your Guidance on Attribution and Reproducibility

Thank you for reading both parts of our response so carefully and for maintaining your positive rating. Your suggestion to evaluate the same backbone before and after dedicated grounding training was especially perceptive. It gave us a direct way to measure what that training adds to manipulation performance on an already strong base. Your questions about “zero-shot” evaluation also helped us state more precisely which splits and data sources were held out.

We agree that repeated rollouts capture only one source of uncertainty. We have already started retraining the manipulation policies with different random seeds. In the camera-ready version, we will report the results and variability across these independent training runs alongside our rollout-based uncertainty estimates. We will also document the full data pipeline, including how we combine teacher annotations field by field, which checks a label must pass for each task, and how we resolve ambiguous cases before using those labels for training.

We are sincerely grateful for the rigor of your questions and the constructive spirit in which you raised them. Your review helped us design a more informative experiment and see more clearly what readers will need to reproduce our work.

Reviewer_wKVPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces GroundingPI, which represents spatial outputs such as points and bounding boxes as quantized coordinate tokens. It develops visual grounding capabilities through multitask perceptual pretraining, supervised fine-tuning, and reinforcement post-training, then transfers the pretrained representations to robotic manipulation and driving trajectory prediction. The paper investigates whether precise perception improves downstream action learning and how pretraining scale and data composition affect transfer performance.

Critical strengths
  • The research question has substantial practical value. I believe grounding is the most important foundational capability for embodied intelligence. The results are very promising, and the experiments are extensive and diverse. The paper focuses on precise spatial perception before action learning, particularly in dense scenes and for tiny objects.

  • The evaluation is broad, covering multiple perceptual tasks as well as out-of-distribution generalization and demonstration efficiency in robotic manipulation. This demonstrates the model’s value from several perspectives.

  • The exploration of the model itself is valuable. Moreover, the downstream comparisons hold the action expert, action data, and training budget fixed, providing a useful comparison of transfer across pretrained foundations. The results with reduced demonstration data are also informative.

Critical weaknesses
  • My primary concern is the presentation of the method. Although I have read all the appendices and appreciate the substantial effort involved, I must emphasize that appendices are optional reading in a conference paper. Nevertheless, the method section occupies only about half a page and is extremely difficult to follow. The paper-specific training procedure, data engine, key reward designs, and model architecture are described very briefly, with information essential to assessing the methodological contribution relegated to the appendices. The main text reads almost like an experimental results report rather than a complete research paper. I consider this a major weakness. The authors should prioritize explaining the key technical differences, controlled variables, and direct supporting evidence in the main text, while clearly distinguishing experimental findings from hypotheses about future system designs.

  • Following the previous point, even after examining the appendices in detail, I would like the authors to define the methodological contribution relative to prior work more clearly. Core components—including quantized coordinates, data engines, and SFT followed by GRPO—are all drawn from Rex-Omni. It is therefore difficult to determine whether GroundingPI’s contribution comes from data construction, reward design, the training recipe, backbone upgrades, or larger-scale training. Building on existing components does not preclude a valuable contribution, but the paper needs to explain what new, verifiable knowledge it provides.

  • The paper lacks comparisons with and discussion of related embodied brain models, such as RoboBrain2.5, Embodied-R1.5, RoboRefer, and MiMo-Embodied. These models and their associated papers also devote substantial effort to improving grounding capabilities.

  • The ablations in Figure 7 are insufficient to establish the independent contributions of different forms of supervision, particularly OCR. The confounding variables make it difficult to attribute the observed improvements clearly.

  • It is difficult to isolate the claim that stronger grounding improves action learning, because the compared models differ in their visual encoders, language models, and other components. Could the authors evaluate controlled variants built on GroundingPI’s own base training recipe?

  • Why do the authors choose a Kimi ViT encoder rather than directly adopting a Qwen VLM architecture? What motivates this design choice?

  • I am curious whether the model retains the VLM’s understanding capabilities under this training recipe. How does it perform on grounding tasks that require multi-hop reasoning? Does it retain broader vision-language understanding, or has it effectively become a specialized visual grounding model?

My current scoring rationale is as follows: I assign a score of 1 for presentation, as the material would be better organized into a full-length journal article. The excellent experimental results provide valuable insights, warranting a score of 3 for informativeness. Taking these aspects together, my current overall score is 2.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer wKVP (1/2)

Thank you for carefully reading all appendices and for the generous assessment of the experiments. Your guidance has been invaluable.

[W1] Presentation

Concern: The method is too brief, with essential technical details left to optional appendices.

We agree. The Rebuttal Revision now presents a full method, addressing the previous impression of a results report. We sincerely appreciate this valuable guidance; if the revision remains unclear or insufficient, we would be very grateful for further suggestions and will continue improving it.

Section 3: a full method section (about 1.5 pages, moved up from the appendix):

  • 3.1 Model architecture and grounding formulation: the encoder and projector, p(Y | I, P), and the vocabulary of labels, protocol markers, and 1,000 coordinate tokens.
  • 3.2 Data construction (App. B): candidate generation, field-level fusion, task-dependent validation, and expert iteration, including acceptance rules.
  • 3.2 Training design (Apps. C, F.3): base VLM training (Stages 1–3, 500B tokens); coordinate-alignment SFT on eight task groups (Stage 4, 221B tokens) with its loss; GRPO with grounding reward  = 0.7·Z(R_set) + 0.3·Z(R_strict) and the OCR reward.

Sections 5.3 and 6: findings separated from outlook.

  • 5.3 Effects of grounding training: pretraining scale and same-base controls.
  • 5.3 What transfers from grounding to action: the data-composition study.
  • 6 Discussion: mechanisms and the System-1/System-2 outlook, explicitly distinguished from measured findings.

Section 4 specifies downstream controls. Key technical differences and direct evidence are now in the main text; full benchmark tables are in Appendix I.

[W2] Contribution relative to Rex-Omni

Concern: Core components overlap with Rex-Omni, leaving the new methodological contribution unclear.

Our main contribution validates grounding for physical intelligence and guides base pretraining. Building on Rex-Omni's ideas, we add larger-scale data construction and training, new RL rewards, and in-depth ViT analysis and experiments:

Shared with Rex-OmniNew in GroundingPI
ModelQuantized coordinate tokens; SFT followed by GRPOOur own MoonViT-V2 + Qwen3-4B base; controlled ViT comparisons and same-base controls
DataData-engine paradigmLarger-scale construction and training across 11 capabilities; multi-teacher fusion with field-level acceptance
RewardsGrounding-oriented GRPOComplementary set-coverage and strict localization/output-quality rewards; joint transcription–geometry OCR reward
QuestionGrounding qualityWhether grounding transfers to action, and which capabilities matter, under one fixed action learner

Performance. With the same interface, GroundingPI outperforms Rex-Omni: 70.86 vs. 55.20 over 20 shared entries (RefSpatial: 75.50 vs. 51.75), and 68.77/20.65 vs. 54.65/14.28 on RoboTwin Full/Clean2Random.

The verifiable knowledge the paper adds:

  1. Additional grounding specialization improves action learning on a fixed base (W5).
  2. The gain grows with grounding pretraining scale (Fig. 7).
  3. Dense grounding is the most beneficial group for transfer under this recipe (Fig. 8).
  4. The coordinate interface matters for transfer (W5).

[W5] Isolating the effect of grounding

Concern: The compared models differ in encoder, LLM, and other components, so the gain may come from a stronger base model.

Same-base control. GroundingPI-BaseVLM shares our encoder, LLM, projector, and full base pretraining. Its roughly 500B base-training tokens include about 300B grounding/OCR tokens for alignment. It precedes dedicated spatial specialization and GRPO. Variants share the action model, demonstrations, budget, and evaluation protocol (App. L.2, Fig. S16; Tables S36, S42–S43).

BackboneRoboTwin FullClean2RandomRoboCasa OOD
Qwen3-VL-4B59.0712.3729.97
GroundingPI-BaseVLM66.2118.5534.44
+ spatial training, textual coordinates66.2618.9934.64
+ quantized coordinates (w/o GRPO)67.4319.7835.16
+ GRPO (GroundingPI)68.7720.6536.10

Other grounding results are in Appendix L.

What the benchmarks test:

  • RoboTwin 2.0: 50 bimanual tasks; Clean2Random transfers from clean to cluttered, re-textured scenes.
  • RoboCasa-GR1: humanoid dexterous-hand tasks, with OOD shifts in appearance, type, or container.

What the ladder shows:

  1. The base is already strong: 3.1–10.5 pp above Qwen3-VL-4B (same LLM family/size), motivating the hypothesis that early ViT alignment affects transfer. BaseVLM is the exact control.
  2. Further specialization helps: all 9 comparisons improve (6 settings, 3 data fractions), by +2.11 pp pooled.
  3. The coordinate interface matters: quantized coordinates transfer better than textual coordinates, and GRPO adds further gains.
AuthorsOpenReview ↗

Response to Reviewer wKVP (2/2)

[W3] Embodied brain models

Concern: The paper does not compare against embodied models that also invest in grounding.

Thank you for highlighting these works. We already compare 8 baseline foundations under one action learner: RynnBrain and RynnBrain1.1, Rex-Omni and LocateAnything, general VLMs, and video world models. Following your suggestion, we add all four named models to grounding and manipulation evaluation (Rebuttal Revision, App. L.4, Tables S36–S42):

ModelGrounding (34-benchmark mean)RoboTwin FullClean2RandomRoboCasa ID
RoboBrain2.5-4B35.9165.0917.7139.28
Embodied-R1.5-8B33.3662.3616.4637.47
RoboRefer-2B-SFT31.8563.8817.0938.03
MiMo-Embodied-7B33.6861.6815.9137.22
GroundingPI74.5768.7720.6541.42

The 34-benchmark mean covers detection, referring, pointing, robot spatial grounding, GUI, OCR, and layout.

RoboBrain2.5 is the strongest added model; GroundingPI retains its lead in grounding and manipulation. Other grounding results are in Appendix L.4, Tables S36–S41. We appreciate these systems' contributions.

[W4] Attributing the OCR effect

Concern: The Fig. 8 leave-one-out changes both data volume and mixture, so the OCR effect is confounded.

Thank you for identifying this confound. The OCR effect is now measured directly:

  • Removing OCR only retains layout/GUI and replaces removed samples with other data at the same training budget. RoboTwin success falls from 68.77 to 67.27 and Clean2Random from 20.65 to 19.78 (App. L.3, Fig. S16).
  • Complementary schedule study (App. D.3.4, Table S6): combining cold-start and batch mixing improves ID/OOD success by 2.58/0.85 pp over the best individual schedules.

These controls support the contribution of OCR and how it is used. The mechanism (sharper local features) is introduced in Section 6 explicitly as "we hypothesize", separate from these measured effects.

[W6] Why MoonViT (Kimi ViT) rather than a Qwen VLM

Concern: The motivation for the encoder choice is unclear.

Our choice has three motivations:

  1. Native-resolution detail. We wanted a ViT sensitive to fine spatial detail at native resolution. This is an empirical design judgment motivated by precise grounding.
  2. Video-input support. GroundingPI is intended as a grounding tool for images and videos. Our pretraining includes video-grounding data and our annotation pipeline supports video input. MoonViT-V2's spatial–temporal design fits this goal. We have not identified a suitable benchmark for the intended video-grounding use, so dedicated evaluation remains future work.
  3. Alignment stability. MoonViT-V2 was trained upstream from scratch with next-token prediction rather than initialized from SigLIP. The Kimi K3 report discusses the resulting training stability (Sec. 2.4, Fig. 6), informing our choice; we do not claim to have isolated this mechanism in our transfer experiments.

Under an identical recipe (App. E.1, Fig. S4), MoonViT-V2 raises the grounding average by +2.03 over the Qwen3 vision encoder and +0.51 over the first MoonViT. Training our own base also gives us checkpoints before dedicated spatial specialization for the exact control in W5. The encoder choice is discussed in Section 3.1.

[W7] General understanding and reasoning

Concern: The recipe may turn the model into a narrow grounding specialist.

Thank you for this important question. The added stage-wise evaluation is in Appendix L.6, Table S44.

MMBenchMMStarRealWorldQAMMMU
GroundingPI-BaseVLM84.5268.4070.7265.67
+ grounding training85.5369.4071.3767.11
+ GRPO (GroundingPI)83.5967.6970.2064.89

What the benchmarks test: MMBench, broad perception and reasoning; MMStar, questions that need the image; RealWorldQA, real-world spatial understanding; MMMU, college-level multi-discipline reasoning.

Grounding training slightly improves general VQA, and the final model remains within one point of BaseVLM. Building on Rex-Omni, we explicitly include understanding/reasoning data and adjust its mixture with grounding, retaining broader capabilities.

HumanRef (multiple attributes), RefCOCOg (long relational expressions), and RefSpatial (spatial relations and placement) require resolving compositional descriptions before localizing. GroundingPI leads similarly sized models on these benchmarks (Tables 1–2). These support reasoning-related capabilities, although dedicated multi-hop grounding evaluation remains future work. GroundingPI targets System-1 perception; longer multi-step reasoning is left to a System-2 planner (Sec. 6).

Thank you for the detailed response from the authors. I believe most concerns have been addressed. I appreciate the big effort the authors have put in to deliver such promising results. One minor suggestion: although I have checked the revised PDF and noticed the main text has been updated, there are no highlight markings, making it difficult for me to identify the newly added content. Regardless, I consider the writing and experiments of this manuscript to be above the acceptance threshold, and I am willing to raise my score to 3.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Feedback and Support

We are deeply grateful for your thoughtful reassessment, your recognition of our writing and experiments, and your willingness to raise your score. We especially appreciate the care and rigor of your review; your constructive suggestions have helped us strengthen the manuscript.

Thank you also for pointing out the missing highlights, and we apologize for the inconvenience. We have uploaded an updated PDF with the revisions in the main text highlighted to make them easier to identify. The newly added appendix content is placed at the end of the appendix, so we have left it unhighlighted.

We warmly welcome any further comments or discussion on aspects that remain unclear or could be improved. We will carefully consider every suggestion and make corresponding improvements with the same dedication. Thank you once again for your time, thoughtful guidance, and encouraging support.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank the AC, PC, and all reviewers. The reviewers agree on the paper's main strengths:

  • Question and evidence: grounding as a foundation for embodied intelligence is of substantial practical value, with extensive, diverse experiments and excellent results (wKVP).
  • Controlled transfer: the manipulation comparison fixes the action expert, data, and budget, making it more informative than comparing complete systems (wKVP, XeWL, Tacd).
  • Results: GroundingPI leads all four OOD settings (Tacd) and shows strong dense-scene grounding across a 34-benchmark evaluation (XeWL).

We are especially grateful for wKVP's careful reading and valuable guidance. In explaining the overall score of 2, the reviewer assigns 1 for presentation while recognizing the excellent experimental results and assigning 3 for informativeness. The review also assigns 3 for soundness. We take the presentation concern seriously. In the Rebuttal Revision, we have rewritten the main-text method, clarified the contribution, and separated experimental findings from design hypotheses.

Each reviewer also receives a complete, self-contained reply.

Shared concerns and what we show

1. Reviewers asked whether the transfer gains come from grounding or from a stronger base model. We show that additional grounding specialization improves action learning on an identical base. (wKVP-W5; XeWL-W1, Q1; Tacd-W1, Q3)

  • The control: GroundingPI-BaseVLM has the same encoder, LLM, projector, and base pretraining, and is already 3.1–10.5 pp above Qwen3-VL-4B. Its roughly 500B-token base training already includes about 300B grounding/OCR-related tokens for alignment. It is the checkpoint before subsequent spatial specialization and GRPO, not before all grounding exposure.
  • The result: on this fixed base, additional grounding specialization improves manipulation in all 9 comparisons (6 settings and 3 reduced-data fractions), by +2.11 pp pooled.
  • The components: textual coordinates achieve strong grounding but limited additional transfer; quantized coordinates improve transfer further, and GRPO adds to it in every setting (Rebuttal Revision, App. L.2, Fig. S16; App. L.5, Table S43).

Qwen3-VL-4B provides a complementary comparison with the same language-model family and size. BaseVLM is the exact multimodal control. Its strong transfer also motivates the hypothesis that early grounding/OCR-based visual alignment may affect embodied learning.

2. Reviewers asked whether the downstream margins are reliable. We show consistent gains across settings and repeated evaluations. (XeWL-W1, Q2; Tacd-W2, Q1)

  • The protocol: each trained policy is evaluated three times with different evaluation seeds, with up to 5,000 rollouts per run.
  • The margins: against the strongest baseline, GroundingPI leads by +3.69 pp on RoboTwin Full, +2.94 pp on Clean2Random, and +2.27 pp on RoboCasa OOD. Approximate rollout-level 95% intervals are ±1.06, ±0.89, and ±1.35 pp, respectively, and all exclude zero under an independent-binomial approximation.
  • The breadth: GroundingPI ranks first against all 12 baselines across six robot settings and all nine available baselines across three reduced-data fractions (Tables S42–S43).

3. Reviewers asked for the method to be presented in the main text. We have rewritten Section 3 as a full method section. (wKVP-W1, W2; XeWL-W2)

  • Section 3: the architecture and coordinate interface; the data engine and its acceptance rules; and base VLM training, coordinate-alignment SFT, and GRPO with its rewards.
  • Sections 5.3 and 6: measured findings separated from hypotheses and the System-1/System-2 outlook.
  • Positioning: what is shared with prior grounding models and what is new, stated explicitly. GroundingPI substantially outperforms prior models built on the same interface. We acknowledge Rex-Omni's coordinate interface and SFT–GRPO paradigm. Our central contribution is controlled evidence informing physical-intelligence pretraining, supported by larger-scale data construction and training, revised grounding/OCR rewards, and encoder investigation.

These changes address the previously blurred technical distinctions, controlled variables, and supporting evidence. If the revised presentation still needs improvement, we would sincerely appreciate further specific suggestions and will continue revising it. We respectfully ask that the revised manuscript and additional evidence be considered when assessing the concerns underlying the initial rating.

AuthorsOpenReview ↗

General Response (2/2)

Reviewer-specific points

Grounding and general capability

  • Embodied brain models (wKVP-W3). The paper already includes RynnBrain and RynnBrain1.1. We further evaluated RoboBrain2.5, Embodied-R1.5, RoboRefer, and MiMo-Embodied on all 34 grounding benchmarks. GroundingPI averages 74.57, against at most 35.91 for these models (Rebuttal Revision, App. L.4, Tables S36–S41).
  • General understanding (wKVP-W7). We evaluated each training stage on MMBench, MMStar, RealWorldQA, and MMMU. GroundingPI stays within one point of its base model on all four, supporting retained general understanding (App. L.6, Table S44). We include understanding/reasoning data and adjust its mixture with grounding. HumanRef, RefCOCOg, and RefSpatial also provide evidence of compositional and spatial reasoning.
  • Encoder choice (wKVP-W6). Our motivations are native-resolution detail, video/spatiotemporal modeling, and alignment stability informed by MoonViT-V2's upstream training from scratch rather than SigLIP initialization. The first is an empirical judgment; our pretraining and annotation pipeline support video, while dedicated video-grounding evaluation remains future work. Under an identical recipe, MoonViT-V2 raises the grounding average by 2.03 over the Qwen3 encoder (App. E.1, Fig. S4).
  • Shared coverage (Tacd-Q2). On the 20 evaluation entries that all six main models cover, GroundingPI still ranks first: its mean is 70.86 vs. 68.98 for GPT-6 Astra, and it wins 12 of the 20 (recomputed from Tables 1–3 with the updated scores).
  • Held-out data sources (XeWL-Q3). GroundingPI leads similarly sized models on HumanRef, RoboSpatial, RefSpatial Unseen, and M6Doc, and surpasses GPT-6 Astra on three. HumanRef, RoboSpatial, and M6Doc have no corresponding training split in our data; RefSpatial Unseen excludes object–relation combinations seen in its training split. Other grounding results are in Appendix L.

Transfer to manipulation and driving

  • Embodied brain models as backbones (wKVP-W3). Under the same action model, GroundingPI reaches 68.77 on RoboTwin Full, against at most 65.09 for the four added models (App. L.4, Table S42). We appreciate their contributions!
  • OCR supervision (wKVP-W4). Removing only OCR, retaining layout/GUI, and replacing the removed samples with other data at the same budget lowers RoboTwin success from 68.77 to 67.27 (App. L.3, Fig. S16). Combining cold-start and batch mixing also improves ID/OOD success by 2.58/0.85 pp over the best individual schedules (App. D.3.4, Table S6). These controls support the contribution of OCR and how it is used.
  • Driving margin (XeWL-W1). GroundingPI has the lowest L2 at every horizon. Changing only the grounding data moves L2 from 0.391 to 0.296 m (Sec. 5.3, Fig. 8). We acknowledge the modest absolute margin and the limitation of open-loop evaluation, and thank the reviewer for motivating dedicated follow-up work on driving, including closed-loop evaluation.

Evaluation protocol and claims

  • "Zero-shot" column and data engine (XeWL-W2, Q3). The column follows the benchmark-specific open-set detection convention in Appendix I, Tables S15–S16: it concerns exposure to that benchmark's detection training set, not all grounding supervision. The engine combines frontier teachers and domain-specialized RL teachers at 4B, 9B, and 27B. Acceptance is task- and case-dependent, with three intended-use/cost tiers: alignment, pretraining, and RL. Engine-produced supervision and public data contribute approximately 4:6 overall. The acceptance rule is now in Section 3.2. Split- and image-level audits found no training–evaluation leakage in the checked data. We will release the detailed audit after the camera-ready stage, if accepted, while preserving review anonymity.
  • Mechanisms (Tacd-W3). The paper states measured results as Findings and introduces mechanisms with "we hypothesize". Our reply maps each claim to its evidence (Secs. 5.3 and 6; Apps. D.3.4 and D.4). Direct validation of the proposed mechanisms is an important direction for subsequent research.

Revision

Section 3 presents the full method. Appendix L.2–L.6 adds the same-base controls, the four embodied baselines, demonstration efficiency, and general-understanding evaluations (Figs. S16–S17; Tables S36–S44). Our replies report the rollout statistics and clarify the zero-shot label in Appendix I, Tables S15–S16. We sincerely thank the reviewers for these constructive suggestions and welcome further guidance on any remaining presentation issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.