acceptodds
Withdrawn from ICLR 2027

GroundProbe: A Benchmark for Diagnosing Language Grounding in Robotic Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models report high success rates on standard manipulation benchmarks, yet recent stress tests reveal that this success often collapses under modest perturbations of objects, viewpoints, or language, suggesting that headline numbers conflate trajectory memorization with functional understanding. We introduce benchmark, a diagnostic benchmark that decomposes policy evaluation into three complementary dimensions, namely (1) language grounding, (2) compositional generalization, and (3) state-conditioned behavior. Within each dimension, training and testing splits are constructed so that the building blocks for the testing distribution are present in training, and each testing condition is matched to a seen one that differs in a single controlled factor, so that, given competence on the matched seen condition, a drop on the held-out one is evidence of a failure to transfer rather than of insufficient supervision. benchmark is built on NVIDIA Isaac Lab-Arena with a Franka Panda manipulator and a curated set of 30 meshes, totaling roughly 2,500 training trajectories and 1,770 testing episodes spread across six sub-tasks. Rather than reducing a policy to one number, benchmark reports closed-loop success rates together with normalized transfer gaps, a gap being the fall from a seen composition to the held-out one that differs from it in a single controlled factor, which distinguishes failure modes such as direction being entangled with category from the absence of an abstract category representation. The release comprises the scenes, instruction sets, teleoperated demonstrations, evaluation configurations and the analysis code, giving VLA designers a sharper diagnostic instrument than overall task success rate. We report a first datapoint rather than a full diagnostic profile. Populating the three axes across a slate of current models is the work this release is meant to enable. The benchmark, the demonstrations and the analysis code will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Robot Manipulation Benchmark, Language Grounding, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Rtn7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a benchmark that attempts to study the language grounding properties of the VLA policies, with protocol designed to distinguish specific failure modes. The benchmark idea is timely and needed, as properly understanding how our wording of the tasks sent to the policies affects their performance has direct implications for the downstream applications of modern large robotic policies. However, the paper has critical issues in execution and validation of the studies.

Critical strengths

None

Critical weaknesses

The paper has following critical weaknesses, which make me chose "Clear rejection" rating.

  1. The paper is very hard to read, and overall reads like a technical report or large supplementary material, rather than research paper. It took me multiple passes over Sections 3.2-3.5 to understand all the details, which are presented in very bloated and convoluted manner.
  2. The most critical issue of the paper is verification of the claims. The paper claims that pi05-DROID and MolmoBot-DROID policies fail to complete manipulation, yet the reason seem to be caused by the embodiment mismatch. While authors try to adapt the viewpoints to those featured in the DROID dataset, in Section 4.2 L414-417 it is stated that "but never completes a grasp and place" in regards to the π0.5\pi_{0.5}-DROID policy. Such a result is expected, as DROID platform utilizes Robotiq 2F-85 gripper, rather than standard Franka Hand gripper used in this benchmark. Hence the bad results are with high probability caused by the embodiment mismatch, where wrist camera sees completely different scene than it is used to from the training.
  3. Applicability of this benchmark is very limited. The benchmark requires it's own training data (recorded in the sim), that are not matched to any established embodiments used in the training of large-scale robotic policies, such as DROID, where questions asked in this benchmark actually make sense. Without it, the questions and answers are confounded with the embodiment mismatch, and small-scale data regime of this framework.
  4. The paper features a limited number of policies, which goes in hand with previous point. Using DROID embodiment would unlock plethora of off-shelf methods to evaluate their capabilities on for free.
Questions for authors
  1. Can you please clarify what adaptations were used in order to bridge the embodiment gap between DROID and "plain" Franka robot with Franka Hand? Why is π0.5\pi_{0.5}-DROID achieving good success rates in RoboLab and MolmoSpaces, is still second best policy on RoboArena, yet fails in this benchmark?
  2. Why was DROID platform not considered in this study, when checkpoints used are DROID-specific?
Reviewer_3TCjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces GroundProbe, a simulation benchmark in Isaac Lab-Arena (Franka Panda, about 30 meshes, about 2,500 teleoperated demos, 1,770 test episodes across six sub-tasks) for diagnosing language grounding in VLA policies. It covers three dimensions: D1, language grounding (attribute binding, spatial modifiers, negation, relative predicates); D2, compositional object–container pairing; and D3, state-conditioned skill selection. Each held-out composition is paired with a matched seen condition that differs in one factor, and the main metric is a normalized transfer gap, which is only interpreted after a policy shows competence on the seen condition. Four scripted policies show that the gap pairs can separate predefined failure modes. The learned-policy evaluation includes a language-free ACT policy, π0.5-DROID and MolmoBot-DROID evaluated zero-shot (both score 0%), and a LoRA-fine-tuned π0.5-DROID, which shows a gap of 0.96 on the spatial-modifier contrast.

Critical strengths
  1. Pairing each held-out composition with a matched seen condition that differs in one factor, and computing gaps only after seen-condition competence is established, is a sound way to separate grounding failures from execution failures. The scripted-policy check (Table 4) confirms that the direction and category gaps respond to the intended defects. 2.The planned release includes scenes, instruction sets, teleoperated demos, layout seeds and analysis code, together with explicit practices such as spawn gating, "not measured" versus zero, and flagging gaps that rest on few successes. This could support careful diagnostic studies of language grounding beyond aggregate success rates.
Critical weaknesses
  1. Only one language-conditioned policy (fine-tuned π0.5) yields any interpretable gap, with one run per dimension. Most contrasts in Tables 3 and 5 are undefined because seen success is 0%, and D2 cannot be measured for any learned policy. The D3 result rests on 2 successes out of 9 episodes, and no gap has a confidence interval. The authors call this "a first datapoint", but a benchmark paper should show that it can rank or separate a range of current models.
  2. ACT drops to 16–21% on fresh layouts. Both released VLAs score 0% zero-shot, with π0.5 at 0 of 167 rollouts. Even fine-tuned π0.5 reaches 0% on trained D2 pairings. This suggests the bottleneck is basic manipulation competence in this particular sim setup (camera, control interface, about 50 demos per cell) rather than language grounding, so most cells cannot yet test what the benchmark is designed to measure.
  3. In the spatial-modifier task, each category is paired with only one direction during training, so direction is perfectly correlated with category. A policy trained this way has no evidence that direction varies independently of category, so a gap near 1 (as for fine-tuned π0.5) may reflect an under-specified training distribution rather than a grounding deficit. Similar concerns apply to negation, where the target is also the visually odd object, and to D2's "in"/bowl confound. Each of these requires extra controls that are deferred to the supplement.
  4. The benchmark has one embodiment, about 30 meshes and only pick-and-place sub-tasks. Its contrastive and compositional evaluation idea is close to VLMbench, ATOM-Bench, ClevrSkills, LIBERO-PRO/Plus and "One Word, Different Action", and the paper does not clearly show what new insight GroundProbe provides that these do not. Key material is also deferred to the supplement, including the comparison table, the control conditions and the per-cell counts, which makes the main paper hard to evaluate on its own.
Questions for authors

please see the weaknesses.

Reviewer_5PyGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GroundProbe, a benchmark for studying how robot policies follow language instructions. It tests language understanding, new object–container combinations, and responses to changes in the scene. The benchmark compares familiar conditions with test conditions that change a specific instruction or scene factor. Experiments show poor transfer for fine-tuned π0.5 on spatial instructions, but evidence across the full benchmark remains limited.

Critical strengths
  1. The paper addresses an important evaluation problem: whether robot policies follow language instructions or rely on familiar visual and action patterns. This is useful for understanding robot behavior beyond overall task success.

  2. The matched comparisons, language-free baseline, and checks on familiar-task performance are useful design choices. These checks provide useful controls for studying failures on new combinations.

Critical weaknesses
  1. The contribution over existing benchmarks needs to be clearer. LIBERO-Plus and LIBERO-PRO already evaluate changes to language instructions, target objects, and task goals. GroundProbe introduces specific training splits and matched comparisons, but the paper does not clearly demonstrate what additional insight these provide. A stronger comparison with existing evaluations would help establish the value of the proposed benchmark.
  2. The low performance on familiar tasks raises concerns about the baseline setup. After fine-tuning, π0.5 achieves only 53% success on familiar direction–category combinations and 0% on familiar object–container pairings. This raises questions about whether the model is sufficiently trained and adapted to the environment. Training curves and a breakdown of object-selection, grasping, and placement errors would help clarify whether basic execution problems limit the evaluation.
  3. The experiments do not clearly establish why the models fail. The failures could come from memorized instructions, visual shortcuts, or difficulty handling unfamiliar objects. Stronger controlled experiments on the same π0.5 model are needed to separate these explanations and support the intended diagnoses.
  4. The experimental coverage and reliability are limited. The overall evaluation is limited, with only one fine-tuned VLA and insufficient validation across the proposed tests. Broader experiments are needed to support the paper’s claims.
Questions for authors
  1. What additional insights does GroundProbe provide beyond existing evaluations such as LIBERO-Plus and LIBERO-PRO? Can the authors demonstrate this difference through direct comparisons?

  2. Why does fine-tuned π0.5 perform poorly even on familiar tasks? Can the authors provide stronger controls to distinguish failures in following instructions from problems with training or basic manipulation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.