acceptodds
Under review as a conference paper at ICLR 2027

Behavior Spatial Benchmark: Evaluating Spatial Understanding for Robotics

Abstract

Robot tasks require grounding spatial language in physical scenes, but task success alone cannot establish whether spatial instructions are followed. We present BEHAVIOR-Spatial, a benchmark built on BEHAVIOR-1K that measures spatial question answering, point-based target grounding, and robot action to examine how VLM spatial understanding relates to VLA behavior. It comprises 3,256 spatial questions across 49 activities, 1,240 grounding queries across 14 manipulation tasks, and action evaluations covering those 14 tasks and 10 navigation tasks. Our manipulation protocol evaluates held-out configurations and instruction following when the requested target changes within a fixed training layout. The baseline achieves 41.4% on held-out configurations but only 16.1% under changed instructions. VLM adaptation and robot-data pretraining provide limited improvements, whereas oracle visual target cues raise these rates to 60.3% and 38.0%, respectively. Perception evaluations reveal policy-dependent correlations between VLM question-answering and grounding accuracy and VLA manipulation performance. Together, these findings motivate evaluating spatial instruction following alongside task completion and examining when spatial perception performance translates into robot behavior.

Primary area:
datasets and benchmarks
Keywords:
vision-language-action models, robot learning, spatial understanding, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Q6UHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BEHAVIOR-Spatial, a benchmark connecting spatial VQA, point-based target grounding, and robotic manipulation/navigation in shared household tasks and scenes. Its prompt-probe setting changes the requested target in familiar layouts to test instruction responsiveness. The experiments distinguish task completion, target-consistent partial progress, and wrong-target commitments, and show benefits from explicit target cues, including VLM-predicted grounding.

Critical strengths
  1. Jointly evaluating understanding, grounding, and action addresses an important gap in VLA evaluation. The shared task structure and prompt probe provide useful diagnostics of when robot behavior follows changed spatial instructions.
  2. The oracle/predicted-grounding comparisons and demonstration-distribution case study offer informative evidence about target selection and execution. Reporting strict success, soft success, and wrong-target rates helps distinguish different failure modes.
Critical weaknesses
  1. The reliability of the perception-action comparisons remains uncertain. Table 3 uses task-level correlations across 14 manipulation tasks, pairing Qwen3.5-4B perception scores with policy variants based on the same backbone. Differences between VQA-success and grounding-success correlations may be sensitive to the small task set and shared task difficulty. Uncertainty or robustness analyses would help assess the strength of these diagnostic comparisons. Appendix F.6 compares three grounding VLMs with a fixed action policy; it usefully tests predicted cues, but does not broaden the task-correlation analysis across action-policy backbones.
  2. The prompt probe changes both the requested target and the action trajectory relative to the demonstrations for that layout. Low success therefore diagnoses limited instruction-responsive behavior, but does not by itself isolate spatial understanding from generalization to undemonstrated trajectories. The paperback augmentation study partially examines this issue, yet also changes trajectories and data exposure, as the authors acknowledge. This limits attribution of the observed failures to particular mechanisms.

Overall, the benchmark is promising and the reported diagnostics are useful. I nevertheless lean toward a weak reject because the uncertainty of the correlation comparisons and the attribution of prompt-probe failures leave important interpretive questions unresolved. I do not interpret the paper as claiming a universal causal relationship between stronger VLMs and better VLAs.

Questions for authors
  1. Could the authors report uncertainty or resampling analyses for Table 3 and the main success-rate differences, accounting for task/scene structure? How robust is the comparison between grounding-success and VQA-success correlations to individual tasks?
  2. Could a controlled subset provide demonstrations for both valid targets in comparable layouts, followed by instruction swaps at evaluation? This would help separate instruction responsiveness from generalization to undemonstrated trajectories. If such a control is unavailable, please clarify the limits of the failure attribution.
4 comments

Thank you for the careful review!

Q1 / W1: Correlation uncertainty and robustness. We resampled the 14 tasks with replacement 10,000 times and computed Spearman correlations and their paired differences. The table reports 95% bootstrap confidence intervals. Holds across tasks’ counts the tasks whose removal leaves the sign of the gap unchanged.

PolicyVQA ρIR ρIR − VQABootstrap fraction with gap > 0Holds across tasks
Baseline0.713 [0.23, 0.93]0.658 [0.19, 0.87]−0.06 [−0.59, +0.43]0.3910/14
LoRA VLM0.468 [−0.09, 0.84]0.644 [0.20, 0.90]+0.18 [−0.37, +0.68]0.7513/14
DROID-pretrain0.424 [−0.16, 0.80]0.552 [0.08, 0.81]+0.13 [−0.45, +0.68]0.6513/14
Dot IR0.560 [0.02, 0.86]0.701 [0.23, 0.93]+0.14 [−0.33, +0.62]0.7314/14
Dot IR + first-frame0.612 [0.12, 0.89]0.618 [0.08, 0.92]+0.01 [−0.55, +0.53]0.517/14
Attention map0.608 [0.07, 0.89]0.753 [0.31, 0.96]+0.15 [−0.28, +0.61]0.7714/14

Grounding has a higher observed correlation with success for four of six policies (LoRA, DROID, dot IR, attention map). For dot IR and attention maps, the ordering survives removal of any individual task, but the difference CIs include zero. The relative predictive strength therefore remains uncertain and policy-dependent. These task-level analyses assess sensitivity to the task set; they do not eliminate shared scene difficulty or quantify training-seed variability. We qualified the correlation interpretation in sec5.3.3. For the main intervention, the oracle-dot soft-success improvement is +18.8 pp [12.5, 25.1], and +21.9 pp [14.3, 29.9] on the probe, with improvements on 13/14 tasks. This provides stronger evidence for the benefit of explicit target conditioning than for the relative predictive strength of grounding versus VQA. The revised matched-exposure results are reported in Table 4 and our response to zqMD. The bootstrap analyses are provided here as additional rebuttal evidence.

Q2 / W2: instruction responsiveness vs undemonstrated trajectories. We apologize for the ambiguity. Evaluation introduces new combinations of object descriptions and spatial requirements, while the constituent descriptions and relations occur during training. Training also includes the same layouts with different instructed targets and corresponding demonstrations, providing supervision for instruction-dependent target selection. (e.g., in reach_toys the same desk layout is demonstrated under both "Reach the smallest toy on the desk" and "Reach the 3rd largest toy on the desk") Thus, training does not consistently associate each layout with a single target. Nevertheless, this does not guarantee demonstration coverage of every probe state–instruction–trajectory combination. We therefore interpret prompt-probe performance as instruction-responsive generalization. The probe measures whether behavior follows changed instructions in familiar layouts. We clarified the training and evaluation construction in sec4.3 and acknowledge this limitation here.

Thank you again for the careful review. Since our reply, we ran additional analyses that address your questions more directly.

1. Q2: instruction responsiveness vs. undemonstrated trajectories.

Within each task, instructions name the same object category and differ only in the spatial relation (e.g., largest / smallest, closest / farthest), and relation variants are balanced in training. Training pairs the same layouts with different instructed targets, so each relation is demonstrated many times across layouts and mapped to different objects. The probe asks the policy to apply these familiar relations in familiar layouts. This is a generalization that needs no new vocabulary, no new scene, and no unattainable motion: the oracle-dot policy, trained on the same data, reaches 38.0% probe soft success versus 16.1% for the baseline.

To identify which failure the probe exposes, we analyzed where wrong commitments go:

Analysis (prompt probe, baseline)ResultChance
Wrong commitments that go to the target demonstrated from that starting point (reach tasks with ≥3 candidates; 7 tasks, 349 episodes)80.6% [68, 91]27.7%
Committed episodes that choose the instructed object (11 reach tasks)12.6% (−17.6 [−26.3, −9.8] vs chance)30.2%
Same, with the oracle target supplied+11.0 [+1.5, +19.9] above chance—

Failures are therefore not random generalization errors. Despite balanced relational supervision, the standard VLA recipe does not ground the spatial relation in the instruction and instead falls back on scene-level associations, and supplying the target reverses this. This is the failure the probe is designed to expose; held-out success alone does not reveal it.

2. Q1: robustness of the correlation ordering to the success metric.

Table 3 uses soft success. We repeated the task-clustered bootstrap (14 tasks, 10,000 resamples) with strict success:

PolicyIR − VQA (strict) [95% CI]Bootstrap fraction with gap > 0Holds across tasks
Baseline+0.23 [−0.41, +0.78]0.7714/14
LoRA VLM+0.28 [−0.28, +0.83]0.8314/14
DROID-pretrain+0.36 [−0.20, +0.89]0.8914/14
Dot IR+0.26 [−0.34, +0.85]0.7914/14
Dot IR + first-frame+0.15 [−0.54, +0.80]0.6612/14
Attention map+0.57 [−0.08, +1.09]0.9614/14

With strict success, grounding correlates more strongly with success than VQA for all six policies, and for five of them the ordering survives removal of any individual task. The individual strict correlations are less certain than the soft ones (VQA ρ 0.02–0.33, grounding ρ 0.38–0.62), so we keep the interpretation in §5.3.3: the ordering is consistent across policies and metrics but suggestive rather than conclusive. This is provided as additional rebuttal evidence.

3. Beyond a single backbone.

In a single-task study (reach_paperback_book, same probe instances), we trained the same frozen-VLM + action-expert recipe on the original π0.5 PaliGemma backbone:

Backbone (frozen VLM + action expert)Probe softProbe strictProbe wrong
PaliGemma (original π0.5)8.72.979.7
Qwen3.5-4B11.610.179.7

The instruction-following failure the probe exposes is therefore not specific to our backbone. Across backbones on the perception side, code-as-policy agents built on five writers show writer VQA and soft success rank-correlated at ρ = 0.70.

Reviewer_Q6UHOpenReview ↗

My main concerns are resolved.

Thank you for the additional analyses and clarifications. The task-level bootstrap and the revised interpretation address my concern about the correlation results. The wrong-target analysis and the oracle-target experiment also clarify what the prompt probe reveals about policies’ tendency to select previously demonstrated targets. These additions resolve my main concerns, and I will raise my rating from 2 to 3.

Thank you for the follow-up and for raising your rating! We are glad the additional analyses clarified the correlation results and what the prompt probe reveals. We will include the wrong-target analysis in the final version. Your feedback substantially improved the paper!

Thank you, Authors

Reviewer_GHZhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To distinguish whether VLAs follow spatial instructions or memorize layouts, the paper proposes a benchmark built on BEHAVIOR-1K that aligns VQA, target location pointing (point grounding), and action evaluation in the same scenes. The baseline VLA achieves 41.4% soft success (counting movement toward the correct target as success even without completion) on unseen layouts, but only 16.1% when the instruction is changed to a different target in a training layout (the paper's prompt-probe set; Table 4).

Critical strengths
  • Aligning VQA, pointing, and action evaluation on the same scenes and objects allows failures to be examined by stage. Unlike comparisons of external VQA scores with policy performance in different environments, this allows perception, grounding, and execution to be measured separately on the same targets.
Critical weaknesses
  1. The policy evaluation does not demonstrate utility as a benchmark. The evaluated policies consist only of one author-developed model (with a frozen VLM and an action expert, the action output component, trained from scratch on this data) and its variants, with no existing VLAs or BEHAVIOR Challenge policies. Since prompt-probe instructs the agent to select a target different from the one demonstrated in a training layout ("when it conflicts with the demonstrated target", p.5), its low success rate may result from training on data that does not require following instructions, rather than an inability of VLAs to follow instructions. Indeed, in the reach paperback book task, where the instructed book was always on the same side in the training data, merely changing the approach direction in the demonstrations raised prompt-probe soft success from 11.6% to 36.2% (p.8). Thus, the current results do not distinguish general limitations of VLAs from limitations of this data–model combination.
  2. The contribution of connecting VQA to action overlaps with prior work, and the design is weaker. The paper states that this connection is "rarely studied directly" (p.1), but Vlaser (arXiv:2510.11027) and VLM4VLA (arXiv:2601.03309) have already studied it directly. Both studies showed that VQA and embodied reasoning scores based on internet images do not predict closed-loop control performance well. Vlaser further showed that training VLMs on QA, grounding, and spatial data constructed in the same domain as robot observations substantially improves closed-loop success rates. This paper's core setup of aligning VQA and action in the same scenes follows the same direction, so the paper should clarify its contribution relative to these two works. By contrast, this paper's analysis is limited to correlations across 14 tasks for a single VLM.
Questions for authors
  1. Does prompt-probe success remain low when training on data demonstrating different targets in the same layout?
  2. How do top BEHAVIOR Challenge policies or existing VLAs such as fine-tuned π0.5 perform under this protocol?
  3. What new insights are gained over Vlaser and VLM4VLA? Do you have results comparing VQA and target pointing (IR) scores with policy success across different backbones?
4 comments

Thank you for the review and the pointers to related work!

W1/Q1: Training distribution and prompt-probe interpretation. We apologize for the ambiguity. Evaluation introduces new combinations of object descriptions and spatial requirements, while the constituent descriptions and relations occur during training. Training also includes layouts with different instructed targets and corresponding demonstrations. For example, reach_toys includes instructions targeting the smallest and third-largest toys in the same desk layout. Thus, training does not consistently associate each layout with a single target. Nevertheless, this does not guarantee demonstration coverage of every probe state–instruction–trajectory combination. We interpret the probe as measuring instruction-responsive generalization, rather than uniquely diagnosing a spatial-understanding failure. The baseline achieves 16.1% soft and 10.0% strict probe success, with 70.0% wrong-target commitments. Oracle-conditioned policies improve these outcomes, while the paperback study shows that demonstration design also matters. These findings concern the tested data–policy combinations, rather than a universal limitation of VLAs. We clarified the evaluation combinations and training supervision in sec4.3.

Q2: existing VLAs / BEHAVIOR Challenge policies. Our VLA study examines the backbone-VLM-to-VLA relation under controlled variations: frozen vs LoRA, initialization, and grounding interface. To show usefulness beyond our policy family, we evaluate code-as-policy agents (following CaP-X) with five writers, three proprietary and two open-weight, on the same held-out episodes and metrics. The benchmark separates them from 5.1% to 51.4% strict. The best (Fable 5.1, 66.1% soft / 51.4% strict) exceeds our oracle-cue VLA (60.3% / 44.8%) yet still commits to the wrong target in 23.5% of episodes. Open-weight writers almost never complete an interaction (5% strict), with 83–87% of episodes ending without a decision. The protocol thus exposes distinct failure profiles: wrong-target commitment for VLAs, and failure to turn perception into control for open-weight agents. We agree that evaluating independent pretrained VLAs would further establish benchmark utility; code-as-policy results do not replace that comparison. We were unable to complete the adaptation and evaluation of additional VLAs within the rebuttal timeline, and leave this as a limitation and direction for future work.

W2 / Q3: relation to Vlaser and VLM4VLA. We agree that connecting spatial perception and robot control, including through in-domain supervision, is not new. Our contribution is the combined evaluation protocol: Aligned measurement. We measure VQA, pointing, and action on the same scenes, objects, and relations, so a failure can be located at the understand, ground, or act stage. Instruction responsiveness as a separate axis. The prompt probe fixes the layout and changes only the target, revealing the wrong-target rate that held-out success hides. Layers from manipulation to navigation. Object-level target and placement grounding in manipulation, and scene-level spatial context in navigation. Our cross-model comparisons address two different questions. Across five code writers, general single-frame VQA and soft success have a descriptive rank correlation of ρ = 0.70. Because the writers use shared perception tools, this does not isolate the contribution of their own spatial perception. Separately, Table S6 changes the grounding source while holding the action policy fixed. Predicted points retain part of the oracle benefit, but the similar rollout results do not establish a reliable ordering by perception accuracy. We therefore do not claim that these experiments demonstrate a general relationship between stronger VLM backbones and better VLA control. We added Vlaser to the related-work discussion and clarified our positioning.

Thank you for the honest and careful response. As it largely acknowledges the concerns rather than resolving them, I keep my score.

Thank you for the reply. We would like to address the remaining concerns more directly with the results we worked on recent days.

1. Prompt-probe failures (W1/Q1).

Within each task, instructions name the same object category and differ only in the spatial relation (e.g., largest / smallest, closest / farthest), and relation variants are balanced in training. Training pairs the same layouts with different instructed targets, so each relation is demonstrated many times across layouts and mapped to different objects. The probe asks the policy to apply these familiar relations in familiar layouts. This is a compositional generalization that needs no new vocabulary, no new scene, and no unattainable motion: the oracle-dot policy, trained on the same data, reaches 38.0% probe soft success versus 16.1% for the baseline.

The failure is also not random:

Analysis (prompt probe, baseline)ResultChance
Wrong commitments that go to the target demonstrated from that starting point (reach tasks with ≥3 candidates; 7 tasks, 349 episodes)80.6% [68, 91]27.7%
Committed episodes that choose the instructed object (11 reach tasks)12.6% (−17.6 [−26.3, −9.8] vs chance)30.2%
Same, with the oracle target supplied+11.0 [+1.5, +19.9] above chance—

Despite balanced relational supervision, the standard VLA recipe does not ground the spatial relation in the instruction and instead falls back on scene-level associations. Supplying the target reverses this. This is the failure the probe is designed to expose; held-out success alone does not reveal it.

2. Existing policies (Q2).

Our VLA study asks whether a given VLM's spatial understanding transfers to action, so we hold the backbone fixed and vary only how it is used: frozen vs. LoRA-adapted, robot-data initialization, and explicit grounding interfaces. We keep the VLM frozen because adapting it does not help: at matched training exposure, LoRA adaptation leaves held-out and probe success unchanged (Table 4). Off-the-shelf VLAs such as pi0.5 change the backbone, pretraining data, and training procedure at once, so their scores could not be attributed to any of these factors and would mainly reflect overall task difficulty. Existing BEHAVIOR Challenge policies are additionally trained for the full long-horizon BEHAVIOR-1K tasks with their own task specifications. The winning policy is conditioned on learned task embeddings rather than language, so it cannot take our spatial instructions.

To show that the benchmark is informative for policies we did not design or train, we evaluate code-as-policy agents built on off-the-shelf frontier models that never see our training data. The benchmark separates them from 5.1% to 51.4% strict success, and even the strongest commits to the wrong target in 23.5% of episodes. The benchmark and evaluation interface are policy-agnostic, so any language-conditioned VLA can be evaluated with the released code.

3. Perception vs. success across backbones (Q3).

Our main VLA study fixes the backbone to isolate adaptation, initialization, and grounding interface. To test whether our conclusions depend on the backbone, we vary the model in three settings while holding everything else fixed.

(i) VLA backbones. In a single-task study (reach_paperback_book, same probe instances), we trained the same frozen-VLM + action-expert recipe on the original pi0.5 PaliGemma backbone:

Backbone (frozen VLM + action expert)Probe softProbe strictProbe wrong
PaliGemma (original pi0.5)8.72.979.7
Qwen3.5-4B11.610.179.7

The instruction-following failure is therefore not specific to our backbone.

(ii) Code-as-policy writers. Across five writers with identical tools, API, and budgets, single-frame VQA accuracy and soft success are rank-correlated (ρ = 0.70), although VQA does not fully determine the ranking (Gemini 3 is second in VQA but third in success).

(iii) Grounding sources for a fixed VLA (Table S6). Three VLMs supply the target point to the same policy. Predicted points recover 29–39% of the oracle gain, and rollout success follows their close tolerant pointing accuracy.

Across these settings, stronger perception helps action but does not by itself determine success, and the probe failure persists across backbones.

We have updated our follow-up above with additional analysis, including a cross-backbone comparison (point 3). Thank you for your time.

Reviewer_zqMDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

BEHAVIOR-Spatial is a benchmark built on BEHAVIOR-1K/OmniGibson that evaluates spatial VQA, point grounding, and action execution using shared scenes, objects, and spatial relations. It includes 3,256 VQA questions across 49 activities, 1,240 grounding queries over 14 manipulation tasks, and an action suite covering those manipulation tasks plus 10 navigation tasks. The experiments examine whether a VLM’s spatial understanding carries over to a VLA policy. The baseline often fails to respond to changed instructions, while explicit target cues substantially improve performance. LoRA adaptation and DROID initialization provide limited gains under the tested settings.

Critical strengths
  1. I found the prompt-probe experiment the most convincing part of the paper. Changing the requested target while keeping the scene fixed is a simple way to check whether a policy follows the instruction. The wrong-target rate rises from 38.2% on held-out episodes to 70.0% under this probe. The separate navigation test, where success drops from 19/20 to 0/20 with new starting poses, also exposes a failure that the original evaluation largely misses.
  2. Evaluating VQA, grounding, and actions on shared scenes and objects makes the benchmark useful for studying where failures occur. It also allows the authors to examine a question that deserves more attention: whether a backbone that answers spatial questions well actually produces a better action policy.
  3. The appendix includes several informative negative results.These experiments help explain why the main results are not simply a matter of adding more data or adapting the backbone.
Critical weaknesses
  1. My main concern is how the paper interprets “transfer.” The task-level correlations are based on 14 manipulation tasks, mostly within one backbone and policy family, with no uncertainty estimates. The strongest correlations involve oracle target cues. This shows that supplying the correct target helps the policy, but says less about whether the policy benefits from its backbone’s own spatial understanding. Table S5 is particularly relevant here: when each backbone supplies its own predicted points, the three models have nearly identical action performance despite their perception differences. This result needs discussion.

  2. The LoRA comparison is difficult to interpret because the training exposure differs substantially. LoRA uses batch size 128 for 110k steps, versus 256 for 100k steps for the baseline, roughly 55% as many training examples. The DROID-initialized variant is also trained for fewer steps. Without a matched comparison, I am not convinced that the limited gains can be attributed to adaptation itself.

  3. The navigation experiments do not adequately test instruction grounding. Each task has only one instruction template, and the evaluation changes the starting pose rather than the requested goal. A policy could therefore succeed without distinguishing between instructions. The new-start failure is useful evidence of poor generalization, but it does not resolve this issue. Direction-question VQA accuracy is also 25%, which makes the correlation with navigation success hard to interpret without uncertainty estimates.

  4. The scope of the conclusions should be narrower. 11/14 manipulation tasks are reaching tasks, and all are single-target, one-step tasks. Navigation covers five scenes, while the held-out manipulation evaluation changes layouts rather than scenes. Together with the template-generated VQA questions, this makes the benchmark primarily a test of target localization in a controlled setting. Broader claims about spatial reasoning in manipulation need additional evidence.

Questions for authors

See weakness.

4 comments

Thank you for the detailed review!

W1: Transfer, correlation uncertainty, and predicted grounding. We distinguish direct use of a backbone’s spatial representations from explicit use of its grounding predictions. First, we provide bootstrap confidence intervals for the task-level correlations in our response to Q6UH. Grounding has higher observed correlations than VQA for four of six policies, including dot IR and attention maps, but the confidence intervals for the differences include zero. We have qualified the interpretation in sec5.3.3. Second, oracle conditioning substantially improves target-directed behavior and completion. Because the baseline and oracle-conditioned policies are trained separately, this comparison only establishes the benefit of explicit target conditioning. Third, original Table S5—now Table S6, sec 5.3.4, Appendix F.7—holds the oracle-trained Qwen3.5-4B action policy fixed and changes only the inference-time dot source. Qwen3.5-4B predictions yield 48.8% held-out and 27.1% prompt-probe soft success. Thus, grounding predictions from the model used as the backbone can support action through an explicit interface. The three grounding sources achieve similar rollout performance, with no model leading on both soft and strict success. We agree that these results do not establish that stronger perception necessarily produces better control. The added code-as-policy experiments broaden policy coverage, but likewise do not isolate direct transfer across VLA backbones.

W2: Matched training exposure. We agree that the original comparison used unequal training exposure. We extended LoRA-VLM and DROID initialization to match the baseline’s training examples.

PolicyHeld-out soft / strictProbe soft / strict / wrong
Baseline41.4 / 27.816.1 / 10.0 / 70.0
LoRA-VLM, 200k × 12839.9 / 25.116.0 / 8.6 / 62.3
DROID initialization, 100k × 25640.5 / 25.418.4 / 11.0 / 63.4

Neither DROID or LORA-VLM improves on held-out. DROID shows a slight improvement over the baseline on the probe, whereas LoRA does not improve probe success. Both retain substantial wrong-target rates. Our conclusion is therefore limited to the tested recipes: matching training exposure does not close the instruction-following gap and our claim holds. We updated Table 4 and sec5.3.1.

W3: Navigation instruction grounding. Several goals share each environment: the diner contains four goals, and the two-floor house’s main floor contains three. From a given start, the scene, context frames, and initial view are identical, and only the instruction identifies the goal. See sec5.4 and C4 figure for more details. A policy ignoring the instruction could reach at most one goal per start, so these tasks directly test instruction following. Our additional trajectory analysis finds that 28 of 143 failed episodes terminate at another trained goal: 17/67 in the house and 11/76 in the diner. The policy follows a training route to the wrong destination, partly relying on memorized scene–route associations.

W4: scope. In manipulation, spatial understanding mainly shows up in two decisions: which object to act on (target localization) and where to put it (placement localization). Our reach and place tasks measure exactly these, across direction, locate, ordinal, distance, and size relations with confusable candidates, and even this core decision often fails. Broader spatial understanding, such as relating the robot to the scene and integrating observations over time, is covered by our navigation suite (five scenes, multi-frame context, multi-room routes). But the benchmark does not cover the full range of spatial reasoning required for general manipulation. We added this limitation in sec6.1.

Thank you again for the detailed review. Since our reply, we ran additional analyses relevant to W1 (interpretation of transfer).

1. The probe failure is not specific to our backbone.

In a single-task study (reach_paperback_book, same probe instances), we trained the same frozen-VLM + action-expert recipe on the original π0.5 PaliGemma backbone:

Backbone (frozen VLM + action expert)Probe softProbe strictProbe wrong
PaliGemma (original π0.5)8.72.979.7
Qwen3.5-4B11.610.179.7

Both backbones show the same instruction-following failure, so it is not an artifact of Qwen3.5-4B.

2. The failure is target selection from language, not execution.

Within each task, instructions name the same object category and differ only in the spatial relation (e.g., largest / smallest, closest / farthest), and relation variants are balanced in training. Training pairs the same layouts with different instructed targets, so each relation is demonstrated many times across layouts and mapped to different objects. The prompt probe asks the policy to apply these familiar relations in familiar layouts, a compositional generalization that needs no new vocabulary, no new scene, and no unattainable motion: the oracle-dot policy, trained on the same data, reaches 38.0% probe soft success versus 16.1% for the baseline.

The failure is also not random. In reach tasks with at least three candidates (7 tasks, 349 episodes), 80.6% [68, 91] of the baseline's probe wrong commitments go to the target demonstrated from that starting point in training (chance 27.7%). Among committed episodes, the policy chooses the instructed object below chance (12.6% vs 30.2%; −17.6 [−26.3, −9.8]), while the oracle-dot policy is above chance (+11.0 [+1.5, +19.9]). Despite balanced relational supervision, the policy does not ground the spatial relation in the instruction and instead falls back on scene-level associations; supplying an explicit target fixes this. This supports our distinction between direct use of a backbone's spatial representations and explicit use of its grounding predictions.

3. Correlation ordering is robust to the success metric.

Repeating the task-clustered bootstrap with strict instead of soft success, grounding correlates more strongly with success than VQA for all six policies (IR − VQA gaps +0.15 to +0.57; sign preserved across 12–14 of 14 tasks). The individual strict correlations are less certain than the soft ones, so we keep the qualified interpretation in §5.3.3. The full table is in our follow-up to Reviewer Q6UH.

Reviewer_zqMDOpenReview ↗

Thank you for the detailed responses and additional experiments.

Thank you for the detailed responses and additional experiments. The reported exposure-matched LoRA and DROID experiments address most of my concern.

Some limitations remain. The experiments do not establish that stronger backbone perception leads to better control, and the additional backbone result covers only one task. The manipulation suite also focuses primarily on single-step target localization. However, the authors have acknowledged these limitations and narrowed their interpretation, which makes the claims better aligned with the evidence.

On balance, I consider the benchmark and the strengthened failure analysis sufficient to support borderline acceptance.

Thank you for the follow-up and for reconsidering your assessment. We will make the remaining limitations explicit in the final version. Your feedback substantially improved the paper!

Thank you, Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response

We thank the reviewers! We address three shared concerns: uncertainty in perception–action correlations, interpretation of prompt-probe failures, and the breadth of evaluated policies.

  • Correlation uncertainty. We provide task-level bootstrap confidence intervals and per tasks analyses below. Grounding has a higher observed correlation with success than VQA for four of six policies (matched-exposure LoRA and DROID, dot IR, attention map); the two are similar for the baseline and first-frame variants. For dot IR and attention maps, this ordering survives removal of any individual task, but the confidence intervals for the differences include zero. We therefore describe the relative ordering as suggestive rather than conclusive. We clarify this in sec5.3.3
  • Matched training exposure. We extended LoRA-VLM and DROID initialization to match other policies’ training examples. Confidence intervals for their soft- and strict-success differences relative to baseline include zero. Neither tested recipe closes the instruction-following gap. We updated Table4 and sec5.3.1
  • Broader policy evaluation. We added code-as-policy agents with five writer models (sec4.4, sec5.3.2, Appendix F.5). Strict success ranges from 5.1% to 51.4%, demonstrating benchmark utility beyond our VLA family.
  • Clarified interpretation. Training includes familiar instruction components and layouts with multiple instructed targets, but the probe can still require generalization to unseen state–instruction–trajectory combinations. We clarify training supervision in sec4.3 and narrow the benchmark’s scope in sec6.1.

Changes in the revised PDF are marked in blue. For newly added sections that span multi-pages (Appendix F.5), only the section title is colored.

Dear Reviewers,

Thank you again for your time and thoughtful feedback throughout the review process. We have provided additional experiments and clarifications in our responses and follow-up replies.

As the discussion period is nearing its end, please let us know if you have any remaining questions or feel that any concern has not been fully addressed. We would be happy to provide further clarification before the discussion closes.

Best regards, The Authors

Dear reviewers, the authors have posted their responses to your reviews and follow-ups. Please have a look and discuss ASAP as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.