- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces BEHAVIOR-Spatial, a benchmark connecting spatial VQA, point-based target grounding, and robotic manipulation/navigation in shared household tasks and scenes. Its prompt-probe setting changes the requested target in familiar layouts to test instruction responsiveness. The experiments distinguish task completion, target-consistent partial progress, and wrong-target commitments, and show benefits from explicit target cues, including VLM-predicted grounding.
Critical strengths
- Jointly evaluating understanding, grounding, and action addresses an important gap in VLA evaluation. The shared task structure and prompt probe provide useful diagnostics of when robot behavior follows changed spatial instructions.
- The oracle/predicted-grounding comparisons and demonstration-distribution case study offer informative evidence about target selection and execution. Reporting strict success, soft success, and wrong-target rates helps distinguish different failure modes.
Critical weaknesses
- The reliability of the perception-action comparisons remains uncertain. Table 3 uses task-level correlations across 14 manipulation tasks, pairing Qwen3.5-4B perception scores with policy variants based on the same backbone. Differences between VQA-success and grounding-success correlations may be sensitive to the small task set and shared task difficulty. Uncertainty or robustness analyses would help assess the strength of these diagnostic comparisons. Appendix F.6 compares three grounding VLMs with a fixed action policy; it usefully tests predicted cues, but does not broaden the task-correlation analysis across action-policy backbones.
- The prompt probe changes both the requested target and the action trajectory relative to the demonstrations for that layout. Low success therefore diagnoses limited instruction-responsive behavior, but does not by itself isolate spatial understanding from generalization to undemonstrated trajectories. The paperback augmentation study partially examines this issue, yet also changes trajectories and data exposure, as the authors acknowledge. This limits attribution of the observed failures to particular mechanisms.
Overall, the benchmark is promising and the reported diagnostics are useful. I nevertheless lean toward a weak reject because the uncertainty of the correlation comparisons and the attribution of prompt-probe failures leave important interpretive questions unresolved. I do not interpret the paper as claiming a universal causal relationship between stronger VLMs and better VLAs.
Questions for authors
- Could the authors report uncertainty or resampling analyses for Table 3 and the main success-rate differences, accounting for task/scene structure? How robust is the comparison between grounding-success and VQA-success correlations to individual tasks?
- Could a controlled subset provide demonstrations for both valid targets in comparable layouts, followed by instruction swaps at evaluation? This would help separate instruction responsiveness from generalization to undemonstrated trajectories. If such a control is unavailable, please clarify the limits of the failure attribution.
4 comments
Thank you again for the careful review. Since our reply, we ran additional analyses that address your questions more directly.
1. Q2: instruction responsiveness vs. undemonstrated trajectories.
Within each task, instructions name the same object category and differ only in the spatial relation (e.g., largest / smallest, closest / farthest), and relation variants are balanced in training. Training pairs the same layouts with different instructed targets, so each relation is demonstrated many times across layouts and mapped to different objects. The probe asks the policy to apply these familiar relations in familiar layouts. This is a generalization that needs no new vocabulary, no new scene, and no unattainable motion: the oracle-dot policy, trained on the same data, reaches 38.0% probe soft success versus 16.1% for the baseline.
To identify which failure the probe exposes, we analyzed where wrong commitments go:
| Analysis (prompt probe, baseline) | Result | Chance |
|---|---|---|
| Wrong commitments that go to the target demonstrated from that starting point (reach tasks with ≥3 candidates; 7 tasks, 349 episodes) | 80.6% [68, 91] | 27.7% |
| Committed episodes that choose the instructed object (11 reach tasks) | 12.6% (−17.6 [−26.3, −9.8] vs chance) | 30.2% |
| Same, with the oracle target supplied | +11.0 [+1.5, +19.9] above chance | — |
Failures are therefore not random generalization errors. Despite balanced relational supervision, the standard VLA recipe does not ground the spatial relation in the instruction and instead falls back on scene-level associations, and supplying the target reverses this. This is the failure the probe is designed to expose; held-out success alone does not reveal it.
2. Q1: robustness of the correlation ordering to the success metric.
Table 3 uses soft success. We repeated the task-clustered bootstrap (14 tasks, 10,000 resamples) with strict success:
| Policy | IR − VQA (strict) [95% CI] | Bootstrap fraction with gap > 0 | Holds across tasks |
|---|---|---|---|
| Baseline | +0.23 [−0.41, +0.78] | 0.77 | 14/14 |
| LoRA VLM | +0.28 [−0.28, +0.83] | 0.83 | 14/14 |
| DROID-pretrain | +0.36 [−0.20, +0.89] | 0.89 | 14/14 |
| Dot IR | +0.26 [−0.34, +0.85] | 0.79 | 14/14 |
| Dot IR + first-frame | +0.15 [−0.54, +0.80] | 0.66 | 12/14 |
| Attention map | +0.57 [−0.08, +1.09] | 0.96 | 14/14 |
With strict success, grounding correlates more strongly with success than VQA for all six policies, and for five of them the ordering survives removal of any individual task. The individual strict correlations are less certain than the soft ones (VQA ρ 0.02–0.33, grounding ρ 0.38–0.62), so we keep the interpretation in §5.3.3: the ordering is consistent across policies and metrics but suggestive rather than conclusive. This is provided as additional rebuttal evidence.
3. Beyond a single backbone.
In a single-task study (reach_paperback_book, same probe instances), we trained the same frozen-VLM + action-expert recipe on the original π0.5 PaliGemma backbone:
| Backbone (frozen VLM + action expert) | Probe soft | Probe strict | Probe wrong |
|---|---|---|---|
| PaliGemma (original π0.5) | 8.7 | 2.9 | 79.7 |
| Qwen3.5-4B | 11.6 | 10.1 | 79.7 |
The instruction-following failure the probe exposes is therefore not specific to our backbone. Across backbones on the perception side, code-as-policy agents built on five writers show writer VQA and soft success rank-correlated at ρ = 0.70.
My main concerns are resolved.
Thank you for the additional analyses and clarifications. The task-level bootstrap and the revised interpretation address my concern about the correlation results. The wrong-target analysis and the oracle-target experiment also clarify what the prompt probe reveals about policies’ tendency to select previously demonstrated targets. These additions resolve my main concerns, and I will raise my rating from 2 to 3.
Thank you for the follow-up and for raising your rating! We are glad the additional analyses clarified the correlation results and what the prompt probe reveals. We will include the wrong-target analysis in the final version. Your feedback substantially improved the paper!
Thank you, Authors
Thank you for the careful review!
Q1 / W1: Correlation uncertainty and robustness. We resampled the 14 tasks with replacement 10,000 times and computed Spearman correlations and their paired differences. The table reports 95% bootstrap confidence intervals. Holds across tasks’ counts the tasks whose removal leaves the sign of the gap unchanged.
Grounding has a higher observed correlation with success for four of six policies (LoRA, DROID, dot IR, attention map). For dot IR and attention maps, the ordering survives removal of any individual task, but the difference CIs include zero. The relative predictive strength therefore remains uncertain and policy-dependent. These task-level analyses assess sensitivity to the task set; they do not eliminate shared scene difficulty or quantify training-seed variability. We qualified the correlation interpretation in sec5.3.3. For the main intervention, the oracle-dot soft-success improvement is +18.8 pp [12.5, 25.1], and +21.9 pp [14.3, 29.9] on the probe, with improvements on 13/14 tasks. This provides stronger evidence for the benefit of explicit target conditioning than for the relative predictive strength of grounding versus VQA. The revised matched-exposure results are reported in Table 4 and our response to zqMD. The bootstrap analyses are provided here as additional rebuttal evidence.
Q2 / W2: instruction responsiveness vs undemonstrated trajectories. We apologize for the ambiguity. Evaluation introduces new combinations of object descriptions and spatial requirements, while the constituent descriptions and relations occur during training. Training also includes the same layouts with different instructed targets and corresponding demonstrations, providing supervision for instruction-dependent target selection. (e.g., in reach_toys the same desk layout is demonstrated under both "Reach the smallest toy on the desk" and "Reach the 3rd largest toy on the desk") Thus, training does not consistently associate each layout with a single target. Nevertheless, this does not guarantee demonstration coverage of every probe state–instruction–trajectory combination. We therefore interpret prompt-probe performance as instruction-responsive generalization. The probe measures whether behavior follows changed instructions in familiar layouts. We clarified the training and evaluation construction in sec4.3 and acknowledge this limitation here.