- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper question the high success rates of current VLA models on standard benchmarks truly reflect correct vision-language-action grounding. The authors introduce SpurDiag-Bench, which only permutes the correspondence between objects and positions. Under this minimal change, the success rates of 12 VLA models drop from 85.4%–99.8% to 10.6%–21.9%. The paper further proposes a chain analysis that decomposes failures into two stages: V-L grounding and (VL)-A action. By combining probing, representation analysis, frozen-action-head fine-tuning, and activation interventions, the authors attempt to localize the source of shortcuts. The experiments suggest that the models often do not lack target or position information; rather, they fail to bind the two correctly. For some models, adapting only the upstream vision-language pathway while keeping the action head frozen is sufficient to substantially recover performance.
Critical strengths
- The evaluation setup is simple but effective. Instead of introducing new tasks or stronger distribution shifts, the paper only permutes the object-position correspondence. This makes the performance drop more directly attributable to reliance on object-position shortcuts.
- The authors further combine probing, representation analysis, frozen-head fine-tuning, and activation interventions to distinguish failures in V-L grounding from failures in the action stage. This is more informative than simply re-evaluating existing models on a new benchmark.
Critical weaknesses
- The frozen-head experiment does not fully localize the source of the shortcut. Freezing the action head shows that performance can be recovered by adapting only the upstream pathway. However, this does not establish that the shortcut originates solely from the V-L pathway, nor does it rule out a contribution from the action module.
- The mechanistic evidence is still limited across models. The strongest activation intervention experiments are mainly conducted on π0.5, while the evaluated models also differ in their training data and starting checkpoints. It therefore remains unclear how well these mechanistic conclusions generalize across VLA architectures.
Questions for authors
- Could the authors include an action-only adaptation setting, where the vision-language pathway is frozen and only the action head is updated, and compare it with upstream-only and joint adaptation? This would provide a cleaner test of where the shortcut mainly arises.
- Could the authors repeat the activation intervention, or a similar causal localization analysis, on more VLA architectures? Reproducing the same pattern across models would substantially strengthen the generality of the mechanistic conclusions.
3 comments
Thank you for the detailed response and the additional experiments. The new action-head-only adaptation results and the activation intervention experiments across more VLA architectures substantially address my main concerns. In particular, these results provide stronger evidence for the localization of shortcuts and the generality of the mechanistic findings.
I also appreciate that this work goes beyond the typical evaluation of language-following capabilities in robotics benchmarks. Rather than simply identifying performance degradation under distribution shifts, the paper further investigates why these failures occur and attempts to localize their underlying mechanisms. I find this deeper analysis particularly valuable and believe it strengthens the paper's contribution.
Overall, I am satisfied with the authors' response, and my concerns have been largely resolved.
Thank you for the follow-up
Dear Reviewer dfzd,
Thank you for the follow-up. We are glad the action-head-only results and the activation substitutions on the additional models addressed your main concerns.
Both are now in the revised PDF, in the sections "Action-head-only fine-tuning" (Appendix L) and "Activation substitution across models and modalities" (Appendix M).
Thank you again for the review and for your time. We hope the weeks ahead go well for you.
Best regards,
Authors of ICLR 2027 Conference Paper 25992
Response to Reviewer dfzd. Part [1/1]
We sincerely thank the reviewer for the time and patience given to this review, and for recognizing the simple but effective design of the benchmark and the chain analysis that separates V-L failures from failures in the action stage.
Thank you for suggesting an action-head-only adaptation setting. We agree that the frozen-head experiment alone does not localize the shortcut. We add action-head-only fine-tuning, in which the vision encoder and reasoning module are frozen and only the action head is updated, on the same demonstrations and training budget as the frozen-head checkpoint (General Response #1). Action-head-only fine-tuning does not exceed the base checkpoint on any model, while the frozen-head checkpoint repairs the same models with the same data. Taken together, the shortcut lies in the representation that reaches the action head fa, and removing it does not require updating fa.
Thank you for suggesting the activation intervention on more VLA architectures. We repeat the substitution protocol of Section 5.4 on six further released checkpoints (General Response #2). The pattern of π0.5 [1] reproduces in every model. Substitution from layer 0 of fr switches the grasp in 65% to 100% of trials, and in every model the switching rate falls at a depth inside fr, with fa unchanged throughout. The depth differs with the backbone, but the pattern does not. The target is selected inside fr, and fa carries out the selection. The models differ in training data and starting checkpoints, as the reviewer notes, and the pattern holds across them.
We thank you again for your valuable comments. If you have any further questions, please let us know, and we will do our best to respond. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation.
References
[1] π0.5: a Vision-Language-Action Model with Open-World Generalization.