acceptodds
Under review as a conference paper at ICLR 2027

SpurDiag-Bench: Diagnosing Spurious Correlations in Vision-Language-Action Models

Abstract

Despite strong benchmark performance, there is growing consensus that many Vision-Language-Action (VLA) models still fail in real-world scenarios, suggesting that benchmark success overstates their real-world generalization. While most previous work attributes it to a data-scale problem, we observe that it can also stem from a spurious correlation phenomenon: through shortcut learning, models associate targets with incidental cues rather than desirable semantics and task logic. To systematically diagnose this phenomenon, we introduce SpurDiag-Bench. SpurDiag-Bench distinguishes itself from existing VLA benchmarks through diagnostic depth: it revisits known classic VLA tasks at fine granularity and reveals that VLA failures do not require harder/broader tasks - simple changes in task configuration alone are sufficient to expose failures hidden behind strong benchmark performance. Precisely, we introduce a chain analysis that decomposes the model along its perception-reasoning-action chain and applies causal interventions to separately examine the grounding of visual-language and actions. Across experiments on 12 state-of-the-art open-source VLAs, we report that spurious correlations are pervasive (even substantial for the best-performing model, ). We further fine-tune on representative models, investigating the mitigation of spurious correlations: some models become less reliant, while others retain persistent shortcut reliance. These findings caution against assuming that shortcut reliance can be readily corrected through additional training. Key properties are also identified in robust VLA models featured for countering spurious correlations (e.g., action-expert architecture, diverse pretraining), providing guidance for future model design for the community. The benchmark and source code will be released, and the leaderboard is available at https://vla-page.netlify.app.

Primary area:
datasets and benchmarks
Keywords:
vision-language-action models, spurious correlations, shortcut learning, benchmark, embodied AI, causal intervention, robot manipulation, LIBERO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dfzdOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper question the high success rates of current VLA models on standard benchmarks truly reflect correct vision-language-action grounding. The authors introduce SpurDiag-Bench, which only permutes the correspondence between objects and positions. Under this minimal change, the success rates of 12 VLA models drop from 85.4%–99.8% to 10.6%–21.9%. The paper further proposes a chain analysis that decomposes failures into two stages: V-L grounding and (VL)-A action. By combining probing, representation analysis, frozen-action-head fine-tuning, and activation interventions, the authors attempt to localize the source of shortcuts. The experiments suggest that the models often do not lack target or position information; rather, they fail to bind the two correctly. For some models, adapting only the upstream vision-language pathway while keeping the action head frozen is sufficient to substantially recover performance.

Critical strengths
  1. The evaluation setup is simple but effective. Instead of introducing new tasks or stronger distribution shifts, the paper only permutes the object-position correspondence. This makes the performance drop more directly attributable to reliance on object-position shortcuts.
  2. The authors further combine probing, representation analysis, frozen-head fine-tuning, and activation interventions to distinguish failures in V-L grounding from failures in the action stage. This is more informative than simply re-evaluating existing models on a new benchmark.
Critical weaknesses
  1. The frozen-head experiment does not fully localize the source of the shortcut. Freezing the action head shows that performance can be recovered by adapting only the upstream pathway. However, this does not establish that the shortcut originates solely from the V-L pathway, nor does it rule out a contribution from the action module.
  2. The mechanistic evidence is still limited across models. The strongest activation intervention experiments are mainly conducted on π0.5, while the evaluated models also differ in their training data and starting checkpoints. It therefore remains unclear how well these mechanistic conclusions generalize across VLA architectures.
Questions for authors
  1. Could the authors include an action-only adaptation setting, where the vision-language pathway is frozen and only the action head is updated, and compare it with upstream-only and joint adaptation? This would provide a cleaner test of where the shortcut mainly arises.
  2. Could the authors repeat the activation intervention, or a similar causal localization analysis, on more VLA architectures? Reproducing the same pattern across models would substantially strengthen the generality of the mechanistic conclusions.
3 comments
AuthorsOpenReview ↗

Response to Reviewer dfzd. Part [1/1]

We sincerely thank the reviewer for the time and patience given to this review, and for recognizing the simple but effective design of the benchmark and the chain analysis that separates V-L failures from failures in the action stage.


W1. The frozen-head experiment does not fully localize the source of the shortcut. Freezing the action head shows that performance can be recovered by adapting only the upstream pathway. However, this does not establish that the shortcut originates solely from the V-L pathway, nor does it rule out a contribution from the action module.

Q1. Could the authors include an action-only adaptation setting, where the vision-language pathway is frozen and only the action head is updated, and compare it with upstream-only and joint adaptation? This would provide a cleaner test of where the shortcut mainly arises.

Thank you for suggesting an action-head-only adaptation setting. We agree that the frozen-head experiment alone does not localize the shortcut. We add action-head-only fine-tuning, in which the vision encoder and reasoning module are frozen and only the action head is updated, on the same demonstrations and training budget as the frozen-head checkpoint (General Response #1). Action-head-only fine-tuning does not exceed the base checkpoint on any model, while the frozen-head checkpoint repairs the same models with the same data. Taken together, the shortcut lies in the representation that reaches the action head faf_a, and removing it does not require updating faf_a.


W2. The mechanistic evidence is still limited across models. The strongest activation intervention experiments are mainly conducted on π0.5, while the evaluated models also differ in their training data and starting checkpoints. It therefore remains unclear how well these mechanistic conclusions generalize across VLA architectures.

Q2. Could the authors repeat the activation intervention, or a similar causal localization analysis, on more VLA architectures? Reproducing the same pattern across models would substantially strengthen the generality of the mechanistic conclusions.

Thank you for suggesting the activation intervention on more VLA architectures. We repeat the substitution protocol of Section 5.4 on six further released checkpoints (General Response #2). The pattern of π0.5\pi_{0.5} [1] reproduces in every model. Substitution from layer 0 of frf_r switches the grasp in 65% to 100% of trials, and in every model the switching rate falls at a depth inside frf_r, with faf_a unchanged throughout. The depth differs with the backbone, but the pattern does not. The target is selected inside frf_r, and faf_a carries out the selection. The models differ in training data and starting checkpoints, as the reviewer notes, and the pattern holds across them.


We thank you again for your valuable comments. If you have any further questions, please let us know, and we will do our best to respond. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation.


References

[1] π0.5\pi_{0.5}: a Vision-Language-Action Model with Open-World Generalization.

Thank you for the detailed response and the additional experiments. The new action-head-only adaptation results and the activation intervention experiments across more VLA architectures substantially address my main concerns. In particular, these results provide stronger evidence for the localization of shortcuts and the generality of the mechanistic findings.

I also appreciate that this work goes beyond the typical evaluation of language-following capabilities in robotics benchmarks. Rather than simply identifying performance degradation under distribution shifts, the paper further investigates why these failures occur and attempts to localize their underlying mechanisms. I find this deeper analysis particularly valuable and believe it strengthens the paper's contribution.

Overall, I am satisfied with the authors' response, and my concerns have been largely resolved.

AuthorsOpenReview ↗

Thank you for the follow-up

Dear Reviewer dfzd,

Thank you for the follow-up. We are glad the action-head-only results and the activation substitutions on the additional models addressed your main concerns.

Both are now in the revised PDF, in the sections "Action-head-only fine-tuning" (Appendix L) and "Activation substitution across models and modalities" (Appendix M).

Thank you again for the review and for your time. We hope the weeks ahead go well for you.

Best regards,

Authors of ICLR 2027 Conference Paper 25992

Reviewer_qzz4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SpurDiag-Bench, a diagnostic benchmark for spurious correlations and shortcut learning in VLA models. Its central claim is that high success on standard benchmarks such as LIBERO-Object can hide reliance on incidental cues, such as fixed object-position pairings, stereotyped trajectories, or default layouts. SpurDiag-Bench keeps the LIBERO-Object objects, scene, and pick-and-place objective unchanged, but permutes the object-to-slot assignment to test whether a policy truly follows the instruction to the current target. The paper further proposes chain analysis, decomposing failures into V-L binding and (VL)-A binding, using frozen-head re-fine-tuning, activation substitution, linear probes, representational geometry, and trajectory-diversity metrics. Across 12 open-source VLAs, success drops from 85.4%-99.8% on official LIBERO-Object to 10.6%-21.9% on SpurDiag-Bench. Follow-up fine-tuning experiments show that some models can repair V-L binding while others retain shortcut reliance, and small-scale SO-100 and UR5 experiments provide real-world validation.

Critical strengths
  • Evaluating 12 open-source VLAs across single-stream, hierarchical, and world-model/representation-augmented families strengthens the generality of the observed phenomenon.
  • The paper goes beyond success rates by decomposing failures into V-L and (VL)-A binding and using frozen-head fine-tuning, activation substitution, linear probes, and representational geometry.
  • Frozen-head fine-tuning suggests that several models mainly fail at V-L binding rather than in the action head, and activation substitution shows how intermediate representations influence target selection. This is more scientifically useful than a leaderboard alone.
  • The paper documents checkpoints, readout layers, fine-tuning hyperparameters, trajectory-diversity computation, failure decomposition, and real-robot protocols.
Critical weaknesses
  • The main task is a 2×3-slot LIBERO-Object pick-and-place setup, mainly targeting position/object-slot shortcuts. Although the paper mentions possible extensions to color and shape, the main experiments do not fully support broad claims about all forms of VLA spurious correlation.
  • Linear probing shows whether information is decodable, not whether the policy uses it. Representational geometry depends on readout-layer and pooling choices. Dvar is not directly comparable across different action representations and coordinate scales. The paper discusses some of these issues, but the main claims sometimes sound stronger than what the metrics alone can prove.
  • The released checkpoints differ in training suites, camera counts, action representations, chunk horizons, and official evaluation code. The authors make reasonable efforts to use official interfaces, but these differences complicate cross-model ranking and architecture-level conclusions.
  • The paper contains a lot of valuable detail, but several figures and tables are dense, some terminology-heavy paragraphs are difficult to parse, and there are minor spelling/grammar issues. Separating metric definitions, diagnostic conclusions, and speculative interpretations more clearly would improve readability.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer qzz4. Part [1/2]

We sincerely thank the reviewer for the time and patience given to this review, and for recognizing the range of the 12 evaluated VLAs and the detail of our documentation.


W1. The main task is a 2×3-slot LIBERO-Object pick-and-place setup, mainly targeting position/object-slot shortcuts. Although the paper mentions possible extensions to color and shape, the main experiments do not fully support broad claims about all forms of VLA spurious correlation.

Thank you for pointing out the risk of broad claims about all forms of VLA spurious correlation. The paper does not claim color or shape shortcuts: they appear only as motivating examples in the Introduction and as possible extensions in the Conclusion, and our main findings concern the object-position shortcut, which we will state explicitly in the Abstract and in the contributions of Section 1 (Introduction). The 2×3 setting is a diagnostic design rather than a claim of coverage: to show that high benchmark success does not imply grounding, one controlled counterexample is enough, and 12 VLAs that reach 85.4% to 99.8% on LIBERO-Object [1] score only 10.6% to 21.9% on it, close to the 16.7% of a policy that ignores where the target is. Because objects, slots, scene and skill are unchanged and only the object-slot pairing is new, the drop can be traced to the shortcut, and the discrete, balanced grid is what makes the factorial geometry, the minimal-pair activation substitution and a clear failure decomposition possible. The same failure also appears beyond the grid and simulation, in the continuous-position OOD splits (Section 5.3) and on the SO-100 and UR5 robots (Appendices B and C).


W2. Linear probing shows whether information is decodable, not whether the policy uses it. Representational geometry depends on readout-layer and pooling choices. Dvar is not directly comparable across different action representations and coordinate scales. The paper discusses some of these issues, but the main claims sometimes sound stronger than what the metrics alone can prove.

Thank you for pointing out the limits of linear probing, representational geometry and DvarD_{\text{var}}. We believe the metric design is sound, but we agree with these limits, so we add the evidence below and will revise the paper accordingly. The probes only claim that instruction identity and object position are present in the representation and where they appear, and the claim that the policy uses this information rests on interventions. For the geometry, Appendix H reports every layer of frf_r, and Appendix H.3 varies the ridge coefficient and the pooling; the orderings the conclusions rest on do not change. We agree that DvarD_{\text{var}} is not comparable across action representations and coordinate scales, and we will use it only to compare checkpoints within one model. Beyond these metrics, activation substitution tests use directly: on seven models, replacing the activations of one instruction with those of another switches the closed-loop grasp to the donor instruction's target (Section 5.4; General Response #2).

AuthorsOpenReview ↗

Response to Reviewer qzz4. Part [2/2]

W3. The released checkpoints differ in training suites, camera counts, action representations, chunk horizons, and official evaluation code. The authors make reasonable efforts to use official interfaces, but these differences complicate cross-model ranking and architecture-level conclusions.

Thank you for raising how the checkpoint differences affect cross-model ranking and architecture-level conclusions. Every model is evaluated through its official interface: we follow its official LIBERO evaluation code, and with this code each model reproduces its reported success on official LIBERO-Object (Appendix E). Inputs and outputs follow the same protocol for all models, with the same layouts, instructions, simulator and success test, while training suites, cameras, action representation and chunk length are each model's own design and are kept as released. Our conclusions compare each model with itself, and despite these different settings all 12 models fall in the same 10.6% to 21.9% range. We will state in Section 5.1 (Performance of state-of-the-art VLAs on SpurDiag-Bench) that the scores are not a ranking across settings, and in Section 5.5 (Real-world validation and architecture design insights) that the design features are hypotheses. To our knowledge, SpurDiag-Bench is the first benchmark that locates where a VLA loses grounding, and we built it as an open platform for the community: we will release the adapters for all 12 models, the layouts and the evaluation and fine-tuning code, and we welcome everyone to reproduce our results and test new models under the same protocol.


W4. The paper contains a lot of valuable detail, but several figures and tables are dense, some terminology-heavy paragraphs are difficult to parse, and there are minor spelling/grammar issues. Separating metric definitions, diagnostic conclusions, and speculative interpretations more clearly would improve readability.

Thank you for these suggestions on readability. In the revision we will mark speculative interpretations as such and separate them from the metric definitions and diagnostic conclusions, rewrite the terminology-heavy paragraphs in plainer language, correct the spelling and grammar issues, and simplify the densest figures and tables.


We thank you again for your valuable comments. If you have any further questions, please let us know, and we will do our best to respond. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation.


References

[1] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning.

AuthorsOpenReview ↗

Invitation to discussion

Dear Reviewer qzz4,

Thank you for the review, and for recognizing the evaluation across 12 VLA models and the documentation. The revised PDF is on OpenReview. The changes for your comments are as follows.

  • W1. The experiments target the object-position shortcut, as now stated in the contributions (Section 1). The same position failure appears in the continuous-position splits (Section 5.3) and on the SO-100 and UR5 robots (Appendix B, Appendix C).
  • W2. We separated the roles of the linear probes, the representational geometry, and DvarD_{\mathrm{var}}. Section 4.3 states that DvarD_{\mathrm{var}} is compared only between checkpoints of the same model. Appendix H reports the probe and the geometry by layer, and Appendix H.3 varies the ridge coefficient and the pooling. Activation substitution on seven models tests use directly (Section 5.4, Appendix M).
  • W3. All 12 models are evaluated through their official interfaces, under the same layouts, instructions, simulator, and success test (Appendix E). Section 5.1 states that the checkpoints differ in training suites, cameras, and action representations, so each score is read against the model's own LIBERO-Object result, and the scores do not rank architectures. Section 5.5 states the design points as hypotheses.
  • W4. The text added in this discussion is shown in blue, and we will continue to improve the readability as described in our response.

We hope you will have a chance to look at the revised PDF. Please let us know if any point is still open, or if you have further questions. We will be glad to answer them.

Thank you again for your time. We hope the weeks ahead go well for you.

Best regards,

Authors of ICLR 2027 Conference Paper 25992

Reviewer_Ek9ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates whether VLA policies follow the instructed object when familiar objects are reassigned among familiar spatial slots, changing the object–position associations.. The principal benchmark contains six objects, six slots, and 180 layouts, evaluated with all six instructions for 1,080 episodes per model. Across 12 released checkpoints, the reported mean success rate falls from 96.8% on official LIBERO-Object evaluation to 14.8% on SpurDiag-Bench. This is a substantial and practically relevant failure pattern. The paper proposes a “chain analysis” separating vision-language binding from vision-language-action binding. It combines linear probes, representational-geometry metrics, frozen-head versus unfrozen-head fine-tuning, activation substitution, and behavioral failure decomposition. The most striking repair result is for pi0.5 success increases from 21.9% to 95.5% while its action expert remains frozen, versus 96.0% when the head is also updated. The study also includes spatial generalization tests and physical-robot experiments. The SO-100 experiment reproduces the direction of the frozen-head recovery, while the UR5 study extends the behavioral evaluation to instruction changes and same-category distractors.

Critical strengths
  1. The paper presents a focused evaluation with a clear practical motivation. Permuting object–slot assignments is an interpretable way to challenge policies that might associate instructions with habitual destinations. Testing every instruction on the same layout is particularly useful: it provides a controlled behavioral comparison without requiring a completely different task family. The benchmark is narrow, but that narrowness is an advantage for diagnosis rather than an automatic weakness

  2. The authors evaluate 12 checkpoints and reproduce their official LIBERO-Object performance using the corresponding evaluation code. The substantial frozen-head improvements also demonstrate that at least some existing action heads can support much better performance after upstream adaptation.

  3. The activation experiments include repeated closed-loop substitution, single-layer residual interventions, random-direction controls, and a first-inference-only control. These are worthwhile attempts to establish causal influence on behavior, even though I disagree with some of the stronger interpretations.

Critical weaknesses
  1. Upstream repair does not uniquely identify the failure source. Frozen-head fine-tuning shows that upstream changes can improve performance, but they could compensate for a fixed downstream mapping. Thus, Section 5.2’s conclusion that the shortcut lies in V-L rather than the action head is stronger than the evidence supports. Activation substitution also transfers image and text activations together, rather than isolating grounded target-location information.

  2. Limited training completion and replication constrain recovery conclusions. X-VLA is evaluated at 20k of a 60k-step schedule, MemoryVLA uses a checkpoint preceding a disk-quota failure, and VLA-JEPA stops at a wall-clock limit. Each adaptation arm is trained once. These results remain valid for the evaluated checkpoints, but without convergence evidence or repeated runs, broader conclusions about persistent shortcut reliance and model-dependent repairability remain uncertain.

Questions for authors
  1. What precisely do the main probe scores establish? Can the authors report target-position decoding from the same action-interface representation used for the geometry, using one shared probe and layout-disjoint evaluation? How are regularization and readout choices selected?

  2. What evidence excludes the possibility that upstream adaptation compensates for a fixed downstream mapping? Can the activation results separate transferring instruction identity or a habitual spatial plan from transferring a grounded target-location representation?

  3. How much of the failure remains after controlling action coverage and task mix? What are the matched six-target original-layout results? How does performance differ between the two original target slots and the remaining slots?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ek9R. Part [1/3]

We sincerely thank the reviewer for the time and patience given to this review, and for recognizing the controlled design of testing every instruction on the same layout and the controls in our activation experiments.


W1. Upstream repair does not uniquely identify the failure source. Frozen-head fine-tuning shows that upstream changes can improve performance, but they could compensate for a fixed downstream mapping. Thus, Section 5.2’s conclusion that the shortcut lies in V-L rather than the action head is stronger than the evidence supports.

Q2.1. What evidence excludes the possibility that upstream adaptation compensates for a fixed downstream mapping?

Thank you for this important comment on upstream adaptation compensating for a fixed downstream mapping. We agree that frozen-head fine-tuning alone cannot uniquely localize the origin of shortcut reliance. Three other results in the paper bear on this point, and we add a fourth as a direct test.

1. Frozen-head fine-tuning raises EposE_{pos} inside frf_r from near zero to between 0.60 and 1.04. This change is the V-L binding itself, and compensation of a fixed downstream mapping alone would not produce it. The rise holds in every model, with ElangE_{lang} lowered and faf_a frozen (Section 5.2, Table 3).

2. Activation substitution locates the layer of frf_r at which the grasp position is decided, and faf_a only executes it. On π0.5 [1] the grasp position is decided between layers 12 and 15 of frf_r (Section 5.4), and each of the other six models has its own deciding depth inside frf_r (General Response #2). A wrong grasp position therefore arises before faf_a.

3. Unfreezing faf_a has limited effect on target selection. Trained on the same demonstrations as the frozen-head checkpoint, the unfrozen-head checkpoint moves the wrong-target failures by at most 2 points on five of the six models, and its gains in success come from fewer not-grasped failures (Section 5.2, Figure 4). A shortcut inside faf_a would not leave target selection unchanged when faf_a is updated.

4. To answer the question directly, we add action-head-only fine-tuning, in which the vision encoder and reasoning module are frozen while only the action head is updated on the same demonstrations and training budget as the frozen-head checkpoint. Action-head-only fine-tuning does not exceed the base checkpoint on any model, while the frozen-head checkpoint repairs the same models with the same data. On π0.5\pi_{0.5}, for example, it reaches 13.3% against 21.9% for the base checkpoint and 95.5% for the frozen-head checkpoint (General Response #1).

Taken together, the shortcut lies in the representation that reaches faf_a, and removing it does not require updating faf_a. We will revise the sentence "The shortcut is in V-L, not in the action head" in Section 5.2 (V-L and (VL)-A binding) accordingly.


W1. Activation substitution also transfers image and text activations together, rather than isolating grounded target-location information.

Q2.2. Can the activation results separate transferring instruction identity or a habitual spatial plan from transferring a grounded target-location representation?

Thank you for this question on separating instruction identity from a grounded target-location representation. Section 5.4 replaces image and text activations together. This transfers the instruction's full effect and localizes the depth at which the target is decided (point 2 above). To separate what is transferred, we add a modality-separated version of the same protocol (Table R3).

ConditionFollows the donor grasp
No substitution (base)0%
Text activations only9.2%
Image activations only85.7%
Image + text100%

Table R3. Modality-separated substitution on π0.5\pi_{0.5}, protocol of Section 5.4 with all layers of frf_r substituted (ℓ=0\ell=0); rates are over the trials where the unpatched donor run grasps its object.

Text activations alone transfer little, and image activations alone transfer most of the effect. The image input is identical in both runs, so what transfers is what the instruction writes into the visual tokens of frf_r. Whether that effect is a habitual plan or the target's position is not what substitution measures, but the geometry in point 1 does.

AuthorsOpenReview ↗

Response to Reviewer Ek9R. Part [2/3]

W2.1. Limited training completion and replication constrain recovery conclusions. X-VLA is evaluated at 20k of a 60k-step schedule, MemoryVLA uses a checkpoint preceding a disk-quota failure, and VLA-JEPA stops at a wall-clock limit.

Thank you for raising the training completion of these runs. We ran additional experiments and report the success rate along training for the three models in Table R4. The reported checkpoints are at or near the peak of each run, and further training does not give better results. We will update the training details in Appendix F (Fine-tuning hyperparameters) accordingly.

ModelSR along training steps
MemoryVLA13.8% (0) → 55.4% (5k) → 65.0% (10k) → 68.6% (14k) → 68.2% (15k) → 65.0% (20k)
VLA-JEPA37.4% (10k) → 38.9% (16k) → 54.2% (18k) → 50.6% (20k) → 50.2% (22k)
X-VLA22.7% (10k) → 24.6% (20k) → 19.4% (40k) → 20.1% (60k)

Table R4. Frozen-head fine-tuning. Bold marks the checkpoint used in the paper.

W2.2. Each adaptation arm is trained once. These results remain valid for the evaluated checkpoints, but without convergence evidence or repeated runs, broader conclusions about persistent shortcut reliance and model-dependent repairability remain uncertain.

On π0.5\pi_{0.5}, we further train the frozen-head and unfrozen-head checkpoints with four additional seeds under the same settings as the paper, each evaluated on all 1,080 episodes. Over five seeds (mean ± standard deviation), the frozen-head checkpoint reaches 94.4% ± 3.1% and the unfrozen-head checkpoint 96.3% ± 1.3%, against 21.9% for the base checkpoint and 95.5% and 96.0% reported in the paper.


Q1.1. What precisely do the main probe scores establish?

Thank you for this question on the main probe scores. The two probe scores establish whether information is present, not whether it is used. Rpos2R^2_{\text{pos}} establishes that the target's position is linearly decodable at the input of frf_r, and Rlang2R^2_{\text{lang}} that the named object is identifiable at the last layer of frf_r, the output that faf_a receives. The design choices behind the probes are answered under the two questions below.


Q1.2. Can the authors report target-position decoding from the same action-interface representation used for the geometry, using one shared probe and layout-disjoint evaluation?

Thank you for suggesting one shared probe and layout-disjoint evaluation. Yes, Table R5 reports it for seven models at the readout layer of the geometry, the representation that faf_a receives. The revised Appendix H reports the full profile by depth, from the first block of fvf_v to the last layer of frf_r, for every checkpoint.

ModelUpstream peak R2R^2, base → frozen-head → unfrozen-headR2R^2 at the readout layer of the geometry, base → frozen-head → unfrozen-head
π0.5\pi_{0.5}0.79 → 0.87 → 0.890.17 → 0.45 → 0.50
π0\pi_00.85 → 0.87 → 0.84−0.02 → 0.29 → 0.23
OpenVLA0.79 → — → 0.890.36 → — → 0.64
UniVLA0.80 → 0.93 → 0.930.47 → 0.67 → 0.70
MemoryVLA0.79 → 0.87 → 0.88−0.05 → 0.49 → 0.50
X-VLA0.69 → 0.77 → 0.770.20 → 0.41 → 0.42
VLA-JEPA0.68 → 0.78 → 0.740.24 → 0.47 → 0.31

Table R5. Target-position decoding with one shared linear probe (ridge, λ=100\lambda = 100), five-fold cross-validation with whole layouts held out, R2R^2 averaged over the six objects. Left: the peak over all depths before the readout layer. Right: the readout layer of the geometry (in Appendix E). OpenVLA [4] has no separate action head and therefore no frozen-head checkpoint.

Table 3 evaluates one probe per object on held-out repeats of the same layouts. Table R5 evaluates one shared probe on held-out layouts, as the reviewer asks. The orderings between checkpoints are the same under both.

According to Table R5, the position is linearly decodable upstream in every base checkpoint. At the readout layer of the geometry, the base checkpoints of π0\pi_0 [2] and MemoryVLA [3] contain almost no linearly decodable position, whereas the remaining base checkpoints still contain part of it. The corresponding geometry (Section 5.2) shows that this information, although present, is not bound to the action.


Q1.3. How are regularization and readout choices selected?

Thank you for this question on the regularization and readout choices. The readout layers follow the chain: Rpos2R^2_{\text{pos}} reads the input of frf_r, the geometry reads its output, and Appendix H reports every layer in between. The ridge coefficient is shared by all probes and was not tuned. All readouts mean-pool their token set. We verified these choices in a sensitivity analysis (Appendix H.3). Varying the ridge coefficient and computing EposE_{\text{pos}} per token instead of pooled leave the orderings the conclusions rest on unchanged.

AuthorsOpenReview ↗

Response to Reviewer Ek9R. Part [3/3]

Q3.1. How much of the failure remains after controlling action coverage and task mix? What are the matched six-target original-layout results?

Thank you for this question on action coverage and task mix. Controlling action coverage and task mix removes only part of the failure. Our pipeline reproduces the official results on the six original tasks (Table R6, columns 1 and 2). Restricting the target to the two official slots, so that every required motion was executed during training, raises success but leaves it 43 to 58 points below the official result (column 3). The next question shows why.

ModelSix original tasks, official harnessSix original tasks, our pipelineAction coverage controlled (official range)Full benchmark
OpenVLA86.0%86.1%31.9%10.6%
OpenVLA-OFT96.7%94.4%44.7%14.9%
UniVLA95.7%92.2%37.5%12.5%
X-VLA97.3%97.8%42.5%14.2%
FLOWER100.0%96.7%45.8%15.3%
π0\pi_097.3%95.0%46.4%15.5%
π0.5\pi_{0.5}99.0%95.6%56.1%21.9%
MemoryVLA97.0%93.3%41.4%13.8%
DreamVLA97.5%94.4%46.1%15.6%
Spatial-Forcing99.3%95.6%42.2%14.3%
RynnVLA-00293.7%91.7%38.1%12.7%
VLA-JEPA99.3%97.8%48.9%16.3%

Table R6. Success rate on the six original tasks on the official harness (column 1) and in our pipeline (column 2), on the SpurDiag-Bench episodes whose target lies in an official slot (column 3) and on all 1,080 episodes (column 4).


Q3.2. How does performance differ between the two original target slots and the remaining slots?

Thank you for this question on the two original target slots. The models rely on the object-position pairing seen in training, not on action coverage. Table R7 splits the 1,080 episodes into three cases: the instructed object is at its own training position, at the training position of another target, or at a slot that was never a target position. The first two cases together are column 3 of Table R6. Success is high only in the first case. In the second case the motion to that slot was trained, yet success is low and zero for five models, close to the third case. This is why controlling action coverage leaves most of the failure in place. This experiment further confirms that the models bind each target to the position it occupied during training, which is the problem the paper diagnoses.

ModelTarget at its training positionTarget at the other original slotTarget at novel slots
OpenVLA64.2%0.0%0.0%
OpenVLA-OFT52.5%37.0%0.0%
UniVLA67.6%7.7%0.0%
X-VLA58.1%27.1%0.0%
FLOWER92.2%0.0%0.0%
π0\pi_093.3%0.0%0.0%
π0.5\pi_{0.5}97.8%14.9%4.9%
MemoryVLA83.2%0.0%0.0%
DreamVLA91.6%1.1%0.4%
Spatial-Forcing53.6%30.9%0.3%
RynnVLA-00276.5%0.0%0.0%
VLA-JEPA97.2%1.1%0.0%

Table R7. Success rate on SpurDiag-Bench by where the instructed object sits: at its training position, at the other original target slot, or at a novel slot.


We thank you again for your valuable comments. If you have any further questions, please let us know, and we will do our best to respond. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation.


References

[1] π0.5\pi_{0.5}: a Vision-Language-Action Model with Open-World Generalization.

[2] π0\pi_0: A Vision-Language-Action Flow Model for General Robot Control.

[3] MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation.

[4] OpenVLA: An Open-Source Vision-Language-Action Model.

[5] Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success.

[6] Learning to Act Anywhere with Task-centric Latent Actions.

[7] X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model.

[8] FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Flow Models.

[9] DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge.

[10] Spatial Forcing: Implicit Spatial Representation Alignment for Vision-Language-Action Model.

[11] RynnVLA-002: A Unified Vision-Language-Action and World Model.

[12] VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model.

AuthorsOpenReview ↗

Invitation to discussion

Dear Reviewer Ek9R,

Thank you for the review. In the rebuttal we ran additional experiments and added them to the revised PDF. They respond to your comments as follows.

  • W1 and Q2. On six models we trained an action-head-only checkpoint, with the vision encoder and reasoning module frozen and only the action head updated. It does not exceed the base checkpoint on any of the six. Together with the frozen-head result, this places the shortcut in the representation that the action head receives (Section 5.2, Appendix L).
  • W1 and Q2. We repeated activation substitution on six further models. On π0.5 we also separated the two modalities: image activations alone switch the grasp in 85.7% of trials, and text activations alone in 9.2% (Section 5.4, Appendix M).
  • W2. We report success along training for MemoryVLA, VLA-JEPA, and X-VLA. The reported checkpoints are at or near the peak of each run. On π0.5 we added four seeds. Over five seeds, frozen-head reaches 94.4% ± 3.1% (Section 5.2, Appendix F).
  • Q1. For seven models we report target-position decoding with one shared probe and layout-disjoint evaluation, plus a sensitivity check on the ridge coefficient and pooling. The probes show whether the target position is decodable, not whether the policy uses it (Appendix H, Appendix H.3).
  • Q3. We report success by target slot and under controlled action coverage. Keeping only the official slots leaves success 43 to 58 points below the official result. Success is highest when the target is at its own training position (Appendix K).

We hope you will have a chance to look at the revised PDF. Please let us know if any point is still open, or if you have further questions. We will be glad to answer them.

Thank you again for your time. We hope the weeks ahead go well for you.

Best regards,

Authors of ICLR 2027 Conference Paper 25992

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response. Part [2/2]

#2 Activation substitution on six more models

We repeat the substitution protocol of Section 5.4 on six further released checkpoints.

Model \ ℓ\ell04812131415161718202223242627
π0.5\pi_{0.5} (18)91.689.793.592.586.984.127.123.40.0———————
π0\pi_0 (18)79.287.587.583.379.275.079.279.20.0———————
UniVLA (32)65.065.070.060.055.020.015.05.00.00.00.00.00.00.00.00.0
OpenVLA (32)100.066.766.750.025.020.84.24.24.20.00.00.00.00.00.00.0
MemoryVLA (32)91.795.891.787.579.212.54.24.24.24.24.28.38.34.20.00.0
VLA-JEPA (28)91.7100.0100.0100.0100.0100.0100.0100.0100.0100.095.895.866.754.241.70.0
X-VLA (24)83.345.825.020.812.58.312.58.34.216.78.38.38.3———

Table R2. Activation substitution with the protocol and scoring of Section 5.4: the activations of frf_r from layer ℓ\ell onward are replaced with those computed under the donor instruction. Cells give the percentage of trials whose closed-loop grasp follows the donor. Parentheses give the number of layers of frf_r. Bold marks where the rate falls below half of its value at ℓ=0\ell = 0.

The pattern of π0.5\pi_{0.5} reproduces in every model. Substitution from ℓ=0\ell = 0 switches the grasp in 65% to 100% of trials, and in every model the switching rate falls at a depth inside frf_r, with faf_a unchanged throughout. The depth differs with the backbone, but the pattern does not. The target is selected inside frf_r, and faf_a carries out the selection. The models differ in training data and starting checkpoints, as Reviewer dfzd notes, and the pattern holds across them.


We respectfully emphasize that SpurDiag-Bench establishes a diagnostic benchmark and analysis framework that reveals spurious associations in VLA models which inspire the community to look beyond whether a VLA succeeds and ask what evidence drives its actions, a crucial step toward trustworthy embodied intelligence. Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed. For any other questions, we are happy to discuss them and make further clarifications.

AuthorsOpenReview ↗

General Response. Part [1/2]

We sincerely thank all reviewers for their careful reading of our manuscript and for their constructive and insightful feedback. We are encouraged by the positive comments, including:

  • Introducing a diagnostic benchmark that goes beyond existing VLA benchmarks, which primarily evaluate task success, generalization, or robustness: our benchmark is specifically designed to diagnose shortcut reliance and its underlying modality-binding mechanisms across the perception-reasoning-action chain of VLA models [Reviewers qzz4 and dfzd].
  • Our main contributions include a chain analysis for diagnosis and an investigation of the mitigation of spurious correlations, which demonstrate an effective analysis and reveal a substantial and practically relevant failure pattern of existing VLA models [Reviewers Ek9R and dfzd].

Meanwhile, we have worked carefully to address the reviewers' concerns and questions, and we provide detailed responses to the individual reviews below. Key additional experiments and planned revisions include:

  • We add action-head-only fine-tuning on six representative VLA models to further investigate where the shortcut mainly arises (#1).
  • We repeat the activation substitution of Section 5.4 on six more released checkpoints (#2), and separate image and text activations on π0.5\pi_{0.5} (Reviewer Ek9R, Q2).
  • We add target-position decoding with one shared probe and layout-disjoint evaluation for seven models, with the full profile by depth in the revised Appendix H (Reviewer Ek9R, Q1).
  • We report success by target slot and under controlled action coverage (Reviewer Ek9R, Q3).

#1 Action-head-only fine-tuning

We add action-head-only fine-tuning, in which the vision encoder and reasoning module are frozen while only the action head is updated on the same demonstrations and training budget as the frozen-head checkpoint. We compare four checkpoints: (1) Base (the released checkpoint), (2) Action-head-only, (3) Frozen-head and (4) Unfrozen-head. Action-head-only fine-tuning does not exceed the base checkpoint on any model, while the frozen-head checkpoint repairs the same models with the same data.

ModelBaseAction-head-onlyFrozen-headUnfrozen-head
UniVLA12.5%2.4%58.4%82.6%
MemoryVLA13.8%4.4%68.6%59.8%
VLA-JEPA16.3%15.7%50.2%70.2%
X-VLA14.2%0.0%24.6%25.9%
π015.5%1.7%89.5%89.6%
π0.521.9%13.3%95.5%96.0%

Table R1. Action-head-only fine-tuning: the vision-language pathway is frozen and only the action head faf_a is updated, on the same demonstrations as the frozen-head checkpoint. Evaluated on all 1,080 episodes. Base, frozen-head and unfrozen-head columns are from the paper. OpenVLA has no separate action head and is omitted.

AuthorsOpenReview ↗

Summary for the Area Chair and Reviewers

Dear Area Chair and Reviewers,

As the discussion closes, we thank the reviewers for their time and briefly summarize what this paper adds.

To our knowledge, SpurDiag-Bench is the first benchmark that not only shows that VLAs fail, but also locates why and where they lose grounding. This matters because a benchmark score alone cannot tell whether a fix removed the shortcut or replaced it:

  • Standard benchmarks hide the failure. Only rearranging the same objects, with no new objects, skills, or harder tasks, drops 12 state-of-the-art VLAs from 85.4%–99.8% to 10.6%–21.9%.
  • The information is there, but not bound. The object name and its position are both decodable, yet the representation does not bind them to each other.
  • The wrong target is chosen before the action head. Activation substitution places the target decision inside the reasoning module, with the same pattern on seven models, and updating only the action head does not exceed the base checkpoint on any of six models. Real SO-100 and UR5 robots show the same failure.
  • The diagnosis separates models a repair fixes from those it does not. With the action head frozen, fine-tuning restores π0.5\pi_{0.5} from 21.9% to 95.5% and π0\pi_0 from 15.5% to 89.5%, while X-VLA and VLA-JEPA recover far less. The pattern suggests, as hypotheses, design features such as dedicated action experts and diverse pretraining, which gives architecture design a testable direction.

Reviewer dfzd noted that the new results "substantially address my main concerns" and found the mechanistic analysis "particularly valuable". All points from Reviewers Ek9R and qzz4 are answered in our responses, with the additions in the revised PDF (in blue), and we would welcome their further comments before the discussion closes.

Best regards,

Authors of ICLR 2027 Conference Paper 25992

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.