- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether language reasoning governs goal selection within VLA policies. The study identifies low-dimensional goal code in the OpenVLA and π0.5 models and tests their behavioral impact through read, write, and delete operations. Results from experiments in the LIBERO benchmark indicate that merely restoring text-based reasoning is insufficient to prompt the model to re-execute rule-based actions; however, explicitly injecting the decoded goal significantly improves task success rates when paired with clear instructions.
Critical strengths
- Causal evidence beyond mere detection. Redirecting the target code resulted in the redirection of 18/20 OpenVLA episodes and 10/10 π0.5 episodes, whereas corresponding control experiments using random subspaces yielded no such redirections. Supplementary ablation experiments further confirm that these representations influence target selection.
- Decoupling of informational capabilities. While the text-recovery strategy enables the model to correctly answer rules, it rarely leads to direct execution; injecting the decoded answers significantly boosts success rates. Experiments with controlled instruction visibility further elucidate how the training process establishes a dependency between behavior and the generation of target text.
Critical weaknesses
- These conclusions extend beyond the scope of the experiments. The core reasoning experiments cover two VLA model families, focusing primarily on rule-based constraints and negation logic within the LIBERO kitchen environment (across three target scenarios). The comparative analysis of ten models includes base models and their respective checkpoints; meanwhile, the broader transfer experiments primarily assess code-manipulation capabilities rather than reasoning abilities. These results only support in-depth investigation into specific mechanisms; they are insufficient to warrant the sweeping conclusion that "VLAs lack reasoning capabilities." The paper’s title and conclusions should be qualified accordingly.
- Causal correlation does not establish the validity of cross-format diagnostics. Since goal prototypes are derived by fitting them to literal instructions, the possibility remains that goal-related information is encoded in other way, even if rule-decoding performance is no better than random chance. Furthermore, while injecting the correct code under rule-based instructions can achieve 93%–95% accuracy in target outputs, OpenVLA failed in all 30 attempts. Consequently, the success observed after switching to literal instructions does not definitively reveal the mechanism behind the initial failure. The analysis should distinguish between failures in reasoning, mismatches in representation, and failures in subsequent execution.
Questions for authors
-
In a decoder trained on rule-based formats, can the model recover the expected target output for previously unseen bindings when "literal-prototype decoding" fails?
-
Even when rule-based instructions are followed and the correct code has been injected, why do execution failures persist? Furthermore, which stages of execution failure can the diagnostic method proposed in this paper reliably distinguish?
We thank Reviewer
qUiefor the positive evaluation (3: Weak acceptance,Presentation: 3,Informativeness: 3) and helpful suggestions. All points are addressed in the revised PDF:1. Qualifying the title and claims to match experimental scope (W1 — Title, Abstract, §1, §9): We agree completely. Our results do not claim that VLAs lack reasoning capability—indeed, text-restored OpenVLA answers unseen rules accurately (
0.90–1.00) and goal-code supervision executes unseen marker-first rules directly (23–28/30, new §8). Rather, standard VLA training inherits lexical word meaning into the goal code without automatically routing in-context reasoning there. We kept the qualified title (VLA Models Inherit the Meaning of Words, Not the Use of Their Reasoning) and explicitly scoped Abstract, §1, and §9 to the tested OpenVLA and π0.5 lineages in LIBERO simulation.2. Rule-format decoder on unseen bindings (W2a, Q1 — §4, App. F, new Table 21): Question: Can a decoder fitted directly on rule formats recover unseen bindings when literal-prototype decoding fails?
0.33).0.39–0.44in OpenVLA,silent, Prismatic, and π0.5). Yet the identical builder recovers0.86–1.00(31–36/36, layer 21) on plain-text rule questions (S1) and0.92(11/12, layer 6) on asked negations. Thus chance decoding on robot rules reflects genuine absence of derived target content, not prototype-format mismatch.3. Raw-rule execution failure, carrier-free training, & 3-stage failure taxonomy (W2b, Q2 — §7, §8, §9): Why unmodified OpenVLA stalls (
0/30) on raw rules even with oracle codes:0->14/20) and OpenVLA negations (16->22/30, App. H).8, 11/30). Adding goal-code supervision Lcode at layers 8–16 raises unseen-marker readout to0.99–1.00and reaches23–28/30closed-loop success directly on raw rule prompts with NO literal carrier (17–27/30when self-distilled from the policy's own text answer).Which failure stages our diagnostic reliably separates (§7 & §9): Combining (i) literal goal-code readout, (ii) format-matched full-depth probing (Table 21), and (iii) closed-loop code writing cleanly separates:
rule-firstorder before deep retrieval).S1text rules at L21 or asked negations at L6), yet it misses Gℓ at action commitment; writing it into Gℓ rescues execution (24–26/30, Table 2).0.87–1.00on descriptions;0.93–0.95on code-injected raw OpenVLA rules), yet rollout execution fails downstream.