- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why vision-language-action (VLA) policies fail at compositional generalization by analyzing their internal representations. The authors propose VLA-PatchLens, which applies activation patching and path patching to matched clean/corrupted rollout pairs and measures recovery with a single-step action metric (R_action) and the closed-loop success-rate gain (R_rollout). They build three compositional generalization suites from LIBERO: paraphrase, seen-goal recombination and seen-target recombination. For π0.5, the analysis suggests that task-critical information in text tokens is spread across layers, and that early-layer text information is concentrated into image tokens in later layers and reaches the action expert through cross-attention. It also finds that R_action tracks R_rollout closely enough to serve as a cheap proxy. In failure cases, the authors identify breaks in this information flow and show that single-layer patching raises success rate from 0% to 3.9–23.5% on average, and up to 66.7% at the best layer. The analysis is also extended to StarVLA-OFT and a real Franka robot.
Critical strengths
-
The question is important and the angle is timely. Most prior work on compositional failures in VLAs proposes behavior-level fixes based on hypothesized mechanisms. This paper instead asks where task information lives inside the model and where it breaks down. Adapting activation and path patching to VLAs, with closed-loop readouts, provides causal rather than correlational evidence.
-
The analysis is systematic and produces clear, interpretable findings: a layer-by-token-group map of task information in π0.5, and a text→image→action flow traced by path patching that agrees with the attention analysis. A further useful observation is that activation similarity is uninformative about patching success, while causal patching is informative.
-
The evaluation is reasonably broad. It covers three suites, several simulator time steps, negative controls (random, cross-task and reverse patching), a second architecture and a real robot. The authors are also upfront about some limitations: they report fixed-layer results and note that the best-layer numbers are descriptive.
Critical weaknesses
-
The patching procedure is underspecified in ways that change how the results should be read. The paper never states whether the backbone layers after the patched layer are recomputed. Fig. 3(a) suggests they are not: patching all prefix tokens at layer 0 yields R_action ≈ 0, whereas with recomputation every later layer would be clean and R_action should approach 1. If only a single layer's KV (as read by the action expert) is replaced, the "evenly distributed and complementary across layers" result is largely expected by design. It would then reflect which layers the action expert reads from, not where information resides in the backbone. It is also unclear which input differs between clean and corrupted runs in the failure setting, since both share the instruction and layout (L206–208).
-
The repair results rest on very few rollouts. R_rollout in Fig. 7(a) only takes the values 0, 33 and 67 pp. The reported averages are consistent with 3 rollouts per layer per suite (e.g., CG-goal: 3.9% = 2/3 ÷ 17 layers), not the N = 15 stated in Appendix B. The 66.7% figure is the best of 17 layers, selected on the evaluation data. The 0% baseline holds by construction, because pairs are selected so that the corrupted run fails. Before the gains can be attributed to restored task information, the paper needs a larger N, confidence intervals, and baselines in the failure setting: the unselected success rate, re-running the same initial state with a different seed, and norm-matched random patching.
-
The experiments do not test the failure mode that motivates the paper. The introduction frames compositional failures mainly as vision shortcuts. Yet no experiment varies the visual scene while keeping the instruction fixed (e.g., swapping target and distractor positions), so the analysis cannot localize where visual input overrides language. In addition, the image input is identical across runs in the blank-prompt setting, so task information in image tokens must come from text by construction. The only informative part of this finding is which layers it happens at.
Questions for authors
n/a