- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how behavior cloning changes visual representations in VLAs. Using π0.5 fine-tuned on LIBERO, the authors compare fixed and refitted probes, representation similarity, and control success throughout training. They report a three-stage pattern of early readout incompatibility, subsequent action learning, and later decline in some checkpoint-local probe scores. The paper also investigates representation alignment and several training interventions.
Critical strengths
- An interesting observation about visual forgetting.
The distinction between losing information and losing the ability to read it is important. The fixed-versus-refit probe results are quite convincing in this setting. In particular, the large gap between the original and refitted localization probes suggests that fine-tuning can change the representation without necessarily removing the information.
- Careful analysis.
The paper follows multiple checkpoints, examines several representation layers, and includes controlled interventions. The low-rank alignment results are interesting, and I like that the authors test the gradient-conflict hypothesis rather than assuming it explains the observed degradation.
Critical weaknesses
- The generality of the three-stage pattern remains unclear.
The main findings are based on two full fine-tuning runs of π0.5 on LIBERO. Although the authors provide additional intervention experiments, the proposed three-stage pattern has not been tested across different VLA architectures or adaptation settings.
The paper describes this pattern as a "three-stage law of adaptation," which seems too strong given the current evidence. Testing another VLA or a substantially different fine-tuning setup would help determine whether this is a general phenomenon or specific to the studied configuration.
- The scope and functional significance of the recovered information remain uncertain.
The paper provides strong evidence that certain spatial quantities remain recoverable after the original readouts fail. However, it is less clear whether this extends to broader vision-language capabilities.
The authors do evaluate GQA, object identity, and spatial relations, but the recoverability results focus mainly on spatial information. It would be useful to examine whether broader semantic capabilities remain recoverable, particularly on held-out or OOD visual-language tasks.
The authors correctly distinguish decodability from actual model use. However, the current experiments do not directly establish whether the recovered information can restore native language functionality or plays a causal role in action generation. The low-rank alignment results are convincing for destination decoding, but their functional implications remain unclear.
Questions for authors
-
The paper shows that spatial information remains recoverable even after the native language head fails. However, does this also hold for broader vision-language capabilities? Could the authors evaluate the adapted checkpoints on held-out VQA, visual grounding, or OOD semantic tasks, rather than relying primarily on spatial probes? This would help clarify whether behavior cloning preserves general semantic information or only the specific spatial information examined here.
-
Can the authors provide more direct evidence that the degradation of native VLM performance is mainly due to readout incompatibility rather than a loss of semantic representations? For example, could aligning the adapted representations with the original language head recover GQA or other vision-language performance? The current low-rank alignment results are convincing for destination decoding, but do not necessarily establish the same conclusion for broader semantic capabilities.
-
Would the same representation changes and three-stage training pattern appear in another VLA architecture or fine-tuning setting? This seems important before drawing broader conclusions about behavior cloning in general VLAs.