- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents a diagnostic study of instruction-following failures in a VLA model under distribution shift. Through in-distribution and out-of-distribution behavioral tests, linear probing, and attention analysis, the authors show that information about the instructed target remains recoverable and attention responds to the instruction, even when the model selects the wrong object. These findings suggest a gap between internal representations and action selection in the tested settings.
Critical strengths
The paper identifies an interesting gap between language-conditioned attention and action selection: in the tested OOD scenes, the model tends to select the same object despite changes in the instruction, even though attention shifts toward the named target. This finding motivates examining how language influences actions beyond attention alone and shows why task success may be insufficient to establish instruction following.
Critical weaknesses
- The behavioral evaluation does not sufficiently isolate the effect of distribution shift. Sections 4.1.1 and 4.1.2 compare settings that differ in scene, object configuration, and instruction content. A matched comparison using the same tasks and instructions while varying the visual environment would provide stronger evidence. The paired control in Section 4.1.3 is useful but covers only the pan–moka-pot pair, limiting the generalizability of its findings.
- The writing is overly conversational, particularly in the descriptions of the experiments, which makes the setup and findings harder to follow.
Questions for authors
- Have you evaluated additional OOD scenes, object pairs, or tasks to assess whether the observed behavior generalizes beyond the reported cases?
- Could you provide spatial attention maps for matched instructions on the same image to show where the reported attention changes occur?