How Should Vision-Language-Action Models Use Proprioceptive
Abstract
Recent Vision–Language–Action (VLA) models almost universally take robot proprioceptive state as input, yet wire it in incompatible ways—serialized into text prompts, projected into the vision–language prefix, or fed directly to the action expert—and almost always as a single current frame. Three questions remain open: (1) whether, and on which tasks, current state actually improves closed-loop control; (2) how much state history helps, and whether its benefit reflects genuine temporal variation rather than added conditioning capacity; and (3) where state should enter the model—the vision–language backbone or the action-generation module. We answer these questions through controlled experiments on a flow-matching VLA, fixing the backbone, training data, action representation, and evaluation protocol throughout. We implement five representative interfaces—discrete state prompt, VLM prefix, action prefix, state expert, and feature modulation—under matched implementation details, and evaluate them on 45 atomic tasks spanning three task families plus 20 composite tasks; we then sweep the state-history length from 1 to 96 frames to examine how historical state information affects model performance. The experiments yield systematic answers to all three questions, distilled into testable design principles for state-aware VLAs.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.