What Survives in a Multi-turn Trace? Dissecting State Maintenance in VLMs

Agents built on vision-language models (VLMs) as vision-language-action (VLA) models often act for many turns. To perform well, they need to keep track of how their own actions change the scene even when it is no longer in view. When such an agent fails, current evaluations rarely tell whether it misread the scene, lost the state or chose a bad action. New observations let the model read the scene again, a question about the state can act as a reminder, and a model's account of its own behavior is not reliable evidence. We present Spatial State Dissection (SSD), a testbed that removes these three problems. The initial and goal states of a Tower of Hanoi or Rubik's cube problem are supplied exactly once at the beginning in one of four encodings that use images, symbols or text. Every probe runs on a discarded copy of the conversation and is scored against a simulator, and two interventions on the history test for cause. Across 19 model settings and more than 470K graded answers, SSD separates four steps that fail in different ways. Symbols are read almost without error while images of the cube are often misread. Models with little reasoning retain a record of their own moves but reconstruct the cube state poorly. With thinking enabled, the largest open-weight models retain more of the state. On Hanoi a model can work out the current state from its own earlier moves. When it tries a move that breaks the rules, such as putting a larger ring on a smaller one, the move is not carried out and the environment still replies only "ok.". The models often assume the move was carried out anyway, so they report that ring in the wrong place. In most of these failures, the model could report the exact state just before making the move but still chose an illegal action. States reached forward survive better than states worked out backward, and giving the state back helps only when the supply included images. SSD will be released upon acceptance.