acceptodds
Under review as a conference paper at ICLR 2027

What Survives in a Multi-turn Trace? Dissecting State Maintenance in VLMs

Abstract

Agents built on vision-language models (VLMs) as vision-language-action (VLA) models often act for many turns. To perform well, they need to keep track of how their own actions change the scene even when it is no longer in view. When such an agent fails, current evaluations rarely tell whether it misread the scene, lost the state or chose a bad action. New observations let the model read the scene again, a question about the state can act as a reminder, and a model's account of its own behavior is not reliable evidence. We present Spatial State Dissection (SSD), a testbed that removes these three problems. The initial and goal states of a Tower of Hanoi or Rubik's cube problem are supplied exactly once at the beginning in one of four encodings that use images, symbols or text. Every probe runs on a discarded copy of the conversation and is scored against a simulator, and two interventions on the history test for cause. Across 19 model settings and more than 470K graded answers, SSD separates four steps that fail in different ways. Symbols are read almost without error while images of the cube are often misread. Models with little reasoning retain a record of their own moves but reconstruct the cube state poorly. With thinking enabled, the largest open-weight models retain more of the state. On Hanoi a model can work out the current state from its own earlier moves. When it tries a move that breaks the rules, such as putting a larger ring on a smaller one, the move is not carried out and the environment still replies only "ok.". The models often assume the move was carried out anyway, so they report that ring in the wrong place. In most of these failures, the model could report the exact state just before making the move but still chose an illegal action. States reached forward survive better than states worked out backward, and giving the state back helps only when the supply included images. SSD will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, state maintenance, state tracking, multi-turn evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CKWROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SSD to study state-tracking failures in multi-turn VLMs using Hanoi and Rubik’s cube tasks. The initial and goal states are shown once. State probes are checked against a simulator and run on discarded conversation copies, so they do not affect later actions. The authors aim to distinguish perception, state updating, failed-action recognition, and action selection. Across 19 model settings and over 470K graded answers, models often recall their moves better than the resulting state and sometimes choose illegal moves despite reporting the correct state on a probe. Thinking-enabled settings improve tracking in the largest open-weight models tested.

Critical strengths
  1. The problem setting is reasonable. Tracking state changes and whether actions succeeded is relevant to multi-turn agents, and task success alone gives limited information about these errors.

  2. The probing design has some merit. Discarded conversation copies avoid feeding probe answers into later actions. Hanoi and Rubik’s cube also cover somewhat different state-tracking challenges.

Critical weaknesses
  1. The experiments show different types of errors, but do not clearly establish their causes. The paper acknowledges that models may reconstruct the state when probed. A correct state answer therefore does not show that the model used the correct state when choosing its action. Image reminders add a fresh observation, and masking irrelevant moves also hurts performance, leaving the causal interpretation unclear.
  2. The additional insight from these findings is limited. The protocol reveals gaps between action recall, state reporting, and action selection, but their relationship remains unclear. For example, most first failures on the cube occur before any state update, while state-reporting accuracy declines later. This leaves unclear how much state-maintenance errors explain task failure.

As a secondary concern, Figure 1 is crowded, and the tables use small text and many abbreviations.

Questions for authors
  • Given that a model may reconstruct the state when probed, what conclusions can SSD support about the cause of an action error?

  • Most first failures on the cube occur before any state update. How does this affect the interpretation of later state-tracking errors and their relationship to task failure?

  • Image reminders add a fresh observation, and masking irrelevant moves also hurts performance. Which effects can the controls specifically attribute to state information?

3 comments

Thank you for the careful review and the three great questions! We took some time to run new experiments before replying. We agree that our wording claimed more causal identification than the interventions provide, and the revised PDF will narrow the claims throughout. We address your questions in order, and we will upload a revised PDF with all changes marked in blue.

Q1. What can SSD conclude about the cause of an action error?: SSD supports a conclusion about availability and not about use. The revised PDF will say this in Sec. 3.4 and 4.4 and rename the step “state available but not acted on” (App. B.5). Four results tie probe answers to the actions of the run which never sees the probes:

  • The fork answer goes with the next move: After an exact answer, the next move was legal in 67 against 33 percent of cases for Terra and 82 against 56 for Flash (+34 [+20, +46] and +21 [+14, +29]).
  • The state is lost after a rejected move: Before the first rejected move, the written state is exact in 86 and 96 percent of probes, and after it in 5 and 25 percent.
  • Writing the state first (new runs): The model writes the state and then its move in one reply. A third of illegal moves for Terra and half for Flash follow a state written exactly in that reply (33 and 48 percent).
  • Reporting the rejection (new runs): When the reply says that a move was not carried out, the state after a rejected move is exact in 63 and 65 percent of probes, against 5 and 28. Illegal moves remain at 41 and 14 percent of turns, against 45 and 18 with “ok.”

Therefore, on Hanoi, the first rejected move usually follows a state the model can write, and knowing the state does not make the next move legal. The errors in the state come after the first rejected move.

Q2. Most cube first failures occur before any update: We agree, and the revised PDF will say so. Among the cube first failures, 368 of 385 (Terra) and 300 of 353 (Flash) occur at turn 1. Most follow an exact reading of the initial state (88 and 86 percent), so they are planning failures (App. B.5). After turn 1, most cube first failures follow a state the model could not write (76 percent of 17 for Terra and 45 percent of 53 for Flash). To measure updating apart from planning, we ran new probes on the cube along scripted move histories where the moves are given and not chosen. Flash then writes the exact cube state at 49 percent of checkpoints with symbols and 14 percent with pictures. Therefore, updating often loses the cube state even when no move is chosen.

How much state errors explain task failure: On Hanoi, little. Restoring the state through the reply changes the share of solved runs by only +2 [−8, +12] and +6 [−1, +11] points. On the cube, the first failure is mostly a planning failure at turn 1. The revised PDF will state both.

Q3. Which effects can the controls attribute to state information?:

  • Remind: In the symbolic encoding, the reminder adds no picture. There the true state makes Flash’s next move legal in 97 percent of cases against 79 after the placebo (+18 [+5, +34]), though not optimal more often. Terra goes the other way in this encoding (−19 [−39, −6] on 22 runs), and we report it. With pictures the placebo shows no picture, so part of that gain may come from seeing a new picture. We are now running a placebo that adds a picture of the initial state, which carries no information about the current state. We will post the result in this thread as soon as it is complete.
  • Mask: We agree. For Terra, hiding a move that cannot change the best decision costs about as much as hiding one that can (+3 [−13, +20]). Only Flash on Hanoi shows a cost specific to the hidden move (10 points [4, 16]). The revised PDF will state that Mask shows dependence on the record as a whole.
  • What the protocol makes visible: The revised introduction will list three findings.

Just after a rejected Hanoi move, models write the state as if the move had been carried out in 65 (Terra) and 41 (Flash) percent of cases. They write the true state in 3 and 31 percent. The state is exact until the first rejected move (86 and 96 percent) and rarely after it (5 and 25 percent). Knowing the state does not make the move legal. Reported rejections restore the state, but illegal moves remain at 41 and 14 percent of turns.

We would welcome any further check you think would help within the discussion period. Thank you!

Thank you for the detailed response and the new experiments. I appreciate the candor about the causal claims. After reading the reply, my main concerns remain:

  1. State maintenance does not explain the failures. Restoring the state on Hanoi leaves illegal moves (41/14 vs. 45/18 percent) and the solve rate essentially unchanged, and most cube first failures are planning failures at turn 1.

  2. The causal tests no longer support a causal reading. Mask is conceded as non-specific, the picture placebo for Remind is still running, and the symbolic Remind result switches the outcome from optimal to legal moves, with the two models moving in opposite directions.

  3. The link between probe answers and the state used for acting is still indirect. The fork result is correlational, and writing the state before the move puts it in context, which the protocol was designed to avoid.

  4. Several numbers are inconsistent.

    • Exact states after a rejected move (Flash): 25 and 28 percent within the reply.
    • Exact states before the first rejected move: 86/96 percent in the reply vs. 90/85 in Appendix B.4.
    • Illegal moves: 45/18 percent in the reply vs. 53/31 in Appendix B.2.
    • Illegal moves following an exact state: 33/48 percent in the new runs vs. 85–90 behind the "most failures" claim in the abstract.

These figures need clear definitions. The changes amount to a substantial reframing that is better suited to a new submission, so I maintain my rating.

Thank you for your follow-up! We answer the four points in order:

  1. We agree, and the revised PDF will state this as a main result. SSD locates where a failure starts. On Hanoi, the state is kept until the first rejected move, and that move follows a state the model could write. Therefore, the failure starts in choosing a move and not in keeping the state. The state is lost after a rejected move (written as if the move had been carried out in 65 and 41 percent of cases), but restoring it through the reply barely reduces illegal moves. On the cube, most first failures are planning failures at turn 1, and the scripted histories show that updating also loses the cube state.

  2. We agree that Mask shows dependence on the record as a whole, and the fianl PDF will not call it causal. For Remind, the outcome stays the optimal next move of Table 4a. The legal-move result is secondary, and Terra's opposite sign will be reported beside it. The picture placebo is now done. It adds a picture of the state to be reached which the model saw at the start and which says nothing about the current state. We did not use the initial state as our first reply said since most forward cube runs fail at turn 1, when the initial state is still the current state. Over the encodings with pictures, the true reminder beats this placebo by 20 points [+16, +25] on Flash's optimal next move. It holds on both tasks (Hanoi +15 [+8, +24], cube +24 [+20, +29]). The picture placebo itself adds only 1 point [−2, +5] over the plain placebo, so a new picture without the current state does not explain the gain. With three realizations of each decision, the gain over the plain placebo is 22 points [+17, +26], against 20 with one. In symbols Flash then also gains 6 points [+1, +11] on the optimal next move.

  3. We agree that the fork result alone is a correlation. Three new results go beyond it, and two of them change the main run itself.

  • Remind changes the state that the model is given and then measures the move. The true reminder beats both placebos by 20 to 22 points on Flash's optimal next move, and the picture placebo rules out a new picture as the cause (point 2). So, the state content itself changes the choice.
  • Reporting rejections in the main run restores the state there (exact in 63 and 65 percent of probes after a rejected move, against 5 and 28). Illegal moves still stay at 41 and 14 percent of turns, against 45 and 18. Restoring the state thus barely changes legality, so the lost state is not what makes these moves illegal.
  • Writing the state first puts the state in context on purpose. This removes the reading that a fork rebuilds a state the main run never had since the main run writes the state right before its move. Even then a third (Terra) and half (Flash) of illegal moves follow an exact state written in the same reply.

Together these show that the state does enter the choice, and that on Hanoi the illegal moves come from choosing a move and not from a lost state. The updated PDF will report state-first as such a check and not as part of SSD.

  1. Each pair counts different events or averages differently. The revised PDF will define each of these figures where it appears.
  • 25 and 28 (Flash, exact after a rejected move). 25 pools every probe after the first rejected move in all Hanoi runs. 28 averages over problems in the runs that match the new runs (Rule, symbols and pictures).
  • 86/96 and 90/85 (exact before). 86 and 96 cover every probe before the first rejected move. 90 and 85 cover only the probe just before a first failure that is a rejected move.
  • 45/18 and 53/31 (illegal moves). 53 and 31 pool all moves up to the optimal length in all Hanoi runs, so long runs on large problems weigh most. 45 and 18 average each run's share over problems under Rule with symbols and pictures.
  • 33/48 and 85 to 90 (illegal after an exact state). 85 to 90 percent concerns the first failure. 33 and 48 percent concern every illegal move in the runs that write the state first. After the first rejected move the state is usually lost, so later illegal moves follow an exact state less often.

Our scope in the revised PDF: The revision will narrow the claims and define the measures. It will add no new claim, and the main numbers come from the submitted runs.

Reviewer_vYr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Spatial State Dissection (SSD) evaluation framework to analyze the state maintenance ability of vision-language models in multi-turn interactions. Based on the Tower of Hanoi and Rubik's Cube tasks, the model's capabilities are divided into four stages: Reading, Updating, Noticing, and Choosing. Through experimental analysis, the failure modes in different stages are identified. The experiments cover 19 model settings, and it is found that the model can recall historical actions well, but has difficulty accurately maintaining the environment state, and correct state reports may not necessarily lead to correct decisions.

Critical strengths
  1. Targeted diagnostic evaluation design. SSD decomposes state maintenance into four steps and employs different probes for evaluation, which is more conducive to distinguishing errors in move recall, state update, and action selection.

  2. Fork-based probing and intervention experiments are innovative. By probing the current state in independent forked conversation histories, it avoids affecting subsequent interactions, and combines Remind and Mask to analyze the impact of state information and historical actions on decision-making.

Critical weaknesses
  1. The diversity and authenticity of the evaluation tasks are insufficient. SSD only includes two synthetic tasks, the Tower of Hanoi and the Rubik's Cube, lacking verification of real visual scenes and other types of interactive tasks, which limits the generalization of the conclusions.

  2. State probes on forked conversations may trigger additional reasoning, allowing models to reconstruct the current state on demand. Therefore, correct state reports do not necessarily imply that the models maintain or use the correct state during action selection

  3. The Thinking experiment lacks strict variable control. Thinking On and Off simultaneously adopt different decoding strategies. Although additional analyses are provided, these factors complicate the attribution of performance improvements specifically to Thinking.

  4. The design of environmental feedback may affect the model's behavior. Responding with "ok." to both legal and illegal actions, although it avoids directly exposing the state, may cause the model to interpret failed operations as having been confirmed or executed, thereby affecting the conclusions related to Noticing.

Questions for authors

see above

3 comments

Thank you for the detailed review and for recognizing the diagnostic design and the probing on forked conversations! We took some time to run new experiments before replying. We address the four weaknesses in order, and we will upload a revised PDF with all changes marked in blue.

W1. Two synthetic tasks: We agree that this limits generality. Every probe needs an exact key at every turn including for states that the model reaches through its own rejected moves. We therefore chose tasks where a simulator provides that key and a renderer can draw any state (App. A.10). The two tasks separate the steps. On Hanoi, 87 percent of first failures follow an exact state. After turn 1, most cube first failures follow an inexact state (76 percent for Terra and 45 percent for Flash). The revised PDF will narrow the claims to puzzles with exact keys. The released code keeps the task simulator apart from the protocol, so any task with a simulator can be added.

W2. Reconstruction on probes: We agree. A correct probe answer shows that the state was available, not that the model used it when it moved. The revised PDF will say this in Sec. 3.4 and 4.4 and rename the step “state available but not acted on” (App. B.5). Three results tie the probes to the run:

  • The fork answer goes with the next move. After an exact answer on a Hanoi fork, the next move of the main run was legal in 67 against 33 percent of cases for Terra (+34 [+20, +46]) and 82 against 56 for Flash (+21 [+14, +29]). The gaps hold at the same turn of the same problem.
  • Giving the state back changes the move. A reminder in symbols makes Flash’s next move legal in 97 against 79 percent of cases after the placebo (+18 [+5, +34]).
  • Writing the state first (new runs). The model writes the state and then its move in one reply. A third of illegal moves for Terra and half for Flash follow a state written exactly in that reply (33 [23, 45] percent on a subset of 79 runs and 48 [38, 59] percent). Writing the state first leaves the rate of illegal moves unchanged (paired −5 [−15, +4] and −1 [−3, +1]).

W3. Thinking and decoding: We agree that the original comparison changed both. The settings with thinking sample at temperature 0.6, and the settings without thinking use temperature 0 (App. A.9). The revised PDF will state this in the main text and describe the result as a comparison of the two settings, not of thinking alone. We are now running thinking off with the same sampling as thinking on (temperature 0.6, top-p 0.95, same seeds) on the same 66 problems for Qwen3.8 27B and Gemma 4 31B. We will post the result in this thread as soon as it is complete.

W4. The reply “ok.” (new runs): The first message states that “ok.” carries no information and no legality feedback (Fig. 6). We tested the reply directly on Hanoi under Rule with the symbolic and image encodings:

  • A reply that confirms nothing: We replaced “ok.” with “Move received. Legality is not reported.” Just after the first rejected move, the written state matches the state with the move carried out in 41 percent of cases for Terra and 50 for Flash. With “ok.” on the same problems it is 45 and 37 percent. So this belief does not come from “ok.” For Flash, the neutral reply even lowers the true state from 34 to 20 percent.
  • A reply that reports the rejection: When the reply says that the move was not carried out, this share falls to 9 and 0 percent. The true state rises to 59 and 83 percent.
  • Legality: Illegal moves fall by only 5 points for both models (Terra −5 [−16, +4], Flash −5 [−8, −1]), so the models can know the state and still choose an illegal move. This supports the distinction between noticing and choosing.

If these results address your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_vYr9OpenReview ↗

response to Authors

Thanks for your reply, it partially solve my questions. However, the lack of the tasks is still an important problem. This could limit the scope of the conclusion. And in thinking experiments, the same decoding strategy has not provided experiments and results. Thich is also a problem for this paper. I tend to keep the initial score 2.

Thank you for reading our response and for sharing us what remains!

Thinking with the same decoding strategy: You are right that these runs on the open weight models are still missing. Our GPUs were not free in time to run them within the discussion period, and the final version will include them. The existing runs already hold one comparison with the same decoding, in which Claude Sonnet 5 ran with thinking off and with thinking on at the provider’s default sampling (App. A.9). With thinking on, Sonnet tracks the cube state better. It places 33 points more of the moved cube pieces correctly with symbols [+11, +58] and 26 points more with pictures [+2, +57]. On Hanoi, there is little room to gain since Sonnet already places 88 to 100 percent of the moved rings correctly without thinking. Gemini Flash shows the same cube gain when its thinking goes from low to high at default sampling (+33 points [+15, +52] with symbols). These pilot runs are small, with 6 to 7 cube problems in each comparison, so we read them for direction only (App. B.8). The final PDF will report them beside the open weight result and will call that result a comparison of two settings.

Scope: We agree that two tasks limit the scope of the conclusions. The revised PDF will state the claims for puzzles with an exact simulator key in the abstract and introduction. Its limitations section will name real scenes and other interactive tasks as open.

Thank you again for the careful review.

Reviewer_GEH1OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at whether virtual logic model (VLM) agents can keep track of spatial state that isn't being observed across their multi-turn actions. The paper introduces a spatial state profiling (SSD) method, which gives just one initial state, probes discarded copies of the dialogue, and uses interventions to tell apart reading, updating, noticing rejected actions, and choosing actions.

Critical strengths

The benchmark design isolates a capability that would otherwise get confounded. The probe dialogues are triggered by the simulator and get thrown away before the main conversation keeps going, so the answers never get written back into context. That's a big methodological contribution, because it lets the paper's distinctions between state reporting, action selection, and history dependence actually be tested behaviorally instead of inferred from self-explanations.

Critical weaknesses

The reported uncertainty doesn't account for the randomness in model outputs. Section 3.6 reports bootstrap intervals over questions, but the full-grid proprietary results seem to use just one realized trajectory per model, question, coding, and direction, with the provider's default sampling settings. That matters a lot for the causal intervention conclusions. Bootstrapping over questions properly captures question heterogeneity, but it doesn't show that effects like the +14 to +28 point-image Remind gains would hold up across repeated calls to the same hosted model. These claims are reasonable and well-qualified in some ways, but the argument would be more convincing if there were repeated-trajectory analysis on at least a representative subset of questions and interventions—or at least a clearer statement that the intervals are based on a single realization per model setup.

Questions for authors

See weakness

2 comments

Thank you for the positive assessment and for recognizing that discarding the probe dialogues allows these distinctions to be tested through behavior rather than inferred from explanations! We took some time to run new experiments before replying.

One realization per setup: You are right. Each problem (one puzzle instance) contributes one sampled run per model, encoding, and direction at the provider’s default settings (App. A.9). The confidence intervals resample problems (Sec. 3.6), so they reflect both the variation from sampling and the differences between problems without separating the two. The revised PDF will state this design at the start of Sec. 3.6 and describe the Remind gains as averages over problems.

How much a single decision varies?: The replay control in Table 4b sends an identical request a second time. The decision matches in 73 [57, 88] percent of replays for Terra on Hanoi and in 84 [77, 89] percent for Flash on Hanoi. For Flash on the cube, agreement drops to only 39 [29, 49] percent. Individual decisions do vary, and accordingly no claim rests on any single decision.

Two independent sets of runs: Forward and backward runs are separate runs on separate problems. Gemini Flash’s gain on the optimal next move with pictures is positive in both directions across all three picture encodings. The forward and backward gains are +18 and +35 with the image alone, +10 and +26 with image and symbols, and +9 and +18 with image and text. The interval lies above zero in four of the six cases. The gain holds most consistently on the cube.

Repeated decisions, in progress: We are now resending the first decision after the reminder and after the placebo until each lane has three realizations, for Flash in all encodings. These correspond to the +14 to +28 point gains you mention. We will post the results in this thread as soon as they are complete.

In reply to the other reviewers, we also report new runs on the reply “ok.”, on writing the state before each move, and on cube probes along scripted move histories. We will upload a revised PDF with all changes marked in blue.

As promised, we sent the first decision after the reminder and after the placebo again until each lane had three realizations. This covers all 565 Flash runs that host both lanes, and none of the new calls failed.

With pictures, the true reminder beats the placebo on the optimal next move by 22 points [+17, +26] with three realizations, against +20 [+13, +27] with the original one. By encoding the gains are +25 (image), +24 (image and symbols) and +16 (image and text). The Table 4a gave +28, +19 and +14. All three intervals exclude zero. Single decisions are noisy, and 61 to 70 percent of their variance comes from sending the same request again. The average gain over problems still stays at 20 to 25 points in each of the three realizations. In the symbolic encoding, the gain is +6 [+1, +11] with three realizations against −1 with one, so the symbolic null result in the Table 4a partly reflects a single draw.

As I replied to reviewer CKWR, we also ran a placebo that adds a picture without the current state. It adds only 1 point [−2, +5] over the plain placebo, and the true reminder still beats it by 20 points [+16, +25]. The revised PDF will report all of this in Sec. 3.6, Sec. 4.4 and App. B.

Thank you again for raising this point!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.