- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SSD to study state-tracking failures in multi-turn VLMs using Hanoi and Rubik’s cube tasks. The initial and goal states are shown once. State probes are checked against a simulator and run on discarded conversation copies, so they do not affect later actions. The authors aim to distinguish perception, state updating, failed-action recognition, and action selection. Across 19 model settings and over 470K graded answers, models often recall their moves better than the resulting state and sometimes choose illegal moves despite reporting the correct state on a probe. Thinking-enabled settings improve tracking in the largest open-weight models tested.
Critical strengths
-
The problem setting is reasonable. Tracking state changes and whether actions succeeded is relevant to multi-turn agents, and task success alone gives limited information about these errors.
-
The probing design has some merit. Discarded conversation copies avoid feeding probe answers into later actions. Hanoi and Rubik’s cube also cover somewhat different state-tracking challenges.
Critical weaknesses
- The experiments show different types of errors, but do not clearly establish their causes. The paper acknowledges that models may reconstruct the state when probed. A correct state answer therefore does not show that the model used the correct state when choosing its action. Image reminders add a fresh observation, and masking irrelevant moves also hurts performance, leaving the causal interpretation unclear.
- The additional insight from these findings is limited. The protocol reveals gaps between action recall, state reporting, and action selection, but their relationship remains unclear. For example, most first failures on the cube occur before any state update, while state-reporting accuracy declines later. This leaves unclear how much state-maintenance errors explain task failure.
As a secondary concern, Figure 1 is crowded, and the tables use small text and many abbreviations.
Questions for authors
-
Given that a model may reconstruct the state when probed, what conclusions can SSD support about the cause of an action error?
-
Most first failures on the cube occur before any state update. How does this affect the interpretation of later state-tracking errors and their relationship to task failure?
-
Image reminders add a fresh observation, and masking irrelevant moves also hurts performance. Which effects can the controls specifically attribute to state information?
3 comments
Thank you for the detailed response and the new experiments. I appreciate the candor about the causal claims. After reading the reply, my main concerns remain:
-
State maintenance does not explain the failures. Restoring the state on Hanoi leaves illegal moves (41/14 vs. 45/18 percent) and the solve rate essentially unchanged, and most cube first failures are planning failures at turn 1.
-
The causal tests no longer support a causal reading. Mask is conceded as non-specific, the picture placebo for Remind is still running, and the symbolic Remind result switches the outcome from optimal to legal moves, with the two models moving in opposite directions.
-
The link between probe answers and the state used for acting is still indirect. The fork result is correlational, and writing the state before the move puts it in context, which the protocol was designed to avoid.
-
Several numbers are inconsistent.
- Exact states after a rejected move (Flash): 25 and 28 percent within the reply.
- Exact states before the first rejected move: 86/96 percent in the reply vs. 90/85 in Appendix B.4.
- Illegal moves: 45/18 percent in the reply vs. 53/31 in Appendix B.2.
- Illegal moves following an exact state: 33/48 percent in the new runs vs. 85–90 behind the "most failures" claim in the abstract.
These figures need clear definitions. The changes amount to a substantial reframing that is better suited to a new submission, so I maintain my rating.
Thank you for your follow-up! We answer the four points in order:
-
We agree, and the revised PDF will state this as a main result. SSD locates where a failure starts. On Hanoi, the state is kept until the first rejected move, and that move follows a state the model could write. Therefore, the failure starts in choosing a move and not in keeping the state. The state is lost after a rejected move (written as if the move had been carried out in 65 and 41 percent of cases), but restoring it through the reply barely reduces illegal moves. On the cube, most first failures are planning failures at turn 1, and the scripted histories show that updating also loses the cube state.
-
We agree that Mask shows dependence on the record as a whole, and the fianl PDF will not call it causal. For Remind, the outcome stays the optimal next move of Table 4a. The legal-move result is secondary, and Terra's opposite sign will be reported beside it. The picture placebo is now done. It adds a picture of the state to be reached which the model saw at the start and which says nothing about the current state. We did not use the initial state as our first reply said since most forward cube runs fail at turn 1, when the initial state is still the current state. Over the encodings with pictures, the true reminder beats this placebo by 20 points [+16, +25] on Flash's optimal next move. It holds on both tasks (Hanoi +15 [+8, +24], cube +24 [+20, +29]). The picture placebo itself adds only 1 point [−2, +5] over the plain placebo, so a new picture without the current state does not explain the gain. With three realizations of each decision, the gain over the plain placebo is 22 points [+17, +26], against 20 with one. In symbols Flash then also gains 6 points [+1, +11] on the optimal next move.
-
We agree that the fork result alone is a correlation. Three new results go beyond it, and two of them change the main run itself.
- Remind changes the state that the model is given and then measures the move. The true reminder beats both placebos by 20 to 22 points on Flash's optimal next move, and the picture placebo rules out a new picture as the cause (point 2). So, the state content itself changes the choice.
- Reporting rejections in the main run restores the state there (exact in 63 and 65 percent of probes after a rejected move, against 5 and 28). Illegal moves still stay at 41 and 14 percent of turns, against 45 and 18. Restoring the state thus barely changes legality, so the lost state is not what makes these moves illegal.
- Writing the state first puts the state in context on purpose. This removes the reading that a fork rebuilds a state the main run never had since the main run writes the state right before its move. Even then a third (Terra) and half (Flash) of illegal moves follow an exact state written in the same reply.
Together these show that the state does enter the choice, and that on Hanoi the illegal moves come from choosing a move and not from a lost state. The updated PDF will report state-first as such a check and not as part of SSD.
- Each pair counts different events or averages differently. The revised PDF will define each of these figures where it appears.
- 25 and 28 (Flash, exact after a rejected move). 25 pools every probe after the first rejected move in all Hanoi runs. 28 averages over problems in the runs that match the new runs (Rule, symbols and pictures).
- 86/96 and 90/85 (exact before). 86 and 96 cover every probe before the first rejected move. 90 and 85 cover only the probe just before a first failure that is a rejected move.
- 45/18 and 53/31 (illegal moves). 53 and 31 pool all moves up to the optimal length in all Hanoi runs, so long runs on large problems weigh most. 45 and 18 average each run's share over problems under Rule with symbols and pictures.
- 33/48 and 85 to 90 (illegal after an exact state). 85 to 90 percent concerns the first failure. 33 and 48 percent concern every illegal move in the runs that write the state first. After the first rejected move the state is usually lost, so later illegal moves follow an exact state less often.
Our scope in the revised PDF: The revision will narrow the claims and define the measures. It will add no new claim, and the main numbers come from the submitted runs.
Thank you for the careful review and the three great questions! We took some time to run new experiments before replying. We agree that our wording claimed more causal identification than the interventions provide, and the revised PDF will narrow the claims throughout. We address your questions in order, and we will upload a revised PDF with all changes marked in blue.
Q1. What can SSD conclude about the cause of an action error?: SSD supports a conclusion about availability and not about use. The revised PDF will say this in Sec. 3.4 and 4.4 and rename the step “state available but not acted on” (App. B.5). Four results tie probe answers to the actions of the run which never sees the probes:
Therefore, on Hanoi, the first rejected move usually follows a state the model can write, and knowing the state does not make the next move legal. The errors in the state come after the first rejected move.
Q2. Most cube first failures occur before any update: We agree, and the revised PDF will say so. Among the cube first failures, 368 of 385 (Terra) and 300 of 353 (Flash) occur at turn 1. Most follow an exact reading of the initial state (88 and 86 percent), so they are planning failures (App. B.5). After turn 1, most cube first failures follow a state the model could not write (76 percent of 17 for Terra and 45 percent of 53 for Flash). To measure updating apart from planning, we ran new probes on the cube along scripted move histories where the moves are given and not chosen. Flash then writes the exact cube state at 49 percent of checkpoints with symbols and 14 percent with pictures. Therefore, updating often loses the cube state even when no move is chosen.
How much state errors explain task failure: On Hanoi, little. Restoring the state through the reply changes the share of solved runs by only +2 [−8, +12] and +6 [−1, +11] points. On the cube, the first failure is mostly a planning failure at turn 1. The revised PDF will state both.
Q3. Which effects can the controls attribute to state information?:
Just after a rejected Hanoi move, models write the state as if the move had been carried out in 65 (Terra) and 41 (Flash) percent of cases. They write the true state in 3 and 31 percent. The state is exact until the first rejected move (86 and 96 percent) and rarely after it (5 and 25 percent). Knowing the state does not make the move legal. Reported rejections restore the state, but illegal moves remain at 41 and 14 percent of turns.
We would welcome any further check you think would help within the discussion period. Thank you!