- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper evaluates small open LLM agents on textual Sokoban by reading each turn of their play rather than only the outcome. The models play through the GamingAgent harness and fail every episode, and a ladder of easier boards shows that they solve some of the easiest boards and none of the harder ones. Each turn is coded with a taxonomy built from blind reading; the main behaviours reported are reasoning about a different board than the one shown and thoughts that do not match the action taken. A pre-registered control finds no difference between solved and unsolved episodes, so the authors present the taxonomy as a description rather than a validated discriminator.
Critical strengths
The authors show that the GamingAgent prompt for Sokoban is a shortened version without the rules, never says that a push requires standing next to the box, and has a single worked example that pairs a plain-move thought with a push action (Section 6). They also rerun one temperature-0 configuration three times, and fewer than half of the model-level cells follow the same path (Section 5.5). Both results change how a benchmark number from this setup should be read.
Critical weaknesses
(1) The evidence does not sufficiently show that turn-level evaluation is needed, and the experiments stay where it could not. The pre-registered positive control found no difference between solved and unsolved episodes (its secondary item even moved the opposite way), and the authors concede that the codebook does not separate competent from incompetent turns (Section 7). I could not find a conclusion in the paper that outcome metrics get wrong and the turn labels get right. Part of the reason is the setting: one game, one harness, six levels and five small models, and every episode in the main corpus fails, most within a few steps. With no partial successes there is nothing to compare the labels against, so reader cannot tell whether the taxonomy describes why agents fail or just how these models play. The board-table finding also comes from a single model (Section 5.4). The ladder, the only result with any successes, is confounded with walking distance. On the first three rungs the worker already stands next to the box, while from the fourth rung on it needs 8 or 9 moves to get into position (supplementary chance_baseline.json), and the prompt never says a push requires adjacency. The drop after three pushes may therefore measure whether the agent can walk around a box, not push depth. Rewording cannot fix this. It needs a setting where some agents partly succeed, such as a stronger open model or easier levels, so labels can be compared between solved and unsolved episodes, and a ladder that keeps the walking distance fixed while the number of pushes grows.
(2) Once we set aside the borrowed parts, the contribution is small, and the main behavioural finding may come from prompts. The environment, levels and prompt come from GamingAgent, reporting kappa with PABAK is standard practice, and run-to-run variation at temperature 0 is already documented (the paper cites He and Thinking Machines Lab, 2025). What is left is a descriptive taxonomy whose discriminating power the control did not show. The thought-action mismatch, one of the two findings the claim rests on, is exactly what the prompt's only worked example demonstrates, and that example is sent on every step of every condition; the authors say nothing in the study bounds its effect (Section 7). The related-work positioning also overstates the gap. Section 2 says failure analysis has not been done on games, yet Appendix G lists TopoBench, an error taxonomy over reasoning traces on puzzles, and Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective (arXiv 2510.15047), which analyses small-model failures in Sokoban, is not cited. To address this, the authors would need a run in which the worked example's thought and action agree, and a direct statement of what their taxonomy adds over these closest works.
(3) On presentation, the research question is hard to find. The Abstract and Introduction are written too densely, with terms and qualifiers piled on top of each other, and the question itself appears only once, in a single clause. The contributions list uses terms that have not been defined yet and mixes in disclosures such as "Blindness to the study's direction was tested, and failed the first test", and Related Work ends with a positioning line made of four items joined by multiplication signs.
Questions for authors
- Is there any conclusion in the paper that solve rate, no-op rate or solver-detected deadlock would get wrong, and that the turn-level labels correct? The argument about ineffective actions in Section 6 is about metrics and does not need the taxonomy.
- In the ladder episodes from the fourth rung on, how often does the worker reach a position from which it can legally push the box? If almost never, the drop after three pushes is about repositioning rather than push depth. Could you add a ladder with fixed walking distance, and a setting where agents partly succeed so that labels can be compared between solved and unsolved episodes?
- What does your taxonomy find that the TopoBench error taxonomy and the World-Modeling Perspective paper (arXiv 2510.15047) do not?
References
[1] Mayug Maniparambil et al. TopoBench: Benchmarking LLMs on Hard Topological Reasoning. In ICLR 2026 Workshop on Logical Reasoning of Large Language Models, 2026. [2] Shiqi Chen et al. Why Do LLM Agents Fail to Explore New Environments? A World-Modeling Perspective. In NeurIPS 2026 Workshop on From Pretrained Representations to Acting Agents (PTA), 2026.