acceptodds
Under review as a conference paper at ICLR 2027

Reading the Board, Playing Another One: Turn-Level Evaluation of Open LLM Agents on Textual Sokoban

Abstract

Aggregate outcomes and no-op rates obscure where an interactive agent fails. We report a turn-level evaluation of five open 3–8B LLM agents on textual Sokoban: 180 episodes and 1,561 steps across six conditions, where no level is solved and no agent docks a box. Two further experiments bound instrument failure and level difficulty. Replaying push-optimal solutions through the harness, repaired to log the solving step, solves all six corpus levels. On a fixed ladder of one to six optimal pushes, five models from three families solve 0 of the 60 episodes at or above four pushes; on the three lower rungs, success is slightly below a pre-registered uniform-random baseline at one push and above it at two and three. With one board per rung, this bounds the fixed ladder rather than establishing a general complexity threshold. We code every corpus turn against a ten-family behavioural taxonomy induced from three blind reading passes, with a contrast set recording locally correct elements of a turn. In 14 episodes of one model across the three reading passes and both temperatures, an accurate board table appears in the model's own output beside prose that reasons from a different board; the clearest instance recurs in all three executions. The move a thought describes and the action it emits also dissociate: a plain move described, a push emitted, or the reverse. On ladder rungs 1–3, a pre-registered blind control of 291 shuffled single turns finds no separation between solved and unsolved episodes on the pre-specified whole-turn contrast item. We therefore present the taxonomy as a description of observed behaviour in this setting, not as a validated discriminator of competent and incompetent turns. A further observation is carried as a metric argument, not a frequency claim: the most irreversibly lost levels are lost to legal, successful pushes, so a count of ineffective actions ranks the most damaging episodes as the best. Both annotators are language models, so we report kappa beside PABAK and prevalence and separately assess sampled label defensibility with 316 human verdicts across three instruments, never pooled. A third language-model annotator over the whole corpus has a substantially lower median in-band kappa with either of the first two (0.319 and 0.381) than they have with each other (0.576), each median over its own item band. Re-running an identical temperature-0 configuration three times, only 14 of 30 cells take the same path: fixed-seed greedy decoding did not reproduce trajectories in this serving configuration.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, turn-level analysis, failure taxonomy, textual Sokoban, annotation reliability, evaluation validity, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_vvpPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates small open LLM agents on textual Sokoban by reading each turn of their play rather than only the outcome. The models play through the GamingAgent harness and fail every episode, and a ladder of easier boards shows that they solve some of the easiest boards and none of the harder ones. Each turn is coded with a taxonomy built from blind reading; the main behaviours reported are reasoning about a different board than the one shown and thoughts that do not match the action taken. A pre-registered control finds no difference between solved and unsolved episodes, so the authors present the taxonomy as a description rather than a validated discriminator.

Critical strengths

The authors show that the GamingAgent prompt for Sokoban is a shortened version without the rules, never says that a push requires standing next to the box, and has a single worked example that pairs a plain-move thought with a push action (Section 6). They also rerun one temperature-0 configuration three times, and fewer than half of the model-level cells follow the same path (Section 5.5). Both results change how a benchmark number from this setup should be read.

Critical weaknesses

(1) The evidence does not sufficiently show that turn-level evaluation is needed, and the experiments stay where it could not. The pre-registered positive control found no difference between solved and unsolved episodes (its secondary item even moved the opposite way), and the authors concede that the codebook does not separate competent from incompetent turns (Section 7). I could not find a conclusion in the paper that outcome metrics get wrong and the turn labels get right. Part of the reason is the setting: one game, one harness, six levels and five small models, and every episode in the main corpus fails, most within a few steps. With no partial successes there is nothing to compare the labels against, so reader cannot tell whether the taxonomy describes why agents fail or just how these models play. The board-table finding also comes from a single model (Section 5.4). The ladder, the only result with any successes, is confounded with walking distance. On the first three rungs the worker already stands next to the box, while from the fourth rung on it needs 8 or 9 moves to get into position (supplementary chance_baseline.json), and the prompt never says a push requires adjacency. The drop after three pushes may therefore measure whether the agent can walk around a box, not push depth. Rewording cannot fix this. It needs a setting where some agents partly succeed, such as a stronger open model or easier levels, so labels can be compared between solved and unsolved episodes, and a ladder that keeps the walking distance fixed while the number of pushes grows.

(2) Once we set aside the borrowed parts, the contribution is small, and the main behavioural finding may come from prompts. The environment, levels and prompt come from GamingAgent, reporting kappa with PABAK is standard practice, and run-to-run variation at temperature 0 is already documented (the paper cites He and Thinking Machines Lab, 2025). What is left is a descriptive taxonomy whose discriminating power the control did not show. The thought-action mismatch, one of the two findings the claim rests on, is exactly what the prompt's only worked example demonstrates, and that example is sent on every step of every condition; the authors say nothing in the study bounds its effect (Section 7). The related-work positioning also overstates the gap. Section 2 says failure analysis has not been done on games, yet Appendix G lists TopoBench, an error taxonomy over reasoning traces on puzzles, and Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective (arXiv 2510.15047), which analyses small-model failures in Sokoban, is not cited. To address this, the authors would need a run in which the worked example's thought and action agree, and a direct statement of what their taxonomy adds over these closest works.

(3) On presentation, the research question is hard to find. The Abstract and Introduction are written too densely, with terms and qualifiers piled on top of each other, and the question itself appears only once, in a single clause. The contributions list uses terms that have not been defined yet and mixes in disclosures such as "Blindness to the study's direction was tested, and failed the first test", and Related Work ends with a positioning line made of four items joined by multiplication signs.

Questions for authors
  1. Is there any conclusion in the paper that solve rate, no-op rate or solver-detected deadlock would get wrong, and that the turn-level labels correct? The argument about ineffective actions in Section 6 is about metrics and does not need the taxonomy.
  2. In the ladder episodes from the fourth rung on, how often does the worker reach a position from which it can legally push the box? If almost never, the drop after three pushes is about repositioning rather than push depth. Could you add a ladder with fixed walking distance, and a setting where agents partly succeed so that labels can be compared between solved and unsolved episodes?
  3. What does your taxonomy find that the TopoBench error taxonomy and the World-Modeling Perspective paper (arXiv 2510.15047) do not?

References

[1] Mayug Maniparambil et al. TopoBench: Benchmarking LLMs on Hard Topological Reasoning. In ICLR 2026 Workshop on Logical Reasoning of Large Language Models, 2026. [2] Shiqi Chen et al. Why Do LLM Agents Fail to Explore New Environments? A World-Modeling Perspective. In NeurIPS 2026 Workshop on From Pretrained Representations to Acting Agents (PTA), 2026.

Reviewer_WEaNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors test how well the different LLM based agents can do on the game of SokoBench (textual verrsion). It is difficult to say if the results are missing or this all there is. In any case, it is very difficult to tell due to very bad writing all around.

Critical strengths

The paper is poorly written and results are not well documented.

Critical weaknesses

I understand AI usage for paper writing is common now but it should lead to better quality writing and not a AI-coded style of writing which is difficult to follow. Frankly, the first sentence itself starts on the wrong foot. For example, the Abstract should have started with giving a context about the “TEXTUAL SOKOBAN” game and the challenges in playing them through AI agents. And then the significance of this work in context of its broader implications. But all we have here is densely written incoherent sentences which just describes the low level details of what experiments were carried out.

Same thing can be observed in Introduction and pretty much in all the sections. Long sentences filled with many jargons, sentences written in 3rd person, negation followed by assertion. The writing is homogenous, almost entirely AI written without much supervision from the authors. In conclusion, the paper is not at all properly written.

It is difficult to understand what exactly the results indicate.

Questions for authors

Please check the strength and weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.