acceptodds
Under review as a conference paper at ICLR 2027

LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

Abstract

GUI agents need long-horizon visual reasoning: they must interpret a changing interface while keeping a multi-step plan viable as earlier actions constrain later ones. Existing benchmarks evaluate grounding, computer use, and game play, but rarely test whether agents stay coherent across long chains of coupled decisions. Long-horizon visual puzzles expose this capability directly: a legal move that looks like progress can make the puzzle unsolvable, and the loss shows only several moves later. We introduce LongPuzzleBench, 114 levels in six puzzle games played through native GUI actions, where one objective can take a human over a thousand actions on persistent boards and dead ends go unannounced. With Native GUI Actions alone, the strongest agents solve most objectives, but success falls sharply on harder, longer boards: seven of ten general-purpose agents solve nothing harder than Medium, and none completes Bolt Unscrew Hard, which a human solves along with every other objective. Code Execution CUA does not close this gap, and its scores mix visual solving with algorithmic search. Controlled diagnostics trace these failures to one limitation that neither rules, state hints, nor failure memory removes: agents judge each move by the visible progress it makes, not by the future options it leaves.

Primary area:
datasets and benchmarks
Keywords:
Interactive Puzzle Solving, Long-Horizon Planning, GUI Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_SyqTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LongPuzzleBench evaluates GUI agents on 114 levels across six puzzle games, grouped into 16 objectives requiring consecutive level completion within a shared time budget. Agents interact through screenshots and coordinate-based actions and manage navigation, progression, and recovery. The evaluation covers 14 agents, includes a human reference, and examines code-assisted interaction for general-purpose agents. Controlled diagnostics investigate capacity-consuming actions, dead-end recognition, failure memory, and rule knowledge. The paper argues that important failures arise from prioritizing immediate visible progress over preserving future options.

Critical strengths
  1. The benchmark is a useful resource for studying autonomous solving and recovery. Persistent puzzle states, agent-controlled progression, unannounced dead ends, and reproducible checkpoints expose failures that aggregate completion rates alone do not explain. The detailed protocol and per-objective results support its value for future work on GUI agents.

  2. The matched-state dead-end intervention provides a concrete empirical finding. In the tested Bolt Unscrew states, explicit state information raises verified recovery from 8/24 to 23/24, while additionally naming the restart control provides little benefit. Solvable controls and the cross-model follow-up strengthen the interpretation that recognizing the need for recovery is a bottleneck in these cases. Together with the benchmark resource, this focused result is sufficient for my weak-acceptance recommendation, even though the broader causal interpretation needs to be narrowed.

Critical weaknesses
  1. The proposed general mechanism is stronger than the diagnostic evidence supports. In Section 5.1, 28 of 55 fatal moves have rationales predicting that the action will release a board. These cases are compatible with incorrect state perception or transition prediction, so they do not uniquely establish a preference for visible progress over preserving future options. The Maze Paint comparison is also unmatched for difficulty. For the memory study, Appendix D.4 states that source histories often omit the final dead-end frame, making it unclear whether the summarizer received enough evidence to infer the failure cause. All 126 retries also fail to complete within the 300-second budget, limiting the sensitivity of completion as an outcome. The results support specific observations about the tested interventions, but do not yet establish a common planning mechanism that explains all of these failures.

  2. The evaluation supports a more limited claim about reasoning horizon and generality. Consecutive level completion and action counts combine reasoning difficulty with per-level reliability and accumulated time costs. H50 covers the eight ten-level objectives and excludes both games with irreversible dead ends. The strongest agent solves 15/16 objectives, and the remaining Bolt Unscrew Hard failure occurs after reaching the first uncleared level with only two minutes left. The principal diagnostics use different models and history management from the main evaluation, and each main model-setting-objective combination has only one scored episode. These limitations constrain the general account of long-horizon failure, while leaving the benchmark's resource value and the matched-state recovery result intact.

Questions for authors
  1. Among the fatal actions in Section 5.1, what evidence distinguishes knowingly sacrificing future options from misperceiving the state or incorrectly predicting the immediate transition? Existing annotations or a small matched-state analysis would help. If this distinction cannot be established, would the authors narrow the causal claim accordingly?

  2. For how many of the 24 memory-study sources was the failure and its cause inferable from the history actually supplied to the summarizer? Do the summary-quality and recurrence findings persist on that subset? Please also clarify how the diagnostic harness affects the scope of the conclusions drawn about agents in the main benchmark; a full rerun with all leading agents is not necessary.

  3. The distributed games/puzzle_suite/NOTICE.md states that the imported Bolt_Unscrew branch had no standalone license and that redistribution of imported code and assets requires authorization from their rights holders. Could you clarify the authorization or licensing basis for these materials? This is a question about the benchmark's availability for reuse; the notice alone does not establish unauthorized use.

Reviewer_t5gUOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper authors introduce LongPuzzleBench, a benchmark that evaluates GUI agents on 6 long-horizon puzzle games, with a total of 114 levels. These 6 games are grouped into 16 objectives. Authors state that these are extremely long horizon games, even for humans, where a human needs a median of 204 GUI actions per objective. An agent sees the screenshots of the game and decides to either click, drag, press keys, and other coordinate-based inputs. Code Execution Computer-Use Agents (CUA) can write and run code over the screenshot to solve it.

Authors evaluate 10 general-purpose LLMs with Codex, while 4 LLMs have a GUI-specialized harness. The 10 general-purpose LLMs are also used in Code Execution CUAs. The main metric is Objective Success Rate, but they have 8 metrics that cover completion of tasks, horizon, difficulty, and behaviour of the agent. The authors analyse results in depth and also look into why the agent fails.

Critical strengths

A very well human-written paper. In-depth and extensive evaluations. Great introduction and related works sections.

Critical weaknesses

Some of the images are not clear what they want to say. I can read the captions and know what they are saying but I should be able to know by looking at the figure. Examples are: Figure 4(a) and (b) (Fig. 4(b) is still better). Figure 3(b) and (c).

Questions for authors

Can you make Fig. 3(b, c) and Fig. 4(a, b) easier to follow? I understand that a lot of effort has been put into them. I can see that, but if this is made a bit more readable, I am willing to move my score from weakly accepted to clearly accepted. Otherwise, I believe this is a great submission.

Reviewer_fa2QOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LongPuzzleBench, which tests GUI agents on six visual puzzle games whose boards persist across many actions. Agents must solve levels, recognize dead ends, and decide when to restart. The paper compares native GUI play with code-assisted play and uses restored states to diagnose failures.

Critical strengths
  • The benchmark creates a concrete and interesting planning problem: an early legal move can consume capacity needed later, without immediate failure feedback. Exact state restoration makes the diagnostic experiments particularly useful.
  • The Bolt Unscrew example is especially interesting: in states with one free hole left, agents frequently fill that hole and end the level; a hint about the lost state improves recovery but does not make subsequent attempts solve it.
Critical weaknesses
  • I am less convinced that one explanation accounts for the full benchmark.
    • Only Bolt Unscrew and Nut and Bolt have irreversible dead ends; in the other four games, poor moves can lengthen the solution but do not make it impossible.
    • The detailed mechanism studies concentrate on Bolt Unscrew and Maze Paint, mainly with three related models.
    • The main comparison also has one episode per agent and objective, so differences on 16 objectives may be sensitive to a single run.
Questions for authors

How stable are the objective rankings over repeated episodes? If agents receive an exact textual board state but must still act through the GUI, does the same failure persist? That would help separate visual state reading from planning.

1 comment

I thank the authors for the response. My concerns on the explanation on the benchmark and stability are addressed, and I keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.