- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LongPuzzleBench evaluates GUI agents on 114 levels across six puzzle games, grouped into 16 objectives requiring consecutive level completion within a shared time budget. Agents interact through screenshots and coordinate-based actions and manage navigation, progression, and recovery. The evaluation covers 14 agents, includes a human reference, and examines code-assisted interaction for general-purpose agents. Controlled diagnostics investigate capacity-consuming actions, dead-end recognition, failure memory, and rule knowledge. The paper argues that important failures arise from prioritizing immediate visible progress over preserving future options.
Critical strengths
-
The benchmark is a useful resource for studying autonomous solving and recovery. Persistent puzzle states, agent-controlled progression, unannounced dead ends, and reproducible checkpoints expose failures that aggregate completion rates alone do not explain. The detailed protocol and per-objective results support its value for future work on GUI agents.
-
The matched-state dead-end intervention provides a concrete empirical finding. In the tested Bolt Unscrew states, explicit state information raises verified recovery from 8/24 to 23/24, while additionally naming the restart control provides little benefit. Solvable controls and the cross-model follow-up strengthen the interpretation that recognizing the need for recovery is a bottleneck in these cases. Together with the benchmark resource, this focused result is sufficient for my weak-acceptance recommendation, even though the broader causal interpretation needs to be narrowed.
Critical weaknesses
-
The proposed general mechanism is stronger than the diagnostic evidence supports. In Section 5.1, 28 of 55 fatal moves have rationales predicting that the action will release a board. These cases are compatible with incorrect state perception or transition prediction, so they do not uniquely establish a preference for visible progress over preserving future options. The Maze Paint comparison is also unmatched for difficulty. For the memory study, Appendix D.4 states that source histories often omit the final dead-end frame, making it unclear whether the summarizer received enough evidence to infer the failure cause. All 126 retries also fail to complete within the 300-second budget, limiting the sensitivity of completion as an outcome. The results support specific observations about the tested interventions, but do not yet establish a common planning mechanism that explains all of these failures.
-
The evaluation supports a more limited claim about reasoning horizon and generality. Consecutive level completion and action counts combine reasoning difficulty with per-level reliability and accumulated time costs. H50 covers the eight ten-level objectives and excludes both games with irreversible dead ends. The strongest agent solves 15/16 objectives, and the remaining Bolt Unscrew Hard failure occurs after reaching the first uncleared level with only two minutes left. The principal diagnostics use different models and history management from the main evaluation, and each main model-setting-objective combination has only one scored episode. These limitations constrain the general account of long-horizon failure, while leaving the benchmark's resource value and the matched-state recovery result intact.
Questions for authors
-
Among the fatal actions in Section 5.1, what evidence distinguishes knowingly sacrificing future options from misperceiving the state or incorrectly predicting the immediate transition? Existing annotations or a small matched-state analysis would help. If this distinction cannot be established, would the authors narrow the causal claim accordingly?
-
For how many of the 24 memory-study sources was the failure and its cause inferable from the history actually supplied to the summarizer? Do the summary-quality and recurrence findings persist on that subset? Please also clarify how the diagnostic harness affects the scope of the conclusions drawn about agents in the main benchmark; a full rerun with all leading agents is not necessary.
-
The distributed games/puzzle_suite/NOTICE.md states that the imported Bolt_Unscrew branch had no standalone license and that redistribution of imported code and assets requires authorization from their rights holders. Could you clarify the authorization or licensing basis for these materials? This is a question about the benchmark's availability for reuse; the notice alone does not establish unauthorized use.
I thank the authors for the response. My concerns on the explanation on the benchmark and stability are addressed, and I keep my positive score.