- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TsuGO, which evaluates "search organization ability" of LLMs via Tsumego problems in GO. This benchmark contains 1500 problems and can serves as a process-level judge for LLMs' search ability. The authors use Gemini-2.5-Pro to parse the CoTs into process search trees, and define the evaluation matric SearchE as a proxy of search efficiency metric. Authors claim that: current LLMs are far from solving this tsumego benchmark; stronger LLMs are earlier to reach the correct solution and invest wisely in the correct reasoning branch; and SearchE is a better evaluation signal compared with TokenE, which fits the accuracy more closely.
Critical strengths
-
Using tsumego as a testbed is novel and interesting. It makes since the solution space is bounded and verifiable. It has a good property of adversarial structure - a move should be able to cope with the best response from the adversary. This makes candidate generation, verification and backtracking necessary and load-bearing in the reasoning traces.
-
The benchmark is well-formatted: it is transformed into SGF/JSON format, and multiple modalities share the same standard states. The label quality is paid enough attention to. It is also of good transparency: it introduces MCTS/KataGO as non-LLM reference. All the prompts are also released in the paper.
-
The benchmark result is far from saturation, which leaves room for improvement in future works.
Critical weaknesses
- It is unclear whether tsumego is a good proxy for general LLM search ability. There could be alternative explanations for poor search behavior:
- Weak domain priors. Go life-and-death is sparsely presented in pretraining data, so LLMs have poor generative ability in this domain; LLMs may fail to propose reasonable candidates before search organization comes into play.
- Generation budget. Kimi-K2.5's mean ITT under K=None is 28.7k–31.8k tokens, against a 32k maximum output length (Table 16). This implies that most of its open-setting responses may be truncated, so its collapse to 4.4% on K=None Hard may reflect the budget rather than search organization. Please report truncation and no-answer rates per model and setting, and re-evaluate with a larger budget.
- Perception and state-tracking failures. The authors' own failure analysis (Fig. 12) attributes 24% of errors to coordinate/state drift (18%) and visual grounding (6%). More fundamentally, the paper does not test whether search ability measured on TsuGO transfers to other search-heavy tasks. Prior work has characterized related failures in other settings: frequent switching between reasoning paths on challenging math problems (Wang et al., 2025), wandering behaviors in search-heavy computation tasks (Lu et al., 2025) and performance collapse as problem size grows (Shojaee et al., 2025). If model rankings on TsuGO disagree with rankings on other adversarial games (e.g., chess endgames, Othello) or on such computational tasks, TsuGO is more likely measuring Go-specific competence than general search ability. Reporting cross-task rank correlations would directly address this concern.
- The contribution is limited to a benchmark plus evaluation, and its claimed utility is untested. The conclusion positions TsuGO as a basis for Search-Efficiency-sensitive training, yet the paper contains no training experiments. Several natural and feasible questions remain unanswered.
- If TsuGO is used as an RL environment with outcome reward only (first-move correctness), does SearchE improve?
- Can SearchE serve as a process reward? Will it introduce reward hacking?
I feel that the contribution would be not so narrow if either (i) authors provide a suite of adversarial structure games, e.g. including other games or computation tasks (ii) authors provide sufficient experiment, results and insights on treating this proposed benchmark as a trainable environment and show how training on this dataset / environment alone can or cannot improve the general search capability of LLMs.
- The process-tree parsing and extraction relies on a single LLM parser (Gemini-2.5-Pro) which may not be faithful. Readers are not convinced whether such parsing introduces hallucination; or omissions of critical solution steps; or the parsing errors. Human examination cannot be scaled up. The paper should at least cross-validate with multiple parsers or CoT monitors from different model families, and report the agreement or stability of the main metrics across parsers.
Reference:
[1] Wang et al., 2025, Thoughts Are All Over the Place, https://arxiv.org/abs/2501.18585
[2] Lu et al. 2025, Reasoning LLMs are Wandering Solution Explorers, https://arxiv.org/abs/2505.20296
[3] Shojaee et al. 2025, The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity, NeurIPS 2025
Questions for authors
-
What evidence supports the claim that TsuGO disentangles domain knowledge from search organization? Does performance change when models are given domain hints, such as liberty counts or a few solved tsumego as in-context examples?
-
What fraction of responses are truncated or contain no parsable answer, per model and setting? How does Kimi-K2.5's K=None accuracy change with a larger output budget?
-
Do model rankings on TsuGO correlate with rankings on other search-heavy tasks, such as other board games or the computational tasks?
-
Under outcome-only RL on TsuGO, does SearchE improve together with accuracy? Does any gain transfer to search tasks outside Go?
-
How stable are the main metrics, and how well do parsers agree, when trees are extracted by parsers from other model families?
-
Does the parser see the reference answer during extraction?
2 comments
Reply_2 to Reviewer tAsp
6. Q6: Reference-answer access
The parser has one task: extracting the visible trace; it does not receive the reference answer or metric-related scoring instructions. Its inputs are the model's visible reasoning and a fixed schema, with no gold key, target metric values, or scoring weights. It extracts expressed candidates, continuations, and judgments without being asked to solve the puzzle or optimize SearchE. Reference answers enter only the later metric-computation stage. The model's own proposed answer may occur in its trace, but this is not the gold answer.
7. Perception and state-maintenance costs
The substantial allocation to perception and verification is itself a finding about search-reasoning behavior. In our trace analysis, most of the examined models allocate approximately 10-50% of their visible reasoning tokens to board perception, including both reasoning and non-reasoning models. This exposes how finite reasoning resources are divided between grounding, state checking, and subsequent search. Our measurement focus is this observable CoT behavior on verifiable adversarial tasks, rather than Go-playing strength alone. Perception, verification, and state maintenance are not unique to Go; their resource costs reveal behavioral preferences relevant to search organization. We will document the token segmentation and distinguish these total-text costs from candidate-branch metrics.
8. Why an adversarial Go testbed?
The benchmark is centered on search within a verifiable adversarial solution space, not on games as the contribution. Go life-and-death problems offer candidate moves, best opponent responses, and checkable refutations across diverse difficulty. An unpruned tree grows roughly as b^d with branching factor b and depth d; local puzzles retain this challenge while making solutions verifiable. The availability of diverse, annotated local problems supports evaluation at scale. Certain forcing-line Gomoku puzzles can have narrow useful candidate sets, so other domains need comparable diversity and solution coverage before being added. Go's broad branching and difficult value estimation also help explain its prominence in neural-guided search research, exemplified by AlphaGo. We welcome extensions, while maintaining the independent value of a well-constructed adversarial testbed.
9. The new efficiency dimension
TsuGO introduces an operational definition of LLM search efficiency in verifiable adversarial environments, adding a distinct evaluation dimension. SearchE measures wrong-branch allocation, the first exploration of a correct candidate, and candidate rank, connecting the organization of reasoning to a verifiable solution space. This complements accuracy and token cost. Prior studies of path switching, wandering, and complexity motivate related questions. Our additional contribution is making search allocation measurable against correct candidates and adversarial refutations, rather than characterizing verbosity alone. We will clarify this relationship to the cited work.
Reply_1 to Reviewer tAsp
Thank you for recognizing the novelty, verifiability, and adversarial structure of TsuGO. We first answer your Questions then address the remaining scope and interpretation concerns.
1. Q1: Domain knowledge, search, and hints
The main-table comparison between K=4 and K=None provides a partial operational separation of domain knowledge and search organization. Supplying candidates reduces the burden of proposing plausible first moves, while the open setting requires candidate generation as well as subsequent verification. Table 3 illustrates this distinction: Kimi scores 52.0/27.8% on Easy and 34.0/4.4% on Hard. The comparison helps identify how candidate support changes search behavior; both settings still require domain knowledge. Knowledge injection is a valuable extension. However, life-and-death solving combines board perception, state maintenance, candidate generation, and adversarial verification. The size and source of gains from liberty counts or solved examples must therefore be examined carefully through paired comparisons of both outcomes and process behavior, rather than attributed to a single component.
2. Q2: Output budgets, truncation, and larger-budget evaluation
The 32k baseline budget is a deliberate choice for fair comparison of search efficiency within a common resource allowance. Table 16 specifies 32k output budgets for Kimi and most evaluated open-model families. Our question is how models organize search and complete reasoning within a specified budget; this constraint is central to the evaluation design. We will report actual output lengths, truncation rates, and no-answer rates by model and setting, distinguishing initial attempts, retries, and final records. Truncation is itself a dimension of budget-conditioned capability: reaching a solution within the available resources is part of effective search. Larger budgets can provide supplementary analysis, but change the evaluation condition and do not replace the fair fixed-budget comparison.
3. Q3: Cross-task model rankings
Thank you for the suggestion; we will consider extending the comparison to suitable general tasks. At submission, to our knowledge, we had not identified a sufficiently matched protocol for fairly comparing these process metrics across tasks. A useful extension should compare a common set of models under explicit task interfaces and resource budgets. The relative ordering observed on TsuGO shows qualitative agreement with some general capability rankings. We will investigate this agreement more systematically through supplementary task comparisons and rank-correlation analysis, making the comparison protocol explicit.
4. Q4: Outcome-only RL, SearchE, and transfer
Training with TsuGO-related objectives is a future direction we are actively exploring. Thank you for suggesting outcome-only RL and transfer experiments. Building on this benchmark, we can compare outcome rewards with objectives involving SearchE and related metrics, examining whether training improves both task success and search organization. Transfer to search tasks outside Go is a natural part of that follow-up evaluation. These are ongoing research questions motivated by the benchmark's verifiable outcomes and process diagnostics.
5. Q5: Extractor reliability and agreement across families
The extractor has been evaluated through human checking and adjudication, as reported in Appendix C. Table 10 (lines 982-992) reports pass rates of 98% for candidate sets, 97% for order, 95% for parent-child assignments, 93% for terminal judgments, and 96% for metric-consistent trees. The protocol in lines 1007-1020 checks the extracted structure against the visible trace, targeting properties that determine the search metrics. This evidence supports the reliability of the current extraction procedure. Comparisons across parser families would complement the human validation, and we will consider applying independent parsers to the same stratified traces and examining the stability of metrics and rankings.